Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
> Quoting Ingo Molnar <[EMAIL PROTECTED]>:
> Subject: Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup
> detected on CPU#0!)
>
>
> * Michael S. Tsirkin <[EMAIL PROTECTED]> wrote:
>
> > > could you turn on CONFIG_SLAB_DEBUG as well?
> > >
> > > that should catch certain types of use-after-free accesses, and
> > > lockdep will also warn if a still locked object is freed.
> >
> > Hmm, no, this does not look like use-after-free. I enabled
> > CONFIG_SLAB_DEBUG, and I still see the same message, so the memory was
> > not overwritten by slab debugger.
>
> that's still not conclusive - the memory might not have been allocated
> by slab again to detect it. Your magic-number check definitely shows
> some sort of corruption going on, right?
Not necessarily in such a direct way.
I currently think we are somehow getting neighbours where
neigh->dev points to a loopback device - that's type 772,
and this seems to make sense.
I printed out the device name and sure enough it is "lo".
Is it true that sticking the following
static int ipoib_neigh_setup_dev(struct net_device *dev,
struct neigh_parms *parms)
{
parms->neigh_destructor = ipoib_neigh_destructor;
return 0;
}
in dev->neigh_setup, as ipoib does, guarantees that neighbour->dev will point to
the current device for any neighbour which ipoib_neigh_destructor gets?
That's the assumption IPoIB makes, and it seems broken in this instance.
How could that be?
--
MST
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
* Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > could you turn on CONFIG_SLAB_DEBUG as well? > > > > that should catch certain types of use-after-free accesses, and > > lockdep will also warn if a still locked object is freed. > > Hmm, no, this does not look like use-after-free. I enabled > CONFIG_SLAB_DEBUG, and I still see the same message, so the memory was > not overwritten by slab debugger. that's still not conclusive - the memory might not have been allocated by slab again to detect it. Your magic-number check definitely shows some sort of corruption going on, right? Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
> Quoting Ingo Molnar <[EMAIL PROTECTED]>: > Subject: Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup > detected on CPU#0!) > > > * Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > > > So either there are other sites that instanciate those objects and > > > forget about the lock init, or the object is corrupted (use after free?) > > > > OK, thanks for the hint. So I added this: > > > And sure enough it triggers: > > > > [ 858.503010] ipoib_neigh_destructor lock c0687880 wrong type 772 > > !! > > could you turn on CONFIG_SLAB_DEBUG as well? > > that should catch certain types of use-after-free accesses, and lockdep > will also warn if a still locked object is freed. Hmm, no, this does not look like use-after-free. I enabled CONFIG_SLAB_DEBUG, and I still see the same message, so the memory was not overwritten by slab debugger. -- MST - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
Quoting Ingo Molnar [EMAIL PROTECTED]: Subject: Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!) * Michael S. Tsirkin [EMAIL PROTECTED] wrote: So either there are other sites that instanciate those objects and forget about the lock init, or the object is corrupted (use after free?) OK, thanks for the hint. So I added this: And sure enough it triggers: [ 858.503010] ipoib_neigh_destructor lock c0687880 wrong type 772 !! could you turn on CONFIG_SLAB_DEBUG as well? that should catch certain types of use-after-free accesses, and lockdep will also warn if a still locked object is freed. Hmm, no, this does not look like use-after-free. I enabled CONFIG_SLAB_DEBUG, and I still see the same message, so the memory was not overwritten by slab debugger. -- MST - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
* Michael S. Tsirkin [EMAIL PROTECTED] wrote: could you turn on CONFIG_SLAB_DEBUG as well? that should catch certain types of use-after-free accesses, and lockdep will also warn if a still locked object is freed. Hmm, no, this does not look like use-after-free. I enabled CONFIG_SLAB_DEBUG, and I still see the same message, so the memory was not overwritten by slab debugger. that's still not conclusive - the memory might not have been allocated by slab again to detect it. Your magic-number check definitely shows some sort of corruption going on, right? Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
Quoting Ingo Molnar [EMAIL PROTECTED]:
Subject: Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup
detected on CPU#0!)
* Michael S. Tsirkin [EMAIL PROTECTED] wrote:
could you turn on CONFIG_SLAB_DEBUG as well?
that should catch certain types of use-after-free accesses, and
lockdep will also warn if a still locked object is freed.
Hmm, no, this does not look like use-after-free. I enabled
CONFIG_SLAB_DEBUG, and I still see the same message, so the memory was
not overwritten by slab debugger.
that's still not conclusive - the memory might not have been allocated
by slab again to detect it. Your magic-number check definitely shows
some sort of corruption going on, right?
Not necessarily in such a direct way.
I currently think we are somehow getting neighbours where
neigh-dev points to a loopback device - that's type 772,
and this seems to make sense.
I printed out the device name and sure enough it is lo.
Is it true that sticking the following
static int ipoib_neigh_setup_dev(struct net_device *dev,
struct neigh_parms *parms)
{
parms-neigh_destructor = ipoib_neigh_destructor;
return 0;
}
in dev-neigh_setup, as ipoib does, guarantees that neighbour-dev will point to
the current device for any neighbour which ipoib_neigh_destructor gets?
That's the assumption IPoIB makes, and it seems broken in this instance.
How could that be?
--
MST
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
* Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > So either there are other sites that instanciate those objects and > > forget about the lock init, or the object is corrupted (use after free?) > > OK, thanks for the hint. So I added this: > And sure enough it triggers: > > [ 858.503010] ipoib_neigh_destructor lock c0687880 wrong type 772 !! could you turn on CONFIG_SLAB_DEBUG as well? that should catch certain types of use-after-free accesses, and lockdep will also warn if a still locked object is freed. Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
Quoting Peter Zijlstra <[EMAIL PROTECTED]>:
Subject: Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup
detected on CPU#0!)
> On Sun, 2007-03-11 at 15:50 +0200, Michael S. Tsirkin wrote:
> > > Quoting Roland Dreier <[EMAIL PROTECTED]>:
> > > Subject: Re: IPoIB caused a kernel: BUG: soft lockup detected on CPU#0!
> > >
> > > >Feb 27 17:47:52 sw169 kernel: []
> > > >_spin_lock_irqsave+0x15/0x24
> > > >Feb 27 17:47:52 sw169 kernel: []
> > > >:ib_ipoib:ipoib_neigh_destructor+0xc2/0x139
> > >
> > > It looks like this is deadlocking trying to take priv->lock in
> > > ipoib_neigh_destructor().
> > > One idea I just had would be to build a kernel with CONFIG_PROVE_LOCKING
> > > turned on, and then rerun this test. There's a good chance that this
> > > would
> > > diagnose the deadlock. (I don't have good access to my test machines
> > > right now, or
> > > else I would do it myself)
> >
> > OK, I did that. But I get
> > [13440.761857] INFO: trying to register non-static key.
> > [13440.766903] the code is fine but needs lockdep annotation.
> > [13440.772455] turning off the locking correctness validator.
> > and I am not sure what triggers this, or how to fix it to have the
> > validator actually do its job.
>
> It usually indicates a spinlock is not properly initialized. Like
> __SPIN_LOCK_UNLOCKED() used in a non-static context, use
> spin_lock_init() in these cases.
>
> However looking at the code, ipoib_neight_destructor only uses
> >lock, and that seems to get properly initialized in ipoib_setup()
> using spin_lock_init().
>
> So either there are other sites that instanciate those objects and
> forget about the lock init, or the object is corrupted (use after free?)
OK, thanks for the hint. So I added this:
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c
b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index f9dbc6f..2eea467 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -821,8 +821,15 @@ static void ipoib_neigh_destructor(struct neighbour *n)
unsigned long flags;
struct ipoib_ah *ah = NULL;
+ if (n->dev->type != ARPHRD_INFINIBAND) {
+ printk(KERN_ERR "ipoib_neigh_destructor lock %p wrong type %d
!!\n",
+ >lock, n->dev->type);
+ BUG_ON(n->dev->type != ARPHRD_INFINIBAND);
+ return;
+ }
+
ipoib_dbg(priv,
"neigh_destructor for %06x " IPOIB_GID_FMT "\n",
IPOIB_QPN(n->ha),
IPOIB_GID_RAW_ARG(n->ha + 4));
And sure enough it triggers:
[ 858.503010] ipoib_neigh_destructor lock c0687880 wrong type 772 !!
[ 858.510036] [ cut here ]
[ 858.514723] kernel BUG at drivers/infiniband/ulp/ipoib/ipoib_main.c:827!
[ 858.521486] invalid opcode: [#1]
[ 858.525212] SMP
[ 858.527173] Modules linked in: rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa
ib_uverbs ibv
[ 858.538736] CPU:0
[ 858.538737] EIP:0060:[]Not tainted VLI
[ 858.538738] EFLAGS: 00010206 (2.6.21-rc3-i686-dbg #4)
[ 858.551755] EIP is at ipoib_neigh_destructor+0x40/0x178 [ib_ipoib]
[ 858.557996] eax: c0687300 ebx: f240e880 ecx: c0223114 edx: c064f280
[ 858.564851] esi: f240e880 edi: f240e880 ebp: c0687880 esp: c06c7e9c
[ 858.571702] ds: 007b es: 007b fs: 00d8 gs: ss: 0068
[ 858.577602] Process swapper (pid: 0, ti=c06c6000 task=c064f280
task.ti=c06c6000)
[ 858.584883] Stack: f89a37be c0687880 0304 c022af6e c064f280
[ 858.593573] c06a2554 c064f280 0001
c064f280
[ 858.602259]c0860be0 c2a1fba0 0246 c06a2554 f240e880
f240e880 c04a
[ 858.610946] Call Trace:
[ 858.613723] [] run_timer_softirq+0x37/0x16b
[ 858.618959] [] dst_run_gc+0x0/0x118
[ 858.623498] [] neigh_destroy+0xbe/0x104
[ 858.628382] [] dst_destroy+0x4d/0xab
[ 858.632998] [] dst_run_gc+0x55/0x118
[ 858.637620] [] run_timer_softirq+0x108/0x16b
[ 858.642934] [] __do_softirq+0x5a/0xd5
[ 858.647648] [] trace_hardirqs_on+0x106/0x141
[ 858.652970] [] __do_softirq+0x69/0xd5
[ 858.657677] [] do_softirq+0x37/0x4d
[ 858.662210] [] smp_apic_timer_interrupt+0x6b/0x77
[ 858.667965] [] default_idle+0x3b/0x54
[ 858.672681] [] default_idle+0x3b/0x54
[ 858.677391] [] apic_timer_interrupt+0x33/0x38
[ 858.682796] [] default_idle+0x3b/0x54
[ 858.687505] [] default_idle+0x3d/0x54
[ 858.692211] [] cpu_idle+0xa2/0xbb
[ 858.696569] [] start_kernel+0x40b/0x413
[ 858.701453] [] unknown_bootoption+0x0/0x205
[ 858.706678] ===
[ 858.710321] Code: 66 83 f8 20 74 29 0f b7 c0 89 44 24 08 89 6c 24 04 c7 04
24 be 37 9a
[ 858.730997] EIP: [] ipoib_neigh_destructor+0x40/0x178 [ib_ipoib]
SS:ESP 0068c
[ 858.740271] Kernel panic - not syncing: Fatal exception in interrupt
Either something is corrupting neighbour dev pointer, or
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
Quoting Peter Zijlstra [EMAIL PROTECTED]:
Subject: Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup
detected on CPU#0!)
On Sun, 2007-03-11 at 15:50 +0200, Michael S. Tsirkin wrote:
Quoting Roland Dreier [EMAIL PROTECTED]:
Subject: Re: IPoIB caused a kernel: BUG: soft lockup detected on CPU#0!
Feb 27 17:47:52 sw169 kernel: [8053aaf1]
_spin_lock_irqsave+0x15/0x24
Feb 27 17:47:52 sw169 kernel: [88067a23]
:ib_ipoib:ipoib_neigh_destructor+0xc2/0x139
It looks like this is deadlocking trying to take priv-lock in
ipoib_neigh_destructor().
One idea I just had would be to build a kernel with CONFIG_PROVE_LOCKING
turned on, and then rerun this test. There's a good chance that this
would
diagnose the deadlock. (I don't have good access to my test machines
right now, or
else I would do it myself)
OK, I did that. But I get
[13440.761857] INFO: trying to register non-static key.
[13440.766903] the code is fine but needs lockdep annotation.
[13440.772455] turning off the locking correctness validator.
and I am not sure what triggers this, or how to fix it to have the
validator actually do its job.
It usually indicates a spinlock is not properly initialized. Like
__SPIN_LOCK_UNLOCKED() used in a non-static context, use
spin_lock_init() in these cases.
However looking at the code, ipoib_neight_destructor only uses
priv-lock, and that seems to get properly initialized in ipoib_setup()
using spin_lock_init().
So either there are other sites that instanciate those objects and
forget about the lock init, or the object is corrupted (use after free?)
OK, thanks for the hint. So I added this:
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c
b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index f9dbc6f..2eea467 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -821,8 +821,15 @@ static void ipoib_neigh_destructor(struct neighbour *n)
unsigned long flags;
struct ipoib_ah *ah = NULL;
+ if (n-dev-type != ARPHRD_INFINIBAND) {
+ printk(KERN_ERR ipoib_neigh_destructor lock %p wrong type %d
!!\n,
+ priv-lock, n-dev-type);
+ BUG_ON(n-dev-type != ARPHRD_INFINIBAND);
+ return;
+ }
+
ipoib_dbg(priv,
neigh_destructor for %06x IPOIB_GID_FMT \n,
IPOIB_QPN(n-ha),
IPOIB_GID_RAW_ARG(n-ha + 4));
And sure enough it triggers:
[ 858.503010] ipoib_neigh_destructor lock c0687880 wrong type 772 !!
[ 858.510036] [ cut here ]
[ 858.514723] kernel BUG at drivers/infiniband/ulp/ipoib/ipoib_main.c:827!
[ 858.521486] invalid opcode: [#1]
[ 858.525212] SMP
[ 858.527173] Modules linked in: rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa
ib_uverbs ibv
[ 858.538736] CPU:0
[ 858.538737] EIP:0060:[f899bfa5]Not tainted VLI
[ 858.538738] EFLAGS: 00010206 (2.6.21-rc3-i686-dbg #4)
[ 858.551755] EIP is at ipoib_neigh_destructor+0x40/0x178 [ib_ipoib]
[ 858.557996] eax: c0687300 ebx: f240e880 ecx: c0223114 edx: c064f280
[ 858.564851] esi: f240e880 edi: f240e880 ebp: c0687880 esp: c06c7e9c
[ 858.571702] ds: 007b es: 007b fs: 00d8 gs: ss: 0068
[ 858.577602] Process swapper (pid: 0, ti=c06c6000 task=c064f280
task.ti=c06c6000)
[ 858.584883] Stack: f89a37be c0687880 0304 c022af6e c064f280
[ 858.593573] c06a2554 c064f280 0001
c064f280
[ 858.602259]c0860be0 c2a1fba0 0246 c06a2554 f240e880
f240e880 c04a
[ 858.610946] Call Trace:
[ 858.613723] [c022af6e] run_timer_softirq+0x37/0x16b
[ 858.618959] [c04a1c0f] dst_run_gc+0x0/0x118
[ 858.623498] [c04a3eab] neigh_destroy+0xbe/0x104
[ 858.628382] [c04a1bb1] dst_destroy+0x4d/0xab
[ 858.632998] [c04a1c64] dst_run_gc+0x55/0x118
[ 858.637620] [c022b03f] run_timer_softirq+0x108/0x16b
[ 858.642934] [c0227634] __do_softirq+0x5a/0xd5
[ 858.647648] [c023b435] trace_hardirqs_on+0x106/0x141
[ 858.652970] [c0227643] __do_softirq+0x69/0xd5
[ 858.657677] [c02276e6] do_softirq+0x37/0x4d
[ 858.662210] [c02167b0] smp_apic_timer_interrupt+0x6b/0x77
[ 858.667965] [c02029ef] default_idle+0x3b/0x54
[ 858.672681] [c02029ef] default_idle+0x3b/0x54
[ 858.677391] [c0204c33] apic_timer_interrupt+0x33/0x38
[ 858.682796] [c02029ef] default_idle+0x3b/0x54
[ 858.687505] [c02029f1] default_idle+0x3d/0x54
[ 858.692211] [c0202aaa] cpu_idle+0xa2/0xbb
[ 858.696569] [c06cd7c3] start_kernel+0x40b/0x413
[ 858.701453] [c06cd1b3] unknown_bootoption+0x0/0x205
[ 858.706678] ===
[ 858.710321] Code: 66 83 f8 20 74 29 0f b7 c0 89 44 24 08 89 6c 24 04 c7 04
24 be 37 9a
[ 858.730997] EIP: [f899bfa5] ipoib_neigh_destructor+0x40/0x178 [ib_ipoib]
SS:ESP 0068c
[ 858.740271] Kernel
Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
* Michael S. Tsirkin [EMAIL PROTECTED] wrote: So either there are other sites that instanciate those objects and forget about the lock init, or the object is corrupted (use after free?) OK, thanks for the hint. So I added this: And sure enough it triggers: [ 858.503010] ipoib_neigh_destructor lock c0687880 wrong type 772 !! could you turn on CONFIG_SLAB_DEBUG as well? that should catch certain types of use-after-free accesses, and lockdep will also warn if a still locked object is freed. Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
