Re: Scheduling in interrupt BUG. [Patch]
On Mon, May 14, 2001 at 08:32:33AM -0400, Michal Ostrowski wrote: > Having looked at the code for locking sockets I am concerned that the > locking procedures for tcp may be wrong. __release_sock releases the > socket spinlock before calling sk->backlog_rcv() (== tcp_v4_do_rcv), > however the comments at the top of tcp_v4_do_rcv() assert that the > socket's spinlock is held (which is definitely not the case). > > Anybody care to comment on this? Looks ok for me. The user socket lock (lock.users>0) is held while __release_sock runs, which is also sufficient to protect it as all new packets will go into backlog. The spinlock comment only applies to bottom halves. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Scheduling in interrupt BUG. [Patch]
Marcell Gal writes:
> Hi,
>
> This patch solved the problem. Should be ready for inclusion in 2.4.
> No more 'Scheduling in interrupt' under those conditions.
> Thanx for the thoughts, solution and the amazing speed.
> You guys are doing a really great job!
>
Alexey pointed out a much nicer/correct/elegant/efficient
solution to this problem and I think that that's the way to go.
New patch below.
Michal Ostrowski
[EMAIL PROTECTED]
--- drivers/net/pppoe.c~Tue Mar 6 22:44:35 2001
+++ drivers/net/pppoe.c Mon May 14 14:10:49 2001
@@ -5,7 +5,7 @@
* PPPoE --- PPP over Ethernet (RFC 2516)
*
*
- * Version:0.6.5
+ * Version:0.6.6
*
* 030700 : Fixed connect logic to allow for disconnect.
* 270700 :Fixed potential SMP problems; we must protect against
@@ -19,6 +19,7 @@
* 051000 :Initialization cleanup.
* 00 :Fix recvmsg.
* 050101 :Fix PADT procesing.
+ * 140501 :Use pppoe_rcv_core to handle all backlog. (Alexey)
*
* Author: Michal Ostrowski <[EMAIL PROTECTED]>
* Contributors:
@@ -376,22 +377,6 @@
return ret;
}
-
-/
- *
- * Receive wrapper called in process context.
- *
- ***/
-int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
-{
- lock_sock(sk);
- pppoe_rcv_core(sk, skb);
- release_sock(sk);
- return 0;
-}
-
-
-
/
*
* Receive a PPPoE Discovery frame.
@@ -481,7 +466,7 @@
sk->protocol = PX_PROTO_OE;
sk->family = PF_PPPOX;
- sk->backlog_rcv = pppoe_backlog_rcv;
+ sk->backlog_rcv = pppoe_rcv_core;
sk->next = NULL;
sk->pprev = NULL;
sk->state = PPPOX_NONE;
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Scheduling in interrupt BUG. [Patch]
Hi,
This patch solved the problem. Should be ready for inclusion in 2.4.
No more 'Scheduling in interrupt' under those conditions.
Thanx for the thoughts, solution and the amazing speed.
You guys are doing a really great job!
I hope we can get the earlier mentioned NULL ptr in all_ppp_units list
straight
soon. (I have a simple workaround - the mentioned hash, that even improves
speed,
but I a real fix would be more satisfaction. The relevant part of
ppp_generic.c
is so simple that it's really strange it is not correct.. ).
thanx:
Cell
Michal Ostrowski wrote:
> Anybody care to comment on this?
> [EMAIL PROTECTED]
--- linuxold/drivers/net/pppoe.cMon May 14 22:06:44 2001
+++ linux/drivers/net/pppoe.c Mon May 14 22:11:25 2001
@@ -4,9 +4,9 @@
* PPPoX --- Generic PPP encapsulation socket family
* PPPoE --- PPP over Ethernet (RFC 2516)
*
*
- * Version:0.6.5
+ * Version:0.6.6
*
* 030700 : Fixed connect logic to allow for disconnect.
* 270700 :Fixed potential SMP problems; we must protect against
* simultaneous invocation of ppp_input
@@ -18,8 +18,9 @@
* in pppoe_release.
* 051000 :Initialization cleanup.
* 00 :Fix recvmsg.
* 050101 :Fix PADT procesing.
+ * 140501 :pppoe_backlog_rcv must call bh_lock_sock, not lock_sock.
*
* Author: Michal Ostrowski <[EMAIL PROTECTED]>
* Contributors:
* Arnaldo Carvalho de Melo <[EMAIL PROTECTED]>
@@ -383,11 +384,11 @@
*
***/
int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
- lock_sock(sk);
+ bh_lock_sock(sk);
pppoe_rcv_core(sk, skb);
- release_sock(sk);
+ bh_unlock_sock(sk);
return 0;
}
--
You'll never see all the places, or read all the books, but fortunately,
they're not all recommended.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Scheduling in interrupt BUG.
Marcell GAL wrote:
>
> int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
> {
> lock_sock(sk);
> pppoe_rcv_core(sk, skb);
> release_sock(sk);
> return 0;
> }
>
The backlog_rcv() method is called inside local_bh_disable()
and so cannot call lock_sock(). Definitely a bug in pppoe.
It looks like pppoe_backlog_rcv() should be using bh_lock_sock().
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Scheduling in interrupt BUG. [Patch]
Marcell GAL writes:
> Hi Guys,
>
> Once upon a time on my
> x86 UP box, UP kernel 2.4.4, (64M ram, 260M swap)
> http://home.sch.bme.hu/~cell/.config
> I hit a reproducable "Scheduling in interrupt" BUG.
> Also reproduced with 128M ram and low memory pressure
> (first I suspected it is related to swapping)
> Running lots of pppd version 2.4.0 (pppoe) sessions almost at the same
> time.
> (before the crash the pppoe sessions work fine)
> It crashed after 89 sessions, 473 another time.. (depending
> on the phase of Jupiter moons I guess .. I still have to verify this),
> usually much before memory is exhausted (30k mem/pppd process).
> To do this you have to patch ppp_generic.c
> http://x-dsl.hu/~cell/ppp_generic_hash/, because
> otherwise we hit 'NULL ptr in all_ppp_units list'
> BUG much _more likely_ than this 'sched.c line 709 thingy'..
>
> EIP: c010faa4 <= sched.c schedule(), line 709:
> which is ~ printk("Scheduling in interrupt");BUG();
>From what I've seen, you have a call chain consisting of:
__release_sock -> pppoe_backlog_rcv -> __lock_sock
This is going to be bad, because when __release_sock calls
sk->backlog_rcv, lock.users is still non-zero and thus the lock_sock
operation will block (leading to deadlock). This problem is fixed
with the patch that I've added below.
You're seeing the "Scheduling in interrupt" message because the
combined effect of the various spin_lock/unlock calls in release_sock
and __release_sock at the time of the call to sk->backlog_rcv is to
increase the local bh count.
Having looked at the code for locking sockets I am concerned that the
locking procedures for tcp may be wrong. __release_sock releases the
socket spinlock before calling sk->backlog_rcv() (== tcp_v4_do_rcv),
however the comments at the top of tcp_v4_do_rcv() assert that the
socket's spinlock is held (which is definitely not the case).
Anybody care to comment on this?
Michal Ostrowski
[EMAIL PROTECTED]
--- drivers/net/pppoe.c~Tue Mar 6 22:44:35 2001
+++ drivers/net/pppoe.c Mon May 14 08:24:06 2001
@@ -5,7 +5,7 @@
* PPPoE --- PPP over Ethernet (RFC 2516)
*
*
- * Version:0.6.5
+ * Version:0.6.6
*
* 030700 : Fixed connect logic to allow for disconnect.
* 270700 :Fixed potential SMP problems; we must protect against
@@ -19,6 +19,7 @@
* 051000 :Initialization cleanup.
* 00 :Fix recvmsg.
* 050101 :Fix PADT procesing.
+ * 140501 :pppoe_backlog_rcv must call bh_lock_sock, not lock_sock.
*
* Author: Michal Ostrowski <[EMAIL PROTECTED]>
* Contributors:
@@ -384,9 +385,9 @@
***/
int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
- lock_sock(sk);
+ bh_lock_sock(sk);
pppoe_rcv_core(sk, skb);
- release_sock(sk);
+ bh_unlock_sock(sk);
return 0;
}
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Scheduling in interrupt BUG.
Hi Guys,
Once upon a time on my
x86 UP box, UP kernel 2.4.4, (64M ram, 260M swap)
http://home.sch.bme.hu/~cell/.config
I hit a reproducable "Scheduling in interrupt" BUG.
Also reproduced with 128M ram and low memory pressure
(first I suspected it is related to swapping)
Running lots of pppd version 2.4.0 (pppoe) sessions almost at the same
time.
(before the crash the pppoe sessions work fine)
It crashed after 89 sessions, 473 another time.. (depending
on the phase of Jupiter moons I guess .. I still have to verify this),
usually much before memory is exhausted (30k mem/pppd process).
To do this you have to patch ppp_generic.c
http://x-dsl.hu/~cell/ppp_generic_hash/, because
otherwise we hit 'NULL ptr in all_ppp_units list'
BUG much _more likely_ than this 'sched.c line 709 thingy'..
EIP: c010faa4<= sched.c schedule(), line 709:
which is ~ printk("Scheduling in interrupt");BUG();
Trace:
0xc01ddac5 <__lock_sock+53>:movl $0x0,0x1c(%esp,1)
0xc01ddacd <__lock_sock+61>:mov%ebx,0x20(%esp,1)
0xc01ddad1 <__lock_sock+65>:movl $0x0,0x24(%esp,1)
0xc01ddad9 <__lock_sock+73>:movl $0x0,0x28(%esp,1)
0xc01ddae1 <__lock_sock+81>:lea0x1c(%esp,1),%esi
0xc01ddae5 <__lock_sock+85>:lea0x34(%edi),%eax
0xc01ddae8 <__lock_sock+88>:mov%esi,%edx
0xc01ddaea <__lock_sock+90>:call 0xc0110598
0xc01ddaef <__lock_sock+95>:nop
0xc01ddaf0 <__lock_sock+96>:movl $0x2,(%ebx)
0xc01ddaf6 <__lock_sock+102>: decl 0xc02f75ec
0xc01ddafc <__lock_sock+108>: call 0xc010f71c
*
0xc01ddb01 <__lock_sock+113>: incl 0xc02f75ec
0xc01ddb07 <__lock_sock+119>: cmpl $0x0,0x30(%edi)
0xc01ddb0b <__lock_sock+123>: jne0xc01ddaf0 <__lock_sock+96>
-
0xc01a315c : push %esi
0xc01a315d : push %ebx
0xc01a315e : mov0xc(%esp,1),%ebx
0xc01a3162 : incl 0xc02f75ec
0xc01a3168 : cmpl $0x0,0x30(%ebx)
0xc01a316c :
je 0xc01a3177
0xc01a316e : push %ebx
0xc01a316f : call 0xc01dda90
<__lock_sock>
0xc01a3174 : add$0x4,%esp
0xc01a3177 : movl $0x1,0x30(%ebx)
0xc01a317e : decl 0xc02f75ec
0xc01a3184 : mov0x10(%esp,1),%eax
0xc01ddb2c <__release_sock>:push %esi
0xc01ddb2d <__release_sock+1>: push %ebx
0xc01ddb2e <__release_sock+2>: mov0xc(%esp,1),%esi
0xc01ddb32 <__release_sock+6>: mov0xb8(%esi),%eax
0xc01ddb38 <__release_sock+12>: movl $0x0,0xbc(%esi)
0xc01ddb42 <__release_sock+22>: movl $0x0,0xb8(%esi)
0xc01ddb4c <__release_sock+32>: lea0x0(%esi,1),%esi
0xc01ddb50 <__release_sock+36>: mov(%eax),%ebx
0xc01ddb52 <__release_sock+38>: movl $0x0,(%eax)
0xc01ddb58 <__release_sock+44>: push %eax
0xc01ddb59 <__release_sock+45>: push %esi
0xc01ddb5a <__release_sock+46>: mov0x31c(%esi),%eax
0xc01ddb60 <__release_sock+52>: call *%eax
0xc01ddb62 <__release_sock+54>: mov%ebx,%eax
0xc01ddb64 <__release_sock+56>: add$0x8,%esp
0xc01ddb67 <__release_sock+59>: test %eax,%eax
int pppoe_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
lock_sock(sk);
pppoe_rcv_core(sk, skb);
release_sock(sk);
return 0;
}
What else should I check? How can we fix it?
PPPoE is more and more frequently used nowadays because of ADSL
services. I think this can effect its stability (guess which direction
;-)
even with one session (though probably not that bad as with many
sessions).
Have a nice week:
Cell
--
Alice? WTFIA?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
