Re: ipc,sem: sysv semaphore scalability

2013-05-06 Thread Jörn Engel
On Sat, 4 May 2013 20:12:45 +0200, Borislav Petkov wrote: > On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: > > Blockconsole currently lives here: > > https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ > > Tja, if only that were upstream... Linus has a pull request. If he

Re: ipc,sem: sysv semaphore scalability

2013-05-06 Thread Jörn Engel
On Sat, 4 May 2013 20:12:45 +0200, Borislav Petkov wrote: On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: Blockconsole currently lives here: https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ Tja, if only that were upstream... Linus has a pull request. If he

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Borislav Petkov
On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: > Blockconsole currently lives here: > https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ Tja, if only that were upstream... -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. -- -- To

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Jörn Engel
On Sat, 23 March 2013 10:19:16 +0700, Emmanuel Benisty wrote: > > I could reproduce it but could you please let me know what would be > the right tools I should use to catch the original oops? > This is what I got but I doubt it will be helpful: > http://i.imgur.com/Mewi1hC.jpg You could use

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Jörn Engel
On Sat, 23 March 2013 10:19:16 +0700, Emmanuel Benisty wrote: I could reproduce it but could you please let me know what would be the right tools I should use to catch the original oops? This is what I got but I doubt it will be helpful: http://i.imgur.com/Mewi1hC.jpg You could use either

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Borislav Petkov
On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: Blockconsole currently lives here: https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ Tja, if only that were upstream... -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. -- -- To

Re: ipc,sem: sysv semaphore scalability

2013-05-03 Thread Peter Hurley
On 03/29/2013 03:01 PM, Dave Jones wrote: On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle

Re: ipc,sem: sysv semaphore scalability

2013-05-03 Thread Peter Hurley
On 03/29/2013 03:01 PM, Dave Jones wrote: On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones da...@redhat.com wrote: On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: Whichever way we go, we should get a

Re: ipc,sem: sysv semaphore scalability

2013-04-16 Thread Andrew Morton
On Tue, 26 Mar 2013 11:35:33 -0700 Andrew Morton wrote: > Do we need the locking at all? What does it actually do? > > sem_lock_and_putref(sma); > if (sma->sem_perm.deleted) { > sem_unlock(sma, -1); >

Re: ipc,sem: sysv semaphore scalability

2013-04-16 Thread Andrew Morton
On Tue, 26 Mar 2013 11:35:33 -0700 Andrew Morton a...@linux-foundation.org wrote: Do we need the locking at all? What does it actually do? sem_lock_and_putref(sma); if (sma-sem_perm.deleted) { sem_unlock(sma, -1);

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Dave Jones
On Tue, Apr 02, 2013 at 03:53:01PM -0400, Sasha Levin wrote: > On 04/02/2013 01:52 PM, Linus Torvalds wrote: > > On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > >> > >> By just playing with the 'msgsz' parameter with MSG_COPY set. > > > > Hmm. Looking closer, I suspect you're testing

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 04/02/2013 01:52 PM, Linus Torvalds wrote: > On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: >> >> By just playing with the 'msgsz' parameter with MSG_COPY set. > > Hmm. Looking closer, I suspect you're testing without commit > 88b9e456b164 ("ipc: don't allocate a copy larger than max").

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > > By just playing with the 'msgsz' parameter with MSG_COPY set. Hmm. Looking closer, I suspect you're testing without commit 88b9e456b164 ("ipc: don't allocate a copy larger than max"). That should limit the size passed in to prepare_copy ->

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > > If you guys are already looking at this, the conversions between size_t, > long and int in the do_msgrcv/load_msg/alloc_msg code are a mess. You could > trigger anything from: Good catch. Let's just change the "(long)bufsz < 0" into "bufsz

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 03/29/2013 03:36 PM, Peter Hurley wrote: > On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: >> On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: >>> >>> Here's an oops I just hit.. >>> >>> BUG: unable to handle kernel NULL pointer dereference at 000f >>> IP: []

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 03/29/2013 03:36 PM, Peter Hurley wrote: On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones da...@redhat.com wrote: Here's an oops I just hit.. BUG: unable to handle kernel NULL pointer dereference at 000f IP:

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin sasha.le...@oracle.com wrote: If you guys are already looking at this, the conversions between size_t, long and int in the do_msgrcv/load_msg/alloc_msg code are a mess. You could trigger anything from: Good catch. Let's just change the (long)bufsz

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin sasha.le...@oracle.com wrote: By just playing with the 'msgsz' parameter with MSG_COPY set. Hmm. Looking closer, I suspect you're testing without commit 88b9e456b164 (ipc: don't allocate a copy larger than max). That should limit the size passed in to

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 04/02/2013 01:52 PM, Linus Torvalds wrote: On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin sasha.le...@oracle.com wrote: By just playing with the 'msgsz' parameter with MSG_COPY set. Hmm. Looking closer, I suspect you're testing without commit 88b9e456b164 (ipc: don't allocate a copy larger

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Dave Jones
On Tue, Apr 02, 2013 at 03:53:01PM -0400, Sasha Levin wrote: On 04/02/2013 01:52 PM, Linus Torvalds wrote: On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin sasha.le...@oracle.com wrote: By just playing with the 'msgsz' parameter with MSG_COPY set. Hmm. Looking closer, I suspect you're

Re: ipc,sem: sysv semaphore scalability

2013-04-01 Thread Stanislav Kinsbursky
29.03.2013 22:43, Linus Torvalds пишет: On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > >Now that I have that reverted, I'm not seeing msgrcv traces any more, but >I've started seeing this.. > >general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have

Re: ipc,sem: sysv semaphore scalability

2013-04-01 Thread Stanislav Kinsbursky
29.03.2013 22:43, Linus Torvalds пишет: On Fri, Mar 29, 2013 at 9:17 AM, Dave Jonesda...@redhat.com wrote: Now that I have that reverted, I'm not seeing msgrcv traces any more, but I've started seeing this.. general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Linus Torvalds
On Sun, Mar 31, 2013 at 6:45 AM, Rik van Riel wrote: > > Should we use "semid" here, like Linus suggested, instead of "un->semid"? As Davidlohr noted, in linux-next the rcu read-lock is held over the whole thing, so no, un->semid should be stable once "un" has been re-looked-up under the

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Emmanuel Benisty
Hi Davidlohr, On Sun, Mar 31, 2013 at 12:01 PM, Davidlohr Bueso wrote: > Specially dropping the rcu read lock before the continue statement > (sorry for not mentioning this in the last email). I was missing this indeed, thanks. Still the same issues however... I'll do some more testing on the

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Rik van Riel
On 03/31/2013 01:01 AM, Davidlohr Bueso wrote: diff --git a/ipc/sem.c b/ipc/sem.c index f257afe..74cedfe 100644 --- a/ipc/sem.c +++ b/ipc/sem.c @@ -1867,8 +1867,7 @@ void exit_sem(struct task_struct *tsk) struct sem_array *sma; struct sem_undo *un;

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Rik van Riel
On 03/31/2013 01:01 AM, Davidlohr Bueso wrote: diff --git a/ipc/sem.c b/ipc/sem.c index f257afe..74cedfe 100644 --- a/ipc/sem.c +++ b/ipc/sem.c @@ -1867,8 +1867,7 @@ void exit_sem(struct task_struct *tsk) struct sem_array *sma; struct sem_undo *un;

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Emmanuel Benisty
Hi Davidlohr, On Sun, Mar 31, 2013 at 12:01 PM, Davidlohr Bueso davidlohr.bu...@hp.com wrote: Specially dropping the rcu read lock before the continue statement (sorry for not mentioning this in the last email). I was missing this indeed, thanks. Still the same issues however... I'll do some

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Linus Torvalds
On Sun, Mar 31, 2013 at 6:45 AM, Rik van Riel r...@surriel.com wrote: Should we use semid here, like Linus suggested, instead of un-semid? As Davidlohr noted, in linux-next the rcu read-lock is held over the whole thing, so no, un-semid should be stable once un has been re-looked-up under the

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 11:33 +0700, Emmanuel Benisty wrote: > On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds > wrote: > > On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty > > wrote: > >> > >> Then I start building a random package and the problems start. They > >> may also happen without

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Emmanuel Benisty
Hi Linus, On Sun, Mar 31, 2013 at 12:22 AM, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty > wrote: >> On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds >>> >>> This came from the gcc build? >> >> yes, very early in the build process, IIRC this line was repeated a >>

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty wrote: > On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds >> >> This came from the gcc build? > > yes, very early in the build process, IIRC this line was repeated a > few times and the build just stalled. Ok, we're bringing out the crazy hacks

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty benist...@gmail.com wrote: On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds This came from the gcc build? yes, very early in the build process, IIRC this line was repeated a few times and the build just stalled. Ok, we're bringing out the

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Emmanuel Benisty
Hi Linus, On Sun, Mar 31, 2013 at 12:22 AM, Linus Torvalds torva...@linux-foundation.org wrote: On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty benist...@gmail.com wrote: On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds This came from the gcc build? yes, very early in the build

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 11:33 +0700, Emmanuel Benisty wrote: On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds torva...@linux-foundation.org wrote: On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty benist...@gmail.com wrote: Then I start building a random package and the problems start.

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds wrote: >> Another shot in >> the dark, I had this weird message when trying to build gcc: >> semop(2): encountered an error: Identifier removed > > This came from the gcc build? yes, very early in the build process, IIRC this line was repeated a

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:33 PM, Emmanuel Benisty wrote: > > I just tried the 7 original patches + the 2 one liners from -next + > modified Linus' patch (attached) .. that patch looks fine. > on the top of 3.9-rc4 using > PREEMPT_NONE and after moving sem_lock(sma, NULL, -1) as explained >

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty wrote: >> >> Then I start building a random package and the problems start. They >> may also happen without compiling but this seems to trigger the bug >> quite quickly. > > I suspect it's

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty wrote: > > Then I start building a random package and the problems start. They > may also happen without compiling but this seems to trigger the bug > quite quickly. I suspect it's about preemption, and the build just results in enough scheduling

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Davidlohr, On Sat, Mar 30, 2013 at 9:08 AM, Davidlohr Bueso wrote: > Not sure which one liner you refer to, but, if you haven't already done > so, please try with these fixes (queued in linux-next): > >

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Fri, 2013-03-29 at 19:09 -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty wrote: > > > > I had to slightly modify the patch since it wouldn't match the changes > > introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, > > hope that was the right

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty wrote: > > I had to slightly modify the patch since it wouldn't match the changes > introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, > hope that was the right thing to do. So, what I tried was: original 7 > patches + the one

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 08:36 +0700, Emmanuel Benisty wrote: > Hi Linus, > > On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds > wrote: > > Emmanuel, can you try the attached patch? I think it applies cleanly > > on top of the scalability series too without any changes, but I didn't > > check if the

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Linus, On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds wrote: > Emmanuel, can you try the attached patch? I think it applies cleanly > on top of the scalability series too without any changes, but I didn't > check if the patches perhaps changed some of the naming or something. I had to

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 2:12 PM, Linus Torvalds wrote: > > I dunno. I'm still not sure this is triggerable, but it looks bad. But > both the semaphore case and the msg cases seem to be solvable by > moving the unlock down, and shm seem to have no getref/putref users to > race with, so this

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 1:41 PM, Linus Torvalds wrote: > > The alternative would be to make sure the thing is always locked (and > in a rcu-read-safe region) before putref/getref. The only place (apart > from the initial allocation, which doesn't matter, because nothing can > see if itf that path

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > RIP: 0010:[] [] free_msg+0x2b/0x40 > Call Trace: > []

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:33 PM, Peter Hurley wrote: > On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: >> I think I foud at least one bug in the MSG_COPY stuff: it leaks the >> "copy" allocation if >> >> mode == SEARCH_LESSEQUAL >> >> but maybe I'm misreading it. > > Yes, you're

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > > > Here's an oops I just hit.. > > > > BUG: unable to handle kernel NULL pointer dereference at 000f > > IP: [] testmsg.isra.5+0x1a/0x60 > > Btw, looking at the code

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: > I think I foud at least one bug in the MSG_COPY stuff: it leaks the > "copy" allocation if > > mode == SEARCH_LESSEQUAL > > but maybe I'm misreading it. Yes, you're misreading it. copy_msg() returns the 'copy' address when copying

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > Here's an oops I just hit.. > > BUG: unable to handle kernel NULL pointer dereference at 000f > IP: [] testmsg.isra.5+0x1a/0x60 Btw, looking at the code leading up to this, what the f*ck is wrong with the IPC stuff? It's using

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > Btw, something that's really bothering me is just how much bogus > 'follow-on' spew we have lately. I'm not sure what changed, but it > seems to have gotten worse. .. we have many more sanity checks, and they tend to trigger in the case of

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:43:25AM -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > > I've started seeing this.. > > > > general protection fault: [#1] PREEMPT SMP

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on - this has been hanging > > > around for

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have CONFIG_CHECKPOINT_RESTORE enabled? Does it go away if you

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 11:04 AM, Dave Jones wrote: > > mainline. Your current tree. Ok, that's what I thought you were generally testing, just wanted to verify. The subject kind of implied otherwise.. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:00:27AM -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > > I've started seeing this.. > > > > general protection fault: [#1] PREEMPT SMP

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > > Looks like seg was already kfree'd. Just to clarify: is this you

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on - this has been hanging > > > around for

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones da...@redhat.com wrote: On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: Whichever way we go, we should get a wiggle on - this has been hanging around

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones da...@redhat.com wrote: Now that I have that reverted, I'm not seeing msgrcv traces any more, but I've started seeing this.. general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Looks like seg was already kfree'd. Just to clarify: is

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:00:27AM -0700, Linus Torvalds wrote: On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones da...@redhat.com wrote: Now that I have that reverted, I'm not seeing msgrcv traces any more, but I've started seeing this.. general protection fault: [#1] PREEMPT SMP

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 11:04 AM, Dave Jones da...@redhat.com wrote: mainline. Your current tree. Ok, that's what I thought you were generally testing, just wanted to verify. The subject kind of implied otherwise.. Linus -- To unsubscribe from this list: send the line unsubscribe

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones da...@redhat.com wrote: Now that I have that reverted, I'm not seeing msgrcv traces any more, but I've started seeing this.. general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have CONFIG_CHECKPOINT_RESTORE enabled? Does it go

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones da...@redhat.com wrote: On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: Whichever way we go, we should get a wiggle on - this has been hanging around

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:43:25AM -0700, Linus Torvalds wrote: On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones da...@redhat.com wrote: Now that I have that reverted, I'm not seeing msgrcv traces any more, but I've started seeing this.. general protection fault: [#1] PREEMPT SMP

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones da...@redhat.com wrote: Btw, something that's really bothering me is just how much bogus 'follow-on' spew we have lately. I'm not sure what changed, but it seems to have gotten worse. .. we have many more sanity checks, and they tend to trigger in

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones da...@redhat.com wrote: Here's an oops I just hit.. BUG: unable to handle kernel NULL pointer dereference at 000f IP: [812c24ca] testmsg.isra.5+0x1a/0x60 Btw, looking at the code leading up to this, what the f*ck is wrong with

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: I think I foud at least one bug in the MSG_COPY stuff: it leaks the copy allocation if mode == SEARCH_LESSEQUAL but maybe I'm misreading it. Yes, you're misreading it. copy_msg() returns the 'copy' address when copying is

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones da...@redhat.com wrote: Here's an oops I just hit.. BUG: unable to handle kernel NULL pointer dereference at 000f IP: [812c24ca] testmsg.isra.5+0x1a/0x60 Btw,

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:33 PM, Peter Hurley pe...@hurleysoftware.com wrote: On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: I think I foud at least one bug in the MSG_COPY stuff: it leaks the copy allocation if mode == SEARCH_LESSEQUAL but maybe I'm misreading it. Yes,

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones da...@redhat.com wrote: Now that I have that reverted, I'm not seeing msgrcv traces any more, but I've started seeing this.. general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC RIP: 0010:[812c20fb] [812c20fb]

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 1:41 PM, Linus Torvalds torva...@linux-foundation.org wrote: The alternative would be to make sure the thing is always locked (and in a rcu-read-safe region) before putref/getref. The only place (apart from the initial allocation, which doesn't matter, because nothing

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 2:12 PM, Linus Torvalds torva...@linux-foundation.org wrote: I dunno. I'm still not sure this is triggerable, but it looks bad. But both the semaphore case and the msg cases seem to be solvable by moving the unlock down, and shm seem to have no getref/putref users to

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Linus, On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds torva...@linux-foundation.org wrote: Emmanuel, can you try the attached patch? I think it applies cleanly on top of the scalability series too without any changes, but I didn't check if the patches perhaps changed some of the naming or

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 08:36 +0700, Emmanuel Benisty wrote: Hi Linus, On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds torva...@linux-foundation.org wrote: Emmanuel, can you try the attached patch? I think it applies cleanly on top of the scalability series too without any changes, but I

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty benist...@gmail.com wrote: I had to slightly modify the patch since it wouldn't match the changes introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, hope that was the right thing to do. So, what I tried was: original 7

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Fri, 2013-03-29 at 19:09 -0700, Linus Torvalds wrote: On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty benist...@gmail.com wrote: I had to slightly modify the patch since it wouldn't match the changes introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, hope that was

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Davidlohr, On Sat, Mar 30, 2013 at 9:08 AM, Davidlohr Bueso davidlohr.bu...@hp.com wrote: Not sure which one liner you refer to, but, if you haven't already done so, please try with these fixes (queued in linux-next):

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty benist...@gmail.com wrote: Then I start building a random package and the problems start. They may also happen without compiling but this seems to trigger the bug quite quickly. I suspect it's about preemption, and the build just results in

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds torva...@linux-foundation.org wrote: On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty benist...@gmail.com wrote: Then I start building a random package and the problems start. They may also happen without compiling but this seems to trigger the

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:33 PM, Emmanuel Benisty benist...@gmail.com wrote: I just tried the 7 original patches + the 2 one liners from -next + modified Linus' patch (attached) .. that patch looks fine. on the top of 3.9-rc4 using PREEMPT_NONE and after moving sem_lock(sma, NULL, -1) as

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds torva...@linux-foundation.org wrote: Another shot in the dark, I had this weird message when trying to build gcc: semop(2): encountered an error: Identifier removed This came from the gcc build? yes, very early in the build process, IIRC this

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Andrew Morton
On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > Whichever way we go, we should get a wiggle on - this has been hanging > > around for too long. Dave, do you have time to determine whether > > reverting

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Dave Jones
On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > Whichever way we go, we should get a wiggle on - this has been hanging > around for too long. Dave, do you have time to determine whether > reverting 88b9e456b1649722673ff ("ipc: don't allocate a copy larger > than max") fixes

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Andrew Morton
On Tue, 26 Mar 2013 10:59:27 -0700 Davidlohr Bueso wrote: > On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: > > On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds > > wrote: > > > And you never see this problem without Rik's patches? > > > > No, never. > > > > > Could you bisect > >

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 02:07 PM, Sasha Levin wrote: On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held,

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 01:59 PM, Davidlohr Bueso wrote: From: Davidlohr Bueso Subject: [PATCH] ipc, sem: prevent possible deadlock In semctl_main(), when cmd == GETALL, we're locking sma->sem_perm.lock (through sem_lock_and_putref), yet after the conditional, we lock it again. Unlock sma right after

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/20/2013 03:55 PM, Rik van Riel wrote: > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. Hi Rik, Another issue that came up is: [ 96.347341]

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: > On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: >> On 03/20/2013 03:55 PM, Rik van Riel wrote: >>> This series makes the sysv semaphore code more scalable, >>> by reducing the time the semaphore lock is held, and making >>> the locking more

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: > On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds > wrote: > > And you never see this problem without Rik's patches? > > No, never. > > > Could you bisect > > *which* patch it starts with? Are the first four ones ok (the moving > > of

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Paul E. McKenney
On Tue, Mar 26, 2013 at 01:33:07PM -0400, Sasha Levin wrote: > On 03/20/2013 03:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with multiple >

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: > On 03/20/2013 03:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with multiple > >

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held, and making the locking more scalable for semaphore arrays with multiple semaphores.

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Paul E. McKenney
On Tue, Mar 26, 2013 at 01:33:07PM -0400, Sasha Levin wrote: On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held, and making the locking more scalable for semaphore arrays with multiple

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds torva...@linux-foundation.org wrote: And you never see this problem without Rik's patches? No, never. Could you bisect *which* patch it starts with? Are the first four ones ok

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held, and making the locking more scalable for

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held, and making the locking more scalable for semaphore arrays with multiple semaphores. Hi Rik, Another issue that came up is: [ 96.347341]

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 01:59 PM, Davidlohr Bueso wrote: From: Davidlohr Bueso davidlohr.bu...@hp.com Subject: [PATCH] ipc, sem: prevent possible deadlock In semctl_main(), when cmd == GETALL, we're locking sma-sem_perm.lock (through sem_lock_and_putref), yet after the conditional, we lock it again.

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 02:07 PM, Sasha Levin wrote: On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held,

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Andrew Morton
On Tue, 26 Mar 2013 10:59:27 -0700 Davidlohr Bueso davidlohr.bu...@hp.com wrote: On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds torva...@linux-foundation.org wrote: And you never see this problem without Rik's patches? No,

  1   2   >