Re: ipc,sem: sysv semaphore scalability

2013-05-06 Thread Jörn Engel
On Sat, 4 May 2013 20:12:45 +0200, Borislav Petkov wrote: > On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: > > Blockconsole currently lives here: > > https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ > > Tja, if only that were upstream... Linus has a pull request. If he

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Borislav Petkov
On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: > Blockconsole currently lives here: > https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ Tja, if only that were upstream... -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. -- -- To unsu

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Jörn Engel
On Sat, 23 March 2013 10:19:16 +0700, Emmanuel Benisty wrote: > > I could reproduce it but could you please let me know what would be > the right tools I should use to catch the original oops? > This is what I got but I doubt it will be helpful: > http://i.imgur.com/Mewi1hC.jpg You could use eith

Re: ipc,sem: sysv semaphore scalability

2013-05-03 Thread Peter Hurley
On 03/29/2013 03:01 PM, Dave Jones wrote: On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on

Re: ipc,sem: sysv semaphore scalability

2013-04-16 Thread Andrew Morton
On Tue, 26 Mar 2013 11:35:33 -0700 Andrew Morton wrote: > Do we need the locking at all? What does it actually do? > > sem_lock_and_putref(sma); > if (sma->sem_perm.deleted) { > sem_unlock(sma, -1); >

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Dave Jones
On Tue, Apr 02, 2013 at 03:53:01PM -0400, Sasha Levin wrote: > On 04/02/2013 01:52 PM, Linus Torvalds wrote: > > On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > >> > >> By just playing with the 'msgsz' parameter with MSG_COPY set. > > > > Hmm. Looking closer, I suspect you're testing w

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 04/02/2013 01:52 PM, Linus Torvalds wrote: > On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: >> >> By just playing with the 'msgsz' parameter with MSG_COPY set. > > Hmm. Looking closer, I suspect you're testing without commit > 88b9e456b164 ("ipc: don't allocate a copy larger than max"). Th

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > > By just playing with the 'msgsz' parameter with MSG_COPY set. Hmm. Looking closer, I suspect you're testing without commit 88b9e456b164 ("ipc: don't allocate a copy larger than max"). That should limit the size passed in to prepare_copy -> lo

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > > If you guys are already looking at this, the conversions between size_t, > long and int in the do_msgrcv/load_msg/alloc_msg code are a mess. You could > trigger anything from: Good catch. Let's just change the "(long)bufsz < 0" into "bufsz >

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 03/29/2013 03:36 PM, Peter Hurley wrote: > On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: >> On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: >>> >>> Here's an oops I just hit.. >>> >>> BUG: unable to handle kernel NULL pointer dereference at 000f >>> IP: [] testmsg.isr

Re: ipc,sem: sysv semaphore scalability

2013-04-01 Thread Stanislav Kinsbursky
29.03.2013 22:43, Linus Torvalds пишет: On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > >Now that I have that reverted, I'm not seeing msgrcv traces any more, but >I've started seeing this.. > >general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have CONFIG_CHECKPOINT_R

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Linus Torvalds
On Sun, Mar 31, 2013 at 6:45 AM, Rik van Riel wrote: > > Should we use "semid" here, like Linus suggested, instead of "un->semid"? As Davidlohr noted, in linux-next the rcu read-lock is held over the whole thing, so no, un->semid should be stable once "un" has been re-looked-up under the semaphor

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Emmanuel Benisty
Hi Davidlohr, On Sun, Mar 31, 2013 at 12:01 PM, Davidlohr Bueso wrote: > Specially dropping the rcu read lock before the continue statement > (sorry for not mentioning this in the last email). I was missing this indeed, thanks. Still the same issues however... I'll do some more testing on the sa

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Rik van Riel
On 03/31/2013 01:01 AM, Davidlohr Bueso wrote: diff --git a/ipc/sem.c b/ipc/sem.c index f257afe..74cedfe 100644 --- a/ipc/sem.c +++ b/ipc/sem.c @@ -1867,8 +1867,7 @@ void exit_sem(struct task_struct *tsk) struct sem_array *sma; struct sem_undo *un;

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 11:33 +0700, Emmanuel Benisty wrote: > On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds > wrote: > > On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty > > wrote: > >> > >> Then I start building a random package and the problems start. They > >> may also happen without compi

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Emmanuel Benisty
Hi Linus, On Sun, Mar 31, 2013 at 12:22 AM, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty > wrote: >> On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds >>> >>> This came from the gcc build? >> >> yes, very early in the build process, IIRC this line was repeated a >>

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty wrote: > On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds >> >> This came from the gcc build? > > yes, very early in the build process, IIRC this line was repeated a > few times and the build just stalled. Ok, we're bringing out the crazy hacks n

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds wrote: >> Another shot in >> the dark, I had this weird message when trying to build gcc: >> semop(2): encountered an error: Identifier removed > > This came from the gcc build? yes, very early in the build process, IIRC this line was repeated a fe

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:33 PM, Emmanuel Benisty wrote: > > I just tried the 7 original patches + the 2 one liners from -next + > modified Linus' patch (attached) .. that patch looks fine. > on the top of 3.9-rc4 using > PREEMPT_NONE and after moving sem_lock(sma, NULL, -1) as explained > above

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty wrote: >> >> Then I start building a random package and the problems start. They >> may also happen without compiling but this seems to trigger the bug >> quite quickly. > > I suspect it's

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty wrote: > > Then I start building a random package and the problems start. They > may also happen without compiling but this seems to trigger the bug > quite quickly. I suspect it's about preemption, and the build just results in enough scheduling

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Davidlohr, On Sat, Mar 30, 2013 at 9:08 AM, Davidlohr Bueso wrote: > Not sure which one liner you refer to, but, if you haven't already done > so, please try with these fixes (queued in linux-next): > > http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=a9cead0347283f3e

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Fri, 2013-03-29 at 19:09 -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty wrote: > > > > I had to slightly modify the patch since it wouldn't match the changes > > introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, > > hope that was the right th

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty wrote: > > I had to slightly modify the patch since it wouldn't match the changes > introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, > hope that was the right thing to do. So, what I tried was: original 7 > patches + the one lin

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 08:36 +0700, Emmanuel Benisty wrote: > Hi Linus, > > On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds > wrote: > > Emmanuel, can you try the attached patch? I think it applies cleanly > > on top of the scalability series too without any changes, but I didn't > > check if the

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Linus, On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds wrote: > Emmanuel, can you try the attached patch? I think it applies cleanly > on top of the scalability series too without any changes, but I didn't > check if the patches perhaps changed some of the naming or something. I had to slight

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 2:12 PM, Linus Torvalds wrote: > > I dunno. I'm still not sure this is triggerable, but it looks bad. But > both the semaphore case and the msg cases seem to be solvable by > moving the unlock down, and shm seem to have no getref/putref users to > race with, so this (whites

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 1:41 PM, Linus Torvalds wrote: > > The alternative would be to make sure the thing is always locked (and > in a rcu-read-safe region) before putref/getref. The only place (apart > from the initial allocation, which doesn't matter, because nothing can > see if itf that path

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > RIP: 0010:[] [] free_msg+0x2b/0x40 > Call Trace: > [] freeque+0xcf/0

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:33 PM, Peter Hurley wrote: > On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: >> I think I foud at least one bug in the MSG_COPY stuff: it leaks the >> "copy" allocation if >> >> mode == SEARCH_LESSEQUAL >> >> but maybe I'm misreading it. > > Yes, you're misr

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > > > Here's an oops I just hit.. > > > > BUG: unable to handle kernel NULL pointer dereference at 000f > > IP: [] testmsg.isra.5+0x1a/0x60 > > Btw, looking at the code lea

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: > I think I foud at least one bug in the MSG_COPY stuff: it leaks the > "copy" allocation if > > mode == SEARCH_LESSEQUAL > > but maybe I'm misreading it. Yes, you're misreading it. copy_msg() returns the 'copy' address when copying is

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > Here's an oops I just hit.. > > BUG: unable to handle kernel NULL pointer dereference at 000f > IP: [] testmsg.isra.5+0x1a/0x60 Btw, looking at the code leading up to this, what the f*ck is wrong with the IPC stuff? It's using t

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > Btw, something that's really bothering me is just how much bogus > 'follow-on' spew we have lately. I'm not sure what changed, but it > seems to have gotten worse. .. we have many more sanity checks, and they tend to trigger in the case of u

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:43:25AM -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > > I've started seeing this.. > > > > general protection fault: [#1] PREEMPT SMP DEBUG_

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on - this has been hanging > > > around for too

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have CONFIG_CHECKPOINT_RESTORE enabled? Does it go away if you d

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 11:04 AM, Dave Jones wrote: > > mainline. Your current tree. Ok, that's what I thought you were generally testing, just wanted to verify. The subject kind of implied otherwise.. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:00:27AM -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > > I've started seeing this.. > > > > general protection fault: [#1] PREEMPT SMP DEBUG_

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > > Looks like seg was already kfree'd. Just to clarify: is this you te

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on - this has been hanging > > > around for too

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Andrew Morton
On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > Whichever way we go, we should get a wiggle on - this has been hanging > > around for too long. Dave, do you have time to determine whether > > reverting 88b9e456b16497

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Dave Jones
On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > Whichever way we go, we should get a wiggle on - this has been hanging > around for too long. Dave, do you have time to determine whether > reverting 88b9e456b1649722673ff ("ipc: don't allocate a copy larger > than max") fixes t

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Andrew Morton
On Tue, 26 Mar 2013 10:59:27 -0700 Davidlohr Bueso wrote: > On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: > > On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds > > wrote: > > > And you never see this problem without Rik's patches? > > > > No, never. > > > > > Could you bisect > >

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 02:07 PM, Sasha Levin wrote: On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held,

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 01:59 PM, Davidlohr Bueso wrote: From: Davidlohr Bueso Subject: [PATCH] ipc, sem: prevent possible deadlock In semctl_main(), when cmd == GETALL, we're locking sma->sem_perm.lock (through sem_lock_and_putref), yet after the conditional, we lock it again. Unlock sma right after ex

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/20/2013 03:55 PM, Rik van Riel wrote: > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. Hi Rik, Another issue that came up is: [ 96.347341]

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: > On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: >> On 03/20/2013 03:55 PM, Rik van Riel wrote: >>> This series makes the sysv semaphore code more scalable, >>> by reducing the time the semaphore lock is held, and making >>> the locking more sca

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: > On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds > wrote: > > And you never see this problem without Rik's patches? > > No, never. > > > Could you bisect > > *which* patch it starts with? Are the first four ones ok (the moving > > of t

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Paul E. McKenney
On Tue, Mar 26, 2013 at 01:33:07PM -0400, Sasha Levin wrote: > On 03/20/2013 03:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with multiple >

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: > On 03/20/2013 03:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with multiple > > semap

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Sasha Levin
On 03/20/2013 03:55 PM, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphore

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 10:53 PM, Rik van Riel wrote: > On 03/25/2013 11:20 AM, Emmanuel Benisty wrote: >> >> On Mon, Mar 25, 2013 at 9:03 PM, Rik van Riel wrote: > > With the first four patches only, I got some X server freeze (just > tried once). Could you try bo

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 11:20 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 9:03 PM, Rik van Riel wrote: With the first four patches only, I got some X server freeze (just tried once). Could you try booting with panic=1 so the kernel panics on the first oops? Sorry that should be "oops=panic"

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 9:03 PM, Rik van Riel wrote: >>> With the first four patches only, I got some X server freeze (just >>> tried once). >> >> >> Could you try booting with panic=1 so the kernel panics on the first >> oops? > > > Sorry that should be "oops=panic" > > >> Maybe that way (if we a

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 9:01 PM, Rik van Riel wrote: > On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: >> >> On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds >> wrote: >>> >>> And you never see this problem without Rik's patches? >> >> >> No, never. >> >>> Could you bisect >>> *which* patch it st

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 10:00 AM, Rik van Riel wrote: On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: And you never see this problem without Rik's patches? No, never. Could you bisect *which* patch it starts with? Are the first four ones ok (th

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: And you never see this problem without Rik's patches? No, never. Could you bisect *which* patch it starts with? Are the first four ones ok (the moving of the locking around, but without t

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: And you never see this problem without Rik's patches? No, never. Could you bisect *which* patch it starts with? Are the first four ones ok (the moving of the locking around, but without t

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: > And you never see this problem without Rik's patches? No, never. > Could you bisect > *which* patch it starts with? Are the first four ones ok (the moving > of the locking around, but without the fine-grained ones), for > example? With t

Re: ipc,sem: sysv semaphore scalability

2013-03-24 Thread Linus Torvalds
On Sun, Mar 24, 2013 at 6:46 AM, Emmanuel Benisty wrote: > > Thanks Linus. I hope I got this right, here's the result (3.9-rc4, 7+1 > patches): http://i.imgur.com/BebGZxV.jpg Ok, that's *slightly* more informative, but not much. At least now we see the actual page fault information, and see what

Re: ipc,sem: sysv semaphore scalability

2013-03-24 Thread Emmanuel Benisty
On Sun, Mar 24, 2013 at 2:45 AM, Linus Torvalds wrote: > On Fri, Mar 22, 2013 at 8:19 PM, Emmanuel Benisty wrote: >> >> I could reproduce it but could you please let me know what would be >> the right tools I should use to catch the original oops? >> This is what I got but I doubt it will be help

Re: ipc,sem: sysv semaphore scalability

2013-03-23 Thread Linus Torvalds
On Fri, Mar 22, 2013 at 8:19 PM, Emmanuel Benisty wrote: > > I could reproduce it but could you please let me know what would be > the right tools I should use to catch the original oops? > This is what I got but I doubt it will be helpful: > http://i.imgur.com/Mewi1hC.jpg In this case, I think t

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Emmanuel Benisty
Hi Linus, On Fri, Mar 22, 2013 at 10:37 PM, Linus Torvalds wrote: > On Fri, Mar 22, 2013 at 4:04 AM, Emmanuel Benisty wrote: >> >> I was trying your patchset and my machine died while building a >> package. I could reproduce the bug the (only) two times I tried. >> There's a poor quality picture

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Davidlohr Bueso
On Wed, 2013-03-20 at 15:55 -0400, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Linus Torvalds
On Fri, Mar 22, 2013 at 4:04 AM, Emmanuel Benisty wrote: > > I was trying your patchset and my machine died while building a > package. I could reproduce the bug the (only) two times I tried. > There's a poor quality picture here: http://i.imgur.com/MuYuyQC.jpg Hmm. The original oops may well hav

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Emmanuel Benisty
Hi Rik, On Thu, Mar 21, 2013 at 2:55 AM, Rik van Riel wrote: > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. I was trying your patchset and my mac

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Mike Galbraith
On Wed, 2013-03-20 at 15:55 -0400, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Rik van Riel
On 03/21/2013 09:23 PM, Linus Torvalds wrote: On Thu, Mar 21, 2013 at 6:12 PM, Davidlohr Bueso wrote: ipc lock contention: 100 users: 8,74% (vanilla)3.17% (v3 patchset) 400 users: 21,86% (vanilla)5.23% (v3 patchset) 800 users 84,35% (vanilla)7.39% (v3 patchset) Ok, I'd call

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Rik van Riel
On 03/21/2013 06:01 PM, Andrew Morton wrote: On Thu, 21 Mar 2013 17:50:05 -0400 Peter Hurley wrote: On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Linus Torvalds
On Thu, Mar 21, 2013 at 6:12 PM, Davidlohr Bueso wrote: > > ipc lock contention: > 100 users: 8,74% (vanilla)3.17% (v3 patchset) > 400 users: 21,86% (vanilla)5.23% (v3 patchset) > 800 users 84,35% (vanilla)7.39% (v3 patchset) Ok, I'd call that pretty much "solved". Sure, it's sti

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Davidlohr Bueso
On Wed, 2013-03-20 at 15:55 -0400, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Andrew Morton
On Thu, 21 Mar 2013 17:50:05 -0400 Peter Hurley wrote: > On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: > > On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > > > > > This series makes the sysv semaphore code more scalable, > > > by reducing the time the semaphore lock is held,

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Peter Hurley
On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: > On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with m

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Peter Hurley
On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: > On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with m

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Andrew Morton
On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. > > The first four patches were written by

Re: ipc,sem: sysv semaphore scalability

2013-03-20 Thread Davidlohr Bueso
On Wed, 2013-03-20 at 13:49 -0700, Linus Torvalds wrote: > On Wed, Mar 20, 2013 at 12:55 PM, Rik van Riel wrote: > > > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with

Re: ipc,sem: sysv semaphore scalability

2013-03-20 Thread Linus Torvalds
On Wed, Mar 20, 2013 at 1:49 PM, Linus Torvalds wrote: > > It *would* be lovely to see this run with the actual Swingbench > numbers. The microbenchmark always looked much nicer. Do the > additional multi-semaphore scalability patches on top of Davidlohr's > patches help with the swingbench issue,

Re: ipc,sem: sysv semaphore scalability

2013-03-20 Thread Linus Torvalds
On Wed, Mar 20, 2013 at 12:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. The series looks sane to me, and I like how