Re: [announce] CFS-devel, performance improvements
Hi Ingo, When compiling, I get: In file included from kernel/sched.c:794: kernel/sched_fair.c: In function 'task_new_fair': kernel/sched_fair.c:857: error: 'sysctl_sched_child_runs_first' undeclared (first use in this function) kernel/sched_fair.c:857: error: (Each undeclared identifier is reported only once kernel/sched_fair.c:857: error: for each function it appears in.) Presumably because sched_fair.c is being included into sched.c before sysctl_sched_child_runs_first is defined. Regards, Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [announce] CFS-devel, performance improvements
On 9/11/07, Rob Hussey <[EMAIL PROTECTED]> wrote: > Hi Ingo, > > When compiling, I get: Yeah, this was my fault :( I've had a chance to test this now, and everything feels great. I did some benchmarks for 2.6.23-rc1, 2.6.23-rc6-cfs, and 2.6.23-rc6-cfs-devel: lat_ctx -s 0 2: 2.6.23-rc1 2.6.23-rc6-cfs 2.6.23-rc6-cfs-devel 5.154.91 5.05 5.235.18 4.85 5.194.89 5.17 5.365.23 4.86 5.355.00 5.13 5.345.05 5.12 5.264.99 5.06 5.115.04 4.96 5.295.19 5.18 5.404.93 5.07 hackbench 50: 2.6.23-rc1 2.6.23-rc6-cfs 2.6.23-rc6-cfs-devel 6.301 5.963 5.837 6.417 5.961 5.814 6.468 5.965 5.757 6.525 5.926 5.840 6.320 5.929 5.751 6.457 5.909 5.825 pipe-test (http://redhat.com/~mingo/cfs-scheduler/tools/pipe-test.c): 2.6.23-rc1 2.6.23-rc6-cfs 2.6.23-rc6-cfs-devel 14.29 14.03 13.89 14.31 14.01 14.10 14.27 13.99 14.15 14.31 14.02 14.16 14.53 14.02 14.14 14.53 14.27 14.16 14.51 14.36 14.12 14.48 14.33 14.16 14.52 14.36 14.17 14.47 14.36 14.15 I turned the results into graphs as well. I'll attach them, but they're also at: http://www.healthcarelinen.com/misc/lat_ctx_benchmark.png http://www.healthcarelinen.com/misc/hackbench_benchmark.png http://www.healthcarelinen.com/misc/pipe-test_benchmark.png The hackbench and pipe-test numbers are very encouraging. The avg between the 2.6.23-rc6-cfs and 2.6.23-rc6-cfs-devel lat_ctx numbers are nearly identical (5.041 and 5.045 respectively). <><><>
Re: [announce] CFS-devel, performance improvements
On 9/13/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > * Rob Hussey <[EMAIL PROTECTED]> wrote: > > > On 9/13/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > > > > > thanks for the numbers! Could you please also post the .config you used? > > > > Sure, .config for 2.6.23-rc1 and 2.6.23-rc6 attached. > > thx! If you've got some time, could you perhaps re-measure with these > disabled: > > CONFIG_SCHED_DEBUG=y Well, I was going over my config myself after you asked for me to post it, and I thought to do the same thing. Except, disabling sched_debug caused the same error as before: In file included from kernel/sched.c:794: kernel/sched_fair.c: In function 'task_new_fair': kernel/sched_fair.c:857: error: 'sysctl_sched_child_runs_first' undeclared (first use in this function) kernel/sched_fair.c:857: error: (Each undeclared identifier is reported only once kernel/sched_fair.c:857: error: for each function it appears in.) make[1]: *** [kernel/sched.o] Error 1 make: *** [kernel] Error 2 It only happens with sched_debug=y. I take it back, it wasn't my fault :) As for everything else, I'd be happy to. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [announce] CFS-devel, performance improvements
On 9/13/07, Rob Hussey <[EMAIL PROTECTED]> wrote: > On 9/13/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > > > * Rob Hussey <[EMAIL PROTECTED]> wrote: > > > > > On 9/13/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > > > > > > > thanks for the numbers! Could you please also post the .config you used? > > > > > > Sure, .config for 2.6.23-rc1 and 2.6.23-rc6 attached. > > > > thx! If you've got some time, could you perhaps re-measure with these > > disabled: > > > > CONFIG_SCHED_DEBUG=y > > Well, I was going over my config myself after you asked for me to post > it, and I thought to do the same thing. Except, disabling sched_debug > caused the same error as before: > In file included from kernel/sched.c:794: > kernel/sched_fair.c: In function 'task_new_fair': > kernel/sched_fair.c:857: error: 'sysctl_sched_child_runs_first' > undeclared (first use in this function) > kernel/sched_fair.c:857: error: (Each undeclared identifier is > reported only once > kernel/sched_fair.c:857: error: for each function it appears in.) > make[1]: *** [kernel/sched.o] Error 1 > make: *** [kernel] Error 2 > > It only happens with sched_debug=y. I take it back, it wasn't my fault :) > I'm trying the patches now to see if they help. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [announce] CFS-devel, performance improvements
On 9/13/07, Rob Hussey <[EMAIL PROTECTED]> wrote: > On 9/13/07, Rob Hussey <[EMAIL PROTECTED]> wrote: > > On 9/13/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > > > > > * Rob Hussey <[EMAIL PROTECTED]> wrote: > > > > > > > On 9/13/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > > > > > > > > > thanks for the numbers! Could you please also post the .config you > > > > > used? > > > > > > > > Sure, .config for 2.6.23-rc1 and 2.6.23-rc6 attached. > > > > > > thx! If you've got some time, could you perhaps re-measure with these > > > disabled: > > > > > > CONFIG_SCHED_DEBUG=y > > > > Well, I was going over my config myself after you asked for me to post > > it, and I thought to do the same thing. Except, disabling sched_debug > > caused the same error as before: > > In file included from kernel/sched.c:794: > > kernel/sched_fair.c: In function 'task_new_fair': > > kernel/sched_fair.c:857: error: 'sysctl_sched_child_runs_first' > > undeclared (first use in this function) > > kernel/sched_fair.c:857: error: (Each undeclared identifier is > > reported only once > > kernel/sched_fair.c:857: error: for each function it appears in.) > > make[1]: *** [kernel/sched.o] Error 1 > > make: *** [kernel] Error 2 > > > > It only happens with sched_debug=y. I take it back, it wasn't my fault :) > > > I'm trying the patches now to see if they help. > Current cfs-devel git compiles fine without sched_debug. Not sure how I broke things, but I need some sleep. I know the 2.6.23-rc1 numbers were good, but not sure about the others. I'll make the changes you suggested, and get some new and hopefully good numbers for 2.6.23-rc6-cfs and 2.6.23-rc6-cfs-devel. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [announce] CFS-devel, performance improvements
On 9/13/07, Rob Hussey <[EMAIL PROTECTED]> wrote: > Bound to single core: ... > hackbench 50 > # rc1 rc6 cfs-devel > 1 7.528 7.950 7.538 > 2 7.649 8.026 7.548 > 3 7.613 8.160 7.580 > 4 7.550 8.054 7.558 > 5 7.563 8.373 7.559 > 6 7.617 8.152 7.550 > 7 7.593 7.831 7.562 > 8 7.602 8.311 7.588 > 9 7.589 8.010 7.552 > 10 7.682 8.059 7.556 > I knew there was no way I'd post all these numbers and not screw something up. Switch rc6 and rc1 for hackbench 50 (bound to single core). Updated graph: http://www.healthcarelinen.com/misc/BOUND_hackbench_benchmark_fixed.png Also attached. <>
Re: 2.6.23-rc6 : crash with RTL8187 USB
On 9/15/07, ポール・ロラン Paul Rolland <[EMAIL PROTECTED]> wrote: > Hello, > > Each time I add the support for this piece of hardware, I have a crash during > the boot process. > Serial console gives the attached boot message... > ... > WARNING: at lib/kref.c:33 kref_get() > Call Trace: > [] kref_get+0x2f/0x36 > [] kobject_get+0x12/0x17 > [] kobject_shadow_add+0x7f/0x1a2 > [] device_add+0xb7/0x631 > [] msleep+0x14/0x1e > [] wiphy_register+0x29/0x80 > [] ieee80211_register_hw+0x11/0x208 > [] rtl8187_probe+0x544/0x624 I had this problem as well. It has to do with mac80211, cfg80211 and the rate control algorithm not initializing early enough in the boot process. These two patches should fix it: [PATCH] mac80211: fix initialisation when built-in http://article.gmane.org/gmane.linux.kernel.wireless.general/5710/match=patch+mac80211+initialisation [PATCH] cfg80211: fix initialisation if built-in http://article.gmane.org/gmane.linux.network/71326/match=patch+cfg80211+initialisation Regards, Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: rtl8187 driver in 2.6.23-rc6-git5: kernel panic if not used as a module. Works as a module.
On 9/15/07, Eric Valette <[EMAIL PROTECTED]> wrote: > Eric Valette wrote: > > > I can probably take a picture of the backtrace if you want. > > Just saw that just above my message in the LKML web interface, someone > posted a backtrace. Mine is different but at least, we are at least two > to have the crash. This is the same thing I said to Paul Rolland, since I think your problems are the same: I had this problem as well. It has to do with mac80211, cfg80211 and the rate control algorithm not initializing early enough in the boot process. These two patches should fix it: [PATCH] mac80211: fix initialisation when built-in http://article.gmane.org/gmane.linux.kernel.wireless.general/5710/match=patch+mac80211+initialisation [PATCH] cfg80211: fix initialisation if built-in http://article.gmane.org/gmane.linux.network/71326/match=patch+cfg80211+initialisation Regards, Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: rtl8187 driver in 2.6.23-rc6-git5: kernel panic if not used as a module. Works as a module.
On 9/15/07, ポール・ロラン Paul Rolland <[EMAIL PROTECTED]> wrote: > Hi Eric, > > On Sat, 15 Sep 2007 20:30:14 +0200 > Eric Valette <[EMAIL PROTECTED]> wrote: > > > Rob Hussey wrote: > > > On 9/15/07, Eric Valette <[EMAIL PROTECTED]> wrote: > > >> Eric Valette wrote: > > >> > > > > Thanks for your help: it does indeed fix the problem. > Nice it works for you too ! > > > Now I have two side questions: > > - the code is no more symetric "subsys_initcall" -> "module_exit". > > Do not know if it is "normal" but I love symmetry in code :-). Did not test > > it still works as a module... > Symmetry is not broken, as we have : > #define subsys_initcall(fn) module_init(fn) > in include/linux/init.h where compiling as a module, and when not compiling > as a module, I doubt the exit function is called unless you are shuting > down your machine... > > > - Who takes the responsability to push a patch to Linus? I guess it > > is urgent unless he plans a rc7 > Good point ! I expect the patches to be already in some queue waiting to be > pulled ! The patches are on their way to making it into 2.6.23: http://marc.info/?l=linux-netdev&m=118986368303529&w=2 Regards, Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Scheduler benchmarks - a follow-up
On 9/17/07, Ingo Molnar <[EMAIL PROTECTED]> wrote:
>
> * Rob Hussey <[EMAIL PROTECTED]> wrote:
>
> > http://www.healthcarelinen.com/misc/benchmarks/BOUND_hackbench_benchmark2.png
>
> heh - am i the only one impressed by the consistency of the blue line in
> this graph? :-) [ and the green line looks a bit like a .. staircase? ]
>
> i've meanwhile tested hackbench 90 and the performance difference
> between -ck and -cfs-devel seems to be mostly down to the more precise
> (but slower) sched_clock() introduced in v2.6.23 and to the startup
> penalty of freshly created tasks.
>
> Putting back the 2.6.22 version and tweaking the startup penalty gives
> this:
>
> [hackbench 90, smaller is better]
>
> sched-devel.git sched-devel.git+lowres-sched-clock+dsp
> --- --
> 5.555 5.149
> 5.641 5.149
> 5.572 5.171
> 5.583 5.155
> 5.532 5.111
> 5.540 5.138
> 5.617 5.176
> 5.542 5.119
> 5.587 5.159
> 5.553 5.177
> --
> avg: 5.572 avg: 5.150 (-8.1%)
>
> ('lowres-sched-clock' is the patch i sent in the previous mail. 'dsp' is
> a disable-startup-penalty patch that is in the latest sched-devel.git)
>
> i have used your .config to conduct this test.
>
> can you reproduce this with the (very-) latest sched-devel git tree:
>
> git-pull
> git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched-devel.git
>
> plus with the low-res-sched-clock patch (re-) attached below?
>
> Ingo
> ---
> arch/i386/kernel/tsc.c |4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> Index: linux/arch/i386/kernel/tsc.c
> ===
> --- linux.orig/arch/i386/kernel/tsc.c
> +++ linux/arch/i386/kernel/tsc.c
> @@ -110,9 +110,9 @@ unsigned long long native_sched_clock(vo
> * very important for it to be as fast as the platform
> * can achive it. )
> */
> - if (unlikely(!tsc_enabled && !tsc_unstable))
> + if (1 || unlikely(!tsc_enabled && !tsc_unstable))
> /* No locking but a rare wrong value is not a big deal: */
> - return (jiffies_64 - INITIAL_JIFFIES) * (10 / HZ);
> + return jiffies_64 * (10 / HZ);
>
> /* read the Time Stamp Counter: */
> rdtscll(this_offset);
> -
Sorry it took so long for me to get back.
Ok, to start the dmesg output for 2.6.22-ck1 is attached. The relevant
lines seem to be:
[ 27.691348] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
[ 27.995427] Time: tsc clocksource has been installed.
I've updated to the latest sched-devel git, and applied the patch
above. I ran it through the same tests, but this time only while bound
to a single core. Some selected numbers:
lat_ctx -s 0 $i (the left most number is $i):
15 3.09
16 3.09
17 3.11
18 3.07
19 2.99
20 3.09
21 3.05
22 3.11
23 3.05
24 3.08
25 3.06
hackbench $i:
80 11.720
81 11.698
82 11.888
83 12.094
84 12.232
85 12.351
86 12.512
87 12.680
88 12.736
89 12.861
90 13.103
pipe-test (the left most number is the run #):
1 8.85
2 8.80
3 8.84
4 8.82
5 8.82
6 8.80
7 8.82
8 8.82
9 8.85
10 8.83
Once again, graphs:
http://www.healthcarelinen.com/misc/benchmarks/BOUND_PATCHED_lat_ctx_benchmark.png
http://www.healthcarelinen.com/misc/benchmarks/BOUND_PATCHED_hackbench_benchmark.png
http://www.healthcarelinen.com/misc/benchmarks/BOUND_PATCHED_pipe-test_benchmark.png
I saw in your other email that you'd like for me to try with
CONFIG_PREEMPT disabled. I should have a chance to try that very soon.
Regards,
Rob
dmesg-2.6.22-ck1.bz2
Description: BZip2 compressed data
<><><>
data_files2.tar.bz2
Description: BZip2 compressed data
Re: Scheduler benchmarks - a follow-up
On 9/18/07, Willy Tarreau <[EMAIL PROTECTED]> wrote: > Hi Rob, > > On Tue, Sep 18, 2007 at 12:30:05AM -0400, Rob Hussey wrote: > > I should have pointed out before that I don't really have a dual-core > > system, just a P4 with Hyper-Threading (I loosely used core to refer > > to processor). > > Just for reference, we call them "siblings", not "cores" on HT. I believe > that a line "Sibling:" appears in /proc/cpuinfo BTW. Thanks, I was searching for the right word but couldn't come up with it. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Scheduler benchmarks - a follow-up
On 9/17/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > * Ingo Molnar <[EMAIL PROTECTED]> wrote: > > > i've meanwhile tested hackbench 90 and the performance difference > > between -ck and -cfs-devel seems to be mostly down to the more precise > > (but slower) sched_clock() introduced in v2.6.23 and to the startup > > penalty of freshly created tasks. > > Rob, another thing i just noticed in your .configs: you have > CONFIG_PREEMPT=y enabled. Would it be possible to get a testrun with > that disabled? That gives the best throughput and context-switch latency > numbers. (CONFIG_PREEMPT might also have preemption artifacts - there's > one report of it having _worse_ desktop latencies on certain hardware > than !CONFIG_PREEMPT.) I reverted the patch from before since it didn't seem to help. Do you think it may have to do with my system having Hyper-Threading enabled? I should have pointed out before that I don't really have a dual-core system, just a P4 with Hyper-Threading (I loosely used core to refer to processor). Some new numbers for 2.6.23-rc6-cfs-devel (!CONFIG_PREEMPT and bound to single processor) lat_ctx: 15 2.73 16 2.74 17 2.81 18 2.74 19 2.74 20 2.73 21 2.60 22 2.74 23 2.72 24 2.74 25 2.74 hackbench: 80 11.578 81 11.991 82 11.914 83 12.026 84 12.226 85 12.347 86 12.552 87 12.655 88 13.011 89 12.941 90 13.237 pipe-test: 1 9.58 2 9.58 3 9.58 4 9.58 5 9.58 6 9.58 7 9.58 8 9.58 9 9.58 10 9.58 The obligatory graphs: http://www.healthcarelinen.com/misc/benchmarks/BOUND_NOPREEMPT_lat_ctx_benchmark.png http://www.healthcarelinen.com/misc/benchmarks/BOUND_NOPREEMPT_hackbench_benchmark.png http://www.healthcarelinen.com/misc/benchmarks/BOUND_NOPREEMPT_pipe-test_benchmark.png A cursory glance suggests that performance wrt lat_ctx and hackbench has increased (lower numbers), but degraded quite a lot for pipe-test. The numbers for pipe-test are extremely stable though, while the numbers for hackbench are more erratic (which isn't saying much since the original numbers gave nearly a straight line). I'm still willing to try out any more ideas. Regards, Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Scheduler benchmarks - a follow-up
On 9/18/07, Ingo Molnar <[EMAIL PROTECTED]> wrote:
>
> * Rob Hussey <[EMAIL PROTECTED]> wrote:
>
> > A cursory glance suggests that performance wrt lat_ctx and hackbench
> > has increased (lower numbers), but degraded quite a lot for pipe-test.
> > The numbers for pipe-test are extremely stable though, while the
> > numbers for hackbench are more erratic (which isn't saying much since
> > the original numbers gave nearly a straight line). I'm still willing
> > to try out any more ideas.
>
> pipe-test is a very stable workload, and is thus quite sensitive to the
> associativity of the CPU cache. Even killing the task and repeating the
> same test isnt enough to get rid of the systematic skew that this can
> cause. I've seen divergence of up to 10% in pipe-test. One way to test
> it is to run pipe-test, then to stop it, then to "ssh localhost" (this
> in itself uses up a couple of pipe objects and file objects and changes
> the cache layout picture), then run pipe-test again, then again "ssh
> localhost", etc. Via this trick one can often see cache-layout
> artifacts. How much 'skew' does pipe-test have on your system if you try
> this manually?
>
I did 7 data sets of 5 runs each using this method. With pipe-test
bound to one sibling, there were 10 unique values in these 7 sets. The
lowest value was 9.22, the highest value was 9.62, and the median of
the unique values was 9.47. So the deviation from mean for the lowest
and highest values was {-0.25, 0.15} The numbers were even tighter for
pipe-test not bound to a single sibling: {-0.07, 0.12}
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Scheduler benchmarks - a follow-up
On 9/18/07, Ingo Molnar <[EMAIL PROTECTED]> wrote: > > * Rob Hussey <[EMAIL PROTECTED]> wrote: > > > The obligatory graphs: > > http://www.healthcarelinen.com/misc/benchmarks/BOUND_NOPREEMPT_lat_ctx_benchmark.png > > http://www.healthcarelinen.com/misc/benchmarks/BOUND_NOPREEMPT_hackbench_benchmark.png > > http://www.healthcarelinen.com/misc/benchmarks/BOUND_NOPREEMPT_pipe-test_benchmark.png > > btw., it's likely that if you turn off CONFIG_PREEMPT for .21 and for > .22-ck1 they'll improve a bit too - so it's not fair to put the .23 > !PREEMPT numbers on the graph as the PREEMPT numbers of the other > kernels. (it shows the .23 scheduler being faster than it really is) > The graphs are really just to show where the new numbers fit in. Plus, I was too lazy to run all the numbers again. > > A cursory glance suggests that performance wrt lat_ctx and hackbench > > has increased (lower numbers), but degraded quite a lot for pipe-test. > > The numbers for pipe-test are extremely stable though, while the > > numbers for hackbench are more erratic (which isn't saying much since > > the original numbers gave nearly a straight line). I'm still willing > > to try out any more ideas. > > the pipe-test behavior looks like an outlier. !PREEMPT only removes code > (which makes the code faster), so this could be a cache layout artifact. > (or perhaps we preempt at a different point which is disadvantageous to > caching?) Pipe-test is equivalent to "lat_ctx -s 0 2" so if there was a > genuine slowdown it would show up in the lat_ctx graph - but the graph > shows a speedup. > Interestingly, every set of lat_ctx -s 0 2 numbers I run on the !PREEMPT kernel are on average higher than with PREEMPT (around 2.84 for !PREEMPT and 2.4 for PREEMPT). Anything higher than around 2 or 3 (such as lat_ctx -s 0 8) gives lower average numbers for !PREEMPT. Regards, Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
