Re: INVARIANTS (was Re: RELENG_4 - 5 - 6: significant performance regression)
Hello! On Sat, 13 May 2006, Matthew D. Fuller wrote: On Sat, May 13, 2006 at 11:58:26AM -0400 I heard the voice of Kris Kennaway, and lo! it spake thus: FYI, INVARIANTS adds checks but does not (is not supposed to) divert code paths. It does at least in UMA; it does a lot of bzero()/NULL'ing out of memory, which might hide later uninitialized-use bugs that could bite you without it (and, of course, probably burns a fair chunk of CPU to do it ;). I know I've heard other cases over the past 5 years or so; that's the only one I've heard recently or can check, but I wouldn't be too surprised if there were others. This is exactly the point of my suggestion: use separate option (e.g., INVARIANTS_EXTENDED) for this additional bzero()/NULL'ing and other performance-expensive checks, and leave only simple asserts under basic INVARIANTS options. IMHO this will encourage (or, at least, not discourage) use of INVARIANTS-enabled kernels in production conditions which for sure will help to analyze latent, heavy load-specific, bugs. Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. Sincerely, Dmitry -- Atlantis ISP, System Administrator e-mail: [EMAIL PROTECTED] nic-hdl: LYNX-RIPE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
Hello! On Sat, 13 May 2006, Jonathan Noack wrote: Have you tried putting I586_CPU in there? See http://lists.freebsd.org/pipermail/freebsd-stable/2005-December/020696.html. Thanks for suggestion. I've just tried it, performance difference is indistinguishable. Also, use the link0 option with your fxp cards if they support it. See the fxp(4) man page for more info. Here is an example /etc/rc.conf entry: ifconfig_fxp0=inet xxx.xxx.xxx.xxx netmask xxx.xxx.xxx.xxx link0 Well, I'm aware of this feature. I'm just evaluating performance of RELENG_6 systems on uniprocessor i386 system, and (for now) it's significantly worse than under RELENG_4. So I've used fxp just for comparison (and to rule out possible specific regression in rl NIC driver), not for anything more serious. Jonathan Noack | [EMAIL PROTECTED] | OpenPGP: 0x991D8195 Sincerely, Dmitry -- Atlantis ISP, System Administrator e-mail: [EMAIL PROTECTED] nic-hdl: LYNX-RIPE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: INVARIANTS (was Re: RELENG_4 - 5 - 6: significant performance regression)
Matthew D. Fuller wrote: On Sat, May 13, 2006 at 11:58:26AM -0400 I heard the voice of Kris Kennaway, and lo! it spake thus: FYI, INVARIANTS adds checks but does not (is not supposed to) divert code paths. It does at least in UMA; it does a lot of bzero()/NULL'ing out of memory, which might hide later uninitialized-use bugs that could bite you without it Shouldn't it write something like 0xdeadc0de or 0x0d0d0d0d to memory -- like user-land memory debugging tools do -- to catch read-before-write bugs? Ulrich Spoerlein -- PGP Key ID: 20FEE9DD Encrypted mail welcome! Fingerprint: AEC9 AF5E 01AC 4EE1 8F70 6CBD E76E 2227 20FE E9DD Which is worse: ignorance or apathy? Don't know. Don't care. pgpC9DTZhM4QV.pgp Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
Hello! On Fri, 12 May 2006, Kris Kennaway wrote: %Sys %Intr %Idl RELENG_6 + rl0 45 40 15 RELENG_6 + fxp0 45 35 20 %Sys %Intr %Idl time md5 -t wall clock time RELENG_6 + rl0 34 24 42 1:43 RELENG_6 + fxp0 30 20 50 1:40 is caused by just these: options INVARIANTS options INVARIANT_SUPPORT So what is the overall status? I am not clear what your results are. Results for RELENG_6+rl0 are %Sys %Intr %Idl 34 24 42 without INVARIANTS, and %Sys %Intr %Idl 45 40 15 with them. Other options like QUOTA and makeoptions CONF_CFLAGS=-fno-builtin make almost no difference. So, under my test conditions, the best % of idle CPU time under RELENG_6 is 42%, while under RELENG_4 we had %Sys %Intr %Idl 14 14 72 under the same conditions (and with INVARIANTS!) ;( available for application under RELENG_5/6 than under RELENG_4 (under identical load pattern). I ran time md5 -t several (3-5 times) just to confirm my assumptions, and results didn't vary more than 3%. So I suppose that ministat isn't necessary in my tests. Perhaps not when the difference is large, but you need to be very careful when differences are below ~10%, because it's easy to make incorrect conclusions. I agree with you. I would make more measurements if my aim was to determine which branch between RELENG_5 and _6 to use. But as these results are close enough, and RELENG_6 is superiour regarding new features (and often stability), IMHO there's no point in using RELENG_5 at all. I'm just trying to understand why performance of RELENG_6 is worse than in RELENG_4 _that much_, and whether this sad situation can be improved somehow. Kris Sincerely, Dmitry -- Atlantis ISP, System Administrator e-mail: [EMAIL PROTECTED] nic-hdl: LYNX-RIPE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
Hello! On Fri, 12 May 2006, Kris Kennaway wrote: So maybe it's time to add, say, options INVARIANTS_EXTENDED for these new and expensive checks, and leave only basic and cheap (yet effective for bug hunting) asserts enabled when only options INVARIANTS is defined? No, they are all effective for bug hunting. You just need to be aware that it is incompatible with performance. But, you know, many bugs can be hunted only under long-term production conditions, while incompatibility between INVARIANTS and performance effectively prevents successful bug hunting under these conditions, because performance is often critical in production. Kris Sincerely, Dmitry -- Atlantis ISP, System Administrator e-mail: [EMAIL PROTECTED] nic-hdl: LYNX-RIPE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On 05/13/06 01:59, Dmitry Pryanishnikov wrote: On Fri, 12 May 2006, Kris Kennaway wrote: %Sys %Intr %Idl RELENG_6 + rl0 45 40 15 RELENG_6 + fxp0 45 35 20 %Sys %Intr %Idl time md5 -t wall clock time RELENG_6 + rl0 34 24 42 1:43 RELENG_6 + fxp0 30 20 50 1:40 is caused by just these: options INVARIANTS options INVARIANT_SUPPORT So what is the overall status? I am not clear what your results are. Results for RELENG_6+rl0 are %Sys %Intr %Idl 34 24 42 without INVARIANTS, and %Sys %Intr %Idl 45 40 15 with them. Other options like QUOTA and makeoptions CONF_CFLAGS=-fno-builtin make almost no difference. So, under my test conditions, the best % of idle CPU time under RELENG_6 is 42%, while under RELENG_4 we had %Sys %Intr %Idl 14 14 72 under the same conditions (and with INVARIANTS!) ;( available for application under RELENG_5/6 than under RELENG_4 (under identical load pattern). I ran time md5 -t several (3-5 times) just to confirm my assumptions, and results didn't vary more than 3%. So I suppose that ministat isn't necessary in my tests. Perhaps not when the difference is large, but you need to be very careful when differences are below ~10%, because it's easy to make incorrect conclusions. I agree with you. I would make more measurements if my aim was to determine which branch between RELENG_5 and _6 to use. But as these results are close enough, and RELENG_6 is superiour regarding new features (and often stability), IMHO there's no point in using RELENG_5 at all. I'm just trying to understand why performance of RELENG_6 is worse than in RELENG_4 _that much_, and whether this sad situation can be improved somehow. Have you tried putting I586_CPU in there? See http://lists.freebsd.org/pipermail/freebsd-stable/2005-December/020696.html. Also, use the link0 option with your fxp cards if they support it. See the fxp(4) man page for more info. Here is an example /etc/rc.conf entry: ifconfig_fxp0=inet xxx.xxx.xxx.xxx netmask xxx.xxx.xxx.xxx link0 -- Jonathan Noack | [EMAIL PROTECTED] | OpenPGP: 0x991D8195 signature.asc Description: OpenPGP digital signature
Re: RELENG_4 - 5 - 6: significant performance regression
On Sat, May 13, 2006 at 03:01:18AM -0400 I heard the voice of Jonathan Noack, and lo! it spake thus: Have you tried putting I586_CPU in there? See http://lists.freebsd.org/pipermail/freebsd-stable/2005-December/020696.html. As Peter Jeremy mentioned in http://lists.freebsd.org/pipermail/freebsd-stable/2005-December/020731.html, the primary suspect (optimized copy/zero routines) would never happen except on a real 586 CPU, and is totally disabled anyway. See sys/i386/isa/npx.c line 424-437 (line numbers from rev 1.163, salt to taste): #ifdef I586_CPU_XXX if (cpu_class == CPUCLASS_586 npx_ex16 npx_exists [...] The #ifdef will never match, and even if it did, the if() would never kick in unless the CPU was actually a 586. The #ifdef has been disabled since rev 1.95 (2001/04/13). (This isn't to say that there isn't something else hiding somewhere that I686_CPU doesn't enable that it should, but just nipping another round of the copy routine discussion in the bud...) -- Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Sat, May 13, 2006 at 08:59:01AM +0300, Dmitry Pryanishnikov wrote: I'm just trying to understand why performance of RELENG_6 is worse than in RELENG_4 _that much_, and whether this sad situation can be improved somehow. The architecture of the system substantially changed in the 5.X timeframe to correctly support SMP. There was not time enough during its lifetime to correct all the problems. The goal (AFAIK) of 6.X is to consolidate those changes and work on bugfixes and performance. (Already 6.X is substantially faster in disk access). There is a great deal of work being done behind-the-scenes (and being discussed here at BSDCan) about performance. Expect to some some dramatic improvements in the 6.2 timeframe, if the hallways discussions are correct :-) mcl ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
INVARIANTS (was Re: RELENG_4 - 5 - 6: significant performance regression)
On Sat, May 13, 2006 at 10:37:40AM -0400 I heard the voice of Kris Kennaway, and lo! it spake thus: With respect to INVARIANTS, you just need to get used to the fact that running thousands of checks for bugs is incompatible with running at optimal speed. (I'm not sure what the point of saying this is, really, but I'll say it anyway.) I've run all my systems with INVARIANTS for at least as long as I've known it was there. While more performance is always good, hardly any of my systems are so constrained as to need every bit of suds all the time; trading off a bit of performance for a better chance of catching a problem before it really screws something up is just a no-brainer. Additionally (and especially on -CURRENT), I run it because I think more people run it than don't, and while theoretically it should just add checks, I know there are places where it changes code paths much more than that. So, the !(INVARIANTS) code paths don't get exercised as much, and I worry about bugs hiding there that don't get found (I think I recall a case or three over the years of just that happening). Like everyone, I'm sure, I'm all for ferreting out bugs and getting them fixed, but I'd rather not have to bust my virtual face on the virtual concrete to do it;) It's totally worth wasting 2% of the system on adding that security. Heck, maybe even 5%. But 10%? 25%? I wonder. It's been a long time since I had a system fall over from a KASSERT, but due to the code path issues, I'm not sure that really means I'm safe without it. Yeah, I was right, there wasn't really much point in there. But I typed it, so y'all have to read it now:p -- Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: INVARIANTS (was Re: RELENG_4 - 5 - 6: significant performance regression)
On Sat, May 13, 2006 at 10:52:32AM -0500, Matthew D. Fuller wrote: On Sat, May 13, 2006 at 10:37:40AM -0400 I heard the voice of Kris Kennaway, and lo! it spake thus: With respect to INVARIANTS, you just need to get used to the fact that running thousands of checks for bugs is incompatible with running at optimal speed. (I'm not sure what the point of saying this is, really, but I'll say it anyway.) I've run all my systems with INVARIANTS for at least as long as I've known it was there. While more performance is always good, hardly any of my systems are so constrained as to need every bit of suds all the time; trading off a bit of performance for a better chance of catching a problem before it really screws something up is just a no-brainer. Additionally (and especially on -CURRENT), I run it because I think more people run it than don't, and while theoretically it should just add checks, I know there are places where it changes code paths much more than that. So, the !(INVARIANTS) code paths don't get exercised as much, and I worry about bugs hiding there that don't get found (I think I recall a case or three over the years of just that happening). Like everyone, I'm sure, I'm all for ferreting out bugs and getting them fixed, but I'd rather not have to bust my virtual face on the virtual concrete to do it;) FYI, INVARIANTS adds checks but does not (is not supposed to) divert code paths. Kris pgp2ibCdGtDyO.pgp Description: PGP signature
Re: INVARIANTS (was Re: RELENG_4 - 5 - 6: significant performance regression)
On Sat, May 13, 2006 at 11:58:26AM -0400 I heard the voice of Kris Kennaway, and lo! it spake thus: FYI, INVARIANTS adds checks but does not (is not supposed to) divert code paths. It does at least in UMA; it does a lot of bzero()/NULL'ing out of memory, which might hide later uninitialized-use bugs that could bite you without it (and, of course, probably burns a fair chunk of CPU to do it ;). I know I've heard other cases over the past 5 years or so; that's the only one I've heard recently or can check, but I wouldn't be too surprised if there were others. -- Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
Hello! On Fri, 28 Apr 2006, Kris Kennaway wrote: makeoptions CONF_CFLAGS=-fno-builtin I don't know, it needs to be tested in your particular case. I've built another kernel, adding back makeoptions CONF_CFLAGS=-fno-builtin options QUOTA Results are almost the same as w/o these 2 options. So the following overhead difference: %Sys %Intr %Idl RELENG_6 + rl0 45 40 15 RELENG_6 + fxp0 45 35 20 %Sys %Intr %Idl time md5 -t wall clock time RELENG_6 + rl0 34 24 42 1:43 RELENG_6 + fxp0 30 20 50 1:40 is caused by just these: options INVARIANTS options INVARIANT_SUPPORT (I'll try to find out which one of these takes which % of overhead when I get free time), but still much worse then under RELENG_4, where this particular (I'd say quote common) usage pattern takes 24-28% of CPU time, while under RELENG_5 / 6 it takes = 50% ;( Thanks. Silly question: the data transfer rate is the same on both 4.x and 6.x, right? i.e. the data transfer itself takes the same time? Yes. I'm transferring a large file (ISO image) from another (much faster, lightly loaded) machine over 10Mbit/s Ethernet link, so the transfer itself is limited only by the wire speed (actual transfer rate is very close to 1000 KBytes/sec according to ftp client and 'systat -vm 1' disk transfer rate in every measurement). The next step is for you to run some profiling tests to see where the kernel is spending time, e.g. with hwpmc. I have to get myself familiar with this new (for me) feature first... Also, hwpmc doesn't exist in RELENG_4, so it'll be impossible to compare results with RELENG_4. It's a pity, because my tests clearly show that main loss of performance (growth of overhead) occured during RELENG_4 - 5 transition. And last, but not least: my test system (Transcend TS-ABX31A motherboard based on Intel BX chipset) does not provide APIC, will hwpmc be useful in this situation? Also, when you are trying to quantify performance differences, you need to run many copies of the test (at least 10) under identical conditions to account for possible variations. The ministat tool (/usr/src/tools/tools/ministat) is good for performing statistically meaningful comparisons of data sets when you have them. As my transfer takes much time (say 10 minutes) I've observed % of time used many times during the transfer - they don't vary more than +/- several (2-3) % during the main transfer phase (when transfer speed is stable). My time md5 -t runs was used only as a confirmation that systat's numbers are trustworthy - they simply confirm that there are _much_ less CPU cycles available for application under RELENG_5/6 than under RELENG_4 (under identical load pattern). I ran time md5 -t several (3-5 times) just to confirm my assumptions, and results didn't vary more than 3%. So I suppose that ministat isn't necessary in my tests. Kris Sincerely, Dmitry -- Atlantis ISP, System Administrator e-mail: [EMAIL PROTECTED] nic-hdl: LYNX-RIPE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
Hello! On Tue, 2 May 2006, Robert Watson wrote: options INVARIANTS options INVARIANT_SUPPORT In FreeBSD 5.x and FreeBSD 6.x, the INVARIANTS option has been significantly expanded to test a much larger set of invariants, and also incorporate kernel use-after-free checking, which involves memory scrubbing. This is great for catching bugs, but it will have a significant performance impact, especially for kernel-intensive loads. So maybe it's time to add, say, options INVARIANTS_EXTENDED for these new and expensive checks, and leave only basic and cheap (yet effective for bug hunting) asserts enabled when only options INVARIANTS is defined? Robert N M Watson Sincerely, Dmitry -- Atlantis ISP, System Administrator e-mail: [EMAIL PROTECTED] nic-hdl: LYNX-RIPE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Fri, May 12, 2006 at 11:32:44PM +0300, Dmitry Pryanishnikov wrote: Hello! On Tue, 2 May 2006, Robert Watson wrote: options INVARIANTS options INVARIANT_SUPPORT In FreeBSD 5.x and FreeBSD 6.x, the INVARIANTS option has been significantly expanded to test a much larger set of invariants, and also incorporate kernel use-after-free checking, which involves memory scrubbing. This is great for catching bugs, but it will have a significant performance impact, especially for kernel-intensive loads. So maybe it's time to add, say, options INVARIANTS_EXTENDED for these new and expensive checks, and leave only basic and cheap (yet effective for bug hunting) asserts enabled when only options INVARIANTS is defined? No, they are all effective for bug hunting. You just need to be aware that it is incompatible with performance. Kris pgpBtm7Pv8jt2.pgp Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
On Fri, May 12, 2006 at 11:25:58PM +0300, Dmitry Pryanishnikov wrote: Hello! On Fri, 28 Apr 2006, Kris Kennaway wrote: makeoptions CONF_CFLAGS=-fno-builtin I don't know, it needs to be tested in your particular case. I've built another kernel, adding back makeoptions CONF_CFLAGS=-fno-builtin options QUOTA Results are almost the same as w/o these 2 options. So the following overhead difference: %Sys %Intr %Idl RELENG_6 + rl0 45 40 15 RELENG_6 + fxp0 45 35 20 %Sys %Intr %Idl time md5 -t wall clock time RELENG_6 + rl0 34 24 42 1:43 RELENG_6 + fxp0 30 20 50 1:40 is caused by just these: options INVARIANTS options INVARIANT_SUPPORT So what is the overall status? I am not clear what your results are. As my transfer takes much time (say 10 minutes) I've observed % of time used many times during the transfer - they don't vary more than +/- several (2-3) % during the main transfer phase (when transfer speed is stable). My time md5 -t runs was used only as a confirmation that systat's numbers are trustworthy - they simply confirm that there are _much_ less CPU cycles available for application under RELENG_5/6 than under RELENG_4 (under identical load pattern). I ran time md5 -t several (3-5 times) just to confirm my assumptions, and results didn't vary more than 3%. So I suppose that ministat isn't necessary in my tests. Perhaps not when the difference is large, but you need to be very careful when differences are below ~10%, because it's easy to make incorrect conclusions. Kris pgpPzVBJ50JAm.pgp Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
On Thu, 27 Apr 2006, Dmitry Pryanishnikov wrote: options INVARIANTS options INVARIANT_SUPPORT In FreeBSD 5.x and FreeBSD 6.x, the INVARIANTS option has been significantly expanded to test a much larger set of invariants, and also incorporate kernel use-after-free checking, which involves memory scrubbing. This is great for catching bugs, but it will have a significant performance impact, especially for kernel-intensive loads. Robert N M Watson ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On 28/04/06, Kris Kennaway [EMAIL PROTECTED] wrote:
On Fri, Apr 28, 2006 at 03:31:17PM +0300, Dmitry Pryanishnikov wrote:
Hello!
On Thu, 27 Apr 2006, Kris Kennaway wrote:
Thanks for your suggestions, they've made a difference (though not as big
as one could hope).
On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote:
makeoptions CONF_CFLAGS=-fno-builtin
Non-default option; this may conceivably affect performance.
The reason why I've initially included this option is the following
comment (NOTES from RELENG_6):
#
# CONF_CFLAGS gives some extra compiler flags that are added to ${CFLAGS}
# after most other flags. Here we use it to inhibit use of non-optimal
# gcc builtin functions (e.g., memcmp).
#
I've read this: using CONF_CFLAGS=-fno-builtin inhibits use of non-optimal
gcc builtin functions, so this option may be useful for getting max.
performance. Are this comment and my interpretation still correct now?
I don't know, it needs to be tested in your particular case.
%Sys %Intr %Idl
RELENG_4 + rl0 14 14 72
RELENG_4 + fxp0 14 10 76
RELENG_5 + rl0 40 30 30
RELENG_5 + fxp0 35 25 40
RELENG_6 + rl0 45 40 15
RELENG_6 + fxp0 45 35 20
%Sys %Intr %Idl time md5 -t wall clock time
RELENG_5 + rl0 33 23 44 1:41
RELENG_5 + fxp0 30 20 50 1:36
RELENG_6 + rl0 34 24 42 1:43
RELENG_6 + fxp0 30 20 50 1:40
So performance now is much better then before removal of
makeoptions CONF_CFLAGS=-fno-builtin
options INVARIANTS
options INVARIANT_SUPPORT
options QUOTA
(I'll try to find out which one of these takes which % of overhead when I
get free time), but still much worse then under RELENG_4, where this
particular (I'd say quote common) usage pattern takes 24-28% of CPU time,
while under RELENG_5 / 6 it takes = 50% ;(
Thanks. Silly question: the data transfer rate is the same on both
4.x and 6.x, right? i.e. the data transfer itself takes the same
time?
The next step is for you to run some profiling tests to see
where the kernel is spending time, e.g. with hwpmc.
Also, when you are trying to quantify performance differences, you
need to run many copies of the test (at least 10) under identical
conditions to account for possible variations. The ministat tool
(/usr/src/tools/tools/ministat) is good for performing statistically
meaningful comparisons of data sets when you have them.
Kris
Does 'makeoptions DEBUG=-g' add any kind of performance hit or
overhead as I noticed it wasnt default in 5.4 but is in 6.0.
Thanks
Chris
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Sun, 2006-Apr-30 10:05:40 +0100, Chris wrote: Does 'makeoptions DEBUG=-g' add any kind of performance hit or overhead as I noticed it wasnt default in 5.4 but is in 6.0. No. It just means that a debug kernel is built in addition to the normal kernel. The major benefit is that if you do get a panic, you can debug it without needing to rebuild the kernel. -- Peter Jeremy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
Peter Jeremy wrote: On Sun, 2006-Apr-30 10:05:40 +0100, Chris wrote: Does 'makeoptions DEBUG=-g' add any kind of performance hit or overhead as I noticed it wasnt default in 5.4 but is in 6.0. No. It just means that a debug kernel is built in addition to the normal kernel. The major benefit is that if you do get a panic, you can debug it without needing to rebuild the kernel. I've often thought that the default should be for this to be on instead of off. Does anyone know why the current state is so? I suppose it incurs a small amount of additional disc space but the benefits are considerable for everyone involved. Cheers, Dominic ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
Dominic Marks wrote: Peter Jeremy wrote: On Sun, 2006-Apr-30 10:05:40 +0100, Chris wrote: Does 'makeoptions DEBUG=-g' add any kind of performance hit or overhead as I noticed it wasnt default in 5.4 but is in 6.0. No. It just means that a debug kernel is built in addition to the normal kernel. The major benefit is that if you do get a panic, you can debug it without needing to rebuild the kernel. I've often thought that the default should be for this to be on instead of off. Does anyone know why the current state is so? I suppose it incurs a small amount of additional disc space but the benefits are considerable for everyone involved. Clarifying what I was getting at (it does default to on) I meant that it has to be turned on with an option not turned off. Cheers, Dominic ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Sun, Apr 30, 2006 at 10:05:40AM +0100, Chris wrote:
On 28/04/06, Kris Kennaway [EMAIL PROTECTED] wrote:
On Fri, Apr 28, 2006 at 03:31:17PM +0300, Dmitry Pryanishnikov wrote:
Hello!
On Thu, 27 Apr 2006, Kris Kennaway wrote:
Thanks for your suggestions, they've made a difference (though not as
big
as one could hope).
On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote:
makeoptions CONF_CFLAGS=-fno-builtin
Non-default option; this may conceivably affect performance.
The reason why I've initially included this option is the following
comment (NOTES from RELENG_6):
#
# CONF_CFLAGS gives some extra compiler flags that are added to ${CFLAGS}
# after most other flags. Here we use it to inhibit use of non-optimal
# gcc builtin functions (e.g., memcmp).
#
I've read this: using CONF_CFLAGS=-fno-builtin inhibits use of
non-optimal
gcc builtin functions, so this option may be useful for getting max.
performance. Are this comment and my interpretation still correct now?
I don't know, it needs to be tested in your particular case.
%Sys %Intr %Idl
RELENG_4 + rl0 14 14 72
RELENG_4 + fxp0 14 10 76
RELENG_5 + rl0 40 30 30
RELENG_5 + fxp0 35 25 40
RELENG_6 + rl0 45 40 15
RELENG_6 + fxp0 45 35 20
%Sys %Intr %Idl time md5 -t wall clock time
RELENG_5 + rl0 33 23 44 1:41
RELENG_5 + fxp0 30 20 50 1:36
RELENG_6 + rl0 34 24 42 1:43
RELENG_6 + fxp0 30 20 50 1:40
So performance now is much better then before removal of
makeoptions CONF_CFLAGS=-fno-builtin
options INVARIANTS
options INVARIANT_SUPPORT
options QUOTA
(I'll try to find out which one of these takes which % of overhead when I
get free time), but still much worse then under RELENG_4, where this
particular (I'd say quote common) usage pattern takes 24-28% of CPU
time,
while under RELENG_5 / 6 it takes = 50% ;(
Thanks. Silly question: the data transfer rate is the same on both
4.x and 6.x, right? i.e. the data transfer itself takes the same
time?
The next step is for you to run some profiling tests to see
where the kernel is spending time, e.g. with hwpmc.
Also, when you are trying to quantify performance differences, you
need to run many copies of the test (at least 10) under identical
conditions to account for possible variations. The ministat tool
(/usr/src/tools/tools/ministat) is good for performing statistically
meaningful comparisons of data sets when you have them.
Kris
Does 'makeoptions DEBUG=-g' add any kind of performance hit or
overhead as I noticed it wasnt default in 5.4 but is in 6.0.
No, the symbols are stripped from the install kernel.
Kris
pgpKLXFmxWWLJ.pgp
Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
Hello!
On Thu, 27 Apr 2006, Kris Kennaway wrote:
Thanks for your suggestions, they've made a difference (though not as big
as one could hope).
On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote:
makeoptions CONF_CFLAGS=-fno-builtin
Non-default option; this may conceivably affect performance.
The reason why I've initially included this option is the following
comment (NOTES from RELENG_6):
#
# CONF_CFLAGS gives some extra compiler flags that are added to ${CFLAGS}
# after most other flags. Here we use it to inhibit use of non-optimal
# gcc builtin functions (e.g., memcmp).
#
I've read this: using CONF_CFLAGS=-fno-builtin inhibits use of non-optimal
gcc builtin functions, so this option may be useful for getting max.
performance. Are this comment and my interpretation still correct now?
options INVARIANTS
options INVARIANT_SUPPORT
These definitely effect performance, much more in 5.x and 6.x (at the
10-20% level) than 4.x.
It's a pity - those help to catch a lot of software bugs ;(
options QUOTA
This definitely effects performance on 6.x since it makes your
filesystem giant-locked, which may also interfere with your network
processing.
It's also a pity.
%Sys %Intr %Idl
RELENG_4 + rl0 14 14 72
RELENG_4 + fxp0 14 10 76
RELENG_5 + rl0 40 30 30
RELENG_5 + fxp0 35 25 40
RELENG_6 + rl0 45 40 15
RELENG_6 + fxp0 45 35 20
Please retry without. Also make sure there are no other diagnostic
messages at boot time about e.g. mpsafenet being forced to 0.
I've removed all mentioned options and redone benchmarks. There are no
diagnostics about mpsafe* (last one I've seen due to IPSEC, and I've
replaced it with FAST_IPSEC+crypto before doing previous benchmarks).
New results are pretty close for RELENG_5 and RELENG_6:
%Sys %Intr %Idl time md5 -t wall clock time
RELENG_5 + rl0 33 23 44 1:41
RELENG_5 + fxp0 30 20 50 1:36
RELENG_6 + rl0 34 24 42 1:43
RELENG_6 + fxp0 30 20 50 1:40
So performance now is much better then before removal of
makeoptions CONF_CFLAGS=-fno-builtin
options INVARIANTS
options INVARIANT_SUPPORT
options QUOTA
(I'll try to find out which one of these takes which % of overhead when I
get free time), but still much worse then under RELENG_4, where this
particular (I'd say quote common) usage pattern takes 24-28% of CPU time,
while under RELENG_5 / 6 it takes = 50% ;(
Kris
Sincerely, Dmitry
--
Atlantis ISP, System Administrator
e-mail: [EMAIL PROTECTED]
nic-hdl: LYNX-RIPE
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Fri, Apr 28, 2006 at 03:31:17PM +0300, Dmitry Pryanishnikov wrote:
Hello!
On Thu, 27 Apr 2006, Kris Kennaway wrote:
Thanks for your suggestions, they've made a difference (though not as big
as one could hope).
On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote:
makeoptions CONF_CFLAGS=-fno-builtin
Non-default option; this may conceivably affect performance.
The reason why I've initially included this option is the following
comment (NOTES from RELENG_6):
#
# CONF_CFLAGS gives some extra compiler flags that are added to ${CFLAGS}
# after most other flags. Here we use it to inhibit use of non-optimal
# gcc builtin functions (e.g., memcmp).
#
I've read this: using CONF_CFLAGS=-fno-builtin inhibits use of non-optimal
gcc builtin functions, so this option may be useful for getting max.
performance. Are this comment and my interpretation still correct now?
I don't know, it needs to be tested in your particular case.
%Sys %Intr %Idl
RELENG_4 + rl0 14 14 72
RELENG_4 + fxp0 14 10 76
RELENG_5 + rl0 40 30 30
RELENG_5 + fxp0 35 25 40
RELENG_6 + rl0 45 40 15
RELENG_6 + fxp0 45 35 20
%Sys %Intr %Idl time md5 -t wall clock time
RELENG_5 + rl0 33 23 44 1:41
RELENG_5 + fxp0 30 20 50 1:36
RELENG_6 + rl0 34 24 42 1:43
RELENG_6 + fxp0 30 20 50 1:40
So performance now is much better then before removal of
makeoptions CONF_CFLAGS=-fno-builtin
options INVARIANTS
options INVARIANT_SUPPORT
options QUOTA
(I'll try to find out which one of these takes which % of overhead when I
get free time), but still much worse then under RELENG_4, where this
particular (I'd say quote common) usage pattern takes 24-28% of CPU time,
while under RELENG_5 / 6 it takes = 50% ;(
Thanks. Silly question: the data transfer rate is the same on both
4.x and 6.x, right? i.e. the data transfer itself takes the same
time?
The next step is for you to run some profiling tests to see
where the kernel is spending time, e.g. with hwpmc.
Also, when you are trying to quantify performance differences, you
need to run many copies of the test (at least 10) under identical
conditions to account for possible variations. The ministat tool
(/usr/src/tools/tools/ministat) is good for performing statistically
meaningful comparisons of data sets when you have them.
Kris
pgpB15jg08sOR.pgp
Description: PGP signature
RELENG_4 - 5 - 6: significant performance regression
Hello! I've done simple (yet, I hope, reality-reflecting) performance benchmarking different STABLE branches (4 vs 5 vs 6) using the following hardware: CPU: Pentium II/Pentium II Xeon/Celeron (334.09-MHz 686-class CPU) Origin = GenuineIntel Id = 0x665 Stepping = 5 Features=0x183f9ffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PA T,PSE36,MMX,FXSR real memory = 134152192 (127 MB) ... rl0: RealTek 8139 10/100BaseTX port 0xe800-0xe8ff mem 0xdc101000-0xdc1010ff irq 5 at device 20.0 on pci0 ... fxp0: Intel 82559 Pro/100 Ethernet port 0xe400-0xe43f mem 0xdc10-0xdc100fff,0xdc00-0xdc0f irq 7 at device 19.0 on pci0 ... ad0: 76351MB SAMSUNG SP0802N TK100-24 at ata0-master UDMA33 and just restoring precompiled 4/5/6-STABLE to the same HDD partition. I've used the following kernel config for 4-STABLE: ident TEST machine i386 maxusers32 makeoptions CONF_CFLAGS=-fno-builtin makeoptions DEBUG=-g options INCLUDE_CONFIG_FILE cpu I686_CPU options COMPAT_43 options USER_LDT options SYSVSHM options SYSVSEM options SYSVMSG options INVARIANTS options INVARIANT_SUPPORT options USERCONFIG options INET options FAST_IPSEC options IPSEC_FILTERGIF pseudo-device ether pseudo-device vlan1 pseudo-device loop pseudo-device bpf pseudo-device ppp 8 options PPP_BSDCOMP options PPP_DEFLATE options PPP_FILTER options IPFIREWALL options IPFW2 options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=100 options IPFIREWALL_FORWARD options IPDIVERT options IPSTEALTH options ICMP_BANDLIM options DUMMYNET options FFS options FFS_ROOT options SOFTUPDATES options QUOTA options P1003_1B options _KPOSIX_PRIORITY_SCHEDULING options _KPOSIX_VERSION=199309L pseudo-device pty pseudo-device crypto device isa device atkbdc0 at isa? port IO_KBD device atkbd0 at atkbdc? irq 1 device psm0at atkbdc? irq 12 device vga0at isa? pseudo-device splash device sc0 at isa? options SC_HISTORY_SIZE=1000 options SC_TWOBUTTON_MOUSE device npx0at nexus? port IO_NPX flags 0x0 irq 13 device ata device atadisk options ATA_STATIC_ID device fdc0at isa? port IO_FD1 irq 6 drq 2 device fd0 at fdc0 drive 0 device fd1 at fdc0 drive 1 device sio0at isa? port IO_COM1 irq 4 device sio1at isa? port IO_COM2 irq 3 device pci and slightly modified it for 5/6-STABLE, here is the diff ( = 4-only option, - 5/6-only): options SCHED_4BSD optionsUSER_LDT optionsUSERCONFIG pseudo-device ether pseudo-device vlan1 pseudo-device loop pseudo-device bpf pseudo-device ppp 8 device ether device loop device bpf optionsIPFW2 options IPFIREWALL_FORWARD_EXTENDED optionsICMP_BANDLIM optionsFFS_ROOT optionsP1003_1B options_KPOSIX_VERSION=199309L pseudo-device pty pseudo-device crypto device pty device crypto device atkbdc0 at isa? port IO_KBD device atkbd0 at atkbdc? irq 1 device psm0at atkbdc? irq 12 device vga0at isa? pseudo-device splash device sc0 at isa? --- device atkbdc device atkbd device psm options KBD_INSTALL_CDEV device vga device splash device sc device npx0at nexus? port IO_NPX flags 0x0 irq 13 device npx device fdc0at isa? port IO_FD1 irq 6 drq 2 device fd0 at fdc0 drive 0 device fd1 at fdc0 drive 1 device sio0at isa? port IO_COM1 irq 4 device sio1at isa? port IO_COM2 irq 3 Also I've set kern.hz=100 in /boot/loader.conf for every system. I've effectively excluded ipfw from the game by using 'add 1 pass all from any to any' rule. I hope, I've compared apples with apples this way. For every x-STABLE, I've received large ISO image via FTP in binary mode twice: using rl NIC and using fxp one, both in 10baseT mode (got approx. 1 Mbyte/s transfer rate). I've noted CPU utilization which gave systat -vm 1 once numbers have stabilized. Here are the results (average numbers, %User and %Nice are close to zero): %Sys %Intr %Idl RELENG_4 + rl0 14 14 72 RELENG_4 + fxp0 14 10 76 RELENG_5 + rl0 40 30 30 RELENG_5 + fxp0 35 25 40 RELENG_6 + rl0 45 40 15 RELENG_6 + fxp0 45 35 20 I've tried to verify these numbers by running 'md5 -t' in parallel with download and
Re: RELENG_4 - 5 - 6: significant performance regression
On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote: makeoptions CONF_CFLAGS=-fno-builtin Non-default option; this may conceivably affect performance. options INVARIANTS options INVARIANT_SUPPORT These definitely effect performance, much more in 5.x and 6.x (at the 10-20% level) than 4.x. options QUOTA This definitely effects performance on 6.x since it makes your filesystem giant-locked, which may also interfere with your network processing. Please retry without. Also make sure there are no other diagnostic messages at boot time about e.g. mpsafenet being forced to 0. Kris pgputya1DQiMW.pgp Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
Kris Kennaway a écrit : On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote: options QUOTA This definitely effects performance on 6.x since it makes your filesystem giant-locked, which may also interfere with your network processing. Why would QUOTA affect performance more on 6.x than 4.x ? I would like to understand because i think a system cannot be secure without QUOTA R. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Thu, Apr 27, 2006 at 08:26:06PM +0200, [EMAIL PROTECTED] wrote: Kris Kennaway a ?crit : On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote: options QUOTA This definitely effects performance on 6.x since it makes your filesystem giant-locked, which may also interfere with your network processing. Why would QUOTA affect performance more on 6.x than 4.x ? I would like to understand because i think a system cannot be secure without QUOTA It makes filesystem writes acquire Giant, which blocks other kernel code that needs to also acquire Giant. When the need to acquire Giant was removed from the mainstream UFS code in 6.0 it was an enormous performance improvement. Kris pgp7sJzhZlUWO.pgp Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
Kris Kennaway a écrit : On Thu, Apr 27, 2006 at 08:26:06PM +0200, [EMAIL PROTECTED] wrote: Kris Kennaway a ?crit : On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote: options QUOTA This definitely effects performance on 6.x since it makes your filesystem giant-locked, which may also interfere with your network processing. Why would QUOTA affect performance more on 6.x than 4.x ? I would like to understand because i think a system cannot be secure without QUOTA It makes filesystem writes acquire Giant, which blocks other kernel code that needs to also acquire Giant. When the need to acquire Giant was removed from the mainstream UFS code in 6.0 it was an enormous performance improvement. Thanks for your anwser, but i'm not sure i understand... You wrote that Giant is needed in 6.0 and now you write it has been removed. So is it removed for 6-STABLE ? 6.1-rc ? Since i already use 6.x boxes, and i do have performance issues, i would upgrade to the right version immediatly R. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
You wrote that Giant is needed in 6.0 and now you write it has been removed. In 4.x, every UFS write requires the Giant lock. In 6.x, Giant is not normally required, making file system operations faster. When you enable QUOTA, you basically get back to the 4.x behavior where Giant is needed for each write. This is why in 6.x UFS will normally be faster but if you enable QUOTA, you lose the newly gained performance again. - Bartosz ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
Bartosz Fabianowski wrote: You wrote that Giant is needed in 6.0 and now you write it has been removed. In 4.x, every UFS write requires the Giant lock. In 6.x, Giant is not normally required, making file system operations faster. When you enable QUOTA, you basically get back to the 4.x behavior where Giant is needed for each write. This is why in 6.x UFS will normally be faster but if you enable QUOTA, you lose the newly gained performance again. Why isn't QUOTA mpsafe then? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Thu, Apr 27, 2006 at 03:47:37PM -0400, Mike Jakubik wrote: Bartosz Fabianowski wrote: You wrote that Giant is needed in 6.0 and now you write it has been removed. In 4.x, every UFS write requires the Giant lock. In 6.x, Giant is not normally required, making file system operations faster. When you enable QUOTA, you basically get back to the 4.x behavior where Giant is needed for each write. This is why in 6.x UFS will normally be faster but if you enable QUOTA, you lose the newly gained performance again. Why isn't QUOTA mpsafe then? Because code doesn't grow on trees. There are uncommitted patches though - perhaps you can test them and get back to the author with your feedback. Kris pgpAdkWuWLHjT.pgp Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
Kris Kennaway wrote: On Thu, Apr 27, 2006 at 03:47:37PM -0400, Mike Jakubik wrote: Why isn't QUOTA mpsafe then? Because code doesn't grow on trees. There are uncommitted patches though - perhaps you can test them and get back to the author with your feedback. What? There is a beautiful PHP tree growing behind me :P If they apply to -CURRENT i will gladly test them, where can i find them? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Thu, Apr 27, 2006 at 04:43:07PM -0400, Mike Jakubik wrote: Kris Kennaway wrote: On Thu, Apr 27, 2006 at 03:47:37PM -0400, Mike Jakubik wrote: Why isn't QUOTA mpsafe then? Because code doesn't grow on trees. There are uncommitted patches though - perhaps you can test them and get back to the author with your feedback. What? There is a beautiful PHP tree growing behind me :P If they apply to -CURRENT i will gladly test them, where can i find them? I think they were posted to current@; check the archives. Kris pgppU2gm0dYIf.pgp Description: PGP signature
Re: RELENG_4 - 5 - 6: significant performance regression
On Apr 27, 2006, at 11:12, Kris Kennaway wrote: On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote: options QUOTA This definitely effects performance on 6.x since it makes your filesystem giant-locked, which may also interfere with your network processing. Any indications when this will be fixed? I need quota for user's files. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: RELENG_4 - 5 - 6: significant performance regression
On Thu, Apr 27, 2006 at 01:57:02PM -0700, Doug Hardie wrote: On Apr 27, 2006, at 11:12, Kris Kennaway wrote: On Thu, Apr 27, 2006 at 05:08:11PM +0300, Dmitry Pryanishnikov wrote: options QUOTA This definitely effects performance on 6.x since it makes your filesystem giant-locked, which may also interfere with your network processing. Any indications when this will be fixed? I need quota for user's files. Some time after users give enough feedback about the mpsafe quota patch. Kris pgpunoj4KhDOf.pgp Description: PGP signature
