Re: 40% slowdown with dynamic /bin/sh

2003-12-03 Thread Leo Bicknell
In a message written on Tue, Dec 02, 2003 at 06:05:40PM -0200, Daniel C. Sobral wrote:
> And if you are using FreeBSD on desktop you should use bash or some 
> other ports-shell, instead of slowing down _the_ shell for shell 
> scripts. But that's IMHO, so I didn't pipe this in until now. But it's 
> the very same argument you are making, so I thought you deserved to hear it.

Except that bash at least is linked statically in ports, supports
none of this, and people don't seem to want to change it.

Gotta love consistency.

-- 
   Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [EMAIL PROTECTED], www.tmbg.org


pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-12-03 Thread Kris Kennaway
On Wed, Dec 03, 2003 at 07:44:16AM -0600, Bob Willcox wrote:

> Nothing specific. I suppose it's just a space-time tradeoff from my
> point of view. With disk sizes what they are today (most of my systems
> have a system disk size of 40 GB or more), in my environment reducing
> the root filesystem size just isn't a priority.

Don't use it then, and let the thread die.

Kris


pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-12-03 Thread Bob Willcox
On Tue, Dec 02, 2003 at 04:48:27PM -0800, Tim Kientzle wrote:
> Bob Willcox wrote:
> >
> >What impact, if any, will this have on those of us that use NIS and
> >still want a statically linked root? I have been using NIS for years ...
> 
> First, let me clarify that I'm advocating moving NIS out of libc in
> the 6.0 timeframe.  Also, I'm not suggesting anyone replace NIS
> with LDAP.  FreeBSD currently has a strong bias for NIS over LDAP; I
> just think we should support both equally.
> 
> How would this affect a static root?
> 
> Make it a lot smaller and faster, for starters.  NIS adds
> as much as 400k each to many programs in /bin and /sbin.
> Over a quarter of a static /bin/sh is from NIS support.
> 
> Does that rule out NIS with a static root?
> 
> Yes, with the current NSS/PAM implementation, although a variety
> of suggestions have been floated around that would make NSS/PAM
> compatible with static binaries.  My personal favorite is to
> implement NSS/PAM daemons to satisfy such requests.  Such daemons
> are surprisingly simple to implement, in my experience.  I'm
> skeptical of efforts to use dlopen() with static binaries; static
> binaries don't have symbol tables, so there's no way to resolve
> references from the dlopen()-ed library back into the executable.
> 
> I'm curious, though.  The single most convincing argument so far in
> favor of a static root has been performance.  Doesn't the NIS network
> overhead swamp any performance gains from static linking?   I suspect
> you have other reasons for wanting a static root.  (Or do you only
> require certain executables to be static, such as /bin/sh?)

Nothing specific. I suppose it's just a space-time tradeoff from my
point of view. With disk sizes what they are today (most of my systems
have a system disk size of 40 GB or more), in my environment reducing
the root filesystem size just isn't a priority.

Bob

> 
> Tim Kientzle

-- 
Bob Willcox  First Law of Procrastination:
[EMAIL PROTECTED]   Procrastination shortens the job and places the
Austin, TX   responsibility for its termination on someone else (i.e.,
 the authority who imposed the deadline).
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-02 Thread Dan Nelson
In the last episode (Dec 02), Tim Kientzle said:
> Does that rule out NIS with a static root?
> 
> Yes, with the current NSS/PAM implementation, although a variety of
> suggestions have been floated around that would make NSS/PAM
> compatible with static binaries.  My personal favorite is to
> implement NSS/PAM daemons to satisfy such requests.  Such daemons are
> surprisingly simple to implement, in my experience.  I'm skeptical of
> efforts to use dlopen() with static binaries; static binaries don't
> have symbol tables, so there's no way to resolve references from the
> dlopen()-ed library back into the executable.
> 
> I'm curious, though.  The single most convincing argument so far in
> favor of a static root has been performance.  Doesn't the NIS network
> overhead swamp any performance gains from static linking?  I suspect
> you have other reasons for wanting a static root.  (Or do you only
> require certain executables to be static, such as /bin/sh?)

The programs in /bin or /sbin that link the NIS stuff in don't use it
all that often (rm, for example, will only do a user_from_uid call in
the -i case for a file not owned by the deleter), and unaccessed pages
in a static executable cause no performance hit.  I'm not sure that any
program in /bin or /sbin really does enough lookups for it to be a
speed factor.  Quite a few of the ones that pull in the NIS code really
do it for resolving hostnames.  /bin/sh is the best argument for an
external NSS lookup agent, since it should be fast but occasionally may
want to do NSS calls.

To list programs that pull in NIS:

ident /bin/* /sbin/* | awk '/^[^ ]/ { cmd=$1 } /nsdispatch/ { print cmd }' 

-- 
Dan Nelson
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-02 Thread Tim Kientzle
Bob Willcox wrote:
What impact, if any, will this have on those of us that use NIS and
still want a statically linked root? I have been using NIS for years ...
First, let me clarify that I'm advocating moving NIS out of libc in
the 6.0 timeframe.  Also, I'm not suggesting anyone replace NIS
with LDAP.  FreeBSD currently has a strong bias for NIS over LDAP; I
just think we should support both equally.
How would this affect a static root?

Make it a lot smaller and faster, for starters.  NIS adds
as much as 400k each to many programs in /bin and /sbin.
Over a quarter of a static /bin/sh is from NIS support.
Does that rule out NIS with a static root?

Yes, with the current NSS/PAM implementation, although a variety
of suggestions have been floated around that would make NSS/PAM
compatible with static binaries.  My personal favorite is to
implement NSS/PAM daemons to satisfy such requests.  Such daemons
are surprisingly simple to implement, in my experience.  I'm
skeptical of efforts to use dlopen() with static binaries; static
binaries don't have symbol tables, so there's no way to resolve
references from the dlopen()-ed library back into the executable.
I'm curious, though.  The single most convincing argument so far in
favor of a static root has been performance.  Doesn't the NIS network
overhead swamp any performance gains from static linking?   I suspect
you have other reasons for wanting a static root.  (Or do you only
require certain executables to be static, such as /bin/sh?)
Tim Kientzle

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-02 Thread Bob Willcox
On Tue, Dec 02, 2003 at 01:43:16PM -0800, Tim Kientzle wrote:
> Daniel C. Sobral wrote:
> >Now, my machines usually get by themselves, but all *I* do on them is 
> >sh(1) intensive, so I'll probably be using the static root option when 
> >it comes time to upgrade them to 5.x.
> 
> The static root option exists for people with special requirements:
>   * Use a lot of shell scripts
>   * Have very high performance requirements for those scripts
>   * Does not use LDAP, NIS, or other networked directory services
> (I want to move NIS out of libc.  Its privileged position
> relative to LDAP and other services is rather dated.)

What impact, if any, will this have on those of us that use NIS and
still want a statically linked root? I have been using NIS for years
and see no compelling reason to switch to LDAP (i.e., NIS satisfies my
needs, is configured, and working).

Bob

-- 
Bob Willcox  First Law of Procrastination:
[EMAIL PROTECTED]   Procrastination shortens the job and places the
Austin, TX   responsibility for its termination on someone else (i.e.,
 the authority who imposed the deadline).
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-02 Thread Tim Kientzle
Daniel C. Sobral wrote:
Now, my machines usually get by themselves, but all *I* do on them is 
sh(1) intensive, so I'll probably be using the static root option when 
it comes time to upgrade them to 5.x.
The static root option exists for people with special requirements:
  * Use a lot of shell scripts
  * Have very high performance requirements for those scripts
  * Does not use LDAP, NIS, or other networked directory services
(I want to move NIS out of libc.  Its privileged position
relative to LDAP and other services is rather dated.)
E.g., someone with a standalone server that's using
/bin/sh to run CGIs would be well advised to compile
a static /bin/sh.  Fortunately, that's easy to do.
And if you are using FreeBSD on desktop you should use bash or some 
other ports-shell, instead of slowing down _the_ shell for shell 
scripts.
User shell scripts do rely on tilde expansion.  Having
that break in any environment relying on networked directory
services is a pretty serious loss of functionality.
Tim Kientzle

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-02 Thread Daniel C. Sobral
Daniel O'Connor wrote:
Why didn't you pipe up when this was discussed _long_ ago?
I don't understand why you can't buildworld with static slash if you feel so 
strongly about it.
As for that, it was discussed on arch@, not [EMAIL PROTECTED] And _I_, 
personally, got the impression the plan was to make dynamic world an 
_option_, not the default.

Now, my machines usually get by themselves, but all *I* do on them is 
sh(1) intensive, so I'll probably be using the static root option when 
it comes time to upgrade them to 5.x.

If you are deploying FreeBSD on servers you should build your own release 
anyway (which is hardly an onerous task).
And if you are using FreeBSD on desktop you should use bash or some 
other ports-shell, instead of slowing down _the_ shell for shell 
scripts. But that's IMHO, so I didn't pipe this in until now. But it's 
the very same argument you are making, so I thought you deserved to hear it.

--
Daniel C. Sobral
Gerência de Operações
Divisão de Comunicação de Dados
Coordenação de Segurança
VIVO Centro Oeste Norte
Fones: 55-61-313-7654/Cel: 55-61-9618-0904
E-mail: [EMAIL PROTECTED]
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-01 Thread Jonathan Mini
On Dec 1, 2003, at 10:15 PM, Scott Long wrote:

Jonathan Mini wrote:
I have found that the cost of printing the spew often
slows down compiles measurably, especially when spewing
to an xterm running on a local XFree86 process.  Even
with syscons, this is noticeable.
I generally tend to run my builds behind the screen
port these days, which  helps (screen implements a
virtual display buffer that disconnects screen updates
from the display client and the slave pty).  Another
optimization worth noting is running make -q, which
silences a lot of that spam (urban legend has it that
the synchronization in parallel makes to write the build
messages causes noticeable amounts of contention).
I regularly use -s.  With the pipe code being completely
Giant-free, I don't see a significant performance difference
anymore on an SMP machine when the output is not supressed.
Ah, I meant -s.  And, that is good news.  I am still running
on a 4.x system most of the time, and (sigh) Linux 2.4.x the
rest of the time.
--
Jonathan Mini
[EMAIL PROTECTED]
http://www.freebsd.org
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-01 Thread Scott Long
Jonathan Mini wrote:
I have found that the cost of printing the spew often
slows down compiles measurably, especially when spewing
to an xterm running on a local XFree86 process.  Even
with syscons, this is noticeable.
I generally tend to run my builds behind the screen
port these days, which  helps (screen implements a
virtual display buffer that disconnects screen updates
from the display client and the slave pty).  Another
optimization worth noting is running make -q, which
silences a lot of that spam (urban legend has it that
the synchronization in parallel makes to write the build
messages causes noticeable amounts of contention).
I regularly use -s.  With the pipe code being completely
Giant-free, I don't see a significant performance difference
anymore on an SMP machine when the output is not supressed.
Scott

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-12-01 Thread Jonathan Mini
I have found that the cost of printing the spew often
slows down compiles measurably, especially when spewing
to an xterm running on a local XFree86 process.  Even
with syscons, this is noticeable.
I generally tend to run my builds behind the screen
port these days, which  helps (screen implements a
virtual display buffer that disconnects screen updates
from the display client and the slave pty).  Another
optimization worth noting is running make -q, which
silences a lot of that spam (urban legend has it that
the synchronization in parallel makes to write the build
messages causes noticeable amounts of contention).
On Nov 29, 2003, at 3:22 PM, Garance A Drosihn wrote:

I also had some benchmarks of doing 'buildworld' over an
ssh connection vs doing it at the console.  Oddly enough,
the ssh connection was faster in some ways and slower in
others.  I wonder if there is a speed-up by writing to a
file instead of the console?
--
Jonathan Mini
[EMAIL PROTECTED]
http://www.freebsd.org
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-29 Thread Garance A Drosihn
At 6:27 PM +1100 11/27/03, Bruce Evans wrote:
On Wed, 26 Nov 2003, Garance A Drosihn wrote:
 >
 > I have reformatted the numbers that Michael reported,
 into the following table:

 >Static /bin/sh: Dynamic /bin/sh:
 >   real385m29.977s real455m44.852s   => 18.22%
 >   user111m58.508s user113m17.807s   =>  1.18%
 >   sys  93m14.450s sys 103m16.509s   => 10.76%
 >   user+sys  =>  5.53%
What are people doing to make buildworld so slow?
Well, I'm not *trying* to make it slow...  :-)

 > Here are some buildworld numbers of my own, from my system.
 In my case, I am running on a single Athlon MP2000, with a
 gig of memory.  It does a buildworld without paging to disk.
I have a similar configuration, except with a single Athlon
XP1600 overclocked by 146/133 and I always benchmark full
makeworlds.  I was unhappy when the gcc pessimizations between
gcc-2.95 and gcc-3.0 increased the makeworld time from about
24 minutes to about 33 minutes.  The time has since increased
to about 38 minutes.  The latter is cheating slightly -- I
leave out the DYNAMICROOT and RESCUE mistakes and the KERBEROS
non-mistake.
I keep DYNAMICROOT, RESCUE, and KERBEROS.  I'm tempted to
drop RESCUE just to see how much of a difference it would
make.
This also shows why -j should not be used on non-SMP machines.
...so it can only possibly help on systems where unbalanced
resources (mainly slow disks) give too much idle time.
I got in the habit if using -j3 back on a system where my
disks were running ATA/33, even though they should have
been faster.  (eventually I figured out that the IDE cable
had been hooked up backwards)
 > Buildworld, static, with no '-j',
   executed /bin/sh  32,308 times.

 Buildworld, static, with '-j2',
   executed /bin/sh  32,802 times.
Turning on accounting must have pessimized things a bit.
I did not turn on accounting.  I added a simple kludge
to /bin/sh so I would keep track of how often it was
called.  I had a benchmark of how much that slowed down
buildworld, but I've deleted that now.  iirc, It wasn't
too much, and it would have been the same amount for all
the builds that I gave numbers on.
I think you are also using a pessimized kernel (with
INVARIANTS and WITNESS).  makeworld times should be
dominated by the gcc hog, but your sys times are almost
as large as your user times.
Ugh.  I forgot to check those.

Yeah, I have:  INVARIANTS, INVARIANT_SUPPORT, and WITNESS.
I used to have them off, but when I bought the newer Althon
I turned them back on.  I also have them off on my sparc.
(I do keep WITNESS_SKIPSPIN, because dropping that gives me
more of a penalty than I want).

 > On all attempts, I started out by doing:
  rm -Rf /usr/obj/usr/src/*
  sync ; sleep 1 ; sync ; sleep 1 ; sync
 > before doing the 'make' command.

I use:
 [...skipping]
# Sometimes: export __MAKE_CONF=/etc/nonesuch
cd /wherever/src || exit 1
DESTDIR=/c/z/root \
MAKEOBJDIRPREFIX=/c/z/obj \
time -l make -s world > /tmp/world.out 2>&1
I also had some benchmarks of doing 'buildworld' over an
ssh connection vs doing it at the console.  Oddly enough,
the ssh connection was faster in some ways and slower in
others.  I wonder if there is a speed-up by writing to a
file instead of the console?
Rebooting doesn't affect the times much in relative terms
(...), but it reduces the variance to less than a second
provided the system is mostly idle.
I had done a few buildworlds before starting any of the
buildworlds that I reported, and they seemed to be fairly
consistent after the first one.  Not "less than a second"
though, I think it was a few seconds difference.  Mainly I
just wanted to avoid the reboots.  These tests had chewed
up enough of my day as it was...
 > Aside: building 5.1-"security" on this same hardware took
 the following times:
 >real54m10.092s   [  71.03% ]
user41m39.121s   [  24.40% ]
sys 10m20.325s   [ 210.69% ]
 And those times *are* with 'script' running, as well as a
 perl-script which I use to summarize "interesting" data from
 the output of a buildworld.  So, those times include extra
 overhead which is not included in the above buildworlds.
 That's from a 'make -j3', and obviously has a static /bin/sh.
Why so much faster?  Now the times are only 20% larger than mine,

Building rescue only accounts for about 2 minutes of the
86-54 difference.
Hmm.  Dunno.  I had assumed it was /rescue, but I did mean
to go back and get a better idea.  It did seem a bit large.
It might very well be that the 5.1-"secure" build was done
on a 5.1-release GENERIC kernel.  Ie, without INVARIANTS,
INVARIANT_SUPPORT, and WITNESS.  Those numbers are from a
left-over log of the build that I just happened to have
around, so it was done not in as controlled a setup as the
other builds.
 > For those who think I'm spoiled by fast hardware, ...
 > ... on my sparc64 box.  So

Re: 40% slowdown with dynamic /bin/sh

2003-11-27 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
Bruce Evans <[EMAIL PROTECTED]> writes:
: What are people doing to make buildworld so slow?

using gcc 3 :-)

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-27 Thread Bruce Evans
On Wed, 26 Nov 2003, Garance A Drosihn wrote:

> At 12:23 AM -0500 11/26/03, Michael Edenfield wrote:
> >
> >Just to provide some real-world numbers, here's what I got
> >out of a buildworld:
>
> I have reformatted the numbers that Michael reported,
> into the following table:
>
> >Static /bin/sh: Dynamic /bin/sh:
> >   real385m29.977s real455m44.852s   => 18.22%
> >   user111m58.508s user113m17.807s   =>  1.18%
> >   sys  93m14.450s sys 103m16.509s   => 10.76%
> >   user+sys  =>  5.53%

What are people doing to make buildworld so slow?  I once optimized
makeworld to take 75 minutes on a K6-233 with 64MB of RAM.  Things
have been pessimized a bit since then, but not signifcantly except for
the 100% slowdown of gcc (we now build large things like secure but
this is partly compensated for by not building large things like perl).
Michael's K7-500 with 320MB (?) of RAM should be serveral times faster
than the K6-233, so I would be unhappy if it took more than 75 minutes
but would expect it to take bit more than 2 hours when well configured.

> Here are some buildworld numbers of my own, from my system.
> In my case, I am running on a single Athlon MP2000, with a
> gig of memory.  It does a buildworld without paging to disk.

I have a similar configuration, except with a single Athlon XP1600
overclocked by 146/133 and I always benchmark full makeworlds.  I
was unhappy when the gcc pessimizations between gcc-2.95 and gcc-3.0
increased the makeworld time from about 24 minutes to about 33 minutes.
The time has since increased to about 38 minutes.  The latter is
cheating slightly -- I leave out the DYNAMICROOT and RESCUE mistakes
and the KERBEROS non-mistake.

> Static sh, No -j:  Dynamic sh, No -j:
>real84m31.366s real86m22.429s   =>  2.04%
>user50m33.013s user51m13.080s   =>  1.32%
>sys 29m59.047s sys 33m04.082s   => 10.29%
>   user+sys =>  4.66%
>
> Static sh, -j2:Dynamic sh, -j2:
>real92m38.656s real95m21.027s   =>  2.92%
>user51m48.970s user52m29.152s   =>  1.29%
>sys 32m07.293s sys 34m40.595s   =>  7.95%
>   user+sys =>  3.84%

This also shows why -j should not be used on non-SMP machines.  Apart
from the make -j bug that causes missed opportunties to run a job,
make -j increases real and user times due to competition for resources,
so it can only possibly help on systems where unbalanced resources (mainly
slow disks) give too much idle time.

My current worst makeworld time is almost twice as small as the fastest
buildworld time in the above (2788 seconds vs 5071 seconds).  From my
collection of makeworld benchmarks:

%%%
Fastest makeworld on a Celeron 366 overclocked by 95/66 (2000/05/15):
3309.30 real  2443.75 user   488.68 sys

Last makeworld on a Celeron 366 overclocked by 95/66 (2001/11/19):
4219.83 real  3253.04 user   667.64 sys

Fastest makeworld on an Athlon XP1600 overclocked by 146/133 (2002/01/03):
1390.18 real   913.56 user   232.63 sys

Last makeworld before gcc-3 on an Athlon XP1600 o/c by 143/133 (2002/05/09)
(overclocking reduced and due to memory problems and some local
memory-related optimizations turned off):
 1532.99 real  1093.08 user   293.15 sys

Early makeworld with gcc-3 on an Athlon XP1600 o/c by 143/133 (2002/05/12):
2268.13 real  1613.25 user   313.56 sys

Fastest makeworld with gcc-3 an Athlon XP1600 overclocked by 146/133
(maximal overclocking recovered; memory increased from 512MB to 1GB, local
memory-related optimizations turned on and tuned) (2003/03/31):
1929.02 real  1576.67 user   205.30 sys

Last makeworld before  on an Athlon XP1600 o/c by 143/133 (2003/04/29:
2012.75 real  1637.59 user   225.07 sys

Makeworld with the defaults (no /etc/make.conf and no local optimizations
in the src tree; mainly no pessimizing for Athlons by optimizing for PII's,
and no building dependencies; only optimizations in the host environment
(mainly no dynamic linkage) on an Athlon as usual (2003/05/06):

Last recorded makeworld with local source and make.conf optimizations
(mainly no dynamic linkage) on an Athlon as usual (2003/10/22):
2225.83 real  1890.64 user   256.33 sys

Last recorded makeworld with the defaults on an Athlon as usual (2003/11/11):
2788.41 real  2316.49 user   357.34 sys
%%%

I don't see such a large slowdown from using a dynamic /bin/sh.  Unrecorded
runs of makeworld gave times like the following:

2262 real ... with local opts including src ones and no dynamic linkage
2290 real ... with same except for /bin/sh (only) dynamically linked

The difference may be because my /usr/bin/true and similar utilities remain
statically linked.  Fork-exec expense depends mor on the exec than the fork.
>From an old 

Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread E.B. Dreger
MD> Date: Tue, 25 Nov 2003 11:50:25 -0800 (PST)
MD> From: Matthew Dillon

MD> (B) the authentication code access an IPC service which *ONLY* allows
MD> challenge/response, does *NOT* give you direct access to the
MD> encrypted contents of the password file, and which limits the challenge
MD> rate to prevent dictionary attacks?

Sounds kind of like SASL...


Eddy
--
Brotsman & Dreger, Inc. - EverQuick Internet Division
Bandwidth, consulting, e-commerce, hosting, and network building
Phone: +1 785 865 5885 Lawrence and [inter]national
Phone: +1 316 794 8922 Wichita
_
  DO NOT send mail to the following addresses :
  [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED]
Sending mail to spambait addresses is a great way to get blocked.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Robert Watson

On Wed, 26 Nov 2003, Terry Lambert wrote:

> I don't know what Matt is planning on delivering, but...
> 
> http://developer.apple.com/darwin/projects/opendirectory/
> 
> [...]
>   lookupd is included with the Darwin project and is
>   documented online in Apple's Support database and
>   as part of Mac OS X and Darwin in the form of "man"
>   pages.
> [...]

Hmm.  Are you offering to port Mach IPC to FreeBSD as well? :-) 

> FreeBSD has also been offered, at least three times, similar proxy code
> from two universities, under BSD license. 

Can't speak to that, not having seen it.  Got some pointers?

I'm fine with the notion of using IPC-based directory services, but it's
not simply a question of finding a tarball and committing it.  There's no
lack of standards, APIs, etc, to pick from.  Since you're pointing at an
Apple solution, it's worth pointing out that Apple has also been in flux
on the topic of directory service APIs over the last two years, trying to
move away from NetInfo and towards DirectoryServices.  One of the things
we're still trying to address for FreeBSD is how to fit into a world of
distributed directory services where, until the last year or so, it's
really been unclear what the most interoperable and easiest to manage
solution would be.

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Michael Edenfield
* M. Warner Losh <[EMAIL PROTECTED]> [031126 14:51]:
> In message: <[EMAIL PROTECTED]>
> Michael Edenfield <[EMAIL PROTECTED]> writes:
> : They were on a single CPU Athlon 500 with 320MB of RAM.
> 
> 320MB is not enough RAM not to swap.
> 
> However, having said that, I think everybody realizes the following:
> 
>   1) Dynamic linking is slower.
>   2) Speed improvements in this area are possible, as
>  demonstrated by other projects.
>   3) PIC code is slower than non-PIC code, in general, and also
>  gcc runs about 5-10% slower depending on if you are running
>  out of a shared library or a static one.  shared libraries
>  must use PIC code (at this time).
>   4) People like to complain.

Just for the record, I've been running WITH_DYNAMICROOT since nearly the
day it came out and don't *notice* any problems.  Mostly because the
noise of dynamic linking overhead gets lost in the noise of "my hardware
sucks so bad I have to take a vacation during buildworlds."  My startup
takes upwards of 5 minutes anyway, another 45 seconds won't even make me
blink.  I'm certainly not complaining about the performance :)

I only posted those numbers to:

1) Give real world numbers, not "interesting but unrealistic numbers"
2) Show that even worst-case numbers weren't on the level of 40% slowdown.
3) Hopefully help someone figure out where to best improve the dynamic linker.

--Mike



pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Matthew Dillon

: > That seems to have the most impact.  We can also expend our efforts
: > to improve dynamic linking performance, since that will improve the
: > performance of the other 99.9% of the universe.
: > 
:
:What happened to mdodd's prebinding efforts?
:
:Drew

Prebinding was put into DFly but the improvement is barely noticeable
when you prebind /bin/sh.  Prebinding is only really useful for much
larger programs like mozilla which have thousands or tens of thousands
of bindings.  These numbers are on DragonFly and will be different on
FBsd, but the relative numbers should be similar with a FBsd prebinding
implementation.

   /bin/sh   
--  
static  freepg -16 zfod 12
dynamic freepg -46 zfod 34
dyn/prebind freepg -46 zfod 26  <<< saves 8*4=32KB worth of
pre-zero'd pages but does
not save any anon memory.

1000 runs of /bin/sh -c exit 0 (program to vfork/exec /bin/sh -c exit 0
1000 times).  This effectively measures startup overhead only and is
not inclusive of what a script might do in addition to starting up.

/bin/sh
static  11023 zfod's00.54s
dynamic 33023 zfod's02.88s
dyn/prebind 25023 zfod's02.36s

There isn't much of a time improvement but there is a significant
improvement in the number of ZFOD's with prebinding.  


leaf:/usr/obj/usr/src/bin/sh# prebind /usr/obj/usr/src/bin/sh/sh
  object /usr/obj/usr/src/bin/sh/sh uniqid 986137809
  object /usr/lib/libedit.so.3  uniqid -1757875926
  object /usr/lib/libncurses.so.5   uniqid 1023436343
  object /usr/lib/libc.so.4 uniqid 2011683737
Non-PLT Prebindings:
  object /usr/lib/libedit.so.3  count 5
  object /usr/lib/libncurses.so.5   count 63
  object /usr/lib/libc.so.4 count 203
PLT Prebindings:
  object /usr/obj/usr/src/bin/sh/sh count 106
  object /usr/lib/libedit.so.3  count 63
  object /usr/lib/libncurses.so.5   count 270
  object /usr/lib/libc.so.4 count 638
Non-PLT COPY Prebindings:
  object /usr/obj/usr/src/bin/sh/sh count 13

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
Michael Edenfield <[EMAIL PROTECTED]> writes:
: time make -j 4 buildworld

Hmmm, more jobs.

: They were on a single CPU Athlon 500 with 320MB of RAM.

320MB is not enough RAM not to swap.

I did some preliminary testing last night (which I lost due to a
crash) that showed that a simple 'make buildworld' slowed down 1-2%
depending on how many times I ran them.

However, make -s buildworld (with or without -j 4 on my dual athlon)
was faster than a normal buildworld, but the dynamic /bin/sh was more
like 5-7% slower.  The difference here is that there are fewer context
switches (and I guess less chance for parallelization).

In all make buildworld, the number of page faults was 10x for the
dynamic case than for the static case.

However, having said that, I think everybody realizes the following:

1) Dynamic linking is slower.
2) Speed improvements in this area are possible, as
   demonstrated by other projects.
3) PIC code is slower than non-PIC code, in general, and also
   gcc runs about 5-10% slower depending on if you are running
   out of a shared library or a static one.  shared libraries
   must use PIC code (at this time).
4) People like to complain.

Warner

P.S.  One interesting note: /bin/sh when linked statically with
libedit and libncurses but dynamically with libc runs about 10% slower
for my /usr/bin/true/true tests than when all three are dynamically
linked.  So it seems that not all dynamic linking is bad for
performance.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Matthew Dillon

:At 00:23 26/11/2003 -0500, Michael Edenfield wrote:
:>Static /bin/sh:
:>   real385m29.977s
:>   user111m58.508s
:>   sys 93m14.450s
:>
:>Dynamic /bin/sh:
:>   real455m44.852s
:>   user113m17.807s
:>   sys 103m16.509s
:
:   Given that user+sys << real in both cases, it looks like you're running 
:out of memory; it's not surprising that dynamic linking has an increased 
:cost in such circumstances, since reading the diverse files into memory 
:will take longer than reading a single static binary.
:   I doubt many systems will experience this sort of performance delta.
:
:Colin Percival

It definitely looks memory related but the system isn't necessarily
'running out' of memory.  It could simply be that the less memory
available for caching files is causing more disk I/O to occur.  It
should be possible to quanity this by doing a full timing of the
build ( /usr/bin/time -l ), which theoretically includes I/O ops.

Dynamically linked code definitely dirties more anonymous memory then
static, and definitely accesses more shared file pages.  The difference
is going to depend on the complexity of the program.  How much this
effects system peformance depends on the situation.  If the system has
significant idle cycles available the impact should not be too serious,
but if it doesn't then the overhead will drag down the pre-zerod pages
(even if the program is exec'd, does something real quick, and exits).

I have included a program below that prints the delta free page count
and the delta zero-fill count once a second.  This can be used to
estimate anonymous memory use.  Run the program and let it stabilize.
Be sure that the system is idle. Then run the target program (it needs
to stick around, it can't just exec and exit), then exit the target
program and repeat.  Leave several seconds in between invocation, exit,
and repeat to allow the system to stabilize.  Note that it may take 
several runs to get reliable information since the program is measuring
anonymous memory use for the whole system.  Also note that shared pages
will not be measured by this program, only the number of dirtied
anonymous pages.  If on an idle system the program is not reporting
'0 0' then your system isn't idle :-).

The main indicator is the 'freepg' negative jump when the target program
is invoked.  The zfod count will be a subset of that, indicating the
number of zero-fill pages requested (verses program text/data COW pages
which do not need zero'd pages but still eat anonymous memory for the
duration of the target program).

When I tested it with a static and dynamic /bin/sh on 4.8 I got 
(looking at 'freepg'), 20 pages for the static binary and 50 pages for
the dynamic binary.  So a dynamic /bin/sh eats 30 * 4K = 120K more 
anonymous memory then a static /bin/sh.  In the same test I got 
12 ZFOD faults for the static binary and 34 ZFOD faults for the 
dynamic binary, which means that 22 additional pre-zero'd pages are
being allocated in the dynamic case (88KB).

If /bin/sh is exec'd a lot in a situation where the system is otherwise
not idle, this will impact the number of pre-zero'd pages available on
the system.  Each exec of a dyanmic /bin/sh eats 22 additional pages
(88K) worth of zero-fill.  Each resident copy of (exec'd) /bin/sh eats
120KB more dirty anonymous memory.  make buildworld -j 1 may have as 
many as a dozen /bin/sh's exec'd at any given moment (impact 120K each)
depending on where in the build it is.  -j 2 and so forth will have
even more.  This will impact your system relative to the amount of total
system memory you have.  The more system memory you have, the less the
percentage impact.

/bin/sh /bin/csh
--  ---
static  freepg -19 zfod 12  freepg -140 zfod 129
dynamic freepg -50 zfod 34  freepg -167 zfod 149

/usr/bin/make  (note that make is static by default)
--
static  freepg -33 zfod 27
dynamic freepg -51 zfod 44


As you can see, the issue becomes less significant on a percentage
basis with larger programs that already allocate more incidental memory.
Also to my surprise I found that 'make' was already static.  It would
seem that this issue was recognized long ago.  bzip2, chflags, make,
and objformat are compiled statically even though they reside in /usr/bin.

-Matt

/*
 * print delta free pages and zfod requests once a second.  Leave running
 * while testing other programs.  Note: ozfod is not displayed.  ozfod is
 * a subset of zfod, just as zfod deltas are a subset of v_free_count
 * allocations.
 */

#include 
#include 
#include 
#include 

int
main(int ac, char 

Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Michael Edenfield
* Garance A Drosihn <[EMAIL PROTECTED]> [031126 06:56]:
> At 12:23 AM -0500 11/26/03, Michael Edenfield wrote:
> >
> >Just to provide some real-world numbers, here's what I got
> >out of a buildworld:
> 
> I have reformatted the numbers that Michael reported,
> into the following table:
> 
> >Static /bin/sh: Dynamic /bin/sh:
> >  real385m29.977s real455m44.852s   => 18.22%
> >  user111m58.508s user113m17.807s   =>  1.18%
> >  sys  93m14.450s sys 103m16.509s   => 10.76%
> >  user+sys  =>  5.53%

Since I forgot to include this information (sorry!):

Both runs were done by doing:

rm -rf /usr/obj
sync
script 
cp -f /bin/sh.{dynamic,static} /bin/sh
file /bin/sh
time make -j 4 buildworld

They were on a single CPU Athlon 500 with 320MB of RAM.  I actually
don't normally do -j 4 on this system, only -j 2, but I'm use to
building on the dual Athlons we use to make production kernels and it
slipped in.  Since it takes hours to run once it's started I just let it
run. :)

--Mike



pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Michael Edenfield
* M. Warner Losh <[EMAIL PROTECTED]> [031126 00:43]:
> In message: <[EMAIL PROTECTED]>
> Michael Edenfield <[EMAIL PROTECTED]> writes:
> : * M. Warner Losh <[EMAIL PROTECTED]> [031125 12:07]:
> : > In message: <[EMAIL PROTECTED]>
> : > "boyd, rounin" <[EMAIL PROTECTED]> writes:
> : > : i see that there some doubt about whether running lots of
> : > : shell scripts ever happens.  what happens when you
> : > : use make?  lots of shells get run and they run small
> : > : (one line?) scripts.
> : > 
> : > make buildworld slows down < 1% when you switch from static /bin/sh to
> : > dynamic.
> : 
> : I'm all for dynamic / and dynamic /bin/sh, but this doesn't even come
> : close to what I observed on my systems.  I see anywhere from 15% to 20%
> : slowdown in buildworld, depending on how bad my hardware already is.  I
> : posted my most recent numbers earlier.
> 
> My dual athlon make -j 4 buildworld differed by about 16-20 seconds on
> a 36 minute buildworld.
> 
> : It's difficult to get lots of these numbers, unfortunately, because it
> : takes a good 6-8 hours just to complete one build.  But the numbers are
> : fairly consistant across the different degrees of old crappy hardware I
> : have.
> 
> So you are seeing a about an hour slowdown (16% slowdown on 6 hours is
> 1 hour) from before/after?  Or are you seeing an hour slowdown from
> 4.x -> 5.2-beta?

I have two 5.x systems, both with dynamic / that were built within the
past month.  One's a bit older, probably a month or so, as I was waiting
for the statfs changes to settle before upgrading it.  The other was built
about 3 days ago.

The first one is pretty old, I only use it for a firewall because no one
will let me spring for a replacement:

CPU: Pentium II/Pentium II Xeon/Celeron (399.10-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x665  Stepping = 5
  
Features=0x183f9ff
real memory  = 66977792 (63 MB)
avail memory = 60477440 (57 MB)
ad0: 19470MB  [39560/16/63] at ata0-master UDMA33


This machine usually takes 10-12 hours to do a full buildworld with -j 3
(which somehow seems to be the fastest).  With static /bin/sh it was
took just about an hour and a half less, but again, I could only do one
pair since that took the whole day :)

The other is slightly better and is my personal FreeBSD workstation,
which I run -CURRENT on for test purposes and cuz I like it better :)

CPU: AMD-K7(tm) Processor (499.04-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x612  Stepping = 2
  Features=0x81f9ff
  AMD Features=0xc040
real memory  = 335478784 (319 MB)
avail memory = 316243968 (301 MB)
ad0: 16603MB  [33735/16/63] at ata0-master UDMA66

This one completes a buildworld in about 7-8 hours, the static /bin/sh
run took about an hour less.  I posted those numbers here earlier.

I don't have any decent hardware running 5.x, all the new machines in
real user are still using 4.8, so these are the only numbers I can come
up with.  Again, I *like* the ability to have NSS in /bin/sh, and the
idea of dynamic linking in general appeals to me.  The hour to
hour-and-a-half slowdown might seem huge, but `make buildworld` really
is the worst case scenario I could come up with, and 15% slowdown in the
*worst* real-world case is certainly much better than 40%.

--Mike



pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread andy-freebsd
obrien wrote @ Tue, 25 Nov 2003 18:55:05 -0800:
> On Tue, Nov 25, 2003 at 03:07:55PM +1030, Daniel O'Connor wrote:
> > What about the newer version of gcc? That is considerably slower than
> > previous versions, but I don't see people screaming to have it removed.
> 
> Uh... you must not know what you are talking about.  GCC *COMPILES*
> slower as it does a better job of optimizing (which adds time to the
> compiling time).  The produced optimzied binaries have quicker
> *RUN-TIME*s.

For C++ maybe, for C compile times (tripled?) stood in no relation
to the run-time "improvements" (minor to negative), but GCC ppl are constantly
improving (3.3 and on) and 2.9x was a dead horse.

I see only very minor relation to the static/dynamic discussion.

I don't know what the reason is for going dynamic by default _now_,
but i am sure it got nothing to do with the unbelievable
points presented till now like space savings or NSS.
Stuff gets default when it's ready, and that would
in this case mean after performance of dynamic
binaries has improved and those ppl that want to introduce it
present a solution, where everything has been thought of,
and discussions like /bin/sh or /sbin/sh have already
been had and solved.

Introduce dynamic option now.
Improve dynamic linking and
make a good solution for a rescue.
Then and only then make it the default.

Everything else leads to threads like these,
because it doesn't make sense to ppl.


aha
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Garance A Drosihn
At 12:23 AM -0500 11/26/03, Michael Edenfield wrote:
Just to provide some real-world numbers, here's what I got
out of a buildworld:
I have reformatted the numbers that Michael reported,
into the following table:
Static /bin/sh: Dynamic /bin/sh:
  real385m29.977s real455m44.852s   => 18.22%
  user111m58.508s user113m17.807s   =>  1.18%
  sys  93m14.450s sys 103m16.509s   => 10.76%
  user+sys  =>  5.53%
Here are some buildworld numbers of my own, from my system.
In my case, I am running on a single Athlon MP2000, with a
gig of memory.  It does a buildworld without paging to disk.
Static sh, No -j:  Dynamic sh, No -j:
  real84m31.366s real86m22.429s   =>  2.04%
  user50m33.013s user51m13.080s   =>  1.32%
  sys 29m59.047s sys 33m04.082s   => 10.29%
 user+sys =>  4.66%
Static sh, -j2:Dynamic sh, -j2:
  real92m38.656s real95m21.027s   =>  2.92%
  user51m48.970s user52m29.152s   =>  1.29%
  sys 32m07.293s sys 34m40.595s   =>  7.95%
 user+sys =>  3.84%
Buildworld, static, with no '-j',
 executed /bin/sh  32,308 times.
Buildworld, static, with '-j2',
 executed /bin/sh  32,802 times.
(I was expecting a much larger difference between the
number of /bin/sh's in a plain 'make buildworld' versus
'make -j2 buildworld'  But apparently I was wrong...).
On all attempts, I started out by doing:
rm -Rf /usr/obj/usr/src/*
sync ; sleep 1 ; sync ; sleep 1 ; sync
before doing the 'make' command.  I usually start up a 'script'
command to capture the output of buildworld, but I did not do
that for these tests.  All of the above buildworlds were typed
in at the console.  Plain console, no X11 running.  All are
running on a snapshot of -current as of sometime Tuesday.  All
are compiling the exact same /usr/src (left over from installing
that snapshot of -current).
Aside: building 5.1-"security" on this same hardware took
the following times:
  real54m10.092s   [  71.03% ]
  user41m39.121s   [  24.40% ]
  sys 10m20.325s   [ 210.69% ]
And those times *are* with 'script' running, as well as a
perl-script which I use to summarize "interesting" data from
the output of a buildworld.  So, those times include extra
overhead which is not included in the above buildworlds.
That's from a 'make -j3', and obviously has a static /bin/sh.
So, if you take that as the base, then the buildworld for
5.2-release (using *static* /bin/sh and -j2) will see the
performance hits that I put in brackets.  That probably seems
like a pretty horrifying hit, but remember that 5.1-release
did *not* build /rescue at all (not for me at least :-), and
that is probably a significant part of the increase.
I add all this just because it is easy to imagine that some
people will do "world-stones" of 5.1-release vs 5.2-release,
and might assume that the entire difference is due to /bin/sh
or the dynamic /bin & /sbin in general.  The above is not a
good enough benchmark to say *where* the time has gone, but
please consider the issue of building /rescue!
I have attempted to do a "decent" job at coming up with all
these numbers, but I'm sure they are not perfect.  It would
be good to use something other than "world-stones" as a
benchmark, but of course I'm not volunteering to do that...
For those who think I'm spoiled by fast hardware, please note
that all of the above has been done while doing just two
buildworlds and one buildkernel+installkernel on my sparc64
box (and that second buildworld is not done yet...).  So I
certainly am interested in how freebsd runs on "slower HW"!
--
Garance Alistair Drosehn=   [EMAIL PROTECTED]
Senior Systems Programmer   or  [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-26 Thread Terry Lambert
Brad Knowles wrote:
> At 2:48 PM -0800 2003/11/25, Matthew Dillon wrote:
> >  What I am advocating is that FreeBSD-5 not marginalize and
> >  restrict (make less flexible) basic infrastructure in order to get other
> >  infrastructure working.
> 
> If you've got working, debugged code that works in the manner you
> are espousing, and still achieves the same goal of making NSS and PAM
> work everywhere, I'm sure we'd all love to see it.
> 
> In the absence of any code contribution to the contrary, I see no
> alternative than the method that has been selected.  Sure, it's not
> great.  Sure, it's slower (more or less, depending on which
> benchmarks you believe).

I don't know what Matt is planning on delivering, but...

http://developer.apple.com/darwin/projects/opendirectory/

[...]
lookupd is included with the Darwin project and is
documented online in Apple's Support database and
as part of Mac OS X and Darwin in the form of "man"
pages.
[...]

FreeBSD has also been offered, at least three times, similar proxy
code from two universities, under BSD license.

I think the ball is in the court of the people asking "Where is
the code?" to take the code which has been made available, and
commit it.

Lest this be considered advocacy, let me emphasize that I think it
is not the canonically correct approach to do this.

FWIW, my personal presference is a libdlopen that can be linked
statically.  I've explained how to make one three times, have
offered my own prototype code (which requires GCC modifications
to constructor parameter lists in the generated code, for C++ to
continue working), and an European fellow posted an implementation
that, while it didn't add full functionality, was good enough to
enable writing PAM and NSS modules, at least, even though it would
not permit a satically linked JRE to run JNI modules.

-- Terry
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Colin Percival
At 00:23 26/11/2003 -0500, Michael Edenfield wrote:
Static /bin/sh:
  real385m29.977s
  user111m58.508s
  sys 93m14.450s
Dynamic /bin/sh:
  real455m44.852s
  user113m17.807s
  sys 103m16.509s
  Given that user+sys << real in both cases, it looks like you're running 
out of memory; it's not surprising that dynamic linking has an increased 
cost in such circumstances, since reading the diverse files into memory 
will take longer than reading a single static binary.
  I doubt many systems will experience this sort of performance delta.

Colin Percival

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
Michael Edenfield <[EMAIL PROTECTED]> writes:
: * M. Warner Losh <[EMAIL PROTECTED]> [031125 12:07]:
: > In message: <[EMAIL PROTECTED]>
: > "boyd, rounin" <[EMAIL PROTECTED]> writes:
: > : i see that there some doubt about whether running lots of
: > : shell scripts ever happens.  what happens when you
: > : use make?  lots of shells get run and they run small
: > : (one line?) scripts.
: > 
: > make buildworld slows down < 1% when you switch from static /bin/sh to
: > dynamic.
: 
: I'm all for dynamic / and dynamic /bin/sh, but this doesn't even come
: close to what I observed on my systems.  I see anywhere from 15% to 20%
: slowdown in buildworld, depending on how bad my hardware already is.  I
: posted my most recent numbers earlier.

My dual athlon make -j 4 buildworld differed by about 16-20 seconds on
a 36 minute buildworld.

: It's difficult to get lots of these numbers, unfortunately, because it
: takes a good 6-8 hours just to complete one build.  But the numbers are
: fairly consistant across the different degrees of old crappy hardware I
: have.

So you are seeing a about an hour slowdown (16% slowdown on 6 hours is
1 hour) from before/after?  Or are you seeing an hour slowdown from
4.x -> 5.2-beta?

Also, so we can characterize where the suckage happens, how much
RAM/CPU does your system have?

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Michael Edenfield
* M. Warner Losh <[EMAIL PROTECTED]> [031125 12:07]:
> In message: <[EMAIL PROTECTED]>
> "boyd, rounin" <[EMAIL PROTECTED]> writes:
> : i see that there some doubt about whether running lots of
> : shell scripts ever happens.  what happens when you
> : use make?  lots of shells get run and they run small
> : (one line?) scripts.
> 
> make buildworld slows down < 1% when you switch from static /bin/sh to
> dynamic.

I'm all for dynamic / and dynamic /bin/sh, but this doesn't even come
close to what I observed on my systems.  I see anywhere from 15% to 20%
slowdown in buildworld, depending on how bad my hardware already is.  I
posted my most recent numbers earlier.

It's difficult to get lots of these numbers, unfortunately, because it
takes a good 6-8 hours just to complete one build.  But the numbers are
fairly consistant across the different degrees of old crappy hardware I
have.

-Mike



pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Michael Edenfield
* boyd, rounin <[EMAIL PROTECTED]> [031125 05:16]:
> i see that there some doubt about whether running lots of
> shell scripts ever happens.  what happens when you
> use make?  lots of shells get run and they run small
> (one line?) scripts.

Just to provide some real-world numbers, here's what I got out of a
buildworld:

Static /bin/sh:
  real385m29.977s
  user111m58.508s
  sys 93m14.450s

Dynamic /bin/sh:
  real455m44.852s
  user113m17.807s
  sys 103m16.509s

The dynamic /bin/sh caused just over an 18% performance hit on the make
process, everything else being equal (the rest of my / is dynamically
linked).  It may seem like a pretty large performance hit, but to my the
difference between a 6-hour and a 7-hour world build is just an extra
hour of Quake :\

--Mike



pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Daniel O'Connor
On Wednesday 26 November 2003 13:25, David O'Brien wrote:
> On Tue, Nov 25, 2003 at 03:07:55PM +1030, Daniel O'Connor wrote:
> > What about the newer version of gcc? That is considerably slower than
> > previous versions, but I don't see people screaming to have it removed.
>
> Uh... you must not know what you are talking about.  GCC *COMPILES*
> slower as it does a better job of optimizing (which adds time to the
> compiling time).  The produced optimzied binaries have quicker
> *RUN-TIME*s.

I'm talking about compile time.

> Why would any one want to call for a compiler to be removed that produces
> faster binaries??

Ahh, why indeed..

-- 
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 9A8C 569F 685A D928 5140  AE4B 319B 41F4 5D17 FDD5

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread David O'Brien
On Tue, Nov 25, 2003 at 03:07:55PM +1030, Daniel O'Connor wrote:
> What about the newer version of gcc? That is considerably slower than
> previous versions, but I don't see people screaming to have it removed.

Uh... you must not know what you are talking about.  GCC *COMPILES*
slower as it does a better job of optimizing (which adds time to the
compiling time).  The produced optimzied binaries have quicker
*RUN-TIME*s.

Why would any one want to call for a compiler to be removed that produces
faster binaries??
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Brad Knowles
At 2:48 PM -0800 2003/11/25, Matthew Dillon wrote:

 What I am advocating is that FreeBSD-5 not marginalize and
 restrict (make less flexible) basic infrastructure in order to get other
 infrastructure working.
	If you've got working, debugged code that works in the manner you 
are espousing, and still achieves the same goal of making NSS and PAM 
work everywhere, I'm sure we'd all love to see it.

	In the absence of any code contribution to the contrary, I see no 
alternative than the method that has been selected.  Sure, it's not 
great.  Sure, it's slower (more or less, depending on which 
benchmarks you believe).

	But it is the best implementation that was available, and this is 
-CURRENT, where things are expected to periodically be in a state of 
flux while major changes are underway.

--
Brad Knowles, <[EMAIL PROTECTED]>
"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
-Benjamin Franklin, Historical Review of Pennsylvania.
GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+() DI+() D+(++) G+() e++> h--- r---(+++)* z(+++)
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Garance A Drosihn
At 11:27 PM +0100 11/25/03, Brad Knowles wrote:
At 11:50 AM -0800 2003/11/25, Matthew Dillon wrote:

 ... Or you can build an IPC mechanism that implements
 the PAM functionality and then have programs which
 would otherwise use PAM instead use the IPC mechanism.
 Which is the whole point of having the IPC mechanism
 in the first place.
	That all sounds wonderful!

So, when are you going to deliver this fully functioning
and debugged code for inclusion in FreeBSD-5.2?
My guess is that he will deliver it to DragonFly BSD, and it
will then be up to someone with FreeBSD commit privs to look
at steal^h^h^h^h^h^h adapting it for our purposes...
Note that we are already in the code-freeze for 5.2-release,
so I will estimate the probability that all this happens in
time for this release is zero.  Absolute zero.  What might
happen for 5.3-release is a different story, of course.
--
Garance Alistair Drosehn=   [EMAIL PROTECTED]
Senior Systems Programmer   or  [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Brad Knowles
At 11:50 AM -0800 2003/11/25, Matthew Dillon wrote:

 ... Or you can build an IPC mechanism that implements the PAM
 functionality and then have programs which would otherwise use PAM
 instead use the IPC mechanism.  Which is the whole point of having
 the IPC mechanism in the first place.
	That all sounds wonderful!

	So, when are you going to deliver this fully functioning and 
debugged code for inclusion in FreeBSD-5.2?

--
Brad Knowles, <[EMAIL PROTECTED]>
"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
-Benjamin Franklin, Historical Review of Pennsylvania.
GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+() DI+() D+(++) G+() e++> h--- r---(+++)* z(+++)
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Matthew Dillon

:No, what you said was "not to tear out..the ability to generate static
:binaries".  That's completely different, and is absolutely not what
:has happened, or what is planned.  Static binaries continue to be
:supported, available, and work with the system NSS and PAM modules as
:before.

I think you are missing the point I made in that response, because
it isn't that cut and dry.  Obviously isn't that cut and dry.  Why
is a dynamic root the default again?  Because statically linking NSS
and PAM is not the direction FreeBSD-5 is going.  So if you are going
to start depending on dynamic loading, and I seem to recall a number
of conversations where that is, in fact, the intention, then you
are marginalizing your static binary support.  The more you use NSS
and operate on the assumption that DLL will be leveraged,
the more you marginalize your static binary support.  FreeBSD-5
has *ALREADY* made major concessions, such as going to the dynamic
root, precisely because it has *ALREADY* marginalized static binary
support.  That is what I'm hearing.

What I am saying is that for something this fundamental to the
infrastructure, it is not appropriate to marginalize static binary support.
That is all I am saying here.  Sure, I think an IPC mechanism is a better
API, but that has nothing at all to do with this DLL / static/dyanmic
binary issue in FreeBSD-5.  They are two separate issues.  Right now,
in FreeBSD-5, the issue is the marginalized static binary support.

:We're not talking about schedulers.  What is at issue is that you
:decided, for no reason appropriate to the topic of discussion, to
:mention that you think an unrelated FreeBSD developer has difficulties
:with logical reasoning.
:
:What the hell, Matt?  By what standards of behaviour is this
:acceptable?
:
:We have rules of conduct on the FreeBSD mailing lists, and people have
:been removed in the past because they were unable to hold themselves
:to them.  Don't think that you're exempt just because you're Matt
:Dillon.

Yes, and apparently you are breaking them as much as you believe I am,
Kris.  You are also seriously misinterpreted my postings.  I am obviously
not advocating that FreeBSD-5 rip everything out and move to an IPC
model.  It takes time and consideration to be able to do something like
that.   What I am advocating is that FreeBSD-5 not marginalize and
restrict (make less flexible) basic infrastructure in order to get other
infrastructure working.

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Kris Kennaway
On Tue, Nov 25, 2003 at 01:15:58PM -0800, Matthew Dillon wrote:
> 
> :> is the path you've chosen to go then you have an obligation not to
> :> tear out major existing system capabilities, such as the ability to
> :> generate static binaries, in the process.
> :
> :If this is what you think has happened, you're living in some parallel
> :fantasy universe.
> 
> I am simply repeating the reasoning being used for going to a dynamic 
> root.  Forgive me if I misread it, but I believe the argument was that
> FreeBSD-5 was migrating to NSS and NSS's DLL mechanism does not work
> in a static world, therefore dynamic becomes the default.  If I am
> wrong and NSS's DLL mechanism can be used in a static world, please
> correct me!

No, what you said was "not to tear out..the ability to generate static
binaries".  That's completely different, and is absolutely not what
has happened, or what is planned.  Static binaries continue to be
supported, available, and work with the system NSS and PAM modules as
before.

> :> There is a lot of circular reasoning going on here... it's the same sort
> :> of circular reasoning that John uses to justify some of the more esoteric
> :> scheduling mechanisms in -current.  A because of B because of A, and
> :> to hell with anyone who wanted to use C.
> :
> :Keep the ad homenim attacks to yourself, buster!  This was uncalled-for.
> :
> :Kris
> 
> Well, the scheduler arguments are more involved but I am not incorrect
> here.

We're not talking about schedulers.  What is at issue is that you
decided, for no reason appropriate to the topic of discussion, to
mention that you think an unrelated FreeBSD developer has difficulties
with logical reasoning.

What the hell, Matt?  By what standards of behaviour is this
acceptable?

We have rules of conduct on the FreeBSD mailing lists, and people have
been removed in the past because they were unable to hold themselves
to them.  Don't think that you're exempt just because you're Matt
Dillon.

  
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/eresources.html#ERESOURCES-MAIL

  Personal attacks and profanity (in the context of an argument) are
  not allowed, and that includes users and developers alike. Gross
  breaches of netiquette, like excerpting or reposting private mail when
  permission to do so was not and would not be forthcoming, are frowned
  upon but not specifically enforced. However, there are also very few
  cases where such content would fit within the charter of a list and it
  would therefore probably rate a warning (or ban) on that basis alone.

Kris



pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Jacques A. Vidrine
On Tue, Nov 25, 2003 at 12:39:11PM -0800, Matthew Dillon wrote:
> My original opinion
> still stands... you guys are using this issue as an excuse to basically
> do away with static binaries, rather then fixing the real problem which
> is an inability to dynamically load modules in a static binary.

No, it is one of the issues that have been pushing FreeBSD (and most
other modern UNIX systems, it seems) towards more dynamically linked
components.  The other issues have also been discussed on this list
recently and in the pass.  I'm not sure why this one draws such
interest.  (I participate in this thread only because I feel like I
know a thing or two about the NSS details.)

Cheers,
-- 
Jacques Vidrine   NTT/Verio SME  FreeBSD UNIX   Heimdal
[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Matthew Dillon

:> is the path you've chosen to go then you have an obligation not to
:> tear out major existing system capabilities, such as the ability to
:> generate static binaries, in the process.
:
:If this is what you think has happened, you're living in some parallel
:fantasy universe.

I am simply repeating the reasoning being used for going to a dynamic 
root.  Forgive me if I misread it, but I believe the argument was that
FreeBSD-5 was migrating to NSS and NSS's DLL mechanism does not work
in a static world, therefore dynamic becomes the default.  If I am
wrong and NSS's DLL mechanism can be used in a static world, please
correct me!

So exactly how far do you intend to go with NSS?  Because it seems to 
me that FreeBSD-5's goal is to start to depend on the DLL capabilities.
If the goal is an intention to depend on DLL then you damn well need to
make it work with static ELF binaries.  It's that simple.

:> There is a lot of circular reasoning going on here... it's the same sort
:> of circular reasoning that John uses to justify some of the more esoteric
:> scheduling mechanisms in -current.  A because of B because of A, and
:> to hell with anyone who wanted to use C.
:
:Keep the ad homenim attacks to yourself, buster!  This was uncalled-for.
:
:Kris

Well, the scheduler arguments are more involved but I am not incorrect
here.  E.G. the priority borrowing exists to work around problems with
the mutex implementation.  Preemption by non-interrupts threads also
exists to work around problems with the mutex implementation.  Preemptive
cpu migration could be turned off fairly easily and doesn't really count.
The priority borrowing is a mess, though.  You may think its the best
thing since sliced bread but I think it unnecessarily complicates both
the scheduler core and the programming model.

-Matt

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Kris Kennaway
On Tue, Nov 25, 2003 at 12:39:11PM -0800, Matthew Dillon wrote:

> So, yes, I do think you guys are being lazy in that regard.  If this
> is the path you've chosen to go then you have an obligation not to
> tear out major existing system capabilities, such as the ability to
> generate static binaries, in the process.

If this is what you think has happened, you're living in some parallel
fantasy universe.

> There is a lot of circular reasoning going on here... it's the same sort
> of circular reasoning that John uses to justify some of the more esoteric
> scheduling mechanisms in -current.  A because of B because of A, and
> to hell with anyone who wanted to use C.

Keep the ad homenim attacks to yourself, buster!  This was uncalled-for.

Kris

pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Matthew Dillon

:Matt, I'm talking about the de facto standard NSS, as found in Solaris
:and Linux; and now FreeBSD 5 [*] and soon NetBSD [**].  You are talking
:about some better mousetrap.  The latter does not have any relevance
:to this thread, which is about dynamic linking in next release of
:FreeBSD.
:
:If you want to talk about FreeBSD 6.x and a better mousetrap, please
:go right ahead with a new thread here on freebsd-current or over on
:freebsd-arch.
:
:If you want to talk about the future of DragonFlyBSD, I'm sure there
:is an appropriate list for that--- this one ain't it.  Parts of your
:message certainly seemed to describe what might be best for some other
:operating system.
:
:Cheers,
:-- 
:Jacques Vidrine   NTT/Verio SME  FreeBSD UNIX   Heimdal
:[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]
:
:
:Side notes:
:
:[*] The actual APIs used by Solaris and Linux are *very* different;
:and the APIs used by FreeBSD are *somewhat* different from Linux.
:However, because the *core concepts* are the same--- dynamic loading,
:in-process modules--- portability issues are not much of a problem.
:
:[**] NetBSD doesn't support dynamic loading yet, but has had
:considerable influence on the FreeBSD implementation.  NetBSD
:developers have indicated to me that they expect to bring in
:the FreeBSD changes so that there will be basically the same
:implementation on FreeBSD and NetBSD.

I'm not sure of the relevance of this comment.  My original opinion
still stands... you guys are using this issue as an excuse to basically
do away with static binaries, rather then fixing the real problem which
is an inability to dynamically load modules in a static binary.

How much do you intend to use NSS for?  I mean, what's the point of
adopting this cool infrastructure if all you are going to do with it
is make a better PAM out of it?  You can use it for basic authentication,
sure, but the more things you try to use it for without static binary
support the fewer things you can compile statically.  You are basically
doing away with the static linking capability of the system for no good
reason that I can see, and coming up with all sorts of extra junk,
like /rescue, to work around that fact.  You are creating a huge mess
*JUST* to be able to port a dynamic load NSS framework and you are 
throwing away functionality left and right to get it.  That's no good
in my book.  If you *REALLY* want dynamic load NSS then you ought to
spend the time to make it work with static binaries rather then just
being lazy and getting rid of static binaries.

So, yes, I do think you guys are being lazy in that regard.  If this
is the path you've chosen to go then you have an obligation not to
tear out major existing system capabilities, such as the ability to
generate static binaries, in the process.

If the APIs are totally different then I don't see any difference
between your direct NSS implementation and my ability, within an 
IPC framework, to implement NSS as a backend service.  I suppose you
can call me work "building a better mousetrap", implying that I am
doing work that has already been done, but it is really no different
then what you are doing in FreeBSD-5 by changing existing API standards
to suit your particular implementation.

There is a lot of circular reasoning going on here... it's the same sort
of circular reasoning that John uses to justify some of the more esoteric
scheduling mechanisms in -current.  A because of B because of A, and
to hell with anyone who wanted to use C.

What I am doing is not reinventing the wheel, I am simply reinventing the
API that the backend of libc uses to access resources.  There is nothing
preventing me from then implementing something like NSS and PAM on the
backend of the new API, as a service rather then as a DLL.  I fully
expect that someone will do that, in fact, possibly even me.  But I also
expect that straight IPC will solve at least as many problems and
in fact solve significantly more problems then the DLL NSS solution
solves.

So, yes, IPC will be the basis used in DFly.  That does not invalidate
my comments vis-a-vie the dynamic/static problem with NSS and PAM
that FreeBSD-5 has.  If you want to do it right then make DLL's work
with static binaries.  If you want to do it wrong then ignore static
binaries.  It is that simple.

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Kris Kennaway
On Tue, Nov 25, 2003 at 07:36:45PM +0100, Dag-Erling Sm?rgrav wrote:
> "Daniel O'Connor" <[EMAIL PROTECTED]> writes:
> > What _REAL WORLD_ task does this slow down?
> 
> I suspect 'make world' takes a serious hit.

It does not (Warner has quoted numbers a few times now).

Kris


pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Jacques A. Vidrine
On Tue, Nov 25, 2003 at 11:50:25AM -0800, Matthew Dillon wrote:
> Just not thinking out of the box, maybe.

Matt, I'm talking about the de facto standard NSS, as found in Solaris
and Linux; and now FreeBSD 5 [*] and soon NetBSD [**].  You are talking
about some better mousetrap.  The latter does not have any relevance
to this thread, which is about dynamic linking in next release of
FreeBSD.

If you want to talk about FreeBSD 6.x and a better mousetrap, please
go right ahead with a new thread here on freebsd-current or over on
freebsd-arch.

If you want to talk about the future of DragonFlyBSD, I'm sure there
is an appropriate list for that--- this one ain't it.  Parts of your
message certainly seemed to describe what might be best for some other
operating system.

Cheers,
-- 
Jacques Vidrine   NTT/Verio SME  FreeBSD UNIX   Heimdal
[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]


Side notes:

[*] The actual APIs used by Solaris and Linux are *very* different;
and the APIs used by FreeBSD are *somewhat* different from Linux.
However, because the *core concepts* are the same--- dynamic loading,
in-process modules--- portability issues are not much of a problem.

[**] NetBSD doesn't support dynamic loading yet, but has had
considerable influence on the FreeBSD implementation.  NetBSD
developers have indicated to me that they expect to bring in
the FreeBSD changes so that there will be basically the same
implementation on FreeBSD and NetBSD.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Matthew Dillon

:IMHO, it makes more sense to write NSS modules that do their own
:proxying than to make things even more complicated in libc.  Those
:that are lightweight don't carry extra baggage; those that do can
:implement proxying in the most efficient manner for that particular
:backend, e.g. some calls can be proxyed while others done in-process.
:You don't have to rewrite existing NSS modules so that they take into
:account that they are really serving multiple processes--- which
:usually means adding credentials management, locking, per-process
:state, and so forth to each NSS module.  Or you could just create a
:whole new NSS API and call it something else and forget about support
:for existing NSS modules.
:
:Caching results (which is different than out-of-process
:implementations, the Linux nscd authors are just confused) does
:require a daemon, but this doesn't really complicate things.  (I
:should get around to that someday :-)
:
:That said, I would not stand in the way of a well-thought out,
:well-written NSS implementation that attempts to proxy every get*()
:call over e.g. RPC.  (Hmm, sounds like NIS to me.  I guess that's
:partially explains why PADL.com's NIS<->LDAP gateway is popular :-)

Well, here's the issue... where do you think the future is?  I
believe the future is in large, extended clusters of machines which
either need to agree on their management resources or which need to
be capable of hierarchically accessing a global resource namespace.

Sure you can do this within the nsswitch framework, by writing 
particular NSS modules that then going out and implementing some other
proxy protocol.  But most NSS modules are not going to be written with IPC
in mind so it would be a fairly difficult and involved job to create 
an integrated framework capable of the above within NSS.  Without doing
that you would be restricted to only those modules which are directly
capable of proxying *AND* you would have to contend with various
proxy-capable modules using different protocols.  In otherwords, it's
a mess.  It seems silly to waste your time on a framework that you are
just going to have to rip out again a year or two from now.

By using an IPC mechanism from the start the framework and centralization
issues go away.  Poof, gone.  No issue.  A module written as an IPC
service doesn't know or care (other then for authentication purpopses)
who is making requests of it.   In DFly it is particularly important
because we are going for an SSI-capable result and you just can't do
that with NSS (at least not without devaluing the NSS mechanism so much
you might as well not have used it in the first place!).

The absolute worst case in an IPC framework is that the program trying 
to access service X and not being able to find it would have to fork/exec
the service binary itself to create the IPC connection.  This, of course,
would only occur under extrodinary circumstances (such as when you are
in single-user mode).  But despite the overhead we are only talking about
two lines of code, really.  fork() and exec().  Well, three... pipe(),
fork(), and exec().

In regards to caching... with an IPC mechanism the client can choose
to cache the results however it pleases.  The IPC mechanism can simply
notify the client asynchronously if a cache invalidation is required.
That's what a real messaging/port protocol gives you the ability to do.
So generaly performance using the IPC mechanism is going to be as good
or better then what we currently have with uncached flat files or uncached
databases.

Which brings up yet another cool result ... when you use an IPC mechanism
you don't need to generate DBM's.  The service process itself will simply
scan /etc/master.passwd, /etc/group, and so forth, and build its own
in-memory database.  Being able to get rid of the DBMs is only part of
the equation because it also means that files which are not otherwise
DBM's, such as /etc/services and /etc/group, enjoy the same feature.

:Um, if you can't trust the authentication code, what can you trust?
:Furthermore, for many many many applications that use getpwnam(3) and
:so on, increased privileges are not needed or wanted.
 
Think out of the box.Consider a multi-layered approach.  Take 
access to master.passwd for example.  Would you have
(A) the authentication code integrated into the potentially buggy program
be able to access the file directly or would you rather have 
(B) the authentication code access an IPC service which *ONLY* allows
challenge/response, does *NOT* give you direct access to the 
encrypted contents of the password file, and which limits the challenge
rate to prevent dictionary attacks?

That's about the best example that I can come up with.  Think about
it... the *ONLY* code that has access to the actual 

Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Dag-Erling Smørgrav
"Daniel O'Connor" <[EMAIL PROTECTED]> writes:
> What _REAL WORLD_ task does this slow down?

I suspect 'make world' takes a serious hit.

DES
-- 
Dag-Erling Smørgrav - [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
"boyd, rounin" <[EMAIL PROTECTED]> writes:
: i see that there some doubt about whether running lots of
: shell scripts ever happens.  what happens when you
: use make?  lots of shells get run and they run small
: (one line?) scripts.

make buildworld slows down < 1% when you switch from static /bin/sh to
dynamic.

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Garance A Drosihn
At 9:19 AM -0600 11/25/03, Jacques A. Vidrine wrote:
On Mon, Nov 24, 2003, Andrew Gallatin wrote:

So can we just have a statically linked /bin/sh and get on
with life?
I still think we would be better off using 5.2-release for
collecting more experience with the *operational* issues of
having a dynamic /bin/sh.  We all know and knew that there
would be a performance hit.  We also all know that a static
/bin/sh will work fine in disaster situations.
That seems to have the most impact.  We can also expend
our efforts to improve dynamic linking performance, since
that will improve the performance of the other 99.9% of
the universe.
This is certainly my hope.  There are more ways to solve the
performance problem than just statically-linking /bin/sh.
If we do not alleviate the performance issues via other means,
then we can certainly statically-link /bin/sh for 5.3-release.
We have run with a statically-linked /bin/sh for years, so
there is nothing much to *learn* by running with it for the
next two months.  Yes, there is a performance benefit, but
nothing to *learn*.
But my fear is that if we *do* address the performance issues,
then we'll still shy off a dynamically-linked /bin/sh simply
because some folks will say "we don't know that we can trust
it", etc.
I have no objection if we want to statically-link some things
like /bin/sh for 5.3-release, but I don't think we need to do
it for 5.2-release -- aka "a snapshot of freebsd-current".
--
Garance Alistair Drosehn=   [EMAIL PROTECTED]
Senior Systems Programmer   or  [EMAIL PROTECTED]
Rensselaer Polytechnic Instituteor  [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Cy Schubert
In message <[EMAIL PROTECTED]>, "Jacques A. Vidrine" 
wri
tes:
> So can we just have a statically linked /bin/sh and get on with life?

I was thinking the same thing myself a few days ago.


Cheers,
--
Cy Schubert <[EMAIL PROTECTED]>http://www.komquats.com/
BC Government .   FreeBSD UNIX
[EMAIL PROTECTED] . [EMAIL PROTECTED]
http://www.gov.bc.ca/ .http://www.FreeBSD.org/



___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Andrew Gallatin

Jacques A. Vidrine writes:
 > 
 > So can we just have a statically linked /bin/sh and get on with life?

That certainly seems like the best compromise.   Then we can end this
thread ;)

 > That seems to have the most impact.  We can also expend our efforts
 > to improve dynamic linking performance, since that will improve the
 > performance of the other 99.9% of the universe.
 > 

What happened to mdodd's prebinding efforts?

Drew
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


RE: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Guy Helmer
Jacques A. Vidrine wrote:
> On Mon, Nov 24, 2003 at 10:06:12PM -0500, Andrew Gallatin wrote:
> > How about Gordon's initial bootstone, which increased by 25%?
> > http://docs.freebsd.org/cgi/mid.cgi?16091.44150.539095.704531
> > 
> > And I just did a "make clean" run in /usr/ports/archivers (by manually
> > mv'ing a static and dynamic sh to /bin in turn):
> > 
> > static:   96.63 real53.45 user39.27 sys
> > dynamic: 112.42 real55.51 user51.62 sys
> > 
> > The wall clock is bad (16% worse) and the system time is worse (31%).
> So can we just have a statically linked /bin/sh and get on with life?
> That seems to have the most impact.  We can also expend our efforts
> to improve dynamic linking performance, since that will improve the
> performance of the other 99.9% of the universe.

Yes, let's do it and get on with it.  /bin/sh is critically important
to the performance of many things in the system, but shared / is very
useful as well - it's allowed me to move my 4.x systems with small /
up to 5-current, and / programs can take advantage of NSS and PAM
modules that exist *today*.
 
> ...
> In any case, I'd really like to see a goal for 5.3-RELEASE that
> includes bringing dynamically-linked /bin/sh performance (*much*)
> closer to statically-linked /bin/sh performance.

Yes -- this is -current: let's get 5.2 out the door and improve on it
for 5.3.

Guy Helmer

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Jacques A. Vidrine
On Mon, Nov 24, 2003 at 08:22:52PM -0600, David Leimbach wrote:
> Yep :).
> 
> I feel like saying "set the default to static and make the dynamic bins 
> the option" so
> the people who can't be bothered to compile their own system even 
> though everyone
> I know does this for tuning purposes anyway can stop whining.
> 
> But I won't say that.

I feel we need to pressure to improve the performance of dynamic
linking.  This is not really different from anything else we do in
-CURRENT: some things we have to throw out there before it is perfect,
in order to reach critical mass.

Cheers,
-- 
Jacques Vidrine   NTT/Verio SME  FreeBSD UNIX   Heimdal
[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Jacques A. Vidrine
On Mon, Nov 24, 2003 at 10:06:12PM -0500, Andrew Gallatin wrote:
> How about Gordon's initial bootstone, which increased by 25%?
> http://docs.freebsd.org/cgi/mid.cgi?16091.44150.539095.704531
> 
> And I just did a "make clean" run in /usr/ports/archivers (by manually
> mv'ing a static and dynamic sh to /bin in turn):
> 
> static:   96.63 real53.45 user39.27 sys
> dynamic: 112.42 real55.51 user51.62 sys
> 
> The wall clock is bad (16% worse) and the system time is worse (31%).
> 
> 
> So.. 
> 
> 1) Microbenchmark:40% worse
> 2) Bootstone(*):  25% worse
> 3) Ports: 16% worse

So can we just have a statically linked /bin/sh and get on with life?
That seems to have the most impact.  We can also expend our efforts
to improve dynamic linking performance, since that will improve the
performance of the other 99.9% of the universe.

Users who REALLY REALLY need /bin/sh to support 3rd-party NSS modules
in the mean time can build /bin/sh dynamically.  Or we can have
/usr/bin/sh as someone else suggested (most of the FreeBSD world's
shell scripts--- which are what we *really* seem to be talking
about--- already have #! /bin/sh).

I prefer to keep as much of the world dynamic, both for dlopen support
and for easier system patching.  But I can also understand the desire
to avoid a penalty for all those short but oft-run scripts.

In any case, I'd really like to see a goal for 5.3-RELEASE that
includes bringing dynamically-linked /bin/sh performance (*much*)
closer to statically-linked /bin/sh performance.

Cheers,
-- 
Jacques Vidrine   NTT/Verio SME  FreeBSD UNIX   Heimdal
[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Jacques A. Vidrine
On Mon, Nov 24, 2003 at 07:11:29PM -0800, Matthew Dillon wrote:
> You don't need dynamic loading to get nsswitch type functionality.  You
> only need dynamic loading if nobody is willing to write an IPC
> model to get the functionality.  It's really silly to create such a
> fundamental restriction on the binary because people are too lazy
> to build an IPC based mechanism.  

Matt, I'm not lazy. (Well, maybe I am, but that's not why I
implemented NSS in the canoncial way instead of using IPC.)  I've
experimented with proxy-based solutions.  Such solutions add a lot of
complexity with little benefit.  The primary NSS backends today are:

 NIS --- lightweight, proxy would just slow things down
 Hesiod --- lightweight
 winbindd --- lightweight (it implements its own proxy)
 nss_ldap --- could benefit

IMHO, it makes more sense to write NSS modules that do their own
proxying than to make things even more complicated in libc.  Those
that are lightweight don't carry extra baggage; those that do can
implement proxying in the most efficient manner for that particular
backend, e.g. some calls can be proxyed while others done in-process.
You don't have to rewrite existing NSS modules so that they take into
account that they are really serving multiple processes--- which
usually means adding credentials management, locking, per-process
state, and so forth to each NSS module.  Or you could just create a
whole new NSS API and call it something else and forget about support
for existing NSS modules.

Caching results (which is different than out-of-process
implementations, the Linux nscd authors are just confused) does
require a daemon, but this doesn't really complicate things.  (I
should get around to that someday :-)

That said, I would not stand in the way of a well-thought out,
well-written NSS implementation that attempts to proxy every get*()
call over e.g. RPC.  (Hmm, sounds like NIS to me.  I guess that's
partially explains why PADL.com's NIS<->LDAP gateway is popular :-)

> Not only silly, but it seems to me
> that it also creates security issues.  At least with a client/server
> model it is possible to isolate the authentication code to, for example,
> disallow exec(), filesystem operations, or the opening of any new files.

Um, if you can't trust the authentication code, what can you trust?
Furthermore, for many many many applications that use getpwnam(3) and
so on, increased privileges are not needed or wanted.

And if you *are* really talking about authentication code (and not
directory services), then you need to get PAM to work in a statically
linked world, also.  (You can compile PAM statically today, but that
means no 3rd-party modules.  The same holds for NSS, of course.)

> The other huge advantage that IPC has over DLL is that you can switchout
> the backend mechanisms on a running system without killing and restarting
> anything other then he one service you want to switch out, and if it
> doesn't work right you can restart the old one or keep live as a fallback.

When using the current NSS implementation, there is no need to
kill/restart anything when you update /etc/nsswitch.conf.  New
modules are dynamically loaded, and any no-longer-referenced ones are
unloaded.

> the IPC model is so much better then the DLL model for this sort of thing
> I don't understand why people are even arguing about it.

Because the rest of us are stupid and lazy, remember?  :-)

Cheers,
-- 
Jacques Vidrine   NTT/Verio SME  FreeBSD UNIX   Heimdal
[EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Operating system advocacy (Was: Re: 40% slowdown with dynamic /bin/sh)

2003-11-25 Thread Mark Murray
Matthew Dillon writes:
> Hmm.  Well, I think there's some confusion here.  While I
> certainly like my vision for DFly better then I like the vision
> for FreeBSD-5, that is simply in the eye of the beholder... of
> course I am going to like my own vision better.  It's my vision,
> after all!  Your vision is obviously different.  In fact, I expect
> that each person has his own vision for the project, so don't
> knock me for mine.

There is a lot of opinion-knocking happening here on both sides, and
tempers are flaring.

Can I please ask that all parties take a step back and do what it takes
to increase light and reduce heat?

> But that has nothing to do with perceived inferiority or
> superiority.

True, inferiority/superiority issues are notoriously fulminative, and we
need to get this track back to the technical level, and away from more
personal achievment issues.

> The issue isn't so much whether one project is better then the
> other as it is whether one is able and willing to borrow a proven
> concept from another project to replace the one that maybe isn't
> so hot in one's own.

No. This thread is about a much more basic issue; the resolution of the
static/dynamic issue in the / volume.

Which operating system has the better solution, while a valid discussion
point, is a side track here, and is serving to add heat, not light.

>  As it happens, I have borrowed quite a bit of code
> from 5.x.  As it also happens, I believe that 5.x would benefit
> by adopting some of the things that have already been proven to
> work quite well in DragonFly.  For example, using a statistical
> time accumulation model instead of calling microtime() in the
> middle of the critical thread switch path, or not preemptively
> switching threads operating in kernelland to another cpu, or the
> implementation of a mutexless scheduler.  Just a few examples.  I
> can only point out the concepts and ideas and point to the code
> in DFly, it is up to FreeBSD-5 developers to take the ball up and
> either throw it away or run with it.

Good points all. Perhaps they need to be discussed in their own right,
and not as a digression to the original thread?

> And, just for the record, I feel quite obligated to try to move
> the FreeBSD project forward along a path that I believe will be
> more beneficial to its users.

Careful. You are working hard on a very admirable project; please can
you continue to do do, and as-and-when issues from DFBSD prove their
worth, they will be adopted by other projects. This is a case where the
separation of strong personalities actually works out rather nicely, and
you can help this by proving how well DFBSD technology is :-).

There have been personality clashes in the past; some of these have been
shown to be unresolvable, but we now have the improved situation where
the talented folks are still developing BSD code without hindering each
other. This way BSD and its users win.

The way you can best help BSD is to continue to develop DragonFlyBSD.

>   Just to be clear:  My obligation is to
> all the people who use FreeBSD, not to the feelings of particular
> developers whos vision(s) I might disagree with.  I have no
> intent or intention of screwing over FreeBSD (how absurd!) but
> you should not mistakenly equate that to me being accomodating to
> FreeBSD's current vision which, yes! it is true! I have serious
> disagreements with.

Sure. There are going to be disagreements, This is why there are 4 BSD's
available for free.

> Over the years I have recommended FreeBSD to hundreds of people
> and I take that responsibility very seriously.

Thank you! I hope that you will also be able to do that with
DragonFlyBSD.

> If it is within the scope of the FreeBSD charter for a person to
> post based on a perceived obligation to the end users of FreeBSD,
> then I certainly still have a right to post to this group.

Sort of. General opinion-mongering, flamage, sidetracking and so on
are off-charter. This is "FreeBSD CURRENT", and it is most likely the
best to keep it somewhat restricted to that as some folks are starting
to get annoyed at the "Dragonfly Advertising". I think that keeping
DFBSD-Advocacy/Discussion on FreeBSD lists to a pretty low level would
help keep blood pressure down all round (No offense intended, DFBSD is
a worthwhile project, its just that inter-project politics are somewhat
rough, and I'm trying to cool things down!)

M
--
Mark Murray
iumop ap!sdn w,I idlaH
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread boyd, rounin
> That's a more interesting result and more comparable to Drew's test.
> It doesn't necessarily invalidate Drew's results - /bin/sh has 3
> shared libraries and is locale-aware whereas /usr/bin/test has 1
> shared library and doesn't rely on the locale.  /usr/bin/true is also
> significantly smaller (implying less relocation requirements).
> /bin/sh could reasonably be expected to take longer to startup then
> /usr/bin/test.

another can of worms.  various shells have test, true and false built in.
so, you have to be very careful in writing a shell comparision benchmark.

to complicate matters, ksh (statically linked) has _always_ been faster than sh.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread boyd, rounin
i see that there some doubt about whether running lots of
shell scripts ever happens.  what happens when you
use make?  lots of shells get run and they run small
(one line?) scripts.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Erik H. Bakke
On Tuesday 25 November 2003 03:07, Don Lewis wrote:
> On 25 Nov, Daniel O'Connor wrote:
> > On Tuesday 25 November 2003 11:52, Dan Nelson wrote:
> >  > > I'd greatly prefer that the the dynamic root default be backed out
> >  > >
> >> > > until a substantial amount of this performance can be recovered.
> >> >
> >> > What _REAL WORLD_ task does this slow down?
> >>
> >> Try timing "cd /usr/ports/www/mozilla-devel ; make clean" with static
> >> and dynamic /bin.  bsd.port.mk spawns many many many /bin/sh processes.
> >
> > OK my bad, it will probably slow down the ports building.
>
> The ports infrastructure is horribly slow even with a static sh, though
> not as glacially slow as installing and patching Solaris 9.
>
But if the change to dynamic root "provokes" this slowdown that people have 
been seeing, it would be much better to address the cause and not the 
symptom.

The change to dynamic root here is not the cause, the extra overhead of using 
dynamically linked tools is, and that should be the main focus point.

If the overhead of dynamic linking is reduced, that will benefit all of us, 
even if we use a dynamic root or not.

-- 
Erik H. Bakke

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Matthew Dillon
:...
:5.x and propaganda about DFBSD doesn't really mean a whole lot, unless you
:are looking for new recruits to your camp.  In any case, you've made your
:point on a nearly daily basis that 5.x is inferior to what DFBSD will be,
:and that you don't have much knowledge or care about 5.x anyways.  So
:please, go do what you do best and make DFBSD the envy of the BSD world.
:I'll be first in line to pat you on the back when you succeed.
:
:Scott

Hmm.  Well, I think there's some confusion here.  While I certainly
like my vision for DFly better then I like the vision for FreeBSD-5,
that is simply in the eye of the beholder... of course I am going
to like my own vision better.  It's my vision, after all!  Your
vision is obviously different.  In fact, I expect that each person
has his own vision for the project, so don't knock me for mine.

But that has nothing to do with perceived inferiority or superiority.
The issue isn't so much whether one project is better then the other
as it is whether one is able and willing to borrow a proven concept
from another project to replace the one that maybe isn't so hot in
one's own.   As it happens, I have borrowed quite a bit of code
from 5.x.  As it also happens, I believe that 5.x would benefit by
adopting some of the things that have already been proven to work
quite well in DragonFly.  For example, using a statistical time 
accumulation model instead of calling microtime() in the middle
of the critical thread switch path, or not preemptively switching
threads operating in kernelland to another cpu, or the implementation
of a mutexless scheduler.  Just a few examples.  I can only point out
the concepts and ideas and point to the code in DFly, it is up to
FreeBSD-5 developers to take the ball up and either throw it away or
run with it. 

I have not been posting daily, but you seem to be frustrated about
something.  I can only suggest that blaming me for your frustrations
is not going to solve any tangible, technical issue in FreeBSD-5.  My
posts are technical and to the point.  Just because it's coming out of
my mouth rather then someone you might respect a bit more doesn't 
make it any more or less valid.  If you cannot address them based
on their technical merit then you've missed the point of the post
entirely.

And, just for the record, I feel quite obligated to try to move
the FreeBSD project forward along a path that I believe will be more
beneficial to its users.  Just to be clear:  My obligation is to all
the people who use FreeBSD, not to the feelings of particular
developers whos vision(s) I might disagree with.  I have no intent or
intention of screwing over FreeBSD (how absurd!) but you should not
mistakenly equate that to me being accomodating to FreeBSD's current
vision which, yes! it is true! I have serious disagreements with.
Over the years I have recommended FreeBSD to hundreds of people and
I take that responsibility very seriously.

If it is within the scope of the FreeBSD charter for a person to
post based on a perceived obligation to the end users of FreeBSD,
then I certainly still have a right to post to this group.

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Andre Guibert de Bruet

On Mon, 24 Nov 2003, Leo Bicknell wrote:

> Process accounting can tell the story:
>
> % lastcomm | wc -l
>47806
> % lastcomm | sed -e 's/ .*.//' | sort | uniq -c | sort -nr | head
> 25281 sendmail
> 4094 sh
> 2987 perl
> 2846 inetd
> 1704 procmail
> 1640 httpd
> 1221 cron
>  814 date
>  732 postgres
>  648 rateup
>
> Looks like sh is the 2nd most frequently executed command on my
> system.  It is 8.5% of all executed programs on this particular
> system.  I think slowing down 8.5% of all the programs the system
> runs is important.

For what it's worth, here's the data that I've taken from the daily
process accounting files of one of our somewhat busy shellboxes:

# lastcomm -f acct.0 | sed -e 's/ .*.//' | sort | uniq -c | sort -nr | head
4829 qpopper
3426 bash
3191 sendmail
1915 sh
1687 httpd
1281 sed
1030 sshd2
 952 rm
 792 procmail
 739 cron
# lastcomm -f acct.1 | sed -e 's/ .*.//' | sort | uniq -c | sort -nr | head
5383 qpopper
3282 bash
2743 sendmail
1617 httpd
1187 sh
1071 sed
 772 rm
 739 cron
 694 procmail
 478 cat
# lastcomm -f acct.2 | sed -e 's/ .*.//' | sort | uniq -c | sort -nr | head
5376 qpopper
2823 bash
2118 sendmail
1674 httpd
1510 sh
 745 procmail
 740 cron
 292 python
 288 atrun
 211 inetd

Though /bin/sh isn't 2nd on the list, it does feature prominently in the
top 10. I would assume that anyone with a fairly busy machine acting as a
shellbox and webserver would see something along these lines...

Regards,

> Andre Guibert de Bruet | Enterprise Software Consultant >
> Silicon Landmark, LLC. | http://siliconlandmark.com/>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Peter Jeremy
On Tue, Nov 25, 2003 at 01:17:34AM -0700, M. Warner Losh wrote:
>True.  However, I get very similar numbers of I change it to
>/usr/bin/true (12% slower).  /bin/sh usually fork+exec things other
>/bin/sh.

That's a more interesting result and more comparable to Drew's test.
It doesn't necessarily invalidate Drew's results - /bin/sh has 3
shared libraries and is locale-aware whereas /usr/bin/test has 1
shared library and doesn't rely on the locale.  /usr/bin/true is also
significantly smaller (implying less relocation requirements).
/bin/sh could reasonably be expected to take longer to startup then
/usr/bin/test.

>Dynamically linked prorgrams tend to be a few percent slower than
>their static counterparts due to PIC code typically being slower than
>non-PIC code.  There's nothing new here.

Except that, on face value, your figures suggested an 11% slowdown
attribute to PIC code - which is way above "a few percent".

>Clearly there are problems to look into, but it isn't the end of the
>world.

Agreed.  I think most people agree that more work needs to be done.  The
arguing seems to be whether the work should be done before or after the
big switch is thrown and how to go about recovering the lost performance.
(And of course one contingent is insisting it be green whilst a different
contingent is insisting that it can't be green and has to be triangular
instead).

Peter
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Peter Jeremy
On Tue, Nov 25, 2003 at 04:54:41AM +, E.B. Dreger wrote:
>What specific aspects of rtld are required to support NSS in
>static binaries?  dlopen(), fixups, and dlsym()?

All of the above.  The underlying problem is how to handle a
library call from within the NSS/PAM/whatever shared library.
This has been discussed in one of the recent threads but it
boils down to:
1) Static executables don't normally have any symbols available at
   runtime so it's difficult for a shared library to resolve symbols
   using definitions in the executable.
2) It is possible (likely?) that a shared library may reference a
   symbol that does not exist within the executable.
3) Loading libc.so etc to resolve a symbol means that there may be
   two distinct (and possibly different) instances of the same object
   associated with a process.  This may create problems where those
   objects have side-effects.

Peter
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
Peter Jeremy <[EMAIL PROTECTED]> writes:
: On Mon, Nov 24, 2003 at 11:16:07PM -0700, M. Warner Losh wrote:
: >H, It looks like the hit is less than 10% in the fork intensive
: >test I just wrote:
: >
: >#!/bin/sh
: >for i in 0 1 2 3 4 5 6 7 8 9; do
: >for j in 0 1 2 3 4 5 6 7 8 9; do
: >for k in 0 1 2 3 4 5 6 7 8 9; do
: > for l in 0 1 2 3 4 5 6 7 8 9; do
: > for m in 0 1 2 3 4 5 6 7 8 9; do
: >  for n in 0 1 2 3 4 5 6 7 8 9; do
: >true;
: >done; done; done; done; done; done;
: 
: Unless you've done something wierd to your /bin/sh, "true" is a
: builtin.  This test just to measures the ongoing runtime overhead
: of a dynamic executable (ie PIC code).  Drew's test was measuring
: the startup overhead.

True.  However, I get very similar numbers of I change it to
/usr/bin/true (12% slower).  /bin/sh usually fork+exec things other
/bin/sh.

: >Clearly dynamic is slower, but it is more like 11% slower (10.67%) on
: >the average than 40% slower.  I think this would be a more typical
: >usage pattern.
: 
: You have measured different things.  Drew's test shows that a dynamic
: /bin/sh tahes about 40% longer to start.  Your test shows that once
: started, it runs about 11% slower.  And the 11% slower is _very_
: worrying since it is probably more widely applicable than just /bin/sh.

Dynamically linked prorgrams tend to be a few percent slower than
their static counterparts due to PIC code typically being slower than
non-PIC code.  There's nothing new here.

Clearly there are problems to look into, but it isn't the end of the
world.

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Peter Jeremy
On Mon, Nov 24, 2003 at 11:16:07PM -0700, M. Warner Losh wrote:
>H, It looks like the hit is less than 10% in the fork intensive
>test I just wrote:
>
>#!/bin/sh
>for i in 0 1 2 3 4 5 6 7 8 9; do
>for j in 0 1 2 3 4 5 6 7 8 9; do
>for k in 0 1 2 3 4 5 6 7 8 9; do
> for l in 0 1 2 3 4 5 6 7 8 9; do
> for m in 0 1 2 3 4 5 6 7 8 9; do
>  for n in 0 1 2 3 4 5 6 7 8 9; do
>true;
>done; done; done; done; done; done;

Unless you've done something wierd to your /bin/sh, "true" is a
builtin.  This test just to measures the ongoing runtime overhead
of a dynamic executable (ie PIC code).  Drew's test was measuring
the startup overhead.

>Clearly dynamic is slower, but it is more like 11% slower (10.67%) on
>the average than 40% slower.  I think this would be a more typical
>usage pattern.

You have measured different things.  Drew's test shows that a dynamic
/bin/sh tahes about 40% longer to start.  Your test shows that once
started, it runs about 11% slower.  And the 11% slower is _very_
worrying since it is probably more widely applicable than just /bin/sh.

Peter
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Miguel Mendez
./Scott Long wrote:

Please, read : http://www.netmeister.org/news/learn2quote.html

> Also, I'm really starting to resent you using the FreeBSD mailing
> lists as an advocacy channel for DragonFly.  I fail to see how FreeBSD
> 4.x and DFBSD relate to FreeBSD 5-current, which is the overall topic
> of this mailing list at the moment.

This comment is so ridiculous I can't resist. He has explained a very
clever and nice solution for the problem, rather than bitching about it
which is what the 95% of the thread is about. By bashing him, not only
you're doing a great job for inter-camp relations, you're making the
absurd assumption that such a solution (once/if it appears in DragonFly)
cannot/will not be easily ported to FreeBSD.

Cheers,
-- 
Miguel Mendez <[EMAIL PROTECTED]>
http://www.energyhq.es.eu.org
PGP Key: 0xDC8514F1

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-25 Thread Maxim M. Kazachek
On Mon, 24 Nov 2003, Andrew Gallatin wrote:

>
>Daniel O'Connor writes:
> >
> > It is _trivial_ to buildworld with a static root.
>
>Then its equally trivial to build with a dynamic root.  Please do so,
>and don't wreck the performance of the OS I've used since 1994.
Then just use OS from 1994 and don't bother about NEW features
that may appear in dumb, slow, dynamically linked OS. Just put your head
in the sand and say "I'm pretty happy with old statically linked OS and I
never need new features in it"... World is going forward, and the payment
of new features is awesome performance of OLD, not so good profiled (due
statical linking) dynamic linking functions. "if it isn't broken - don't
fix it". But if you don't use dynamic linking - it's DEFINITELY NOT
broken. And will never be fixed.


   Sincerely, Maxim M. Kazachek
   mailto:[EMAIL PROTECTED]
   mailto:[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Gordon Tetlow
On Mon, Nov 24, 2003 at 08:55:31PM -0500, Andrew Gallatin wrote:
> 
> Daniel O'Connor writes:
> 
>  > Why didn't you pipe up when this was discussed _long_ ago?
> 
> In the orginal thread, there was an agreement that the performance
> would be measured BEFORE the default was changed, and the default
> would only be changed if there was no measurable performance impact.
> I believe sam@ asked for this.  As far as I know, performance
> measurments were never done.  Instead, the switch was thrown just
> before the code freeze.

That's not true. I was asked to present numbers so we could make a
determination as to what the impact was. It was never said that it
would only be the default iff there was no performance impact.

FWIW, I did find that the boot process took a performance hit, I
also found that the average worldstone did not increase appreciably
(ie, less than 1%). I took these numbers to re@ when I was asked to
flip the dynamic switch and the feeling was that the overhead was
worth the tradeoff for functionality.

Finally, I must ask if anyone has evidence that this has slowed down
anything other than microbenchmarks? My point of view was it did
slow down the boot, but so did rcNG and no one seemed to mind about
that. Also, you don't write time-sensitive applications in shell
so the dynamic link overhead is not noticed there. People asked me
about the affect on periodic. My response is why do you care if your
periodic took 1 extra second to run (on the outside) due to dynamic
linking overhead. It's just crazy.

In summary, I have yet to see a compelling arguement to consider
backing out the dynamic linking changes I've put in. I've read all of
the messages in all of the 3+ huge threads and I'm still as resolved
today as I was when I made the commit. Frankly, I'm surprised people
didn't yell at me when I massively restructured the tree to put
libraries in /lib. Turning on dynamic linking was the most minor part
from the architectural point of view but is getting the most vitriol.
How typical.

-gordon


pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread M. Warner Losh
> And I just did a "make clean" run in /usr/ports/archivers (by manually
> mv'ing a static and dynamic sh to /bin in turn):
>
> static:   96.63 real53.45 user39.27 sys
> dynamic: 112.42 real55.51 user51.62 sys
>
> The wall clock is bad (16% worse) and the system time is worse (31%).

So all the worstness is in system time, or nealy all.  However, you
rant this test only once.

: >
: > So..
: >
: > 1) Microbenchmark:  40% worse
: > 2) Bootstone(*):25% worse
: > 3) Ports:   16% worse

H, It looks like the hit is less than 10% in the fork intensive
test I just wrote:

#!/bin/sh
for i in 0 1 2 3 4 5 6 7 8 9; do
for j in 0 1 2 3 4 5 6 7 8 9; do
for k in 0 1 2 3 4 5 6 7 8 9; do
 for l in 0 1 2 3 4 5 6 7 8 9; do
 for m in 0 1 2 3 4 5 6 7 8 9; do
  for n in 0 1 2 3 4 5 6 7 8 9; do
true;
done; done; done; done; done; done;

Here's the numbers I got:

x sh-s static sh
+ sh-d dynamic sh
+--+
|x  x  +   |
|x  x   +  +   |
|x  x   x   x   +   +  +   |
|x  x   x   x   +   +   +  +   +  +|
|x   x  x   x   x   +   +   +  +   +   +  +   +|
||___A__M_||A__M|  |
+--+
N   Min   MaxMedian   AvgStddev
x  20  1.27  1.31   1.31.2925   0.012085224
+  20   1.4  1.47  1.44 1.431   0.023597502
Difference at 95.0% confidence
0.1385 +/- 0.0119989
10.7157% +/- 0.928346%
(Student's t, pooled s = 0.0187469)

As you can see, I ran each of the tests 10 times.  I timed it using a
the tcsh time built-in.  I ran each command once before I started
timing the commands to reduce cache effects.

Clearly dynamic is slower, but it is more like 11% slower (10.67%) on
the average than 40% slower.  I think this would be a more typical
usage pattern.

So things are a little bad, but it isn't the end of the world,
especially for a 5.2-beta that's going out.

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Sam Leffler
On Monday 24 November 2003 07:06 pm, Andrew Gallatin wrote:
> M. Warner Losh writes:
>  > In message: <[EMAIL PROTECTED]>
>  > I'm just saying that most of the developers I'm talking to on IRC say
>  > this tread is insane, has no content and they are blowing it off
>  > because of that.  A concrete, real benchmark will go a long way
>  > towards changing that.  Until then, you are as good as kill filed.
>
> How about Gordon's initial bootstone, which increased by 25%?
> http://docs.freebsd.org/cgi/mid.cgi?16091.44150.539095.704531
>
> And I just did a "make clean" run in /usr/ports/archivers (by manually
> mv'ing a static and dynamic sh to /bin in turn):
>
> static:   96.63 real53.45 user39.27 sys
> dynamic: 112.42 real55.51 user51.62 sys
>
> The wall clock is bad (16% worse) and the system time is worse (31%).
>
>
> So..
>
> 1) Microbenchmark:40% worse
> 2) Bootstone(*):  25% worse
> 3) Ports: 16% worse

I don't believe it was ever demonstrated there was no significant performance 
loss.  I think the switch should not be made until this is resolved.  netbsd 
went through this recently and made an effort to bring performance of a 
dynamic root in line with a static root before making the change.

Sam

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread E.B. Dreger
PW> Date: Mon, 24 Nov 2003 18:56:21 -0800
PW> From: Peter Wemm

PW> We need nsswitch type functionality in /bin/sh.  To the
PW> people who want to make it static, lets see some static
PW> binary dlopen() support or a nsswitch proxy system.

I started a new thread inquiring about the challenges involved...
which received no responses.  I assume that not much has changed
since the mid- to late-90s list archives I found via Google...

What specific aspects of rtld are required to support NSS in
static binaries?  dlopen(), fixups, and dlsym()?


Eddy
--
Brotsman & Dreger, Inc. - EverQuick Internet Division
Bandwidth, consulting, e-commerce, hosting, and network building
Phone: +1 785 865 5885 Lawrence and [inter]national
Phone: +1 316 794 8922 Wichita
_
  DO NOT send mail to the following addresses :
  [EMAIL PROTECTED] -or- [EMAIL PROTECTED] -or- [EMAIL PROTECTED]
Sending mail to spambait addresses is a great way to get blocked.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Daniel O'Connor
On Tuesday 25 November 2003 12:20, Andrew Gallatin wrote:
>  > OK my bad, it will probably slow down the ports building.
>
> I'll bet a larger percentage of our users build ports than need nss or
> ldap.

Err, yes..
Of course you are claiming it should be either/or, which is not very 
reasonable.

What about the newer version of gcc? That is considerably slower than previous 
versions, but I don't see people screaming to have it removed.

Most things are tradeoffs, this is one of them.

-- 
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 9A8C 569F 685A D928 5140  AE4B 319B 41F4 5D17 FDD5

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Scott Long
On Mon, 24 Nov 2003, Matthew Dillon wrote:
> :I think that you forgot to attach the patches that demonstrate all of
> :this.
> :
> :Also, I'm really starting to resent you using the FreeBSD mailing lists as
> :an advocacy channel for DragonFly.  I fail to see how FreeBSD 4.x and
> :DFBSD relate to FreeBSD 5-current, which is the overall topic of this
> :mailing list at the moment.
> :
> :Scott
>
> That's too bad, Scott.  You are just going to have to live with it.
>
> From my point of view anything related to BSD is fair and reasonable
> game, and regardless of anything else this mailing list is discussing
> topics and things that are near and dear to my heart and I have a perfect
> right to comment on them.  I've contributed a considerable amount of
> time and effort to FreeBSD and I have a right to be on this list.
>
> From my point of view, it's all under the same open-source umbrella
> anyway.  The moment you start thinking of a project in isolation is
> the moment you start to disconnect from your responsibility to the
> project and to all the users and developers who depend on the project.
>
> I still run a lot of FreeBSD boxes and know people who run a lot of
> FreeBSD boxes and, frankly, that gives me just as much right to comment
> on the future directions of FreeBSD as you or anyone else.
>
> So you are just going to have to live with it.  Maybe instead of fuming
> over DFly you should consider the work from a more technical standpoint.
>
>   -Matt
>   Matthew Dillon
>   <[EMAIL PROTECTED]>
>
>

I am eager to see what you can do with DFBSD.  I really, truly, am.  Until
then, trolling the FreeBSD mailing lists with uninformed statements about
5.x and propaganda about DFBSD doesn't really mean a whole lot, unless you
are looking for new recruits to your camp.  In any case, you've made your
point on a nearly daily basis that 5.x is inferior to what DFBSD will be,
and that you don't have much knowledge or care about 5.x anyways.  So
please, go do what you do best and make DFBSD the envy of the BSD world.
I'll be first in line to pat you on the back when you succeed.

Scott
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
Andrew Gallatin <[EMAIL PROTECTED]> writes:
: 1) Microbenchmark:40% worse
: 2) Bootstone(*):  25% worse
: 3) Ports: 16% worse

Thanks for the real numbers.

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Matthew Dillon
:I think that you forgot to attach the patches that demonstrate all of
:this.
:
:Also, I'm really starting to resent you using the FreeBSD mailing lists as
:an advocacy channel for DragonFly.  I fail to see how FreeBSD 4.x and
:DFBSD relate to FreeBSD 5-current, which is the overall topic of this
:mailing list at the moment.
:
:Scott

That's too bad, Scott.  You are just going to have to live with it.

From my point of view anything related to BSD is fair and reasonable
game, and regardless of anything else this mailing list is discussing
topics and things that are near and dear to my heart and I have a perfect
right to comment on them.  I've contributed a considerable amount of
time and effort to FreeBSD and I have a right to be on this list.

From my point of view, it's all under the same open-source umbrella
anyway.  The moment you start thinking of a project in isolation is
the moment you start to disconnect from your responsibility to the
project and to all the users and developers who depend on the project.

I still run a lot of FreeBSD boxes and know people who run a lot of 
FreeBSD boxes and, frankly, that gives me just as much right to comment
on the future directions of FreeBSD as you or anyone else.

So you are just going to have to live with it.  Maybe instead of fuming
over DFly you should consider the work from a more technical standpoint.

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Matthew Dillon
:>
:> :I supported the decision because:
:> :
:> :1.  It has been requested for years
:> :2.  It benefits PAM and NSS.
:> :3.  It is easy to revert.
:>
:> Easy to revert?  You are talking about depending on mechanisms for
:> authentication and other things that WILL NOT WORK with static binaries
:> as they currently stand and, apparently, will not work in the
:> future either.  Easy to revert?  I don't think so.
:>
:> More like "Lets do away with support for static binaries entirely".
:> Because that is precisely what is happening here.
:
:What the hell are you talking about

What I am talking about is that if the intent in -CURRENT is to start
to depend on things like NSS... and it really does make sense to
be able to depend on something like NSS, then it will become less
and less feasible to compile programs in a way that cannot use NSS.

/bin/sh is an excellent example of this.  Why is /bin/sh now dynamic
again?  Why can't it be static?  I'm being retorical, but I think
it demonstrates the problem and my point quite succinctly.

Regardless of whether you have a dynamic root or a static root,
FreeBSD is digging itself into a hole if it cannot use these spiffy
new mechanisms with static binaries.

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Scott Long
On Mon, 24 Nov 2003, Matthew Dillon wrote:
>
> :I supported the decision because:
> :
> :1.  It has been requested for years
> :2.  It benefits PAM and NSS.
> :3.  It is easy to revert.
>
> Easy to revert?  You are talking about depending on mechanisms for
> authentication and other things that WILL NOT WORK with static binaries
> as they currently stand and, apparently, will not work in the
> future either.  Easy to revert?  I don't think so.
>
> More like "Lets do away with support for static binaries entirely".
> Because that is precisely what is happening here.

What the hell are you talking about


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Dan Nelson
In the last episode (Nov 24), Scott Long said:
> I think that you forgot to attach the patches that demonstrate all of
> this.
> 
> Also, I'm really starting to resent you using the FreeBSD mailing
> lists as an advocacy channel for DragonFly.  I fail to see how
> FreeBSD 4.x and DFBSD relate to FreeBSD 5-current, which is the
> overall topic of this mailing list at the moment.

I think he's pointing out that within a short time there will be a
BSD-licensed nscd we can use.

-- 
Dan Nelson
[EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Scott Long
On Mon, 24 Nov 2003, Matthew Dillon wrote:
> :Ding!  "Oh god, not another one!" *plonk*
> :
> :We need nsswitch type functionality in /bin/sh.  To the people who want to
> :make it static, lets see some static binary dlopen() support or a nsswitch
> :proxy system.
> :
> :If half as much effort had been spent on implementing such a thing as there
> :has been hand wringing, then we'd have the best of both worlds by now.
> :
> :I'm sorry to sound harsh, but its the reality of the situation.  Code
> :speaks louder than words.
> :
> :Cheers,
> :-Peter
> :--
> :Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
> :"All of this is for nothing if we don't go to the stars" - JMS/B5
>
> You don't need dynamic loading to get nsswitch type functionality.  You
> only need dynamic loading if nobody is willing to write an IPC
> model to get the functionality.  It's really silly to create such a
> fundamental restriction on the binary because people are too lazy
> to build an IPC based mechanism.  Not only silly, but it seems to me
> that it also creates security issues.  At least with a client/server
> model it is possible to isolate the authentication code to, for example,
> disallow exec(), filesystem operations, or the opening of any new files.
>
> The other huge advantage that IPC has over DLL is that you can switchout
> the backend mechanisms on a running system without killing and restarting
> anything other then he one service you want to switch out, and if it
> doesn't work right you can restart the old one or keep live as a fallback.
> the IPC model is so much better then the DLL model for this sort of thing
> I don't understand why people are even arguing about it.
>
> And, of course, moving all of this junk out means that the libc
> implementation becomes a lot smaller... it just becomes an IPC interface
> and may be a small local cache rather then the full blown algorithm.
> The result?  Static binaries become a lot smaller too (not that that
> is really a problem anyway).
>
> This 'it has to be static so dlopen works' argument is just not a good
> argument.  It's really more of an excuse then an argument.  If you
> really want to use dlopen then make it work with static binaries.
> If you want to do it right, develop an IPC model or use an existing
> IPC model.
>
> That said, an IPC model is precisely what I am developing for DragonFly
> (a 'money where your mouth is' response).  :-)  Right now I'm building
> it as a userspace library but I intend to move the rendezvous code
> directly into the kernel.  Unix domain sockets are so icky(!).  They work,
> but they are icky.  I intend to use it for all resource and
> authentication lookups... password, group, services, pam equivalent,
> kerberos, resolver, and so on and so forth.  Hell, I think I might use
> it for C locale too just to be able to rip locale out of libc.
>

I think that you forgot to attach the patches that demonstrate all of
this.

Also, I'm really starting to resent you using the FreeBSD mailing lists as
an advocacy channel for DragonFly.  I fail to see how FreeBSD 4.x and
DFBSD relate to FreeBSD 5-current, which is the overall topic of this
mailing list at the moment.

Scott
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Matthew Dillon

:I supported the decision because:
:
:1.  It has been requested for years
:2.  It benefits PAM and NSS.
:3.  It is easy to revert.

Easy to revert?  You are talking about depending on mechanisms for
authentication and other things that WILL NOT WORK with static binaries
as they currently stand and, apparently, will not work in the
future either.  Easy to revert?  I don't think so.

More like "Lets do away with support for static binaries entirely".
Because that is precisely what is happening here.

:Now please move along and revert it on your local system.  There are far
:too many REAL problems out there that need to be addressed so that 5.2 can
:go out the door.  This is just wasting time and energy.
:
:Scott

This is a real problem.  I have no problem with people who want dynamic
roots to get dynamic roots.  My problem is with this intention to not
fix PAM or NSS in a way that works with static binaries, and my problem
is with changing the default from static to dynamic.  The result
is, down the line, that either (A) it will become impossible to compile
anything static, or (B) there will be things you WON'T be able to use
NSS for because it will break static binaries.

It is a serious logistical and planning mistake, IMHO.

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Andrew Gallatin

Peter Wemm writes:

 > We need nsswitch type functionality in /bin/sh.  To the people who want to
 > make it static, lets see some static binary dlopen() support or a nsswitch
 > proxy system.

Maybe this is just nieve, but I always thought that it was the
responsibility of the party introducing the creeping feature to
underdatand and then minimize a potential performance impact.  But now
we're treading over ground brought up by John Dyson last week.

I've at least partially quantified the performance impact, which is
all I have time for.



Drew
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Matthew Dillon
:Ding!  "Oh god, not another one!" *plonk*
:
:We need nsswitch type functionality in /bin/sh.  To the people who want to
:make it static, lets see some static binary dlopen() support or a nsswitch
:proxy system.
:
:If half as much effort had been spent on implementing such a thing as there
:has been hand wringing, then we'd have the best of both worlds by now.
:
:I'm sorry to sound harsh, but its the reality of the situation.  Code
:speaks louder than words.
:
:Cheers,
:-Peter
:--
:Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
:"All of this is for nothing if we don't go to the stars" - JMS/B5

You don't need dynamic loading to get nsswitch type functionality.  You
only need dynamic loading if nobody is willing to write an IPC
model to get the functionality.  It's really silly to create such a
fundamental restriction on the binary because people are too lazy
to build an IPC based mechanism.  Not only silly, but it seems to me
that it also creates security issues.  At least with a client/server
model it is possible to isolate the authentication code to, for example,
disallow exec(), filesystem operations, or the opening of any new files.

The other huge advantage that IPC has over DLL is that you can switchout
the backend mechanisms on a running system without killing and restarting
anything other then he one service you want to switch out, and if it
doesn't work right you can restart the old one or keep live as a fallback.
the IPC model is so much better then the DLL model for this sort of thing
I don't understand why people are even arguing about it.

And, of course, moving all of this junk out means that the libc 
implementation becomes a lot smaller... it just becomes an IPC interface
and may be a small local cache rather then the full blown algorithm. 
The result?  Static binaries become a lot smaller too (not that that 
is really a problem anyway).

This 'it has to be static so dlopen works' argument is just not a good 
argument.  It's really more of an excuse then an argument.  If you
really want to use dlopen then make it work with static binaries. 
If you want to do it right, develop an IPC model or use an existing
IPC model.

That said, an IPC model is precisely what I am developing for DragonFly
(a 'money where your mouth is' response).  :-)  Right now I'm building
it as a userspace library but I intend to move the rendezvous code
directly into the kernel.  Unix domain sockets are so icky(!).  They work,
but they are icky.  I intend to use it for all resource and
authentication lookups... password, group, services, pam equivalent,
kerberos, resolver, and so on and so forth.  Hell, I think I might use
it for C locale too just to be able to rip locale out of libc. 

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Colin Percival

  Would it be possible to ship a static /bin/sh and a dynamic 
/bin/dynamic-sh, with /bin/sh execing /bin/dynamic-sh if it is invoked 
interactively?  If I'm understanding the issues correctly, a dynamic 
/bin/sh is desired for the benefit of interactive users, while the 
performance of a static /bin/sh is only an issue in non-interactive cases.


Colin Percival

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Andrew Gallatin

M. Warner Losh writes:
 > In message: <[EMAIL PROTECTED]>
 > I'm just saying that most of the developers I'm talking to on IRC say
 > this tread is insane, has no content and they are blowing it off
 > because of that.  A concrete, real benchmark will go a long way
 > towards changing that.  Until then, you are as good as kill filed.

How about Gordon's initial bootstone, which increased by 25%?
http://docs.freebsd.org/cgi/mid.cgi?16091.44150.539095.704531

And I just did a "make clean" run in /usr/ports/archivers (by manually
mv'ing a static and dynamic sh to /bin in turn):

static:   96.63 real53.45 user39.27 sys
dynamic: 112.42 real55.51 user51.62 sys

The wall clock is bad (16% worse) and the system time is worse (31%).


So.. 

1) Microbenchmark:  40% worse
2) Bootstone(*):25% worse
3) Ports:   16% worse


Drew










___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Geoff Speicher
On Mon, Nov 24, 2003 at 03:15:57PM -0500, Andrew Gallatin wrote:
> 
> % /usr/bin/time ./harness.sh ./sh.dynamic 100
> 1.60 real 0.21 user 1.18 sys
> 
> % ./harness.sh ./sh.static 100
> 1.12 real 0.08 user 0.87 sys
> 
> So.. forking a dynamic sh is roughly 40% more expensive than
> forking a static copy of sh.  This is embarrassing.

All this noise because, for every 1,000 invocations of dynamic
/bin/sh, you have to wait nearly an extra 5 seconds.

Truly embarrassing, indeed.  Totally unacceptable for a BETA release.

We all could have collectively spawned millions of dynamic shells
in the time that it has taken to compile this thread.  But we didn't.
Can't be all that important, can it?  :)

Geoff

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Peter Wemm
"M. Warner Losh" wrote:
> In message: <[EMAIL PROTECTED]>
> Andrew Gallatin <[EMAIL PROTECTED]> writes:
> : 
> : M. Warner Losh writes:
> :  > In message: <[EMAIL PROTECTED]>
> :  > Andrew Gallatin <[EMAIL PROTECTED]> writes:
> :  > : I'll bet a larger percentage of our users build ports than need nss or
> :  > : ldap.
> :  > 
> :  > I'll bet a larger percentage of the people are ignoring this thread
> :  > than reading it since it has been so devoid of concrete numbers.
> : 
> : Aside from $SUBJECT..
> 
> I'm just saying that most of the developers I'm talking to on IRC say
> this tread is insane, has no content and they are blowing it off
> because of that.  A concrete, real benchmark will go a long way
> towards changing that.  Until then, you are as good as kill filed.

Ding!  "Oh god, not another one!" *plonk*

We need nsswitch type functionality in /bin/sh.  To the people who want to
make it static, lets see some static binary dlopen() support or a nsswitch
proxy system.

If half as much effort had been spent on implementing such a thing as there
has been hand wringing, then we'd have the best of both worlds by now.

I'm sorry to sound harsh, but its the reality of the situation.  Code
speaks louder than words.

Cheers,
-Peter
--
Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
"All of this is for nothing if we don't go to the stars" - JMS/B5

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Daniel O'Connor
On Tuesday 25 November 2003 12:44, Frank Mayhar wrote:
> _This_ is the issue.  You assert that this change "benefits a fair number
> of users."  I and others assert that it hurts performance and makes
> disaster recovery more complex (while the existence of /rescue is a great
> idea, it still adds complexity).  There's proof for our assertions, but all
> I'm hearing from you guys is handwaving.

I don't think any one has said dynamic / is faster.

> And I'm _not_ trying to be insulting or condescending.  I've done
> handwaving myself in the past, but I try to be aware of it and only do it
> when I can justify it.  In this case, the handwaving is in place of real
> evidence. So, how much does it help?  How _many_ users will it benefit, in
> general? Sure, it doesn't matter for a webserver that runs httpd or for a
> database server that does nothing but run Postgresql, but those cases are
> irrelevant to the issue of a dynamically-linked root.  They are affected
> neither way. It is people who run a variety of applications that will be
> affected, either good or bad.
> So, we've seen data about the performance hit.  What about data about
> improved performance or improved function in some other way?  What is
> the compelling reason to move to a dynamic root?
>
> So far I've seen no argument that was even convincing, let alone
> compelling.

Re-read -current, and the commit messages around this whole thing.

-- 
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 9A8C 569F 685A D928 5140  AE4B 319B 41F4 5D17 FDD5

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
"David O'Brien" <[EMAIL PROTECTED]> writes:
: What qualifies as a "concrete, real benchmark"?  I take it you don't
: think Drew's qualifies.

No.  forkbomds are realworld.

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread David O'Brien
On Mon, Nov 24, 2003 at 07:19:31PM -0700, M. Warner Losh wrote:
> In message: <[EMAIL PROTECTED]>
> Andrew Gallatin <[EMAIL PROTECTED]> writes:
> : 
> : M. Warner Losh writes:
> :  > In message: <[EMAIL PROTECTED]>
> :  > Andrew Gallatin <[EMAIL PROTECTED]> writes:
> :  > : I'll bet a larger percentage of our users build ports than need nss or
> :  > : ldap.
> :  > 
> :  > I'll bet a larger percentage of the people are ignoring this thread
> :  > than reading it since it has been so devoid of concrete numbers.
> : 
> : Aside from $SUBJECT..
> 
> I'm just saying that most of the developers I'm talking to on IRC say
> this tread is insane, has no content and they are blowing it off
> because of that.  A concrete, real benchmark will go a long way
> towards changing that.  Until then, you are as good as kill filed.

What qualifies as a "concrete, real benchmark"?  I take it you don't
think Drew's qualifies.

-- 
-- David  ([EMAIL PROTECTED])
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Scott Long
On Mon, 24 Nov 2003, Frank Mayhar wrote:
> Daniel O'Connor wrote:
> > You DO know FreeBSD is a cooperative project right?
>
> Of course I do.  I was using it when it was just 386BSD 0.1 and a patchkit.
> I've watched it through a lot of changes and while I've never been a part
> of the team, mostly due to lack of time, I try to throw whatever I can at
> it when I'm able.
>
> On the _other_ hand:
>
> > I hardly think you're in a position to complain about a (probably very minor)
> > performance loss which has a trivial work around, which also benefits a fair
> > number of users.
>
> _This_ is the issue.  You assert that this change "benefits a fair number
> of users."  I and others assert that it hurts performance and makes disaster
> recovery more complex (while the existence of /rescue is a great idea, it
> still adds complexity).  There's proof for our assertions, but all I'm
> hearing from you guys is handwaving.
>
> And I'm _not_ trying to be insulting or condescending.  I've done handwaving
> myself in the past, but I try to be aware of it and only do it when I can
> justify it.  In this case, the handwaving is in place of real evidence.
> So, how much does it help?  How _many_ users will it benefit, in general?
> Sure, it doesn't matter for a webserver that runs httpd or for a database
> server that does nothing but run Postgresql, but those cases are irrelevant
> to the issue of a dynamically-linked root.  They are affected neither way.
> It is people who run a variety of applications that will be affected,
> either good or bad.
>
> So, we've seen data about the performance hit.  What about data about
> improved performance or improved function in some other way?  What is
> the compelling reason to move to a dynamic root?

Ok, I've about had it with this thread.  Please look at the beginning of
the argument last week where I CLEARLY layed out the benefits of dynamic
root.  If you want me to go around the globe and take a count of every
person that might be benfited by it, please offer me a salary to do so.
Otherwise, I really don't want to hear any more statements like above.

>
> So far I've seen no argument that was even convincing, let alone compelling.

I supported the decision because:

1.  It has been requested for years
2.  It benefits PAM and NSS.
3.  It is easy to revert.

If you want more details on those three points, please go back and re-read
your email.

Now please move along and revert it on your local system.  There are far
too many REAL problems out there that need to be addressed so that 5.2 can
go out the door.  This is just wasting time and energy.

Scott
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Clement Laforet
On Tue, 25 Nov 2003 03:26:14 +0100
Clement Laforet <[EMAIL PROTECTED]> wrote:
:27 CET 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/LUCIFER  i386

Forget about it :-)
Next time i should think befor posting ;-)

sorry for the noise
clem
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Lars Eggert
M. Warner Losh wrote:

In message: <[EMAIL PROTECTED]>
Andrew Gallatin <[EMAIL PROTECTED]> writes:
: I'll bet a larger percentage of our users build ports than need nss or
: ldap.
I'll bet a larger percentage of the people are ignoring this thread
than reading it since it has been so devoid of concrete numbers.
Amen! More benchmarks, less speculation, please. Since both variants are 
already implemented, any argument lacking measurements is weak.

Lars
--
Lars Eggert <[EMAIL PROTECTED]>   USC Information Sciences Institute


smime.p7s
Description: S/MIME Cryptographic Signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Clement Laforet
On Tue, 25 Nov 2003 12:14:23 +1030
"Daniel O'Connor" <[EMAIL PROTECTED]> wrote:

> > Try timing "cd /usr/ports/www/mozilla-devel ; make clean" with static
> > and dynamic /bin.  bsd.port.mk spawns many many many /bin/sh processes.
> 
> OK my bad, it will probably slow down the ports building.

you can use put SH=/rescue/sh in your make.conf if you think it can have an impact...

[EMAIL PROTECTED]|(501)| kde3]# ldd /bin/sh 
/bin/sh:
libedit.so.4 => /lib/libedit.so.4 (0x28094000)
libncurses.so.5 => /lib/libncurses.so.5 (0x280a9000)
libc.so.5 => /lib/libc.so.5 (0x280e8000)
[EMAIL PROTECTED]|(502)| kde3]# ldd /rescue/sh
ldd: /rescue/sh: not a dynamic executable
[EMAIL PROTECTED]|(503)| kde3]# time make clean SH=/bin/sh

real0m45.019s
user0m20.657s
sys 0m18.339s
[EMAIL PROTECTED]|(504)| kde3]# time make clean SH=/rescue/sh

real0m45.476s
user0m20.892s
sys 0m18.131s
[EMAIL PROTECTED]|(505)| kde3]# uname -ar
FreeBSD lucifer.cultdeadsheep.org 5.2-BETA FreeBSD 5.2-BETA #0: Sat Nov 22 21:09:27 
CET 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/LUCIFER  i386

clem
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread David Leimbach
On Nov 24, 2003, at 8:09 PM, M. Warner Losh wrote:

In message: <[EMAIL PROTECTED]>
Andrew Gallatin <[EMAIL PROTECTED]> writes:
: I'll bet a larger percentage of our users build ports than need nss 
or
: ldap.

I'll bet a larger percentage of the people are ignoring this thread
than reading it since it has been so devoid of concrete numbers.
Yep :).

I feel like saying "set the default to static and make the dynamic bins 
the option" so
the people who can't be bothered to compile their own system even 
though everyone
I know does this for tuning purposes anyway can stop whining.

But I won't say that.

Dave

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"[EMAIL PROTECTED]"
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
Andrew Gallatin <[EMAIL PROTECTED]> writes:
: 
: M. Warner Losh writes:
:  > In message: <[EMAIL PROTECTED]>
:  > Andrew Gallatin <[EMAIL PROTECTED]> writes:
:  > : I'll bet a larger percentage of our users build ports than need nss or
:  > : ldap.
:  > 
:  > I'll bet a larger percentage of the people are ignoring this thread
:  > than reading it since it has been so devoid of concrete numbers.
: 
: Aside from $SUBJECT..

I'm just saying that most of the developers I'm talking to on IRC say
this tread is insane, has no content and they are blowing it off
because of that.  A concrete, real benchmark will go a long way
towards changing that.  Until then, you are as good as kill filed.

I don't say this to be mean or nasty.  I'm just saying that people are
sick to death of the arguing without any real facts.

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Frank Mayhar
Daniel O'Connor wrote:
> You DO know FreeBSD is a cooperative project right?

Of course I do.  I was using it when it was just 386BSD 0.1 and a patchkit.
I've watched it through a lot of changes and while I've never been a part
of the team, mostly due to lack of time, I try to throw whatever I can at
it when I'm able.

On the _other_ hand:

> I hardly think you're in a position to complain about a (probably very minor) 
> performance loss which has a trivial work around, which also benefits a fair 
> number of users.

_This_ is the issue.  You assert that this change "benefits a fair number
of users."  I and others assert that it hurts performance and makes disaster
recovery more complex (while the existence of /rescue is a great idea, it
still adds complexity).  There's proof for our assertions, but all I'm
hearing from you guys is handwaving.

And I'm _not_ trying to be insulting or condescending.  I've done handwaving
myself in the past, but I try to be aware of it and only do it when I can
justify it.  In this case, the handwaving is in place of real evidence.
So, how much does it help?  How _many_ users will it benefit, in general?
Sure, it doesn't matter for a webserver that runs httpd or for a database
server that does nothing but run Postgresql, but those cases are irrelevant
to the issue of a dynamically-linked root.  They are affected neither way.
It is people who run a variety of applications that will be affected,
either good or bad.

So, we've seen data about the performance hit.  What about data about
improved performance or improved function in some other way?  What is
the compelling reason to move to a dynamic root?

So far I've seen no argument that was even convincing, let alone compelling.
-- 
Frank Mayhar [EMAIL PROTECTED]  http://www.exit.com/
Exit Consulting http://www.gpsclock.com/
http://www.exit.com/blog/frank/
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Andrew Gallatin

M. Warner Losh writes:
 > In message: <[EMAIL PROTECTED]>
 > Andrew Gallatin <[EMAIL PROTECTED]> writes:
 > : I'll bet a larger percentage of our users build ports than need nss or
 > : ldap.
 > 
 > I'll bet a larger percentage of the people are ignoring this thread
 > than reading it since it has been so devoid of concrete numbers.

Aside from $SUBJECT..

Drew
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread M. Warner Losh
In message: <[EMAIL PROTECTED]>
Andrew Gallatin <[EMAIL PROTECTED]> writes:
: I'll bet a larger percentage of our users build ports than need nss or
: ldap.

I'll bet a larger percentage of the people are ignoring this thread
than reading it since it has been so devoid of concrete numbers.

Warner
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Leo Bicknell
In a message written on Tue, Nov 25, 2003 at 12:12:59PM +1030, Daniel O'Connor wrote:
> If you have a file, web, mail, database, etc server it's predominant 
> application is already dynamically linked.

It just occured to me what bothers me about this line of thinking,
since several people have brought it up.  When I run kwrite, or
Mozilla, or any number of other dynamic apps they are relatively
long lived.  My database loads (eg, pays the dynamic link penalty)
once at startup.  By contrast /bin/sh is run often.

Process accounting can tell the story:

% lastcomm | wc -l
   47806
% lastcomm | sed -e 's/ .*.//' | sort | uniq -c | sort -nr | head
25281 sendmail
4094 sh
2987 perl
2846 inetd
1704 procmail
1640 httpd
1221 cron
 814 date
 732 postgres
 648 rateup

Looks like sh is the 2nd most frequently executed command on my
system.  It is 8.5% of all executed programs on this particular
system.  I think slowing down 8.5% of all the programs the system
runs is important.

I don't suggest I am representative, but for all those with process
accounting turned on you have the commands above, check it out.

> If you are deploying FreeBSD on servers you should build your own release 
> anyway (which is hardly an onerous task).

What?  Did you read what you wrote?  It was a stand alone paragraph,
I didn't take it out of context.  People who use FreeBSD on servers
should build their own release?  That's so nutz I don't know where
to start to attack it.  I think I'll leave it to the third point from
www.freebsd.org:

] FreeBSD makes an ideal Internet or Intranet server. It provides robust
] network services under the heaviest loads and uses memory efficiently to
] maintain good response times for thousands of simultaneous user
] processes. Visit our gallery for examples of FreeBSD powered
] applications and services.

-- 
   Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - [EMAIL PROTECTED], www.tmbg.org


pgp0.pgp
Description: PGP signature


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Don Lewis
On 25 Nov, Daniel O'Connor wrote:
> On Tuesday 25 November 2003 11:52, Dan Nelson wrote:
>  > > I'd greatly prefer that the the dynamic root default be backed out
>> > > until a substantial amount of this performance can be recovered.
>> >
>> > What _REAL WORLD_ task does this slow down?
>>
>> Try timing "cd /usr/ports/www/mozilla-devel ; make clean" with static
>> and dynamic /bin.  bsd.port.mk spawns many many many /bin/sh processes.
> 
> OK my bad, it will probably slow down the ports building.

The ports infrastructure is horribly slow even with a static sh, though
not as glacially slow as installing and patching Solaris 9.

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Daniel O'Connor
On Tuesday 25 November 2003 12:23, Frank Mayhar wrote:
> > Let's see.  You dislike the dynamic root decision enough that
> > you are considering the abandonment of FreeBSD.  Then when
> > you're told that you can still build a static root if you
> > need/want it, you make a sarcastic remark.
>
> It wasn't sarcastic, it was serious.  Needing to have special configuration
> defeats the purpose of running FreeBSD for me.  I'm busy enough that I
> don't have time to deal with everything _now_; adding something else I have
> to keep track of for a half-dozen systems is something else I just don't
> need.

You DO know FreeBSD is a cooperative project right?

I hardly think you're in a position to complain about a (probably very minor) 
performance loss which has a trivial work around, which also benefits a fair 
number of users.

-- 
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 9A8C 569F 685A D928 5140  AE4B 319B 41F4 5D17 FDD5

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Frank Mayhar
I'm not going to add to the heat in the rest of the email, but this is
a very good question:

Daniel O'Connor wrote:
> Why didn't you pipe up when this was discussed _long_ ago?

Honestly, I don't remember the discussion.  It's certainly possible that I
may have missed it.  I just dug around in the -current archives and couldn't
find the it, but my life has been kind of chaotic over the last year or
so.  In fact, if it happened about a year ago, it could well have taken
place while I was in China for three weeks.

Believe me, if I had encountered the discussion when it took place, I would
have made my feelings known.
-- 
Frank Mayhar [EMAIL PROTECTED]  http://www.exit.com/
Exit Consulting http://www.gpsclock.com/
http://www.exit.com/blog/frank/
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: 40% slowdown with dynamic /bin/sh

2003-11-24 Thread Andrew Gallatin

Daniel O'Connor writes:
 > 
 > It is _trivial_ to buildworld with a static root.

Then its equally trivial to build with a dynamic root.  Please do so,
and don't wreck the performance of the OS I've used since 1994.


 > Why didn't you pipe up when this was discussed _long_ ago?

In the orginal thread, there was an agreement that the performance
would be measured BEFORE the default was changed, and the default
would only be changed if there was no measurable performance impact.
I believe sam@ asked for this.  As far as I know, performance
measurments were never done.  Instead, the switch was thrown just
before the code freeze.

 > 
 > If you are deploying FreeBSD on servers you should build your own release 
 > anyway (which is hardly an onerous task).
 > 

Its pretty hard to use the binary update tools that way.

Drew
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


  1   2   >