Wes Peters writes:
> "Albert D. Cahalan" wrote:
>> No, no, no. You have to tune the systems EQUALLY. Um, how? :-)
>>
>> What if some random admin was picked to tune the systems?
>> Maybe he is a Solaris admin, but he honestly tries to tune
>> the other systems. Sure you wouldn't complain that he did a
>> bad job if FreeBSD lost?
>>
>> Driver quality varies too, so hardware choice matters. It is
>> not OK to test on identical hardware, unless the purchaser
>> selects random off-the-shelf hardware to avoid any bias.
>>
>> There are 2 sane ways to benchmark:
>>
>> 1. Use an out-of-the box config on randomly selected hardware.
>> This is what a typical low-paid admin will throw together,
>> so it certainly is a valid test. It is best to run this test
>> many times, since an OS may get unlucky with hardware selection.
>> (tuning is equal: none at all)
>
> But this is not a valid test. I certainly wouldn't hire someone
> who knows NOTHING about the platform to run a critical service on
> it, why would I accept a benchmark run in such a manner? This is
> a completely ludicrous statement.
Not. Lots of places don't have the time, money, or judgement
to hire an expert. Even if they do, they often don't want to
be stuck relying on that expert too much. Maybe he quits one
day, and soon after that his manager gets stuck rebuilding
the system. It's nice to have an OS doesn't require serious
hacking and careful hardware selection to operate with reasonable
performance.
> The other problem is the impossibility of any such benchmark to
> discover the underlying reasons behind the default configuration.
> Re-run the same test, pulling the power cord once an hour (pretend
> you're in California here) and see which spends most of it's time
> in fsck.
I don't have a problem with that test, even if I may dislike
the results. It is a perfectly reasonable test to run.
>> In the Sysadmin article, the biggest error was that the admin
>> crudely tuned the FreeBSD and Linux boxes.
>
> No, he crudely tuned the FreeBSD and Solaris boxes, while proving his
> foregone conclusion that Linux was the cat's ass. Gee, that was a
> surprise.
Oh sorry, Linux got the same treatment as FreeBSD and Solaris.
Only the NT box was untuned, and it beat FreeBSD BTW.
He did "ulimit -n 8192" on all three UNIX-like systems, and...
Linux:
echo 65536 > /proc/sys/fs/file-max
FreeBSD:
kern.maxfiles=65536
kern.maxfilesperproc=32768
Solaris:
set rlim_fd_max=0x8000
set rlim_fd_cur=0x8000
Hey, no fair! FreeBSD and Solaris got twice as much tuning as
the Linux box, and NT got none. But you don't like the results,
so you say this was somehow unfair.
I'd say the real winner was NT. It mostly kept up with Linux,
trashed FreeBSD and Solaris, and didn't need any tuning to do it.
>> He should have left
>> both with out-of-the-box limits to be fair to NT and Solaris.
>
> No, he should have configured all of them as close to equally as
> possible.
That is pretty much what he did.
Oh, you mean he should fairly tune them for performance?
Let's see you tune an NT box as well as your FreeBSD box.
Except for an open competition, benchmarking on tuned
boxes is crap. There just isn't any way to be fair.
>> It is absurd to suggest that he should have been hacking away
>> at compile-time constants. Every OS had a default kernel.
>
> And nobody on the planet, other than you, would use it for this
> or any other application.
I'd rather not, but I might if I was pressed for time.
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
