----- Original Message ----- From: "Robert G. Brown" <[EMAIL PROTECTED]>
To: "Vincent Diepeveen" <[EMAIL PROTECTED]>
Cc: "Geoff Jacobs" <[EMAIL PROTECTED]>; <[email protected]>
Sent: Saturday, August 26, 2006 1:43 PM
Subject: Re: [Beowulf] SPEC CPU 2006 released
On Sat, 26 Aug 2006, Vincent Diepeveen wrote:
Find me 1 site that 'tests' hardware that's objective. Spec is the
best compromise.
I don't know about site testing hardware, but there are many objective
hardware tests on sites. In particular, lmbench is an excellent and
unbiased toolset (in my personal belief, having communicated fairly
extensively with Larry and Carl, knowing e.g. that Linus uses lmbench
routinely to test and tune the linux kernel). benchmaster (my own
lmbench is a great achievement to make, let's be clear there.
i don't want to spit at it at all.
lmbench is totally useless for 99% of the programmers who program for
speed (of course
the group of programmers who needs speed is a small subgroup of the
total world, we just
talk about that last group here which needs performance when you
scroll down).
The latencies it reports are simply not the latencies THEY get because
worst cases of memory
work different. Worst is always a random lookup to memory and majority
of software needs random
lookups. You'll argue that on paper certain software you can rewrite
to not using it.
That's just not reality.
Basically what 99% needs is a good random latency, when randomly reading
to memory with all the cpu's busy at the same time.
Both AMD and Intel engineers clearly understand this lucky a lot
better than some simplistic lmbenches show here.
What has improved at latest incarnations of AMD&Intel is the random
lookup latency, which typically does NOT
fit in L2 cache like specint2000. For a short while specint2006 is
good. Sjeng for example uses 150MB ram,
this where specint2000 crafty was lobotomized to using 2MB ram.
You see now suddenly P4 is 2 times slower than AMD64. That didn't used
to be the case. So even studying
improvement of benchmarking is showing the reality there.
lmbench already historically has it wrong there. I remember past 10
years this problem repeating.
What has a faster latency at this moment, a DDR ram Opteron (160-200
ns latency random memory lookups; 8 bytes in fact
i lookup in own test) or a woodcrest system DDRII (100-140 ns latency
for random lookups through entire memory)?
LMBENCH will just fool you there.
Thanks to Bill here for testing my program at one of his woodcrest
systems.
To slowly move to your next subject:
This really is the times of the brilliant programmer. Most
universities that need a lot of crunching speed do not realize it yet,
but should. If you want to do a simplistic modulo using CMOV's at the
processor
of a tiny prime my code went like this. This is already a quite fast
way to do it, not used by most implementations:
if( i >= prime ) // 1x
i -= prime;
mask = i-3;
i = (i<<1)|1;
if( i >= prime ) // 2x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 3x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 4x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 5x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 6x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 7x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 8x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 9x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( i >= prime ) // 10x
i -= prime;
mask |= (i-3);
i = (i<<1)|1;
if( mask == 0xffffffff ) { ... verification of mask
So now we took up to 10 modulo's of 1 prime. However the problem of
all such codes is that it is totally sequential.
I managed to get the program 50% faster however than the above code.
How?
if( i >= prime ) // 1x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask = i-3;
mask2 = i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 2x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 3x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 4x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 5x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 6x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 7x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 8x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 9x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( i >= prime ) // 10x
i -= prime;
if( i2 >= prime2 )
i2 -= prime2;
mask |= i-3;
mask2 |= i2-3;
i = (i<<1)|1;
i2 = (i2<<1)|1;
if( mask == 0xffffffff ) { ...verfication for prime
...
}
if( mask2 == 0xffffffff ) { ...verfication for prime2
...
}
Now i basically give the processor a program with a higher ILP
(instruction level parallellism,
most writing in this list already realize that majority of her readers
and all the people who find this list at google,
do not know what the abbreviations stand for), so it has a chance to
reschedule code internally now and
execute effectively at a higher IPC (instruction per cycle).
Effective speedup at K8 is nearly exactly 50% of this first brute
force layer factorisation by sieving.
Such toying is more difficult at K7. The same code doesn't run 50%
faster at K7. Just a small % it runs faster
at k7.
I compared with 32 bits code, so the comparision of k7 vs k8 is very
fair, though in fact my sieve is basically doing layer 1
factorisation with a brute force sieving by primes up to 64 bits.
The real cause of that 50% speedup, whether k8 has for example better
rescheduling, or whether k7 has only a single execution unit that
can do CMOV and k8 has 2, or whether it takes the branches faster,
that's not real clear to me. AMD for sure isn't going to comment on that.
Of course i'll do a try with 3 primes simultaneously as well. But 50%
speedup kicks major butt.
However just getting the idea of doing an optimization of this, any
non-commercial researcher ever done optimization like this?
I bet out of a total of tens of thousands you can count the number of
programmers capable and carrying it out at 1 hand.
Majority of the highend software simply needs highend because the
persons in questions are too lazy to either give the software to a good
programmer to optimize it, don't want to pay someone for it, but in
the meantime their government spends millions to hardware
that runs their software factor 50 slower and compensates by that
factor 50 slowdown by throwing factor 50 more cpu's into battle.
The best optimized software typically runs single cpu in commercial
interfaces and the worst optimized software usually runs at a couple
of hundreds
of nodes.
Of course for those reading this posting, not YOUR software :)
microbenchmark suite, ex. cpu_rate) isn't finished, really, but it is
definitely objective. netpipe is quite objective. I've never heard
bonnie accused of being biased. Stream is objective, and damn simple
code at that. I really dunno about the new code being developed for
top500 testing, but I do trust to SOME extent the folks developing it --
the bias if any will be in that it is focussed on HPC and perhaps
certain classes of code WITHIN HPC.
Well here is the big problem, what do you want to test with it?
Like IMHO one important feature of highend is the effective latency of
node to
node when the entire network is running the same program, even more than
the effective bandwidth you have from node to node when entire machine
functioning.
A real problem is that totally embarrassingly software is the easiest
to donate.
As soon as latency is a problem then it helps so so much to optimize
to the hardware
architecture, that a testset could be called biassed.
Like the parallellism i use in Diep is not biassed (it's not taking
advantage of figuring out
which processes are less hops away than others), whereas i easily
could have done that,
but that would only work for SGI in that case.
Yet the program profits a lot from such things.
On other hand i'm busy with a small attempt to see whether it's
possible to find large primes with
other than mersenne numbers. For that i'm busy porting in my spare
time some FFT code in order
to run it parallel.
I'll have no option but to do that over a cluster with highend cards
having a good one way pingpong
latency, because a single core just can't deliver enough calculation
power coming 10 years.
So the parallellisation will be 2 layers. Both over the number of
cores that share the same memory
in a rather fast manner, what i'm still studying at, is how to
parallellize such a thing over a fast latency
network.
This is rather interesting for a benchmark as we plan to opensource
the code.
However we soon figured out that calculating in integers is way faster
for FFT than calcultaing with
floating point.
To give concrete examples:
a) in floating point you have at most 53 bits significance in a 64
bits double precision floating point
b) a multiplication in floating point gives 53 x 53 bits = 53 bits.
Means effectively you lost 53 bits in significance. Just 26 bits
of the original 53 bits could be
used. Whereas in 64 x 64 bits integers you get a 128 bits result.
Means you can use 64 bits
effectively. An imul (reg x reg) is at least as fast, if not
normally faster than a floating point multiply.
c) simple instructions like moving a value from 1 register to another
in the normal GPR (general programming registers)
can be executed at 3 instructions a cycle at K8 and 4 at core2.
At MMX/SSE/SSE2 this is roughly 4 cycles.
The above will hold true for future even more, because the tiny
processors are winning bigtime. A dinosaur like itanium2
of course can't compete with a k8 or core2. Just the production price
of such a giant chip like itanium2 is factors more of course,
as its sheer size is factors bigger than the tiny sizes of an A64 dual
core (183mm^2) or a core2 dual core (141mm^2).
This where the factory is doubling in price to build, so you simply
can't use a dedicated factory for a highend processor,
as the revenue out of a highend processor is less than 1/10th of the
factory cost. So production of a highend giant dinosaur
processor will be always done on outdated process technologies.
itanium2 will be using 0.09 when every pc processor is already 0.065 nm
technology.
So by Apollo as the Greeks would say, please let it be an integer
benchmark this time. It's all going to be pc processors in those
supercomputers anyway.
Many FFT codes, yes even when calculating at 10^-19 can be done with
integer codes a lot faster.
Just no one got paid so far to do it.
Measuring number of flops is totally useless with tiny pc processors.
Number of integer operations a second is far more useful. Such codes
can run factor 6 faster.
Macro benchmarks are much tougher, as they are really in the category of
"is this program "like" my program" for the component(s) of the suite or
tool. They also tend to be relatively "rich" code environments, where
compilers are tested as much or more than just the hardware. This is
always true, I supposed, but it makes it difficult to separate
comparative analysis of different hardware from the compiler even when
the same compiler is used, especially across architectures. Leaving
aside the issue of whether or not one SHOULD try to differentiate
compiler from the hardware. Microbenchmarks are probably better for
that purpose as in many cases the core code fragments they time are
small enough that there isn't that much variation in their assembler
implementation, at any rate.
I am a longstanding, fairly passionate advocate of totally open (GPL)
benchmarking code. I was one of the folks who talked John into opening
Sure, but who pays good programmers. A good programmer makes 100x more
difference. It's real hard to find good codes to benchmark, because a
compiler
team can totally screw you and show results that in reality just
aren't the case.
I remember when P4 Xeon 1.7Ghz and K7 1.2Ghz MP released. Every single
person on planet
earth the K7 was always faster than a P4.
True, the memory subsystem of K7 was totally screwed, SO YOU PROGRAM
AROUND THAT.
Somehow all kind of P4's managed to land as pretty good in benchmarks.
This because intel shipped
those testers P4EE type chips with a 2 cycle or 3 cycle L1, and sold
in reality majority of P4 prescotts
with a 4 cycle L1.
History will repeat itself there always.
I remember many hardware homepages who asked for Diep, to in the end
not use it in a test, because
they wanted to promote a certain new processor (be it AMD or Intel or
some other manufacturer) and
my software simply didn't run fast at it. Selective testing.
That'll happen once again here of course.
up lmbench and liberalizing its fairly restrictive original license,
pointing out that the synergy obtained by freeing the code was of
greater importance than the control he was trying to maintain to prevent
vendor abuse. In the end, I don't think anyone has abused lmbench in
part because it has never become a major marketing tool (unlike certain
other benchmark suites I can name).
One of the things that has LONG irritated me is that SPEC isn't open
source, isn't extensible, isn't freely available. I'm certain that
there are reasons for this -- they could even be "good" reasons. That
doesn't mean I have to like it, or think that the world doesn't need a
truly open alternative. However, this needs at least one human to "own"
the project of creating the alternative. I myself cannot do it -- I
already own a languishing microbenchmark suite, a cluster monitoring
toolset, a random number tester, and an XML-based flashcard presentation
program, and none of them get enough attention as it is. If somebody
DID put one together and own it, though, I'd contribute, help, clap
loudly, cheer.
rgb
Hah when running XML parsers you have to pay microsoft first as they
claim to have some rights there :)
Vincent
