SPEC self-limits its relevance by refusing to recognize that it should
be open-source. being open-hostile means that it has very limited
numbers
of data points,
Yup, only 9,335 submissions indexed on this page:
http://www.spec.org/cpu2006/results/cpu2006.html
I think SPEC is worthwhile, just don't understand why the organization
has persisted in self-limiting its accessibility/relevance.
that number is also a bit disingenuous, as it lumps together
int/fp and the rate versions. further, you see many cases
where the results are shown for systems that differ only in
packaging (who would guess that a dell 1950 performs the same
as a 2950 when configured identically!)
actually, the latter is one of SPEC's uses: allowing a vendor
to confirm in a public way that a particular model is not broken.
it's also nice to see a particular model/config with a range
of different CPUs. and it's sometimes possible to compare across
vendors, as well. these are all wonderful, and I value SPEC
for them. what I don't understand is why it helps SPEC or consumers
of SPEC to keep the rest of the world from running the tests.
very minimalistic UI (let alone data mining tools),
It is a limited text-based UI -- that runs on Linux, Windows, and
proprietary Unixes. Portability was/is a major goal of SPEC.
sure, everyone understands portability. but wider availability
of the source (and the vast increase in data that would result)
would make it far more interesting to do real data mining.
The search form:
http://www.spec.org/cgi-bin/osgresults?conf=cpu2006&op=form
is useful for data mining.
I use it, but let's not kid ourselves - it's not data mining by any
meaningful definition. and I definitely mean no slight to whoever
put together the interface - it's nice given its mandate.
and perhaps most importantly, slow adaptation to changes in how
machines
are used (memory footprint, etc).
True. But SPEC MPI2007 v2.0 is a 2.4 GB package of software [larger
datasets, and a 128 GB minimum RAM / 64 cores (min) requirement to run the
Large suite; the Medium suite (16 GB, ~8 core minimum) is still part of
2.0].
I meant SPECCPU, of course. actually, I'd like to ask you how to think
about SPECMPI results. I spent some time staring at them just now, and
am not sure how to draw conclusions.
for instance, with SPECCPU, one of the first things you have to do is
trim the results:
http://www.spec.org/cpu2006/results/res2008q2/cpu2006-20080328-03888.html
that cactusADM result is not informative.
also, of the 187 SPECMPI results, there are only a handful of vendors:
given that there are hundreds of CPU centers that would love to be able to
profile their clusters wrt other clusters, don't you think that being
open-source would fundamentally change the value of the benchmark?
Like HPC Challenge, and NAS Parallel, it does not provide a single number
as a metric of performance. There are always compromises and knashing of
teeth in coming up with a formula for that single number, but
SPEC and the Linpack/Top500 maintainers have found that people like it.
I guess it's a question of what your goals are. the single scalar result
is good for the marketing folk (and I would claim that SPEC is pretty
driven by them). for customers, I don't think the single scalar is much
used or wanted, since it's too hard to tell what it means. with top500,
it's perfectly clear what the number means: raw incache flops with a minor
adjustment for interconnect. yes, it's entirely possible to extract the
raw component-level data from SPEC and produce your own (trimmed, perhaps
more discipline-focused) metric. but how many people do it? I did it
for our last major hardware refresh ~5 years ago, but if it was possible
to have community involvement (variety, eyes, fertilization) in SPEC,
the benchmarks could be an entirely different kind of garden kind of garden.
_______________________________________________
Beowulf mailing list, [email protected] sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
