Re: Benchmark Suite

2004-01-30 Thread Dan Sugalski 
At 6:38 PM -0500 1/25/04, Gordon Henriksen wrote:
On Sunday, January 25, 2004, at 06:01 , Matt Fowles wrote:

Of late it seems that everybody has been throwing around their own 
little homegrown benchmarks to support their points.  But many 
people frequently point out that these benchmarks are flawed on one 
way or another.

I suggest that we add a benchmark/ subdirectory and create a 
canonical suite of benchmarks that exercise things well (and 
hopefully fully). Then we can all post relative times for runs on 
this benchmark suite, and we will know exactly what is being tested 
and how valid it is.
Well, there's already examples/benchmarks. If those programs are not 
at all realistic, then more realistic benchmarks should be added.

Would be nice if there were a convenient way to run the lot of them 
and collect the timing information, though.
Sounds like a good plan. I've thrown an item into the todo list :)
--
Dan
--it's like this---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Benchmark Suite

2004-01-25 Thread Matt Fowles 
All~

Of late it seems that everybody has been throwing around their own 
little homegrown benchmarks to support their points.  But many people 
frequently point out that these benchmarks are flawed on one way or another.

I suggest that we add a benchmark/ subdirectory and create a canonical 
suite of benchmarks that exercise things well (and hopefully fully). 
Then we can all post relative times for runs on this benchmark suite, 
and we will know exactly what is being tested and how valid it is.

Matt

Re: Benchmark Suite

2004-01-25 Thread Luke Palmer 
Matt Fowles writes:
 All~
 
 Of late it seems that everybody has been throwing around their own 
 little homegrown benchmarks to support their points.  But many people 
 frequently point out that these benchmarks are flawed on one way or another.
 
 I suggest that we add a benchmark/ subdirectory and create a canonical 
 suite of benchmarks that exercise things well (and hopefully fully). 
 Then we can all post relative times for runs on this benchmark suite, 
 and we will know exactly what is being tested and how valid it is.

Like, for example, examples/benchmarks ?

It's quite difficult to create benchmarks that test *everything*.  But
any time someone posts a good benchmark, it really should go in here.
Hopefully with some documentation describing what it tests.

Luke

Re: Benchmark Suite

2004-01-25 Thread Michael Scott 
A few notes on the benchmarks can be found/added here

http://www.vendian.org/parrot/wiki/bin/view.cgi/Main/ 
ParrotDistributionExamples#benchmarking

Mike

On 26 Jan 2004, at 00:14, Luke Palmer wrote:

Matt Fowles writes:
All~

Of late it seems that everybody has been throwing around their own
little homegrown benchmarks to support their points.  But many people
frequently point out that these benchmarks are flawed on one way or  
another.

I suggest that we add a benchmark/ subdirectory and create a canonical
suite of benchmarks that exercise things well (and hopefully fully).
Then we can all post relative times for runs on this benchmark suite,
and we will know exactly what is being tested and how valid it is.
Like, for example, examples/benchmarks ?

It's quite difficult to create benchmarks that test *everything*.  But
any time someone posts a good benchmark, it really should go in here.
Hopefully with some documentation describing what it tests.
Luke

Re: Benchmark Suite

2004-01-25 Thread Gordon Henriksen 
On Sunday, January 25, 2004, at 06:01 , Matt Fowles wrote:

Of late it seems that everybody has been throwing around their own 
little homegrown benchmarks to support their points.  But many people 
frequently point out that these benchmarks are flawed on one way or 
another.

I suggest that we add a benchmark/ subdirectory and create a canonical 
suite of benchmarks that exercise things well (and hopefully fully). 
Then we can all post relative times for runs on this benchmark suite, 
and we will know exactly what is being tested and how valid it is.
Well, there's already examples/benchmarks. If those programs are not at 
all realistic, then more realistic benchmarks should be added.

Would be nice if there were a convenient way to run the lot of them and 
collect the timing information, though.



Gordon Henriksen
[EMAIL PROTECTED]

Re: Benchmark Suite

2004-01-25 Thread Leopold Toetsch 
Gordon Henriksen [EMAIL PROTECTED] wrote:

 Would be nice if there were a convenient way to run the lot of them
 collect the timing information, though.

Yep. That would be really great. That is: have per platform numbers over
time (correlated to patches) about performance of current and a lot of
*TODO* benchmarks.

 =97

a!

 Gordon Henriksen

 --Apple-Mail-2-129918875--

[ Still sucks - SCNR ]

leo