On Mon, 2011-14-02 at 18:16 +0300, Richard Hainsworth wrote:
> On 02/14/11 16:47, Guy Hulbert wrote:
> > Some comments on what I plan to do after my 2 week hibernation.
[snip]
> > I am interested first in developing a generic framework around the work
> > already done for 'the benchmark game' (TBG*). I will pretend that I am
> > starting from scratch and define a protocol for adding algorithms and
> > exchanging information.
> TBG was designed to test languages and assumes a stable implementation
> of the language. There is a possibility of different language
> implementations, but in a sense they are treated as different languages.
Yes. I'd like to make things a bit more abstract. I want to remove
their goals but keep the useful parts of their framework.
[snip]
> Actually, all of the intelligence is hidden in the
> bencher/makefiles/xxxxxx.ini file
> Explanation is in the .ini and in the bencher/readme file. However, it
> can be very obscure.
That's very useful to know.
[snip]
> > (A) Protocol to specify algorithm.
> >
> > 1. Define an algorithm and provide a reference for it.
> > 2. Define standard inputs and implement algorithm in 2 languages.
> > 3. Generate and verify outputs corresponding to inputs.
> > 4. Make code, input and output available.
> TBG has directories for each benchmark, since the idea is to make it
> easy to add another language to a benchmark.
I would change their structure but that should be independent of the
protocol.
>
> Also a directory for input files and output files to be diffed against
>
> I think that we should have directories for each language, since the
> number of languages is small, and if a new implementation of a benchmark
> is added, we want to see when it appears in the historical record.
Keeping a historical record is useful for perl6. I want to address the
following use cases:
1. perl6 development
2. TBG
3. idle curiosity
4. adding algorithms
By (3), I mean it should be as easy as:
$ apt-get install BM
$ apt-get install C C++ perl6
$ run-bm
lang: .all
algo: nbody
algo: pidigits
algo: .done
... run ...
Algorithm C C++ perl6
nbody 1 1.2 1.5
pidigits 2 1.9 3.7
The program would download examples on request and automatically find
installed languages. There would be a filtered command-line
presentation and the user could build his own database of results.
If you can do that then you can do pretty much anything you want.
[snip]
--
--gh
