On Tue, 30 Jul 2002, Andrew J Bromage wrote: [snip] > In the end, though, benchmarks ignore one of the most important rules > of software performance: "throughput" (i.e. the amount of processing > that your system can do just prior to being overloaded) is almost never > the most important consideration. Other considerations such as > flexibility, robustness, responsiveness and scalability are almost > always more important.
Mmm, such statements really assume that there's a sensible meaning to `almost always' when applied to the set of all programmers, whereas I think a much more realistic assumption is that `there's lots of people out there, all with different priorities' and present things in way which lets people perform their own evaluations. (In cases where I've reason to believe how I code I can simply, reliably and significantly affect throughput I care very much about it.) The problem with language benchmarks is not that they `over-rate' the importance of performance but that they assume per se that choice of language is a single-variable (execution speed) optimization problem; there's no attempt to measure the other items in your list, most especially flexibility. (I'm assuming you mean flexibility of the programmer rewriting, retargeting, refactoring and re-engineering exisiting code.) Of course, I don't have any good ideas about how to measure these, particularly flexibility, in a practicallty implementable and accurate way :-) > I've thought for a while that what we need is more benchmarks like > pseudoknot: Real tasks which real people want to do. Computing > Ackermann's function is all well and good, but when's the last time you > actually needed to compute it in a real program? I suspect there probably are things that make Ackermann's function a bad `test-case' (eg, computationally simple and regular => good cache utilisation after optimzation which don't extrapolate?) but, for the purposes I'd want to use benchmarks for these deficiencies compared to things like BLAS performance -- which are things that _some_ real people do all day -- are probably don't affect the results all that much. Of more concern to me is, when's the last time you actually got a well specified computational problem and a reasonable amount of time to write a carefully crafted program to solve it, (particularly when you had some reassurance that the very specification of what to solve wouldn't change after the first time you ran the code :-) )? > Off the top of my head, some "real" tasks which could be benchmarked > include: > > - MPEG video compression. Here you really want to measure `glue-code' overhead (both performance wise and `human rewriting'-wise) of linking together core processing elements either written in low-level code (MMX, etc) or available on DSP chips, in a way which would allow shifting components and algorithms about. (I.e., in my opinion it's an informative task to have `benchmark-type' data about because it's complicated with many ways to solve the problem, not because it's `real world'.) ___cheers,_dave_________________________________________________________ www.cs.bris.ac.uk/~tweed/ | `It's no good going home to practise email:[EMAIL PROTECTED] | a Special Outdoor Song which Has To Be work tel:(0117) 954-5250 | Sung In The Snow' -- Winnie the Pooh _______________________________________________ Haskell-Cafe mailing list [EMAIL PROTECTED] http://www.haskell.org/mailman/listinfo/haskell-cafe
