On Tue, 25 Jan 2005 14:39:42 -0500, renard <[EMAIL PROTECTED]> wrote: > I am a newbie and have been following this thread since I am interested in > benchmarking. > > So I copied the code and ran it on my machine. I have a 3.5 MHz system > runing Windows XP. I am using ActivePerl 3.8.6. > > On my machine, benchmark complained about too few iterations. So I modified > the script in this manner: > 1. Changed the iterations to 10,000,000. > 2. Used :hireswallclock. > 3. Added the pos-regex code. > 4. Also called on cmpthese since the display is easier to read and gives a > more intuitive feel for the results. > > The bench2 code and results are shown below. > > The results essential agree with the results of the original script. > > I did some experimentation. In the man page, the examples have the tested > code inside single quotes while the suggested benchmarking script uses > anonymous subroutines. So I edited the script to eliminate the subroutines > and placed the code to be tested inside single quotes. > > The bench3 code and results are also shown below. > > The results are totally different. Every code segment run faster and the > regexs are way ahead of substrings. > > I ran other experiments and I have come to the following conclusions: > > 1. number of iterations (time that test code runs) is an important factor. > Too short and the results vary > greatly between runs. I found that too many iteations tended to provide > inconsistent results between runs. > > 2. running too many segment codes in one benchmark has the tendency to > produce inconsistent results. > it is best to only tests two to three code segments within one benchmark. > > 3.try to run the code segments within the same environments they will be > run. > The anonymous subroutines proved to be very costly and produce totally > different results. > I leave this to experts to provide an explanation. > > 4. how busy your machine is will greatly affect the benchmark results. > > It is not my intent to cause a flame war. I was just trying to learn and > verified the results. > > My final 2 cents: It appears that the simplest regex perfomed best.
I think you've got a pretty good handle on the different factors that can affect benchmarking, and its dangers and uses. Subroutine invocation is, indeed, expensive, and the results will reflect that. Different machines will also perform the operations at different speeds: your 3.5 GHz PIV is about 40% faster than my PPC/750 in this case and orders of magnitude faster than my PII/166. It also seems that different oberators and functions optimize differently for different OS/architecture combinations. And that, if it turns out to be correct, is the most significant observation. It's a truism that the language is implemented differently on different platforms, but I think most people don't often stop to think about the broader implications of that fact. Normally we encounter OS artifacts in I/O, and if we can get that to work and get the modules we need to compile, we go on to ignore the OS. Maybe we shouldn't. The most important thing to remember about benchmarking, though, is that code is always executed in context, and it's difficult to get a good handle on actual performance. For instance: the 'plain_subtr' solution was slower than the regex, but testing it also incurred an extra variable assignment. If any of the functions were being used differently, they might very well test differently. So now you know that regex worked best for you, this time, implemented this way. And you know that half the time it worked best for me, this time, implemented in this way. And that gives us both something to think about the next time encounter a similar problem, a little exttra knowledge to combine with what we already know and deepen out understanding of the language. That is the real use of benchmarking. For some othe thoughs, you might take a look at the current thread 'hash slice/exists vs. grep benchmark weirdness...' HTH & YMMV, --jay -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] <http://learn.perl.org/> <http://learn.perl.org/first-response>
