Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-12 Thread Vinícius Zavam
2011/3/12 Poul-Henning Kamp : > In message <4d7b44af.7040...@freebsd.org>, Martin Matuska writes: > > > Thanks a lot for doing this properly. > >>What significance level should I take? > > I think I set ministat(1) to use 95 % confidence level by default > and that is in general a pretty safe bet (

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-12 Thread Poul-Henning Kamp
In message <4d7b44af.7040...@freebsd.org>, Martin Matuska writes: Thanks a lot for doing this properly. >What significance level should I take? I think I set ministat(1) to use 95 % confidence level by default and that is in general a pretty safe bet (1 in 20 chance) >I hope this approach is b

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-12 Thread Mehmet Erol Sanliturk
2011/3/12 Martin Matuska > Hi Poul-Henning, > > I have redone the test for majority of the processors, this time taking > 5 samples of each whole testrun, calculating the average, standard > deviation, relative standard deviation, standard error and relative > standard error. > > The relative sta

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-12 Thread Mehmet Erol Sanliturk
2011/3/12 Martin Matuska > Hi Poul-Henning, > > I have redone the test for majority of the processors, this time taking > 5 samples of each whole testrun, calculating the average, standard > deviation, relative standard deviation, standard error and relative > standard error. > > The relative sta

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-12 Thread Martin Matuska
Hi Poul-Henning, I have redone the test for majority of the processors, this time taking 5 samples of each whole testrun, calculating the average, standard deviation, relative standard deviation, standard error and relative standard error. The relative standard error is below 0.25% for ~91%, betw

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-11 Thread Poul-Henning Kamp
In message <4d7a42cc.8020...@freebsd.org>, Martin Matuska writes: >But what I can say, e.g. for the Intel Atom processor, if there are >performance gains in all but one test (that falls 2% behind), generic >perl code (the routines benchmarked) on this processor is very likely to >run faster with t

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-11 Thread Martin Matuska
I don't take this personally and fully understand your point. But even if all conditions you described are met, I am still not able to say "this is better" as I am not doing a microbenchmark. The +x% score is just an average of all test scores weightened by factor 1 - this does not reflect any rea

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-11 Thread Alexander Leidinger
Quoting Martin Matuska (from Thu, 10 Mar 2011 22:33:37 +0100): Hi everyone, we have performed a benchmark of the perl binary compiled with base gcc, ports gcc and ports clang using the perlbench benchmark suite. Our benchmark was performed solely on amd64 with 10 different processors and we

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-11 Thread Poul-Henning Kamp
In message <4d7943b1.1030...@freebsd.org>, Martin Matuska writes: >More information, detailed test results and test configuration are at >our blog: >http://blog.vx.sk/archives/25-FreeBSD-Compiler-Benchmark-gcc-base-vs-gcc-ports-vs-clang.html Please don't take this personally Martin, but you have

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-11 Thread Kostik Belousov
On Thu, Mar 10, 2011 at 10:33:37PM +0100, Martin Matuska wrote: > Hi everyone, > > we have performed a benchmark of the perl binary compiled with base gcc, > ports gcc and ports clang using the perlbench benchmark suite. > Our benchmark was performed solely on amd64 with 10 different processors >

Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-10 Thread Pedro F. Giffuni
FWIW .. I think the phoronix benchmarks got similar results. http://www.phoronix.com/scan.php?page=article&item=gcc_llvm_clang&num=1 http://www.phoronix.com/scan.php?page=article&item=llvm_gcc_atom&num=1 IMHO, 10% is not a huge performance difference though. Pedro.

FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

2011-03-10 Thread Martin Matuska
Hi everyone, we have performed a benchmark of the perl binary compiled with base gcc, ports gcc and ports clang using the perlbench benchmark suite. Our benchmark was performed solely on amd64 with 10 different processors and we have tried different -march= flags to compare binary performance of t