On Mon, Sep 17, 2012 at 8:17 PM, Brett Cannon <br...@python.org> wrote: > > > On Mon, Sep 17, 2012 at 11:36 AM, Maciej Fijalkowski <fij...@gmail.com> > wrote: >> >> On Mon, Sep 17, 2012 at 5:00 PM, Brett Cannon <br...@python.org> wrote: >> > >> > >> > On Sun, Sep 16, 2012 at 10:54 AM, Maciej Fijalkowski <fij...@gmail.com> >> > wrote: >> >> >> >> On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon <br...@python.org> wrote: >> >> > Quick question about the hexiom2 benchmark: what does it measure? It >> >> > is >> >> > by >> >> > far the slowest benchmark I ported, and considering it isn't a >> >> > real-world >> >> > app benchmark I want to make sure the slowness of it is worth it. >> >> > Otherwise >> >> > I would rather drop it since having something run 1/25 as many >> >> > iterations >> >> > compared to the other simple benchmarks seems to water down its >> >> > robustness. >> >> >> >> It's a puzzle solver. It got included because PyPy 1.9 got slower than >> >> 1.8 on this particular benchmark that people were actually running >> >> somewhere, so it has *some* value. >> > >> > >> > Fair enough. Just wanted to make sure that it was worth having a slow >> > execution over. >> > >> >> >> >> I wonder, does adding a fixed >> >> random number seed help the distribution? >> > >> > >> > Fix how? hexiom2 doesn't use a random value for anything. >> >> Ok, then please explain why having 1/25th of iterations kill robustness? > > > Less iterations to help smooth out any bumps in the measurements. E.g 4 > iterations compared to 100 doesn't lead to as even of a measurement. I mean > you would hope because the benchmark goes for so long that it would just > level out within a single run instead of needing multiple runs to get the > same evening out.
Yes precisely :) I think the term "iterations" is a bit overloaded. The "stability" is more important. > > -Brett > >> >> >> > >> > -Brett >> > >> >> >> >> >> >> > >> >> > >> >> > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski >> >> > <fij...@gmail.com> >> >> > wrote: >> >> >> >> >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <br...@python.org> >> >> >> wrote: >> >> >> > So I managed to get the following benchmarks moved into the >> >> >> > unladen >> >> >> > repo >> >> >> > (not pushed yet until I figure out some reasonable scaling values >> >> >> > as >> >> >> > some >> >> >> > finish probably too fast and others go for a while): >> >> >> > >> >> >> > chaos >> >> >> > fannkuch >> >> >> > meteor-contest (renamed meteor_contest) >> >> >> > spectral-norm (renamed spectral_norm) >> >> >> > telco >> >> >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this >> >> >> > benchmark) >> >> >> > go >> >> >> > hexiom2 >> >> >> > json_bench (renamed json_dump_v2) >> >> >> > raytrace_simple (renamed raytrace) >> >> >> > >> >> >> > Most of the porting was range/xrange related. After that is was >> >> >> > str/unicode. >> >> >> > I also stopped having the benchmarks write out files as it was >> >> >> > always >> >> >> > to >> >> >> > verify results and not a core part of the benchmark. >> >> >> > >> >> >> > That leaves us with the benchmarks that rely on third-party >> >> >> > projects. >> >> >> > The >> >> >> > chameleon benchmark can probably be ported as chameleon has a >> >> >> > version >> >> >> > released running on Python 3. But django and html5lib have only >> >> >> > in-development versions that support Python 3. If we want to pull >> >> >> > in >> >> >> > the >> >> >> > tip >> >> >> > of their repos then those benchmarks can also be ported now rather >> >> >> > than >> >> >> > later. People have opinions on in-dev code vs. released for >> >> >> > benchmarking? >> >> >> > >> >> >> > There is also the sphinx benchmark, but that requires getting >> >> >> > CPython's >> >> >> > docs >> >> >> > building under Python 3 (see http://bugs.python.org/issue10224). >> >> >> > >> >> >> > _______________________________________________ >> >> >> > Speed mailing list >> >> >> > Speed@python.org >> >> >> > http://mail.python.org/mailman/listinfo/speed >> >> >> > >> >> >> >> >> >> great job! >> >> > >> >> > >> > >> > > > _______________________________________________ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed