[Summary: the recursion_and_inlining branch stops us tracing arbitrarily deep
in recursive functions. Comments welcome. If we think this is a good idea, we
have to decide if/how many levels deep we unroll before stopping tracing. My
current thinking is that 7 levels looks about right, though this is heavily
dependent on which benchmarks one considers.]
I recently pushed a branch recursion_and_inlining which aims to tackle the
problem of controlling tracing in the face of recursion. I have been vaguely
aware of an issue here for a while, but never quite managed to nail the
problem down. Then Kevin Modzelewski pointed me to a small benchmark called
"polymorphism" [1] he'd sent to the list back in April which shows the core
problem.
Some simple benchmarking shows that we have a performance problem -- PyPy is
4.4x slower than CPython on this benchmark:
$ multitime -n 10 python polymorphism.py
===> multitime results
1: python /tmp/polymorphism.py
Mean Std.Dev. Min Median Max
real 1.649 0.058 1.595 1.632 1.763
user 1.643 0.057 1.592 1.624 1.756
sys 0.002 0.003 0.000 0.002 0.008
$ multitime -n 10 ./pypy-c-orig polymorphism.py
===> multitime results
1: ./pypy-c-orig /tmp/polymorphism.py
Mean Std.Dev. Min Median Max
real 7.198 0.047 7.131 7.203 7.274
user 7.156 0.051 7.076 7.154 7.232
sys 0.033 0.011 0.012 0.032 0.048
The problem is that RPython's naturally aggressive inlining also inlines
recursive functions. So while we don't unroll explicit loops (while/for), we
end up unrolling recursive functions. Sometimes this works out OK, but it's
more likely to end in aborts, or traces which end up with lots of side
traces. Both latter cases are inefficient. As far as I can see, the most
frequent use of aborts is as a heuristic that recursion has been encountered.
There is one way in which recursion doesn't inline, which is if the
recursive function happens to be the start of the trace, in which case (I
think) myjitpl.MIFrame.opimpl_jit_merge_point turns it into a nice loop.
However, if the recursion starts part way into a trace, all bets are off.
Kevin's benchmark has two types of recursion: one in the make_random
function; and another in Poly1.score. make_random is (obviously) directly
recursive; Poly1.score is indirectly recursive. Both cause the problem noted
above.
The recursion_and_inlining branch [2] tries to dynamically spot recursion by
trapping function calls in pyjitpl.MIFrame._opimpl_recursive_call and seeing
if they're currently on the meta-interpreter stack. If they are then it can
choose to stop tracing and turn the recursive call into an actual function
call (it then sets the function as JC_DONT_TRACE_HERE so that, if it hasn't
been traced separately already, it will then be traced, with the recursion
handled by the existing case in opimpl_jit_merge_point). Doing this speeds
Kevin's benchmark up significantly:
$ multitime -n 10 ./pypy-c-level1 /tmp/polymorphism.py
===> multitime results
1: ./pypy-c-level1 /tmp/polymorphism.py
Mean Std.Dev. Min Median Max
real 0.535 0.013 0.516 0.535 0.560
user 0.517 0.018 0.476 0.522 0.544
sys 0.016 0.008 0.008 0.012 0.036
We've gone from 4.4x slower than CPython to over 3x faster (i.e. PyPy in
recursion_and_inlining is 13.5x faster than normal PyPy). Which is nice.
However, if I run the branch on the PyPy benchmark suite, we see some
significant slowdown in a handful of benchmarks relative to normal PyPy.
[Full data is attached as results1.txt]. e.g. hexiom2 is 1.3x slower;
raytrace-simple is 3.3x slower; spectral-norm 3.3x slower; sympy_str 1.5x
slower; and telco 4x slower. A few benchmarks speed up in a meaningful way;
most run so quickly that any differences are lost in the noise (IMHO any
benchmark running for 0.1s or less is too short to draw many conclusions
from). The translate test doesn't seem to be impacted by the change either
way. Nevertheless -- and even taking into account that the current benchmark
suite and PyPy optimisations have sort-of evolved hand-in-hand -- the slow
benchmarks aren't good.
So, fortunately, our branch can trivially be extended to identify not just
recursion, but how deeply we've recursed. Put another way, we can choose to
allow a function to be unrolled a fixed number of times before stopping
tracing. How many unrollings should we choose? Well, I've spent time putting
together some rough data (note: this is not perfect benchmarking, but it's
probably good enough). Let's take our slow-coaches (to one decimal place,
because there's quite a bit of noise) and Kevin's benchmark (to 2 decimal
places, because it runs long enough to make that sensible) and see how they
change relative to normal PyPy as we crank up the unrollings:
#unrollings | 1 | 2 | 3 | 5 | 7 | 10 |
-----------------+------+------+------+------+------+------+
hexiom2 | 1.3 | 1.4 | 1.1 | 1.0 | 1.0 | 1.0 |
raytrace-simple | 3.3 | 3.1 | 2.8 | 1.4 | 1.0 | 1.0 |
spectral-norm | 3.3 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
sympy_str | 1.5 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
telco | 4 | 2.5 | 2.0 | 1.0 | 1.0 | 1.0 |
-----------------+------+------+------+------+------+------+
polymorphism | 0.07 | 0.07 | 0.07 | 0.07 | 0.08 | 0.09 |
Lower is better in this table. What you can't quite see from that is that
Kevin's benchmark starts getting a teeny bit slower from 5 unrollings upwards
(we'd need another decimal place to see it). All data is attached in
results<x>.txt files if you want to see the raw data.
What I take from the above is that a reasonable guess as to a sensible number
of unrollings seems to be 7. It makes most of the standard benchmarks behave
as they always have, while not doing too much to benchmarks which are
recursive in a different style. Of course, this is all *highly* dependent on
the benchmarks chosen, and if one day we have more benchmarks, we should
revisit this number accordingly.
Note also that this is a patch to *RPython*. It should effect (for better or
worse) all RPython VMs, not just PyPy. e.g. it speeds the Converge compiler
up by a bit over 5% (I haven't tried much more than that).
So there you have it. How would you folks like to proceed? I welcome
comments, as this is not an area of RPython that I've looked at before.
e.g. how many unrollings do you think are sensible? Note that a few tests are
currently broken, because they depend on how many unrollings we choose to do.
Once we've fixed that behaviour, I'll unbreak the tests. I also would very
much welcome tests on other recursive and non-recursive programs/benchmarks
you may have to see how this branch impacts upon performance.
I'd like to say thanks to Kevin for pointing out the problem to me; Carl
Friedrich for helping me understand that tracing recursion was the problem;
and Tanzim Hoque for helping pick into some of the details of pyjitpl.
Laurie
[1]
https://raw.githubusercontent.com/dropbox/pyston/master/microbenchmarks/polymorphism.py
[2] The key patch is:
https://bitbucket.org/pypy/pypy/commits/b60064f55316ceb4a3bd784c00a467253a197c4c
Report on Linux bencher3 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64
Total CPU cores: 4
### ai ###
Min: 0.023896 -> 0.023524: 1.0158x faster
Avg: 0.024042 -> 0.023743: 1.0126x faster
Significant (t=4.300444, a=0.95)
Stddev: 0.00031 -> 0.00038: 1.2092x larger
### bm_chameleon ###
Min: 0.007552 -> 0.008992: 1.1906x slower
Avg: 0.010259 -> 0.011695: 1.1399x slower
Significant (t=-2.958955, a=0.95)
Stddev: 0.00422 -> 0.00418: 1.0098x smaller
### bm_dulwich_log ###
Min: 0.093526 -> 0.108061: 1.1554x slower
Avg: 0.132285 -> 0.143125: 1.0819x slower
Not significant
Stddev: 0.11097 -> 0.09810: 1.1312x smaller
### bm_mako ###
Min: 0.008855 -> 0.008900: 1.0051x slower
Avg: 0.011058 -> 0.010998: 1.0055x faster
Not significant
Stddev: 0.00367 -> 0.00355: 1.0325x smaller
### chaos ###
Min: 0.003802 -> 0.006283: 1.6525x slower
Avg: 0.005174 -> 0.007426: 1.4354x slower
Not significant
Stddev: 0.00888 -> 0.00728: 1.2189x smaller
### sphinx ###
Raw results: [76.860211134] [75.8158118725]
### crypto_pyaes ###
Min: 0.024762 -> 0.025725: 1.0389x slower
Avg: 0.027516 -> 0.028323: 1.0293x slower
Not significant
Stddev: 0.01021 -> 0.00950: 1.0752x smaller
### deltablue ###
Min: 0.000666 -> 0.001062: 1.5941x slower
Avg: 0.006572 -> 0.006668: 1.0147x slower
Not significant
Stddev: 0.00912 -> 0.00978: 1.0721x larger
### django ###
Min: 0.015844 -> 0.020710: 1.3071x slower
Avg: 0.017266 -> 0.022298: 1.2915x slower
Significant (t=-11.193663, a=0.95)
Stddev: 0.00226 -> 0.00223: 1.0132x smaller
### eparse ###
Min: 0.167241 -> 0.182543: 1.0915x slower
Avg: 0.226103 -> 0.237554: 1.0506x slower
Not significant
Stddev: 0.05000 -> 0.04521: 1.1060x smaller
### fannkuch ###
Min: 0.092299 -> 0.092014: 1.0031x faster
Avg: 0.093250 -> 0.092878: 1.0040x faster
Not significant
Stddev: 0.00440 -> 0.00437: 1.0066x smaller
### float ###
Min: 0.015519 -> 0.015288: 1.0151x faster
Avg: 0.023097 -> 0.022767: 1.0145x faster
Not significant
Stddev: 0.00619 -> 0.00609: 1.0163x smaller
### genshi_text ###
Min: 0.007438 -> 0.007730: 1.0392x slower
Avg: 0.009406 -> 0.010046: 1.0681x slower
Not significant
Stddev: 0.00894 -> 0.00907: 1.0144x larger
### genshi_xml ###
Min: 0.023765 -> 0.023280: 1.0208x faster
Avg: 0.027038 -> 0.026623: 1.0156x faster
Not significant
Stddev: 0.01481 -> 0.01475: 1.0039x smaller
### go ###
Min: 0.055311 -> 0.053643: 1.0311x faster
Avg: 0.082657 -> 0.062297: 1.3268x faster
Significant (t=4.947673, a=0.95)
Stddev: 0.02728 -> 0.01013: 2.6918x smaller
### hexiom2 ###
Min: 5.799438 -> 7.763072: 1.3386x slower
Avg: 6.019426 -> 7.979220: 1.3256x slower
Significant (t=-44.482017, a=0.95)
Stddev: 0.22222 -> 0.21834: 1.0178x smaller
### html5lib ###
Min: 1.317897 -> 1.255603: 1.0496x faster
Avg: 1.772609 -> 1.587764: 1.1164x faster
Not significant
Stddev: 0.56009 -> 0.49102: 1.1407x smaller
### json_bench ###
Min: 0.400927 -> 0.438493: 1.0937x slower
Avg: 0.404558 -> 0.443308: 1.0958x slower
Significant (t=-19.128974, a=0.95)
Stddev: 0.01106 -> 0.00911: 1.2142x smaller
### meteor-contest ###
Min: 0.059441 -> 0.058887: 1.0094x faster
Avg: 0.060555 -> 0.059946: 1.0102x faster
Not significant
Stddev: 0.00321 -> 0.00319: 1.0071x smaller
### nbody_modified ###
Min: 0.018606 -> 0.018764: 1.0085x slower
Avg: 0.019102 -> 0.019194: 1.0048x slower
Not significant
Stddev: 0.00201 -> 0.00198: 1.0122x smaller
### pidigits ###
Min: 5.363477 -> 5.149447: 1.0416x faster
Avg: 5.383062 -> 5.170978: 1.0410x faster
Significant (t=8.117698, a=0.95)
Stddev: 0.04099 -> 0.04163: 1.0155x larger
### pyflate-fast ###
Min: 0.185501 -> 0.215078: 1.1594x slower
Avg: 0.188702 -> 0.216523: 1.1474x slower
Significant (t=-65.827533, a=0.95)
Stddev: 0.00281 -> 0.00101: 2.7794x smaller
### raytrace-simple ###
Min: 0.017988 -> 0.059219: 3.2922x slower
Avg: 0.018873 -> 0.060078: 3.1833x slower
Significant (t=-141.219274, a=0.95)
Stddev: 0.00113 -> 0.00173: 1.5266x larger
### richards ###
Min: 0.001870 -> 0.002248: 1.2022x slower
Avg: 0.002008 -> 0.002368: 1.1792x slower
Significant (t=-4.065223, a=0.95)
Stddev: 0.00043 -> 0.00045: 1.0413x larger
### rietveld ###
Min: 0.043556 -> 0.050356: 1.1561x slower
Avg: 0.116072 -> 0.120596: 1.0390x slower
Not significant
Stddev: 0.09529 -> 0.08353: 1.1408x smaller
### scimark_fft ###
0.136257 -> -1.000000: -1
### scimark_lu ###
0.300565 -> -1.000000: -1
### scimark_montecarlo ###
0.139376 -> -1.000000: -1
### scimark_sor ###
0.226458 -> -1.000000: -1
### scimark_sparsematmult ###
0.147132 -> -1.000000: -1
### slowspitfire ###
Min: 0.150665 -> 0.150214: 1.0030x faster
Avg: 0.161504 -> 0.161734: 1.0014x slower
Not significant
Stddev: 0.00558 -> 0.00722: 1.2952x larger
### spambayes ###
Min: 0.029400 -> 0.029954: 1.0188x slower
Avg: 0.049929 -> 0.051126: 1.0240x slower
Not significant
Stddev: 0.01489 -> 0.01562: 1.0489x larger
### spectral-norm ###
Min: 0.009344 -> 0.031196: 3.3386x slower
Avg: 0.010038 -> 0.031759: 3.1640x slower
Significant (t=-37.808674, a=0.95)
Stddev: 0.00286 -> 0.00289: 1.0089x larger
### spitfire ###
Min: 1.150000 -> 1.150000: no change
Avg: 1.186200 -> 1.191600: 1.0046x slower
Not significant
Stddev: 0.03181 -> 0.03080: 1.0329x smaller
### spitfire_cstringio ###
[29/1765]
Min: 0.530000 -> 0.410000: 1.2927x faster
Avg: 0.558800 -> 0.433800: 1.2882x faster
Significant (t=25.891200, a=0.95)
Stddev: 0.02480 -> 0.02346: 1.0568x smaller
### sympy_expand ###
Min: 0.210130 -> 0.212947: 1.0134x slower
Avg: 0.317229 -> 0.308200: 1.0293x faster
Not significant
Stddev: 0.19828 -> 0.17363: 1.1420x smaller
### sympy_integrate ###
Min: 0.776576 -> 0.764212: 1.0162x faster
Avg: 1.237801 -> 1.147388: 1.0788x faster
Not significant
Stddev: 0.70127 -> 0.56203: 1.2477x smaller
### sympy_str ###
Min: 0.153791 -> 0.104809: 1.4673x faster
Avg: 0.297872 -> 0.277957: 1.0716x faster
Not significant
Stddev: 0.20000 -> 0.20368: 1.0184x larger
### sympy_sum ###
Min: 0.207153 -> 0.224769: 1.0850x slower
Avg: 0.293553 -> 0.302402: 1.0301x slower
Not significant
Stddev: 0.12345 -> 0.10984: 1.1239x smaller
### telco ###
Min: 0.008000 -> 0.032002: 4.0003x slower
Avg: 0.014001 -> 0.036482: 2.6057x slower
Significant (t=-12.647330, a=0.95)
Stddev: 0.00915 -> 0.00862: 1.0623x smaller
### trans2_annotate ###
Raw results: [301.8] None
### trans2_rtype ###
Raw results: [586.4] None
### trans2_backendopt ###
Raw results: [82.0] None
### trans2_database ###
Raw results: [112.6] None
### trans2_source ###
Raw results: [120.2] None
### twisted_iteration ###
Min: 0.002044 -> 0.002382: 1.1655x slower
Avg: 0.002076 -> 0.002406: 1.1589x slower
Significant (t=-90.456630, a=0.95)
Stddev: 0.00002 -> 0.00001: 1.5006x smaller
### twisted_names ###
Min: 0.000758 -> 0.000808: 1.0667x slower
Avg: 0.000767 -> 0.000816: 1.0634x slower
Significant (t=-41.869786, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.1888x smaller
### twisted_pb ###
Min: 0.003379 -> 0.003722: 1.1016x slower
Avg: 0.003661 -> 0.004049: 1.1059x slower
Significant (t=-11.815989, a=0.95)
Stddev: 0.00016 -> 0.00017: 1.1098x larger
### twisted_tcp ###
Min: 0.085662 -> 0.089216: 1.0415x slower
Avg: 0.087151 -> 0.090960: 1.0437x slower
Significant (t=-11.383010, a=0.95)
Stddev: 0.00155 -> 0.00179: 1.1552x larger
### ai ###
Min: 0.024519 -> 0.023830: 1.0289x faster
Avg: 0.024719 -> 0.024149: 1.0236x faster
Significant (t=5.841142, a=0.95)
Stddev: 0.00032 -> 0.00061: 1.8921x larger
### bm_chameleon ###
Min: 0.007607 -> 0.009337: 1.2274x slower
Avg: 0.010243 -> 0.012217: 1.1928x slower
Significant (t=-4.033700, a=0.95)
Stddev: 0.00420 -> 0.00428: 1.0183x larger
### bm_dulwich_log ###
Min: 0.097236 -> 0.097346: 1.0011x slower
Avg: 0.135518 -> 0.138144: 1.0194x slower
Not significant
Stddev: 0.11196 -> 0.11137: 1.0053x smaller
### bm_mako ###
Min: 0.008915 -> 0.009264: 1.0392x slower
Avg: 0.011081 -> 0.011375: 1.0265x slower
Not significant
Stddev: 0.00370 -> 0.00361: 1.0249x smaller
### chaos ###
Min: 0.003758 -> 0.004400: 1.1709x slower
Avg: 0.005148 -> 0.005729: 1.1129x slower
Not significant
Stddev: 0.00897 -> 0.00854: 1.0504x smaller
### sphinx ###
Raw results: [78.2611758709] [77.0095219612]
### crypto_pyaes ###
Min: 0.025186 -> 0.024932: 1.0102x faster
Avg: 0.027842 -> 0.027758: 1.0030x faster
Not significant
Stddev: 0.01012 -> 0.01023: 1.0108x larger
### deltablue ###
Min: 0.000680 -> 0.000923: 1.3573x slower
Avg: 0.006624 -> 0.006197: 1.0688x faster
Not significant
Stddev: 0.00915 -> 0.00887: 1.0316x smaller
### django ###
Min: 0.015887 -> 0.020281: 1.2766x slower
Avg: 0.017195 -> 0.021805: 1.2681x slower
Significant (t=-10.510799, a=0.95)
Stddev: 0.00215 -> 0.00224: 1.0428x larger
### eparse ###
Min: 0.158280 -> 0.174940: 1.1053x slower
Avg: 0.221771 -> 0.222904: 1.0051x slower
Not significant
Stddev: 0.04991 -> 0.05164: 1.0348x larger
### fannkuch ###
Min: 0.091970 -> 0.092063: 1.0010x slower
Avg: 0.092925 -> 0.093031: 1.0011x slower
Not significant
Stddev: 0.00431 -> 0.00438: 1.0169x larger
### float ###
Min: 0.015624 -> 0.015870: 1.0157x slower
Avg: 0.023262 -> 0.023572: 1.0133x slower
Not significant
Stddev: 0.00613 -> 0.00618: 1.0074x larger
### genshi_text ###
Min: 0.007519 -> 0.007399: 1.0162x faster
Avg: 0.009532 -> 0.009198: 1.0363x faster
Not significant
Stddev: 0.00875 -> 0.00774: 1.1306x smaller
### genshi_xml ###
Min: 0.022098 -> 0.024445: 1.1062x slower
Avg: 0.025456 -> 0.027746: 1.0899x slower
Not significant
Stddev: 0.01513 -> 0.01386: 1.0915x smaller
### go ###
Min: 0.054469 -> 0.049704: 1.0959x faster
Avg: 0.081538 -> 0.062104: 1.3129x faster
Significant (t=4.477202, a=0.95)
Stddev: 0.02725 -> 0.01412: 1.9304x smaller
### hexiom2 ###
Min: 5.765985 -> 7.972130: 1.3826x slower
Avg: 5.997823 -> 8.225795: 1.3715x slower
Significant (t=-45.382973, a=0.95)
Stddev: 0.23419 -> 0.25624: 1.0941x larger
### html5lib ###
Min: 1.251295 -> 1.210693: 1.0335x faster
Avg: 1.732520 -> 1.672716: 1.0358x faster
Not significant
Stddev: 0.58002 -> 0.55705: 1.0412x smaller
### json_bench ###
Min: 0.410679 -> 0.528665: 1.2873x slower
Avg: 0.414892 -> 0.533759: 1.2865x slower
Significant (t=-53.306479, a=0.95)
Stddev: 0.01118 -> 0.01112: 1.0059x smaller
### meteor-contest ###
Min: 0.059319 -> 0.058654: 1.0113x faster
Avg: 0.060485 -> 0.059748: 1.0123x faster
Not significant
Stddev: 0.00321 -> 0.00323: 1.0058x larger
### nbody_modified ###
Min: 0.018536 -> 0.018806: 1.0146x slower
Avg: 0.019032 -> 0.019321: 1.0152x slower
Not significant
Stddev: 0.00204 -> 0.00197: 1.0325x smaller
### pidigits ###
Min: 5.366191 -> 5.344685: 1.0040x faster
Avg: 5.392111 -> 5.366315: 1.0048x faster
Not significant
Stddev: 0.04388 -> 0.04192: 1.0467x smaller
### pyflate-fast ###
Min: 0.193320 -> 0.212552: 1.0995x slower
Avg: 0.196630 -> 0.213785: 1.0872x slower
Significant (t=-54.781980, a=0.95)
Stddev: 0.00197 -> 0.00101: 1.9416x smaller
### raytrace-simple ###
Min: 0.018000 -> 0.056360: 3.1311x slower
Avg: 0.019035 -> 0.057023: 2.9957x slower
Significant (t=-140.523775, a=0.95)
Stddev: 0.00116 -> 0.00152: 1.3046x larger
### richards ###
Min: 0.001856 -> 0.001877: 1.0113x slower
Avg: 0.001988 -> 0.001997: 1.0047x slower
Not significant
Stddev: 0.00044 -> 0.00044: 1.0091x larger
### rietveld ###
Min: 0.041706 -> 0.042024: 1.0076x slower
Avg: 0.114850 -> 0.115267: 1.0036x slower
Not significant
Stddev: 0.09532 -> 0.09368: 1.0175x smaller
### scimark_fft ###
0.136390 -> -1.000000: -1
### scimark_lu ###
0.298607 -> -1.000000: -1
### scimark_montecarlo ###
0.141685 -> -1.000000: -1
### scimark_sor ###
0.226635 -> -1.000000: -1
### scimark_sparsematmult ###
0.147101 -> -1.000000: -1
### slowspitfire ###
Min: 0.152183 -> 0.152311: 1.0008x slower
Avg: 0.162501 -> 0.163124: 1.0038x slower
Not significant
Stddev: 0.00551 -> 0.00570: 1.0344x larger
### spambayes ###
Min: 0.028572 -> 0.029657: 1.0380x slower
Avg: 0.049160 -> 0.051171: 1.0409x slower
Not significant
Stddev: 0.01736 -> 0.01518: 1.1434x smaller
### spectral-norm ###
Min: 0.009342 -> 0.009381: 1.0042x slower
Avg: 0.010040 -> 0.010042: 1.0002x slower
Not significant
Stddev: 0.00285 -> 0.00289: 1.0140x larger
### spitfire ###
Min: 1.160000 -> 1.140000: 1.0175x faster
Avg: 1.194200 -> 1.181400: 1.0108x faster
Significant (t=2.040372, a=0.95)
Stddev: 0.03302 -> 0.02962: 1.1145x smaller
### spitfire_cstringio ###
Min: 0.400000 -> 0.390000: 1.0256x faster
Avg: 0.426800 -> 0.410000: 1.0410x faster
Significant (t=3.168258, a=0.95)
Stddev: 0.02691 -> 0.02611: 1.0308x smaller
### sympy_expand ###
Min: 0.206214 -> 0.240614: 1.1668x slower
Avg: 0.316202 -> 0.333418: 1.0544x slower
Not significant
Stddev: 0.19306 -> 0.19010: 1.0156x smaller
### sympy_integrate ###
Min: 0.801874 -> 0.781623: 1.0259x faster
Avg: 1.271010 -> 1.176734: 1.0801x faster
Not significant
Stddev: 0.67044 -> 0.64535: 1.0389x smaller
### sympy_str ###
Min: 0.156287 -> 0.160201: 1.0250x slower
Avg: 0.303686 -> 0.326078: 1.0737x slower
Not significant
Stddev: 0.20273 -> 0.22414: 1.1056x larger
### sympy_sum ###
Min: 0.198554 -> 0.201798: 1.0163x slower
Avg: 0.290399 -> 0.298513: 1.0279x slower
Not significant
Stddev: 0.12515 -> 0.13811: 1.1036x larger
### telco ###
Min: 0.008000 -> 0.020001: 2.5001x slower
Avg: 0.014241 -> 0.025362: 1.7809x slower
Significant (t=-6.447078, a=0.95)
Stddev: 0.00943 -> 0.00774: 1.2192x smaller
### trans2_annotate ###
Raw results: [303.2] None
### trans2_rtype ###
Raw results: [612.2] None
### trans2_backendopt ###
Raw results: [83.8] None
### trans2_database ###
Raw results: [115.0] None
### trans2_source ###
Raw results: [120.7] None
### twisted_iteration ###
Min: 0.002040 -> 0.002048: 1.0040x slower
Avg: 0.002054 -> 0.002068: 1.0066x slower
Significant (t=-6.838156, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.0082x smaller
### twisted_names ###
Min: 0.000737 -> 0.000749: 1.0166x slower
Avg: 0.000747 -> 0.000759: 1.0148x slower
Significant (t=-8.378163, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.2645x larger
### twisted_pb ###
Min: 0.003442 -> 0.003522: 1.0232x slower
Avg: 0.003739 -> 0.003842: 1.0276x slower
Significant (t=-3.155522, a=0.95)
Stddev: 0.00016 -> 0.00016: 1.0193x smaller
### twisted_tcp ###
Min: 0.081666 -> 0.089769: 1.0992x slower
Avg: 0.083214 -> 0.090795: 1.0911x slower
Significant (t=-31.579612, a=0.95)
Stddev: 0.00152 -> 0.00076: 1.9919x smaller
### ai ###
Min: 0.024530 -> 0.023645: 1.0374x faster
Avg: 0.024767 -> 0.023963: 1.0336x faster
Significant (t=11.078261, a=0.95)
Stddev: 0.00033 -> 0.00039: 1.2038x larger
### bm_chameleon ###
Min: 0.007709 -> 0.009386: 1.2175x slower
Avg: 0.010223 -> 0.011915: 1.1655x slower
Significant (t=-3.516124, a=0.95)
Stddev: 0.00424 -> 0.00410: 1.0344x smaller
### bm_dulwich_log ###
Min: 0.111125 -> 0.107174: 1.0369x faster
Avg: 0.151440 -> 0.145219: 1.0428x faster
Not significant
Stddev: 0.11254 -> 0.11114: 1.0125x smaller
### bm_mako ###
Min: 0.009072 -> 0.008993: 1.0088x faster
Avg: 0.011201 -> 0.011180: 1.0019x faster
Not significant
Stddev: 0.00366 -> 0.00364: 1.0047x smaller
### chaos ###
Min: 0.003846 -> 0.004008: 1.0421x slower
Avg: 0.005232 -> 0.005390: 1.0303x slower
Not significant
Stddev: 0.00900 -> 0.00896: 1.0047x smaller
### sphinx ###
Raw results: [78.2049300671] [76.9637699127]
### crypto_pyaes ###
Min: 0.025174 -> 0.024822: 1.0142x faster
Avg: 0.027979 -> 0.027446: 1.0194x faster
Not significant
Stddev: 0.01029 -> 0.01014: 1.0148x smaller
### deltablue ###
Min: 0.000685 -> 0.000915: 1.3355x slower
Avg: 0.006643 -> 0.006407: 1.0369x faster
Not significant
Stddev: 0.00920 -> 0.00861: 1.0692x smaller
### django ###
Min: 0.016882 -> 0.015822: 1.0670x faster
Avg: 0.018295 -> 0.017131: 1.0679x faster
Significant (t=2.678836, a=0.95)
Stddev: 0.00218 -> 0.00217: 1.0056x smaller
### eparse ###
Min: 0.178579 -> 0.182563: 1.0223x slower
Avg: 0.227297 -> 0.224844: 1.0109x faster
Not significant
Stddev: 0.04975 -> 0.04936: 1.0078x smaller
### fannkuch ###
Min: 0.092023 -> 0.092981: 1.0104x slower
Avg: 0.093026 -> 0.093839: 1.0087x slower
Not significant
Stddev: 0.00459 -> 0.00436: 1.0523x smaller
### float ###
Min: 0.016049 -> 0.016316: 1.0166x slower
Avg: 0.023619 -> 0.023923: 1.0129x slower
Not significant
Stddev: 0.00619 -> 0.00619: 1.0006x larger
### genshi_text ###
Min: 0.007739 -> 0.007923: 1.0238x slower
Avg: 0.009736 -> 0.009785: 1.0050x slower
Not significant
Stddev: 0.00889 -> 0.00759: 1.1709x smaller
### genshi_xml ###
Min: 0.022961 -> 0.023157: 1.0085x slower
Avg: 0.026239 -> 0.026411: 1.0066x slower
Not significant
Stddev: 0.01519 -> 0.01359: 1.1178x smaller
### go ###
Min: 0.053826 -> 0.044093: 1.2207x faster
Avg: 0.082227 -> 0.055605: 1.4788x faster
Significant (t=6.157912, a=0.95)
Stddev: 0.02767 -> 0.01299: 2.1303x smaller
### hexiom2 ###
Min: 5.772802 -> 6.185860: 1.0716x slower
Avg: 6.008467 -> 6.430217: 1.0702x slower
Significant (t=-8.696341, a=0.95)
Stddev: 0.23806 -> 0.24684: 1.0369x larger
### html5lib ###
Min: 1.276636 -> 1.201707: 1.0624x faster
Avg: 1.742303 -> 1.666700: 1.0454x faster
Not significant
Stddev: 0.57257 -> 0.54649: 1.0477x smaller
### json_bench ###
Min: 0.403071 -> 0.405566: 1.0062x slower
Avg: 0.407013 -> 0.410124: 1.0076x slower
Not significant
Stddev: 0.01099 -> 0.01134: 1.0322x larger
### meteor-contest ###
Min: 0.059418 -> 0.058484: 1.0160x faster
Avg: 0.060614 -> 0.059640: 1.0163x faster
Not significant
Stddev: 0.00318 -> 0.00324: 1.0179x larger
### nbody_modified ###
Min: 0.018613 -> 0.018432: 1.0098x faster
Avg: 0.019143 -> 0.018961: 1.0096x faster
Not significant
Stddev: 0.00203 -> 0.00202: 1.0075x smaller
### pidigits ###
Min: 5.398082 -> 5.244479: 1.0293x faster
Avg: 5.421811 -> 5.267033: 1.0294x faster
Significant (t=5.504659, a=0.95)
Stddev: 0.04472 -> 0.04419: 1.0119x smaller
### pyflate-fast ###
Min: 0.185665 -> 0.213436: 1.1496x slower
Avg: 0.191272 -> 0.215193: 1.1251x slower
Significant (t=-25.983353, a=0.95)
Stddev: 0.00577 -> 0.00302: 1.9133x smaller
### raytrace-simple ###
Min: 0.018075 -> 0.050743: 2.8074x slower
Avg: 0.018910 -> 0.051445: 2.7206x slower
Significant (t=-124.054463, a=0.95)
Stddev: 0.00120 -> 0.00142: 1.1829x larger
### richards ###
Min: 0.001887 -> 0.001873: 1.0074x faster
Avg: 0.002089 -> 0.002055: 1.0167x faster
Not significant
Stddev: 0.00044 -> 0.00044: 1.0147x smaller
### rietveld ###
Min: 0.043351 -> 0.044205: 1.0197x slower
Avg: 0.114839 -> 0.114658: 1.0016x faster
Not significant
Stddev: 0.09476 -> 0.09160: 1.0345x smaller
### scimark_fft ###
0.134479 -> -1.000000: -1
### scimark_lu ###
0.305992 -> -1.000000: -1
### scimark_montecarlo ###
0.139750 -> -1.000000: -1
### scimark_sor ###
0.225868 -> -1.000000: -1
### scimark_sparsematmult ###
0.147379 -> -1.000000: -1
### slowspitfire ###
Min: 0.151334 -> 0.152800: 1.0097x slower
Avg: 0.161073 -> 0.163202: 1.0132x slower
Not significant
Stddev: 0.00620 -> 0.00555: 1.1162x smaller
### spambayes ###
Min: 0.027756 -> 0.027993: 1.0085x slower
Avg: 0.048300 -> 0.046956: 1.0286x faster
Not significant
Stddev: 0.01600 -> 0.01517: 1.0543x smaller
### spectral-norm ###
Min: 0.009358 -> 0.009377: 1.0020x slower
Avg: 0.010026 -> 0.010028: 1.0002x slower
Not significant
Stddev: 0.00287 -> 0.00284: 1.0110x smaller
### spitfire ###
Min: 1.160000 -> 1.140000: 1.0175x faster
Avg: 1.198400 -> 1.178200: 1.0171x faster
Significant (t=3.149072, a=0.95)
Stddev: 0.03377 -> 0.03028: 1.1150x smaller
### spitfire_cstringio ###
Min: 0.490000 -> 0.480000: 1.0208x faster
Avg: 0.511000 -> 0.499800: 1.0224x faster
Significant (t=2.213100, a=0.95)
Stddev: 0.02435 -> 0.02622: 1.0770x larger
### sympy_expand ###
Min: 0.216114 -> 0.226401: 1.0476x slower
Avg: 0.325487 -> 0.327479: 1.0061x slower
Not significant
Stddev: 0.20156 -> 0.19519: 1.0326x smaller
### sympy_integrate ###
Min: 0.863688 -> 0.724006: 1.1929x faster
Avg: 1.298924 -> 1.175341: 1.1051x faster
Not significant
Stddev: 0.71017 -> 0.70017: 1.0143x smaller
### sympy_str ###
Min: 0.154443 -> 0.158522: 1.0264x slower
Avg: 0.302207 -> 0.309182: 1.0231x slower
Not significant
Stddev: 0.20272 -> 0.21334: 1.0524x larger
### sympy_sum ###
Min: 0.206235 -> 0.214458: 1.0399x slower
Avg: 0.291689 -> 0.296673: 1.0171x slower
Not significant
Stddev: 0.11763 -> 0.12348: 1.0497x larger
### telco ###
Min: 0.008000 -> 0.016001: 2.0001x slower
Avg: 0.014161 -> 0.021841: 1.5424x slower
Significant (t=-4.700125, a=0.95)
Stddev: 0.00908 -> 0.00715: 1.2703x smaller
### trans2_annotate ###
Raw results: [296.3] None
### trans2_rtype ###
Raw results: [593.8] None
### trans2_backendopt ###
Raw results: [81.8] None
### trans2_database ###
Raw results: [109.5] None
### trans2_source ###
Raw results: [118.0] None
### twisted_iteration ###
Min: 0.002045 -> 0.002054: 1.0042x slower
Avg: 0.002061 -> 0.002071: 1.0048x slower
Significant (t=-4.479936, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.1017x smaller
### twisted_names ###
Min: 0.000726 -> 0.000764: 1.0514x slower
Avg: 0.000735 -> 0.000776: 1.0552x slower
Significant (t=-31.900206, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.1643x larger
### twisted_pb ###
Min: 0.003398 -> 0.003665: 1.0786x slower
Avg: 0.003714 -> 0.003966: 1.0679x slower
Significant (t=-6.633568, a=0.95)
Stddev: 0.00022 -> 0.00016: 1.3333x smaller
### twisted_tcp ###
Min: 0.082466 -> 0.082349: 1.0014x faster
Avg: 0.084178 -> 0.083816: 1.0043x faster
Not significant
Stddev: 0.00170 -> 0.00145: 1.1724x smaller
### ai ###
Min: 0.023883 -> 0.023692: 1.0081x faster
Avg: 0.024102 -> 0.024008: 1.0039x faster
Not significant
Stddev: 0.00034 -> 0.00033: 1.0268x smaller
### bm_chameleon ###
Min: 0.007408 -> 0.009028: 1.2187x slower
Avg: 0.009830 -> 0.012773: 1.2994x slower
Significant (t=-6.177826, a=0.95)
Stddev: 0.00419 -> 0.00406: 1.0334x smaller
### bm_dulwich_log ###
Min: 0.103286 -> 0.100674: 1.0259x faster
Avg: 0.142535 -> 0.139344: 1.0229x faster
Not significant
Stddev: 0.11222 -> 0.11305: 1.0074x larger
### bm_mako ###
Min: 0.009023 -> 0.010273: 1.1385x slower
Avg: 0.011059 -> 0.012495: 1.1298x slower
Significant (t=-2.013881, a=0.95)
Stddev: 0.00354 -> 0.00359: 1.0165x larger
### chaos ###
Min: 0.003792 -> 0.003755: 1.0099x faster
Avg: 0.005160 -> 0.005149: 1.0021x faster
Not significant
Stddev: 0.00889 -> 0.00897: 1.0090x larger
### sphinx ###
Raw results: [77.5520658493] [78.3562920094]
### crypto_pyaes ###
Min: 0.024972 -> 0.024917: 1.0022x faster
Avg: 0.027624 -> 0.027582: 1.0015x faster
Not significant
Stddev: 0.01015 -> 0.01019: 1.0041x larger
### deltablue ###
Min: 0.000656 -> 0.000906: 1.3808x slower
Avg: 0.006466 -> 0.006557: 1.0140x slower
Not significant
Stddev: 0.00901 -> 0.00845: 1.0656x smaller
### django ###
Min: 0.015847 -> 0.015596: 1.0161x faster
Avg: 0.017174 -> 0.016892: 1.0167x faster
Not significant
Stddev: 0.00218 -> 0.00218: 1.0006x smaller
### eparse ###
Min: 0.176189 -> 0.182437: 1.0355x slower
Avg: 0.222830 -> 0.229030: 1.0278x slower
Not significant
Stddev: 0.04900 -> 0.04850: 1.0103x smaller
### fannkuch ###
Min: 0.091812 -> 0.092155: 1.0037x slower
Avg: 0.092707 -> 0.093207: 1.0054x slower
Not significant
Stddev: 0.00437 -> 0.00439: 1.0044x larger
### float ###
Min: 0.015223 -> 0.015089: 1.0089x faster
Avg: 0.022767 -> 0.022708: 1.0026x faster
Not significant
Stddev: 0.00609 -> 0.00617: 1.0140x larger
### genshi_text ###
Min: 0.007651 -> 0.007503: 1.0197x faster
Avg: 0.009658 -> 0.009628: 1.0031x faster
Not significant
Stddev: 0.00878 -> 0.00913: 1.0408x larger
### genshi_xml ###
Min: 0.022101 -> 0.024660: 1.1158x slower
Avg: 0.025370 -> 0.028154: 1.1097x slower
Not significant
Stddev: 0.01471 -> 0.01500: 1.0199x larger
### go ###
Min: 0.054293 -> 0.044338: 1.2245x faster
Avg: 0.082098 -> 0.062413: 1.3154x faster
Significant (t=3.986552, a=0.95)
Stddev: 0.02746 -> 0.02157: 1.2734x smaller
### hexiom2 ###
Min: 5.799802 -> 5.791338: 1.0015x faster
Avg: 6.026409 -> 6.035714: 1.0015x slower
Not significant
Stddev: 0.22891 -> 0.24686: 1.0784x larger
### html5lib ###
Min: 1.244801 -> 1.226899: 1.0146x faster
Avg: 1.704128 -> 1.653622: 1.0305x faster
Not significant
Stddev: 0.57008 -> 0.54916: 1.0381x smaller
### json_bench ###
Min: 0.402525 -> 0.426142: 1.0587x slower
Avg: 0.408857 -> 0.430575: 1.0531x slower
Significant (t=-9.744694, a=0.95)
Stddev: 0.01106 -> 0.01123: 1.0153x larger
### meteor-contest ###
Min: 0.059208 -> 0.059024: 1.0031x faster
Avg: 0.060232 -> 0.060214: 1.0003x faster
Not significant
Stddev: 0.00318 -> 0.00323: 1.0186x larger
### nbody_modified ###
Min: 0.018770 -> 0.018680: 1.0048x faster
Avg: 0.019336 -> 0.019231: 1.0054x faster
Not significant
Stddev: 0.00203 -> 0.00202: 1.0055x smaller
### pidigits ###
Min: 5.358582 -> 4.985167: 1.0749x faster
Avg: 5.380902 -> 5.005724: 1.0749x faster
Significant (t=14.099518, a=0.95)
Stddev: 0.04282 -> 0.04131: 1.0364x smaller
### pyflate-fast ###
Min: 0.186429 -> 0.213897: 1.1473x slower
Avg: 0.188376 -> 0.215966: 1.1465x slower
Significant (t=-55.643026, a=0.95)
Stddev: 0.00280 -> 0.00211: 1.3304x smaller
### raytrace-simple ###
Min: 0.017904 -> 0.024924: 1.3921x slower
Avg: 0.018745 -> 0.025592: 1.3652x slower
Significant (t=-34.649821, a=0.95)
Stddev: 0.00120 -> 0.00071: 1.6899x smaller
### richards ###
Min: 0.001859 -> 0.001879: 1.0108x slower
Avg: 0.002026 -> 0.002007: 1.0096x faster
Not significant
Stddev: 0.00043 -> 0.00043: 1.0134x larger
### rietveld ###
Min: 0.042592 -> 0.045415: 1.0663x slower
Avg: 0.112874 -> 0.115784: 1.0258x slower
Not significant
Stddev: 0.09324 -> 0.08764: 1.0640x smaller
### scimark_fft ###
0.136233 -> -1.000000: -1
### scimark_lu ###
0.305460 -> -1.000000: -1
### scimark_montecarlo ###
0.141590 -> -1.000000: -1
### scimark_sor ###
0.225836 -> -1.000000: -1
### scimark_sparsematmult ###
0.149314 -> -1.000000: -1
### slowspitfire ###
Min: 0.153071 -> 0.152174: 1.0059x faster
Avg: 0.163220 -> 0.166867: 1.0223x slower
Significant (t=-2.677061, a=0.95)
Stddev: 0.00532 -> 0.00803: 1.5087x larger
### spambayes ###
Min: 0.028488 -> 0.027426: 1.0387x faster
Avg: 0.052744 -> 0.049873: 1.0576x faster
Not significant
Stddev: 0.01536 -> 0.01530: 1.0045x smaller
### spectral-norm ###
Min: 0.009338 -> 0.009289: 1.0053x faster
Avg: 0.010012 -> 0.009947: 1.0064x faster
Not significant
Stddev: 0.00286 -> 0.00286: 1.0013x larger
### spitfire ###
Min: 1.150000 -> 1.150000: no change
Avg: 1.187400 -> 1.190200: 1.0024x slower
Not significant
Stddev: 0.03337 -> 0.03204: 1.0415x smaller
### spitfire_cstringio ###
Min: 0.540000 -> 0.520000: 1.0385x faster
Avg: 0.559400 -> 0.544800: 1.0268x faster
Significant (t=2.810196, a=0.95)
Stddev: 0.02583 -> 0.02613: 1.0117x larger
### sympy_expand ###
Min: 0.224202 -> 0.214263: 1.0464x faster
Avg: 0.315703 -> 0.322553: 1.0217x slower
Not significant
Stddev: 0.19840 -> 0.20339: 1.0252x larger
### sympy_integrate ###
Min: 0.835360 -> 0.855213: 1.0238x slower
Avg: 1.265200 -> 1.301990: 1.0291x slower
Not significant
Stddev: 0.69199 -> 0.68222: 1.0143x smaller
### sympy_str ###
Min: 0.158076 -> 0.166441: 1.0529x slower
Avg: 0.306286 -> 0.315436: 1.0299x slower
Not significant
Stddev: 0.20398 -> 0.21630: 1.0604x larger
### sympy_sum ###
Min: 0.199477 -> 0.211708: 1.0613x slower
Avg: 0.292236 -> 0.296414: 1.0143x slower
Not significant
Stddev: 0.12355 -> 0.12324: 1.0025x smaller
### telco ###
Min: 0.008000 -> 0.008000: no change
Avg: 0.013441 -> 0.013441: no change
Not significant
Stddev: 0.00931 -> 0.00899: 1.0357x smaller
### trans2_annotate ###
Raw results: [297.3] None
### trans2_rtype ###
Raw results: [605.0] None
### trans2_backendopt ###
Raw results: [83.1] None
### trans2_database ###
Raw results: [112.9] None
### trans2_source ###
Raw results: [116.6] None
### twisted_iteration ###
Min: 0.002048 -> 0.002089: 1.0203x slower
Avg: 0.002075 -> 0.002115: 1.0194x slower
Significant (t=-12.246309, a=0.95)
Stddev: 0.00002 -> 0.00001: 1.1937x smaller
### twisted_names ###
Min: 0.000725 -> 0.000708: 1.0247x faster
Avg: 0.000735 -> 0.000718: 1.0237x faster
Significant (t=13.454979, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.1215x larger
### twisted_pb ###
Min: 0.003328 -> 0.003357: 1.0086x slower
Avg: 0.003669 -> 0.003640: 1.0080x faster
Not significant
Stddev: 0.00019 -> 0.00016: 1.1619x smaller
### twisted_tcp ###
Min: 0.081791 -> 0.083739: 1.0238x slower
Avg: 0.083348 -> 0.087015: 1.0440x slower
Significant (t=-9.432635, a=0.95)
Stddev: 0.00162 -> 0.00222: 1.3698x larger
### ai ###
Min: 0.023917 -> 0.024226: 1.0129x slower
Avg: 0.024064 -> 0.024386: 1.0134x slower
Significant (t=-5.215923, a=0.95)
Stddev: 0.00032 -> 0.00029: 1.0885x smaller
### bm_chameleon ###
Min: 0.007497 -> 0.007378: 1.0161x faster
Avg: 0.010038 -> 0.009876: 1.0164x faster
Not significant
Stddev: 0.00415 -> 0.00415: 1.0009x larger
### bm_dulwich_log ###
Min: 0.105978 -> 0.109689: 1.0350x slower
Avg: 0.144271 -> 0.147769: 1.0242x slower
Not significant
Stddev: 0.11168 -> 0.11125: 1.0039x smaller
### bm_mako ###
Min: 0.008896 -> 0.008730: 1.0190x faster
Avg: 0.011094 -> 0.010903: 1.0175x faster
Not significant
Stddev: 0.00372 -> 0.00365: 1.0193x smaller
### chaos ###
Min: 0.003693 -> 0.003763: 1.0190x slower
Avg: 0.005095 -> 0.005169: 1.0146x slower
Not significant
Stddev: 0.00900 -> 0.00910: 1.0113x larger
### sphinx ###
Raw results: [78.4002809525] [77.5027208328]
### crypto_pyaes ###
Min: 0.025048 -> 0.024925: 1.0049x faster
Avg: 0.027692 -> 0.027490: 1.0073x faster
Not significant
Stddev: 0.01008 -> 0.01002: 1.0055x smaller
### deltablue ###
Min: 0.000682 -> 0.000665: 1.0255x faster
Avg: 0.006535 -> 0.006583: 1.0073x slower
Not significant
Stddev: 0.00902 -> 0.00912: 1.0106x larger
### django ###
Min: 0.016845 -> 0.015642: 1.0769x faster
Avg: 0.018176 -> 0.016943: 1.0728x faster
Significant (t=2.813002, a=0.95)
Stddev: 0.00225 -> 0.00214: 1.0519x smaller
### eparse ###
Min: 0.186282 -> 0.188626: 1.0126x slower
Avg: 0.227064 -> 0.228694: 1.0072x slower
Not significant
Stddev: 0.04855 -> 0.04853: 1.0004x smaller
### fannkuch ###
Min: 0.092230 -> 0.093653: 1.0154x slower
Avg: 0.093131 -> 0.094608: 1.0159x slower
Not significant
Stddev: 0.00437 -> 0.00439: 1.0028x larger
### float ###
Min: 0.015930 -> 0.015764: 1.0105x faster
Avg: 0.023582 -> 0.023495: 1.0037x faster
Not significant
Stddev: 0.00617 -> 0.00618: 1.0022x larger
### genshi_text ###
Min: 0.007442 -> 0.007406: 1.0049x faster
Avg: 0.009456 -> 0.009357: 1.0106x faster
Not significant
Stddev: 0.00878 -> 0.00890: 1.0131x larger
### genshi_xml ###
Min: 0.022542 -> 0.023450: 1.0403x slower
Avg: 0.025766 -> 0.026843: 1.0418x slower
Not significant
Stddev: 0.01495 -> 0.01482: 1.0089x smaller
### go ###
Min: 0.056117 -> 0.049464: 1.1345x faster
Avg: 0.080922 -> 0.068242: 1.1858x faster
Significant (t=2.939745, a=0.95)
Stddev: 0.02612 -> 0.01576: 1.6576x smaller
### hexiom2 ###
Min: 5.785369 -> 5.858442: 1.0126x slower
Avg: 6.014472 -> 6.099259: 1.0141x slower
Not significant
Stddev: 0.23143 -> 0.24326: 1.0511x larger
### html5lib ###
Min: 1.237429 -> 1.285687: 1.0390x slower
Avg: 1.653288 -> 1.719167: 1.0398x slower
Not significant
Stddev: 0.53921 -> 0.54721: 1.0148x larger
### json_bench ###
Min: 0.402597 -> 0.399106: 1.0087x faster
Avg: 0.408401 -> 0.403247: 1.0128x faster
Significant (t=2.262900, a=0.95)
Stddev: 0.01175 -> 0.01102: 1.0665x smaller
### meteor-contest ###
Min: 0.059306 -> 0.059382: 1.0013x slower
Avg: 0.060412 -> 0.060674: 1.0043x slower
Not significant
Stddev: 0.00317 -> 0.00324: 1.0238x larger
### nbody_modified ###
Min: 0.018762 -> 0.018587: 1.0094x faster
Avg: 0.019270 -> 0.019129: 1.0074x faster
Not significant
Stddev: 0.00200 -> 0.00203: 1.0125x larger
### pidigits ###
Min: 5.394168 -> 5.105118: 1.0566x faster
Avg: 5.415438 -> 5.125833: 1.0565x faster
Significant (t=11.121185, a=0.95)
Stddev: 0.04079 -> 0.04155: 1.0186x larger
### pyflate-fast ###
Min: 0.187713 -> 0.187266: 1.0024x faster
Avg: 0.189677 -> 0.189938: 1.0014x slower
Not significant
Stddev: 0.00297 -> 0.00309: 1.0401x larger
### raytrace-simple ###
Min: 0.018063 -> 0.018088: 1.0014x slower
Avg: 0.018894 -> 0.018949: 1.0029x slower
Not significant
Stddev: 0.00125 -> 0.00115: 1.0834x smaller
### richards ###
Min: 0.001877 -> 0.001877: no change
Avg: 0.002013 -> 0.002014: 1.0007x slower
Not significant
Stddev: 0.00046 -> 0.00045: 1.0394x smaller
### rietveld ###
Min: 0.042445 -> 0.042944: 1.0118x slower
Avg: 0.114897 -> 0.114120: 1.0068x faster
Not significant
Stddev: 0.09465 -> 0.09446: 1.0021x smaller
### scimark_fft ###
0.133715 -> -1.000000: -1
### scimark_lu ###
0.300824 -> -1.000000: -1
### scimark_montecarlo ###
0.140648 -> -1.000000: -1
### scimark_sor ###
0.225912 -> -1.000000: -1
### scimark_sparsematmult ###
0.149313 -> -1.000000: -1
### slowspitfire ###
Min: 0.151268 -> 0.151540: 1.0018x slower
Avg: 0.161153 -> 0.160559: 1.0037x faster
Not significant
Stddev: 0.00626 -> 0.00589: 1.0629x smaller
### spambayes ###
Min: 0.028285 -> 0.027390: 1.0327x faster
Avg: 0.052711 -> 0.048411: 1.0888x faster
Not significant
Stddev: 0.01551 -> 0.01610: 1.0378x larger
### spectral-norm ###
Min: 0.009358 -> 0.009379: 1.0022x slower
Avg: 0.010043 -> 0.010044: 1.0001x slower
Not significant
Stddev: 0.00287 -> 0.00288: 1.0033x larger
### spitfire ###
Min: 1.140000 -> 1.170000: 1.0263x slower
Avg: 1.182600 -> 1.204400: 1.0184x slower
Significant (t=-3.422626, a=0.95)
Stddev: 0.03256 -> 0.03111: 1.0467x smaller
### spitfire_cstringio ###
Min: 0.400000 -> 0.490000: 1.2250x slower
Avg: 0.421600 -> 0.514600: 1.2206x slower
Significant (t=-19.295518, a=0.95)
Stddev: 0.02333 -> 0.02484: 1.0648x larger
### sympy_expand ###
Min: 0.213237 -> 0.214561: 1.0062x slower
Avg: 0.324671 -> 0.326383: 1.0053x slower
Not significant
Stddev: 0.20330 -> 0.20373: 1.0021x larger
### sympy_integrate ###
Min: 0.861252 -> 0.771418: 1.1165x faster
Avg: 1.295300 -> 1.231274: 1.0520x faster
Not significant
Stddev: 0.70704 -> 0.70340: 1.0052x smaller
### sympy_str ###
Min: 0.157927 -> 0.160802: 1.0182x slower
Avg: 0.305096 -> 0.305706: 1.0020x slower
Not significant
Stddev: 0.20053 -> 0.20028: 1.0012x smaller
### sympy_sum ###
Min: 0.206980 -> 0.211585: 1.0222x slower
Avg: 0.293323 -> 0.297996: 1.0159x slower
Not significant
Stddev: 0.12204 -> 0.12069: 1.0112x smaller
### telco ###
Min: 0.008000 -> 0.008000: no change
Avg: 0.013761 -> 0.014081: 1.0233x slower
Not significant
Stddev: 0.00897 -> 0.00901: 1.0044x larger
### trans2_annotate ###
Raw results: [285.6] None
### trans2_rtype ###
Raw results: [632.3] None
### trans2_backendopt ###
Raw results: [82.2] None
### trans2_database ###
Raw results: [114.2] None
### trans2_source ###
Raw results: [121.6] None
### twisted_iteration ###
Min: 0.002044 -> 0.002047: 1.0013x slower
Avg: 0.002063 -> 0.002064: 1.0006x slower
Not significant
Stddev: 0.00001 -> 0.00001: 1.1070x larger
### twisted_names ###
Min: 0.000731 -> 0.000719: 1.0171x faster
Avg: 0.000741 -> 0.000731: 1.0136x faster
Significant (t=7.793531, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.0432x smaller
### twisted_pb ###
Min: 0.003384 -> 0.003489: 1.0309x slower
Avg: 0.003645 -> 0.003819: 1.0477x slower
Significant (t=-5.022045, a=0.95)
Stddev: 0.00015 -> 0.00020: 1.3186x larger
### twisted_tcp ###
Min: 0.085805 -> 0.082353: 1.0419x faster
Avg: 0.087671 -> 0.083905: 1.0449x faster
Significant (t=11.604523, a=0.95)
Stddev: 0.00163 -> 0.00162: 1.0033x smaller
### ai ###
Min: 0.023829 -> 0.023632: 1.0083x faster
Avg: 0.024003 -> 0.023850: 1.0064x faster
Significant (t=2.400409, a=0.95)
Stddev: 0.00031 -> 0.00033: 1.0729x larger
### bm_chameleon ###
Min: 0.007460 -> 0.007549: 1.0119x slower
Avg: 0.009940 -> 0.010032: 1.0093x slower
Not significant
Stddev: 0.00414 -> 0.00417: 1.0057x larger
### bm_dulwich_log ###
Min: 0.108187 -> 0.105768: 1.0229x faster
Avg: 0.146484 -> 0.144371: 1.0146x faster
Not significant
Stddev: 0.11170 -> 0.11066: 1.0094x smaller
### bm_mako ###
Min: 0.008846 -> 0.008903: 1.0064x slower
Avg: 0.010970 -> 0.011081: 1.0101x slower
Not significant
Stddev: 0.00357 -> 0.00355: 1.0058x smaller
### chaos ###
Min: 0.003697 -> 0.003737: 1.0109x slower
Avg: 0.005094 -> 0.005151: 1.0111x slower
Not significant
Stddev: 0.00888 -> 0.00911: 1.0260x larger
### sphinx ###
Raw results: [78.9604752064] [74.3792219162]
### crypto_pyaes ###
Min: 0.024770 -> 0.024693: 1.0031x faster
Avg: 0.027501 -> 0.027404: 1.0035x faster
Not significant
Stddev: 0.01004 -> 0.01013: 1.0089x larger
### deltablue ###
Min: 0.000669 -> 0.000681: 1.0178x slower
Avg: 0.006649 -> 0.006617: 1.0049x faster
Not significant
Stddev: 0.00922 -> 0.00917: 1.0050x smaller
### django ###
Min: 0.015834 -> 0.015616: 1.0140x faster
Avg: 0.017296 -> 0.017001: 1.0173x faster
Not significant
Stddev: 0.00230 -> 0.00224: 1.0289x smaller
### eparse ###
Min: 0.181033 -> 0.176031: 1.0284x faster
Avg: 0.224494 -> 0.223065: 1.0064x faster
Not significant
Stddev: 0.04944 -> 0.05042: 1.0197x larger
### fannkuch ###
Min: 0.092375 -> 0.093036: 1.0072x slower
Avg: 0.093316 -> 0.093966: 1.0070x slower
Not significant
Stddev: 0.00436 -> 0.00446: 1.0234x larger
### float ###
Min: 0.015118 -> 0.015959: 1.0556x slower
Avg: 0.022721 -> 0.023674: 1.0419x slower
Not significant
Stddev: 0.00612 -> 0.00622: 1.0162x larger
### genshi_text ###
Min: 0.007559 -> 0.007537: 1.0029x faster
Avg: 0.009615 -> 0.009507: 1.0114x faster
Not significant
Stddev: 0.00890 -> 0.00887: 1.0035x smaller
### genshi_xml ###
Min: 0.023032 -> 0.023132: 1.0043x slower
Avg: 0.026363 -> 0.026608: 1.0093x slower
Not significant
Stddev: 0.01486 -> 0.01503: 1.0115x larger
### go ###
Min: 0.054099 -> 0.047412: 1.1410x faster
Avg: 0.083018 -> 0.073164: 1.1347x faster
Not significant
Stddev: 0.02759 -> 0.02562: 1.0772x smaller
### hexiom2 ###
Min: 5.791835 -> 5.827246: 1.0061x slower
Avg: 6.022151 -> 6.058549: 1.0060x slower
Not significant
Stddev: 0.23265 -> 0.23365: 1.0043x larger
### html5lib ###
Min: 1.223819 -> 1.200750: 1.0192x faster
Avg: 1.638997 -> 1.682524: 1.0266x slower
Not significant
Stddev: 0.53759 -> 0.57438: 1.0684x larger
### json_bench ###
Min: 0.404723 -> 0.404303: 1.0010x faster
Avg: 0.409006 -> 0.408657: 1.0009x faster
Not significant
Stddev: 0.01094 -> 0.01136: 1.0388x larger
### meteor-contest ###
Min: 0.059269 -> 0.058557: 1.0122x faster
Avg: 0.060447 -> 0.059726: 1.0121x faster
Not significant
Stddev: 0.00318 -> 0.00324: 1.0200x larger
### nbody_modified ###
Min: 0.018903 -> 0.018290: 1.0335x faster
Avg: 0.019407 -> 0.018800: 1.0323x faster
Not significant
Stddev: 0.00200 -> 0.00203: 1.0171x larger
### pidigits ###
Min: 5.362328 -> 5.104557: 1.0505x faster
Avg: 5.384472 -> 5.125910: 1.0504x faster
Significant (t=9.835787, a=0.95)
Stddev: 0.04288 -> 0.04021: 1.0663x smaller
### pyflate-fast ###
Min: 0.187195 -> 0.187777: 1.0031x slower
Avg: 0.189842 -> 0.190292: 1.0024x slower
Not significant
Stddev: 0.00232 -> 0.00282: 1.2171x larger
### raytrace-simple ###
Min: 0.018084 -> 0.018125: 1.0023x slower
Avg: 0.019183 -> 0.018926: 1.0136x faster
Not significant
Stddev: 0.00203 -> 0.00114: 1.7810x smaller
### richards ###
Min: 0.001875 -> 0.001858: 1.0091x faster
Avg: 0.002055 -> 0.001983: 1.0366x faster
Not significant
Stddev: 0.00042 -> 0.00044: 1.0505x larger
### rietveld ###
Min: 0.043779 -> 0.042596: 1.0278x faster
Avg: 0.116975 -> 0.113526: 1.0304x faster
Not significant
Stddev: 0.09618 -> 0.09382: 1.0252x smaller
### scimark_fft ###
0.133822 -> -1.000000: -1
### scimark_lu ###
0.302480 -> -1.000000: -1
### scimark_montecarlo ###
0.141413 -> -1.000000: -1
### scimark_sor ###
0.226131 -> -1.000000: -1
### scimark_sparsematmult ###
0.147616 -> -1.000000: -1
### slowspitfire ###
Min: 0.151880 -> 0.151957: 1.0005x slower
Avg: 0.162099 -> 0.162365: 1.0016x slower
Not significant
Stddev: 0.00540 -> 0.00552: 1.0231x larger
### spambayes ###
Min: 0.027543 -> 0.027913: 1.0134x slower
Avg: 0.046907 -> 0.047533: 1.0134x slower
Not significant
Stddev: 0.01568 -> 0.01523: 1.0298x smaller
### spectral-norm ###
Min: 0.009342 -> 0.009278: 1.0069x faster
Avg: 0.010035 -> 0.009935: 1.0100x faster
Not significant
Stddev: 0.00284 -> 0.00288: 1.0136x larger
### spitfire ###
Min: 1.140000 -> 1.120000: 1.0179x faster
Avg: 1.174400 -> 1.161200: 1.0114x faster
Significant (t=2.175956, a=0.95)
Stddev: 0.03011 -> 0.03055: 1.0145x larger
### spitfire_cstringio ###
Min: 0.420000 -> 0.390000: 1.0769x faster
Avg: 0.438600 -> 0.412600: 1.0630x faster
Significant (t=5.469437, a=0.95)
Stddev: 0.02339 -> 0.02414: 1.0323x larger
### sympy_expand ###
Min: 0.212689 -> 0.214927: 1.0105x slower
Avg: 0.324190 -> 0.322393: 1.0056x faster
Not significant
Stddev: 0.20323 -> 0.19896: 1.0215x smaller
### sympy_integrate ###
Min: 0.827287 -> 0.740015: 1.1179x faster
Avg: 1.276936 -> 1.232548: 1.0360x faster
Not significant
Stddev: 0.74238 -> 0.73077: 1.0159x smaller
### sympy_str ###
Min: 0.158311 -> 0.157475: 1.0053x faster
Avg: 0.303297 -> 0.307362: 1.0134x slower
Not significant
Stddev: 0.20213 -> 0.20610: 1.0197x larger
### sympy_sum ###
Min: 0.199854 -> 0.231293: 1.1573x slower
Avg: 0.290365 -> 0.307076: 1.0575x slower
Not significant
Stddev: 0.12262 -> 0.11703: 1.0477x smaller
### telco ###
Min: 0.008000 -> 0.008000: no change
Avg: 0.013761 -> 0.013681: 1.0058x faster
Not significant
Stddev: 0.00890 -> 0.00897: 1.0079x larger
### trans2_annotate ###
Raw results: [298.1] None
### trans2_rtype ###
Raw results: [626.3] None
### trans2_backendopt ###
Raw results: [82.3] None
### trans2_database ###
Raw results: [113.8] None
### trans2_source ###
Raw results: [118.6] None
### twisted_iteration ###
Min: 0.002048 -> 0.002035: 1.0064x faster
Avg: 0.002062 -> 0.002048: 1.0068x faster
Significant (t=7.056602, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.0306x smaller
### twisted_names ###
Min: 0.000738 -> 0.000702: 1.0515x faster
Avg: 0.000748 -> 0.000710: 1.0528x faster
Significant (t=32.935112, a=0.95)
Stddev: 0.00001 -> 0.00001: 1.0987x smaller
### twisted_pb ###
Min: 0.003419 -> 0.003405: 1.0041x faster
Avg: 0.003723 -> 0.003762: 1.0105x slower
Not significant
Stddev: 0.00015 -> 0.00019: 1.2125x larger
### twisted_tcp ###
Min: 0.083069 -> 0.084500: 1.0172x slower
Avg: 0.084602 -> 0.085989: 1.0164x slower
Significant (t=-4.166700, a=0.95)
Stddev: 0.00164 -> 0.00169: 1.0357x larger
_______________________________________________
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev
