On the 0x22A day of Apache Harmony Alexey Varlamov wrote: > [snip] > > We can try to tune DRLVM default mode for HelloWorld + Eclipse + > > SciMark + DaCapo, would it be enough for users to be happy with > > Harmony on their first glance? I dunno, actually. IMHO, almost all > > java users taking care of performance know about client and server > > modes and what applications are best for each mode. > > Err, SciMark looks quite specific in this list, and I suspect it will > provoke that "short blanket" scenario. Probably we should not include > it for the default mode tuning.
What can we do: * agree that SciMark is not for DRLVM default mode (=do nothing) * take SciMark into account, make a weighted metric (with SciMark at low priority, of course) We should be "reasonably" fast, is 4x slowdown on SciMark reasonable? Tuning decisions are always not easy... > As a side note, any user cares about performance but not all dare to > tune it. Thank you for the side note)) There is going to be the "short blanket" story forever or at least for a while. And options are all we can do for those who care. I think, we all know that. > This is the main motivation for measuring OOB at the first place... Now I know, OOB=OutOfBox :) > > > > > > Currently we have in mind the following list: > > > > - HWA (Hello World Application) > > > > - SciMark > > > > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and > > > > xalan) > > > > - Anything else? > > > > > > > > What do you think about this? Any additions to the list? Comments? > > > > Questions? > > > > > > The problem I have with this is that I feel that each one of such > > > scenario might require different tuning parameters... and if that is the > > > case, you end up with the 'short blanket' problem: you improve here and > > > you decrease there. > > > > > > An 'adaptive' scenario, on the other hand, would allow us to: > > > > > > 1) avoid trying to find the optimal defaults (since we can't possible > > > test every scenario that will be useful in a way that is consistent with > > > real world usage) > > > > > > 2) avoid the blanket problem, each VM can adapt to the scenario of use > > > > > > 3) avoid the 'stiffness' problem, each VM can adapt to machine resource > > > changes and 'retune' itself if the environment changes. > > > > > > Of course, there is a price to pay in such 'fedback variability' systems > > > since they have to find the minima over and over again. > > > > > > So, another solution is to have a JVM "tuning parameters discovery mode" > > > that you can run and you turn such "parameter finding" autoprofiling > > > on... and the JVM dumps the tuning results for you on disk which you can > > > later use to initialize the JVM on your own. > > > > > > Not sure how feasible or complicated to write this is, but wow does this > > > sound on paper? > > > > > > > > > > Thanks, > > > > --- > > > > Sergey Kuksenko > > > > Intel Enterprise Solutions Software Division > > > > > > > > > > > > On 11/17/06, Stefano Mazzocchi <[EMAIL PROTECTED]> wrote: > > > >> > > > >> Alexey Varlamov wrote: > > > >> > Stefano, > > > >> > > > > >> > It is a bit unfair to compare *debug* build of Harmony with other > > > >> > release versions :) > > > >> > > > >> I'm simulating what a journalist with a developer could do. > > > >> > > > >> If there is a way to make it compile in 'release mode' (if such a thing > > > >> exists), I'll be very glad to redo the benchmarks. > > > >> > > > >> > I suppose all VMs where run in default mode (i.e. no special cmd-line > > > >> > switches)? > > > >> > > > >> Right. No switches. I'm simulating what users do when they get the JVM: > > > >> they run "java"... and if it's now fast enough they buy a new box. > > > >> > > > >> Having command line tuning parameters is mostly useless since most > > > >> people don't know the internals of a JVM well enough to guess what > > > >> parameters to tune anyway. > > > >> > > > >> So, what people will do once they get an harmony snapshot is "java > > > >> my.class.Name <http://my.class.name/>" and see the results. > > > >> > > > >> I want to simulate that and compare it to the same exact experience > > > >> they > > > >> will get with other virtual machines for a variety of common scenarios > > > >> (number crunching, xml processing, http serving, database load, etc...) > > > >> > > > >> I will focus on the server because that's there the apache action (and > > > >> my personal interest) is. > > > >> > > > >> So, like I said, if there are 'compile time' switches that I can use to > > > >> turn 'release mode' on, please tell me and I'll re-do the tests. > > > >> > > > >> > 2006/11/17, Stefano Mazzocchi <[EMAIL PROTECTED]>: > > > >> >> There are lies, damn lies and benchmarks.... which don't really tell > > > >> you > > > >> >> if an implementation of a program is *faster* but at least it tells > > > >> you > > > >> >> where you're at. > > > >> >> > > > >> >> So, as Geir managed to get the DSO linking problem go away in > > > >> >> DRLVM, I > > > >> >> was able to start running some benchmarks. > > > >> >> > > > >> >> The machine is the following: > > > >> >> > > > >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep > > > >> >> 16 > > > >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux > > > >> >> > > > >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz > > > >> >> bogomips 6410.31 (per CPU) > > > >> >> > > > >> >> There is nothing else running on the machine (load is 0.04 at the > > > >> >> time > > > >> >> of testing). > > > >> >> > > > >> >> The various virtual machines tested are: > > > >> >> > > > >> >> harmony > > > >> >> ------- > > > >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache > > > >> >> Software > > > >> >> Foundation or its licensors, as applicable. > > > >> >> java version " 1.5.0" > > > >> >> pre-alpha : not complete or compatible > > > >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build > > > >> >> > > > >> >> sun5 > > > >> >> --- > > > >> >> java version "1.5.0_09 " > > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build > > > >> >> 1.5.0_09-b03) > > > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode) > > > >> >> > > > >> >> sun6 > > > >> >> ---- > > > >> >> java version " 1.6.0-rc" > > > >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104) > > > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode) > > > >> >> > > > >> >> ibm > > > >> >> --- > > > >> >> java version " 1.5.0" > > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build > > > >> >> pxa64dev-20061002a (SR3) ) > > > >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64 > > > >> >> j9vmxa6423-20061001 (JIT enabled) > > > >> >> J9VM - 20060915_08260_LHdSMr > > > >> >> JIT - 20060908_1811_r8 > > > >> >> GC - 20060906_AA) > > > >> >> JCL - 20061002 > > > >> >> > > > >> >> bea > > > >> >> --- > > > >> >> java version "1.5.0_06 " > > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build > > > >> >> 1.5.0_06-b05) > > > >> >> BEA JRockit(R) (build > > > >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, ) > > > >> >> > > > >> >> > > > >> >> > > > >> -------------------------------------------------------------------------- > > > >> > > > >> >> > > > >> >> > > > >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/) > > > >> >> > > > >> >> command: java jnt.scimark2.commandline > > > >> >> > > > >> >> NOTE: bigger number is better > > > >> >> > > > >> >> Sun6 > > > >> >> Composite Score: 364.5832265230057 > > > >> >> FFT (1024): 220.8458713892794 > > > >> >> SOR (100x100): 696.1542342357722 > > > >> >> Monte Carlo : 149.37978088875656 > > > >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845 > > > >> >> LU (100x100): 430.1617273683819 > > > >> >> > > > >> >> BEA > > > >> >> Composite Score: 359.13480378697835 > > > >> >> FFT (1024): 303.8746880751562 > > > >> >> SOR (100x100): 454.25628897202307 > > > >> >> Monte Carlo : 93.23913192138497 > > > >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391 > > > >> >> LU (100x100): 413.8627835924175 > > > >> >> > > > >> >> Sun5 > > > >> >> Composite Score: 332.84987587548574 > > > >> >> FFT (1024): 216.5144595799027 > > > >> >> SOR (100x100): 689.429322146947 > > > >> >> Monte Carlo : 25.791262124978065 > > > >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373 > > > >> >> LU (100x100): 414.99493895566377 > > > >> >> > > > >> >> IBM > > > >> >> Composite Score: 259.8249218693683 > > > >> >> FFT (1024): 296.8415012789055 > > > >> >> SOR (100x100): 428.974881649179 > > > >> >> Monte Carlo : 89.15159857584082 > > > >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982 > > > >> >> LU (100x100): 339.8042037225181 > > > >> >> > > > >> >> Harmony > > > >> >> Composite Score: 113.65082278962575 > > > >> >> FFT (1024): 203.76641991778123 > > > >> >> SOR (100x100): 224.37761309236748 > > > >> >> Monte Carlo : 9.063866256533116 > > > >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227 > > > >> >> LU (100x100): 65.6410280487242 > > > >> >> > > > >> >> In this test harmony is clearly lagging behind... at about 30% > > > >> >> performance of the best JVM, it's a little crappy. Please note how > > > >> FFT's > > > >> >> performance is not so bad awhile monte carlo is pretty bad compared > > > >> >> to > > > >> >> BEA or IBM. > > > >> >> > > > >> >> Overall, it seems like there is some serious work to do here to > > > >> >> catch > > > >> up. > > > >> >> > > > >> >> > > > >> -------------------------------------------------------------------------- > > > >> > > > >> >> > > > >> >> > > > >> >> Test 2: Dhrystones > > > >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/ > > > >> ) > > > >> >> > > > >> >> command: java dhry 100000000 > > > >> >> > > > >> >> NOTE: bigger is better > > > >> >> > > > >> >> NB: I modified the code to accept the count at input from the > > > >> >> command > > > >> >> line! > > > >> >> > > > >> >> sun6: 8552856 dhrystones/sec > > > >> >> sun5: 6605892 > > > >> >> bea: 5678914 > > > >> >> harmony: 669734 > > > >> >> ibm: 501562 > > > >> >> > > > >> >> The performance here is horrific but what's surprising is that J9 is > > > >> >> even worse. No idea what's going on but it seems like something is > > > >> >> not > > > >> >> working as it should (in both harmony and J9) > > > >> >> > > > >> >> > > > >> -------------------------------------------------------------------------- > > > >> > > > >> >> > > > >> >> > > > >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz) > > > >> >> > > > >> >> command: java Sieve 30 > > > >> >> > > > >> >> NB: I modified the test to run for a configurable amount of seconds. > > > >> >> > > > >> >> sun6 8545 sieves/sec > > > >> >> sun5 8364 > > > >> >> bea 6174 > > > >> >> harmony 1836 > > > >> >> ibm 225 > > > >> >> > > > >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly > > > >> >> lagging behind. > > > >> >> > > > >> >> Stay tuned for more tests. > > > >> >> > > > >> >> -- > > > >> >> Stefano. > > > >> >> > > > >> >> > > > >> > > > >> > > > >> -- > > > >> Stefano. > > > >> > > > >> > > > > > > > > > > > > > -- > > > Stefano. > > > > > > > > > > -- > > Egor Pasko > > > > > -- Egor Pasko
