Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-18 Thread Pavel Ozhdikhin
On 18 Sep 2006 12:00:57 +0700, Egor Pasko <[EMAIL PROTECTED]> wrote: On the 0x1E5 day of Apache Harmony Pavel Ozhdikhin wrote: > Thanks for explaining. This is another variant of the bytecode-based > regression tests. This variant is also adoptable to Java-based and IR-based regression tests.

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-17 Thread Egor Pasko
On the 0x1E5 day of Apache Harmony Pavel Ozhdikhin wrote: > Thanks for explaining. This is another variant of the bytecode-based > regression tests. This variant is also adoptable to Java-based and IR-based regression tests. > On 15 Sep 2006 17:53:00 +0700, Egor Pasko <[EMAIL PROTECTED]> wrote: >

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-15 Thread Rana Dasgupta
On 15 Sep 2006 11:26:40 +0700, Egor Pasko <[EMAIL PROTECTED]> wrote: >>On the 0x1E4 day of Apache Harmony Mikhail Fursov wrote: >> This would be the best solution to test if an optimization works as > >expected. > >We can create the following framework inside Jitrino compiler to test > >individua

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-15 Thread Rana Dasgupta
Hi Mikhail, I save the same observation as Egor, that jit optimizations should be much more deterministic and their impact predictable. I understand the unpredictability of the situation you described. Thanks, Rana On 14 Sep 2006 21:55:16 +0700, Egor Pasko <[EMAIL PROTECTED]> wrote: > > On

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-15 Thread Pavel Ozhdikhin
Thanks for explaining. This is another variant of the bytecode-based regression tests. -Pavel On 15 Sep 2006 17:53:00 +0700, Egor Pasko <[EMAIL PROTECTED]> wrote: On the 0x1E5 day of Apache Harmony Pavel Ozhdikhin wrote: > Egor, > > How Nullstone tests differ from what Rana proposed and Mikh

[general] change the subject line... (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-15 Thread Tim Ellison
at least, or better yet move to a new thread. This conversation has moved way beyond discussing the results of a contribution vote. For the poor people who have hundreds of mails to read ;-) Tim Pavel Ozhdikhin wrote: > Rana, > > To cover most of the performance regressions ideally we might ha

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-15 Thread Egor Pasko
On the 0x1E5 day of Apache Harmony Pavel Ozhdikhin wrote: > Egor, > > How Nullstone tests differ from what Rana proposed and Mikhail L. prototyped > - could you please elaborate? the idea is simple. You have two versions of a test. First -- unoptimized, second -- same algorithm, but optimized by

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-15 Thread Pavel Ozhdikhin
Rana, To cover most of the performance regressions ideally we might have three types of tests: 1. High-level benchmarks (like SPECs or DaCapo benchmarks) - cover significant performance issues or issues that are not related to a particular optimization 2. Performance regression tests or

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Pavel Ozhdikhin
Egor, How Nullstone tests differ from what Rana proposed and Mikhail L. prototyped - could you please elaborate? Thanks, Pavel Any other ideas or experience how to test compiler optimizations > predictably? although having performance measurements, NULLSTONE-like tests are quite predictable

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Egor Pasko
On the 0x1E4 day of Apache Harmony Mikhail Fursov wrote: > This would be the best solution to test if an optimization works as > expected. > We can create the following framework inside Jitrino compiler to test > individual optimizations and optimizations inter-dependencies: > > Create a special o

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-14 Thread Rana Dasgupta
Hi Pavel, Platform specific optimizations can be accomodated in the scheme described by doing a cpuid check in the test and automatically passing it or disabling on all other platforms. That shouldn't be too hard. I understand that some jit optimizations are deeper and more abstract, but ultim

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Egor Pasko
On the 0x1E4 day of Apache Harmony Mikhail Loenko wrote: > In the example i've mentioned before the difference between optimized and > non-optimized calls was about 1000x. But the test sometimes failed anyway Yet, I think, pure Java performance is more predictable than network performance. I am al

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Mikhail Fursov
This would be the best solution to test if an optimization works as expected. We can create the following framework inside Jitrino compiler to test individual optimizations and optimizations inter-dependencies: Create a special optimization ("test") that that works only for "special" Java method

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-14 Thread Weldon Washburn
Sorry for picking on Harmony-1417. It happened to be what I was looking at when I wrote the email. The issue is more general than JIT optimizations. I realize it will be impossible to supply rigorous tests for BBC.patch and 14xx.patches in the next few days. However, would it be possible for de

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Mikhail Loenko
In the example i've mentioned before the difference between optimized and non-optimized calls was about 1000x. But the test sometimes failed anyway Thanks, Mikhail 14 Sep 2006 17:59:44 +0700, Egor Pasko <[EMAIL PROTECTED]>: On the 0x1E4 day of Apache Harmony Pavel Ozhdikhin wrote: > When I thin

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Egor Pasko
On the 0x1E4 day of Apache Harmony Pavel Ozhdikhin wrote: > When I think of an optimization which gives 1% improvement on some simple > workload or 3% improvement on EM64T platforms only I doubt this can be > easily detected with a general-purpose test suite. IMO the performance > regression testin

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-14 Thread Pavel Ozhdikhin
Hello Rana, When I think of an optimization which gives 1% improvement on some simple workload or 3% improvement on EM64T platforms only I doubt this can be easily detected with a general-purpose test suite. IMO the performance regression testing should have a specialized framework and a stable e

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Pavel Ozhdikhin
*Re-sending to the new thread:* Hello Rana, When I think of an optimization which gives 1% improvement on some simple workload or 3% improvement on EM64T platforms only I doubt this can be easily detected with a general-purpose test suite. IMO the performance regression testing should have a spe

[testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Mikhail Loenko
Hi Rana 2006/9/14, Rana Dasgupta <[EMAIL PROTECTED]>: One way to write the test would be to loop N times on a scenario that kicks in the optimization say, array bounds check elimination and then loop N times a very similar scenario but such that the bounds check does not get eliminated. Then t

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-13 Thread Rana Dasgupta
Hi Egor, An optimization is a functionality that can regress like anything else. The functionality is the perf gain, which is the point of the optimization. How would any committer confirm that the submitted code does perform the optimization ...other than the developer's word that it is happe

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-13 Thread Geir Magnusson Jr.
Heh - I'm working on this very problem - I'm putting the 1363 patch into head, and ran across that one. Thanks for the hint and the patch. geir Gregory Shimansky wrote: On Thursday 14 September 2006 00:12 Weldon Washburn wrote: Anyone having problems building patch 1363 on Linux? I can get

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-13 Thread Egor Pasko
On the 0x1E4 day of Apache Harmony Weldon Washburn wrote: > Where do I find regression tests for the recent drlvm patches? Right now, > I am trying to drill down on Harmony-1417. I look in drlvm/trunk/vm/tests > but don't see regression tests that look like they verify that lazy > exception optim

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-13 Thread Weldon Washburn
I looked closer at the output. It turned out to be an ld error that was caused by stale object files. Now that I think about it, the rollback replaced newer source files with older source files. I somehow expected the make system to recompile oops. A "build clean" followed by a build fixed

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-13 Thread Gregory Shimansky
On Thursday 14 September 2006 00:12 Weldon Washburn wrote: > Anyone having problems building patch 1363 on Linux? > > I can get 1363 to work on WinXP without problem. I first do: > > svn update -r "{2006-08-31}" > > Then a: > > patch -p0 -i BBC.patch > > The build is successful and "build

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-13 Thread Weldon Washburn
Where do I find regression tests for the recent drlvm patches? Right now, I am trying to drill down on Harmony-1417. I look in drlvm/trunk/vm/tests but don't see regression tests that look like they verify that lazy exception optimization works correctly. Maybe I overlooked something? -- Weldon

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-13 Thread Weldon Washburn
Anyone having problems building patch 1363 on Linux? I can get 1363 to work on WinXP without problem. I first do: svn update -r "{2006-08-31}" Then a: patch -p0 -i BBC.patch The build is successful and "build test" is successful When I roll back the Linux tree to {2006-08-31} the bui

RE: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-11 Thread Magnusson, Geir
M > To: harmony-dev@incubator.apache.org; [EMAIL PROTECTED] > Subject: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes > and additions > > On 9/10/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote: > > > > +1 from Geir, Alexey V, Vladimir, Mikhail L, Mikhail F, Gregory > &g

Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-11 Thread Weldon Washburn
On 9/10/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote: +1 from Geir, Alexey V, Vladimir, Mikhail L, Mikhail F, Gregory I'd have preferred a third committer, +1 based on a a quick scan of the source code. I did an "svn update" about 12 hours ago. Then a " patch -p1 -i BBC.patch". The bu

[result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions

2006-09-10 Thread Geir Magnusson Jr.
+1 from Geir, Alexey V, Vladimir, Mikhail L, Mikhail F, Gregory I'd have preferred a third committer, but as it is a set of patches and enhancements to an existing codebase, and there was no opposition, I think we're ok. geir Geir Magnusson Jr. wrote: All is in order and in SVN for Harmony-