Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-18 Thread Pavel Ozhdikhin
On 18 Sep 2006 12:00:57 +0700, Egor Pasko [EMAIL PROTECTED] wrote: On the 0x1E5 day of Apache Harmony Pavel Ozhdikhin wrote: Thanks for explaining. This is another variant of the bytecode-based regression tests. This variant is also adoptable to Java-based and IR-based regression tests.

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-17 Thread Egor Pasko
On the 0x1E5 day of Apache Harmony Pavel Ozhdikhin wrote: Thanks for explaining. This is another variant of the bytecode-based regression tests. This variant is also adoptable to Java-based and IR-based regression tests. On 15 Sep 2006 17:53:00 +0700, Egor Pasko [EMAIL PROTECTED] wrote:

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-15 Thread Egor Pasko
On the 0x1E5 day of Apache Harmony Pavel Ozhdikhin wrote: Egor, How Nullstone tests differ from what Rana proposed and Mikhail L. prototyped - could you please elaborate? the idea is simple. You have two versions of a test. First -- unoptimized, second -- same algorithm, but optimized by

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-15 Thread Rana Dasgupta
On 15 Sep 2006 11:26:40 +0700, Egor Pasko [EMAIL PROTECTED] wrote: On the 0x1E4 day of Apache Harmony Mikhail Fursov wrote: This would be the best solution to test if an optimization works as expected. We can create the following framework inside Jitrino compiler to test individual

[testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Mikhail Loenko
Hi Rana 2006/9/14, Rana Dasgupta [EMAIL PROTECTED]: SNIP One way to write the test would be to loop N times on a scenario that kicks in the optimization say, array bounds check elimination and then loop N times a very similar scenario but such that the bounds check does not get eliminated.

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Pavel Ozhdikhin
*Re-sending to the new thread:* Hello Rana, When I think of an optimization which gives 1% improvement on some simple workload or 3% improvement on EM64T platforms only I doubt this can be easily detected with a general-purpose test suite. IMO the performance regression testing should have a

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Egor Pasko
On the 0x1E4 day of Apache Harmony Pavel Ozhdikhin wrote: When I think of an optimization which gives 1% improvement on some simple workload or 3% improvement on EM64T platforms only I doubt this can be easily detected with a general-purpose test suite. IMO the performance regression testing

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Mikhail Loenko
In the example i've mentioned before the difference between optimized and non-optimized calls was about 1000x. But the test sometimes failed anyway Thanks, Mikhail 14 Sep 2006 17:59:44 +0700, Egor Pasko [EMAIL PROTECTED]: On the 0x1E4 day of Apache Harmony Pavel Ozhdikhin wrote: When I think

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Mikhail Fursov
This would be the best solution to test if an optimization works as expected. We can create the following framework inside Jitrino compiler to test individual optimizations and optimizations inter-dependencies: Create a special optimization (test) that that works only for special Java method

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Egor Pasko
On the 0x1E4 day of Apache Harmony Mikhail Loenko wrote: In the example i've mentioned before the difference between optimized and non-optimized calls was about 1000x. But the test sometimes failed anyway Yet, I think, pure Java performance is more predictable than network performance. I am

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Egor Pasko
On the 0x1E4 day of Apache Harmony Mikhail Fursov wrote: This would be the best solution to test if an optimization works as expected. We can create the following framework inside Jitrino compiler to test individual optimizations and optimizations inter-dependencies: Create a special

Re: [testing] optimization regressions (was: Re: [result] Re: [vote] HARMONY-1363 - DRLVM fixes and additions)

2006-09-14 Thread Pavel Ozhdikhin
Egor, How Nullstone tests differ from what Rana proposed and Mikhail L. prototyped - could you please elaborate? Thanks, Pavel Any other ideas or experience how to test compiler optimizations predictably? although having performance measurements, NULLSTONE-like tests are quite predictable