The problem is unstable execution time of java programs: If you consequently run the same java program on the same computer in the same conditions, execution time may vary by 20% or even more
If you have Config1 such that a loop invoking some security function runs consiquently; 10sec, 11sec, 9sec and Config2 that executes the same loop 20sec, 19secc, 23sec Then you may conclude that Config1 is faster. If you have to call via reflection that might take 100x more than execution of the test method itself then you'll see times for Config1: 1010sec, 912sec, 1183sec and for Config2: 1020sec, 835sec, 1190sec Which one is faser? And if you want to compare performance of just a single method and your configs imply usage of different reflection implementations, than those "performance results" would be useless at all - you will always compare perfromance of reflection rather then performance of functionality of interest. On 1/19/06, George Harley1 <[EMAIL PROTECTED]> wrote: > Hi Mikhail, > > > > When we run the tests (those ones that we call unit tests :) in > performance mode > > we start a timer, then run the same testcase in a loop millions of > > times (and we do NOT call them via reflection) then we stop a timer > > Here is a flavour of what works for a lot of people... > > Use the JUnit type RepeatedTest (see [1]) to wrap any concrete JUnit Test > (e.g. a TestSuite, a TestCase etc) so that the test (or tests if it is a > TestSuite) get run as many million times as you want. Then, take the very > same RepeatedTest object and wrap that inside a JUnitPerf TimedTest (see > [2]), also passing in a maximum elapsed time that the RepeatedTest has to > run in. If the RepeatedTest (which is running the original test case or > suite millions of times for you) doesn't complete in the specified maximum > time then the test is a failure. > > > > As I understand your point, you are going to measure a single call of > > a test method and that call is over reflection, or multiple calls but > > each of them over reflection. > > No, that was not my point. My point is that, at the unit test level, it is > possible to carry out performance timings using *established* JUnit > techniques that are *dynamic* in nature and do *not* require adding in a > superclass into the inheritance hierarchy meaning that every child class > "is a" performance test spouting out lots of "== test Foo passed OK ==" > messages. What happens if you want to carry out some stress testing with > the unit tests ? Does it mean the addition of a new superclass at the top > of the hierarchy so that every unit test class "is a" stress test as well As you might notice I suggested to remove super class and add a simple predictable Logger class whose execution time would me minor comparing to test case's time. If other modes require usage of other functionality - it might be considered also. Thanks, Mikhail > ? And so on for interop tests ... etc .. etc ... ? > > > > Just wanted to say that reflection part of the call might take 99% of > method > > execution and results would not be reliable. Please correct me if I > wrong. > > You could adjust your target timings to take into account the effects of > the test framework. What would be the problem in doing that ? If you are > looking for the introduction of performance problems then isn't it the > relative timings (e.g. "before versus after the introduction of new code" > or "test on JRE1 versus identical test on JRE2") that matter ? > > > Best regards, > George > > [1] > http://www.junit.org/junit/javadoc/3.8.1/junit/extensions/RepeatedTest.html > [2] http://clarkware.com/software/JUnitPerf.html#howtouse > ________________________________________ > George C. Harley > > > > > > Mikhail Loenko <[EMAIL PROTECTED]> > 18/01/2006 17:06 > Please respond to > harmony-dev@incubator.apache.org > > > To > harmony-dev@incubator.apache.org > cc > > Subject > Re: [jira] Commented: (HARMONY-31) Move peformance timing of unit tests > into a decorator class. > > > > > > > Hi George > > When we run the tests (those ones that we call unit tests :) in > performance mode > we start a timer, then run the same testcase in a loop millions of > times (and we do > NOT call them via reflection) then we stop a timer > > As I understand your point, you are going to measure a single call of > a test method and that call is over reflection, or multiple calls but > each of them over reflection. > Just wanted to say that reflection part of the call might take 99% of > method > execution and results would not be reliable. Please correct me if I wrong. > > Thanks, > Mikhail > > On 1/18/06, George Harley1 <[EMAIL PROTECTED]> wrote: > > Hi Mikhail, > > > > > The messages are important to analyze failures also. > > > > What is JUnit's own failure reporting mechanism not providing to you ? > > > > > > > And the possibility to test perfromance is also important > > > > Yes it is. But - to return to the original point of the HARMONY-31 JIRA > > issue - this can be done without the need to either bring in a > superclass > > in the test hierarchy and/or scatter logging calls around the test code. > > Performance testing using JUnit should be done in a transparent manner > > using a decorator. As an example of what I mean please take a look at > the > > JUnitPerf site at http://clarkware.com/software/JUnitPerf.html . > JUnitPerf > > is an extension to JUnit that helps with the creation of performance > tests > > that match existing unit tests. It does this using a decorator design. > > This provides for a separation of concerns that benefits developers and > > performance engineers. > > > > The decorator approach means that we effectively *extend* JUnit with > > simple wrapper classes in which we can make full use of the JUnit API to > > give us the additional behaviour needed when running the tests (maybe > this > > could be used to give whatever extra failure analysis data you say is > > lacking ?). And if somebody doesn't want the extension behaviour ? They > > just run the suite without the custom wrappers. > > > > > > > When we have a powerful performance suite we may revisit this. > > > > Don't you mean when we have a powerful unit test suite ? > > > > Take care, > > George > > ________________________________________ > > George C. Harley > > > > > > > > > > > > Mikhail Loenko <[EMAIL PROTECTED]> > > 18/01/2006 12:57 > > Please respond to > > harmony-dev@incubator.apache.org > > > > > > To > > harmony-dev@incubator.apache.org > > cc > > > > Subject > > Re: [jira] Commented: (HARMONY-31) Move peformance timing of unit tests > > into a decorator class. > > > > > > > > > > > > > > Well, those messages were important for developers when > > they were writing code and tests. Then tests came to repository 'as is'. > > > > The messages are important to analyze failures also. > > And the possibility to test perfromance is also important > > > > For me any option that does not break functionality in favor of beauty > > looks > > good. I suggest swithcing off logging by default and ether keep > > PerformanceTest > > super class as is or replace it with a simple Logger class. > > > > When we have a powerful performance suite we may revisit this. > > > > Thanks, > > Mikhail. > > > > On 1/18/06, Tim Ellison <[EMAIL PROTECTED]> wrote: > > > Absolutely right -- writing meaningful performance tests is hard. > > > Implementing your own Logger would not solve the problem though<g>. > > > > > > Best to avoid the 'This test worked OK' log messages altogether, and > > > stick to assertions. > > > > > > Regards, > > > Tim > > > > > > Mikhail Loenko wrote: > > > > It might be a problem... > > > > > > > > When we use java.util.logging we do not just compare performance of > > security > > > > API functions, the result is also depends on difference in > performance > > of > > > > java.util.logging in standard classes vs. Harmony classes. So if we > > use > > > > non-trivial functionality from there then our results will be > spoiled > > a little. > > > > > > > > Will investigate more... > > > > > > > > Thanks, > > > > Mikhail. > > > > > > > > On 1/17/06, Tim Ellison <[EMAIL PROTECTED]> wrote: > > > >> neither is the Logger class -- so my point is if you are going to > > write > > > >> some logging code why not do it in java.util.logging? You may > choose > > to > > > >> only do simple stubs for now until somebody steps up to do a real > > impl. > > > >> > > > >> Regards, > > > >> Tim > > > >> > > > >> Mikhail Loenko wrote: > > > >>> It's not yet implemented. > > > >>> > > > >>> thanks, > > > >>> Mikhail > > > >>> > > > >>> On 1/17/06, Tim Ellison <[EMAIL PROTECTED]> wrote: > > > >>>> Why not use java.util.logging? > > > >>>> > > > >>>> Regards, > > > >>>> Tim > > > >>>> > > > >>>> Mikhail Loenko (JIRA) wrote: > > > >>>>> [ > > > http://issues.apache.org/jira/browse/HARMONY-31?page=comments#action_12362910 > > > ] > > > >>>>> > > > >>>>> Mikhail Loenko commented on HARMONY-31: > > > >>>>> --------------------------------------- > > > >>>>> > > > >>>>> This is not what I meant. > > > >>>>> > > > >>>>> I was going to create a Logger class at this point like this: > > > >>>>> > > > >>>>> public class Logger { > > > >>>>> public static boolean printAllowed = false; > > > >>>>> public static void log(String message) { > > > >>>>> if (printAllowed) System.out.print(message); > > > >>>>> } > > > >>>>> public static void logln(String message) { > > > >>>>> if (printAllowed) System.out.println(message); > > > >>>>> } > > > >>>>> public static void logError(String message) { > > > >>>>> if (printAllowed) System.err.print(message); > > > >>>>> } > > > >>>>> public static void loglnError(String message) { > > > >>>>> if (printAllowed) System.err.println(message); > > > >>>>> } > > > >>>>> } > > > >>>>> > > > >>>>> And replace log() with Logger.log() everywhere in the tests. > > > >>>>> > > > >>>>> All the remaining functionality in the PerformanceTest is > > obsolete. > > > >>>>> > > > >>>>> > > > >>>>>> Move peformance timing of unit tests into a decorator class. > > > >>>>>> ------------------------------------------------------------ > > > >>>>>> > > > >>>>>> Key: HARMONY-31 > > > >>>>>> URL: http://issues.apache.org/jira/browse/HARMONY-31 > > > >>>>>> Project: Harmony > > > >>>>>> Type: Improvement > > > >>>>>> Reporter: George Harley > > > >>>>>> Assignee: Geir Magnusson Jr > > > >>>>>> Priority: Minor > > > >>>>>> Attachments: PerfDecorator.java > > > >>>>>> > > > >>>>>> There has been some low-level discussion on the dev mailing > list > > recently about the inclusion of performance-related logging code near > the > > top of a unit test class inheritance hierarchy (see > > com.openintel.drl.security.test.PerformanceTest in the HARMONY-16 > > contribution). This particular issue suggests an alternative way of > adding > > in timing code but without making it the responsibility of the unit > tests > > themselves and without the need to introduce a class in the inheritance > > hierarchy. > > > >>>>>> The basic approach is to exploit the > > junit.extensions.TestDecorator type in the JUnit API to add in timing > > behaviour before and after each test method runs. This will be > > demonstrated with some simple sample code. > > > >>>> -- > > > >>>> > > > >>>> Tim Ellison ([EMAIL PROTECTED]) > > > >>>> IBM Java technology centre, UK. > > > >>>> > > > >> -- > > > >> > > > >> Tim Ellison ([EMAIL PROTECTED]) > > > >> IBM Java technology centre, UK. > > > >> > > > > > > > > > > -- > > > > > > Tim Ellison ([EMAIL PROTECTED]) > > > IBM Java technology centre, UK. > > > > > > > > > > > >
