I'd agree in general that it's the naming convention that would be key to readily being able to exercise the tests by another framework. With regards to refactoring the parameterized tests, I like the concept of being able to have a battery of data sets that can be used to exercise tests. Maybe we can put in place some simple bespoke function for this kind of behaviour. I've had this in the back of my mind while looking at the code.
Another compilcation is that there's no precedent in junit 3.8 for the @BeforeClass type of calls, which some of the new tests are using, so we'll need to establish a convention for that. A frustration that I find is that the current structure doesn't permit running/debugging individual tests. If you want breakpoints deep in the SDO /EMF code and then have to run 50 tests before getting to the one you are interested in then that's a bit of a pain. Often in eclipse in the SDO implementation tests, I right click in the Junit panel on a failing test and click run/debug, to exercise a single failing test. I think the restriction is introduced into the CTS primarily because of the one time initialization of the implementation specific test helper. I would imagine it could be very low cost to initialize this once per setUp() in a superclass, the first initialization triggering some static code that performed any real startup overhead and cached the helper. This all leads me to believing that to get true agnosticism wrt the test harness we should perhaps introduce bespoke function, some of which replicates the junit 4.1features, either by creating an abstract specialization of TestCase or TestRunner or both. -- Kelvin On 20/04/07, Andy Grove <[EMAIL PROTECTED]> wrote:
Just for clarification, I think we're saying that the important thing here is the method naming convention, rather than requiring the tests to extend TestCase? If we follow the junit 3.8 naming convention and always use the method names setUp / teardown (and make sure they are public methods) and have all test methods start with "test" then it won't matter if the tests extend TestCase or have junit 4.1 annotations. However, just to complicate matters, the tests in the parameterizedTests package are making use of new junit 4.1 features for providing parameters to the tests and these tests don't currently fit into the simple junit 3.8 style and it will be much harder to re-use these tests from other frameworks in their current form. If we want to stick to the simple junit 3.8 style then these tests will need some refactoring. Regards, Andy. -----Original Message----- From: kelvin goodson [mailto:[EMAIL PROTECTED] Sent: 19 April 2007 11:03 To: tuscany-dev@ws.apache.org Subject: Re: [Java SDO CTS] Junit 4.1 pattern for calling setUp when classes don't inherit from TestCase In fact I'd say for the purposed of introspection by some other harness the old style is far preferable, since it's easy to examine the method names/signatures to determine what is a test and what is a setup method. I was about to start cleaning these up, but I'd like to complete this discussion and decide whether we should be making everything use the old 3.8style or the new 4.1 annotations. What I will do in the meantime is add setup methods to all the files in their existing style in order to fix up the issues with reusing type helpers between tests, and then revisit the style after the discussion has completed. For simplicity I will use the same method signatures for setup methods as are used in 3.8 when using 4.1 annotations. Regards, Kelvin. On 18/04/07, Andy Grove <[EMAIL PROTECTED]> wrote: > > > Frank, > > You're absolutely right. I guess I'd forgotten that you could override > a protected method and make it public. > > In that case, it doesn't seem to matter if we use "old-style" junit or > annotations - it should still be possible to call the tests without > using the junit test runners. > > Andy. > > -----Original Message----- > From: Frank Budinsky [mailto:[EMAIL PROTECTED] > Sent: 17 April 2007 18:01 > To: tuscany-dev@ws.apache.org > Subject: RE: [Java SDO CTS] Junit 4.1 pattern for calling setUp when > classes don't inherit from TestCase > > Hi Andy, > > Java allows you make something more visible in a derived class than in > the base, so declaring setUp() public in MyTest wouldn't seem to be a > problem. > > > Frank > > "Andy Grove" <[EMAIL PROTECTED]> wrote on 04/17/2007 12:19:37 PM: > > > Hi Frank, > > > > The TestCase class defines setUp and tearDown as protected methods, > > forcing the child class to also declare them as protected methods > > and this means they can't be loaded using reflection. > > > > Using junit 4.1 means we can declare the methods as public. > > > > Thanks, > > > > Andy. > > > > -----Original Message----- > > From: Frank Budinsky [mailto:[EMAIL PROTECTED] > > Sent: 17 April 2007 17:03 > > To: tuscany-dev@ws.apache.org > > Subject: RE: [Java SDO CTS] Junit 4.1 pattern for calling setUp when > > classes don't inherit from TestCase > > > > Hi Andy, > > > > Maybe this is a stupid question (my junit ignorance showing through > :-), > > but couldn't you have run your simple test harness (main) even if > MyTest > > extended from TestCase? Is there something about having the base > > class that prevents you from simply invoking the test methods directly? > > > > Frank. > > > > "Andy Grove" <[EMAIL PROTECTED]> wrote on 04/17/2007 11:21:49 AM: > > > > > > > > To better understand this myself, I just put a simple test case > > > together using junit 4.1 with annotations and made use of the > > > junit assertion calls e.g. > > > > > > public class MyTest { > > > @Test > > > public void testSomething() { > > > // this test will fail > > > assertEquals( "numbers are same", 1, 2 ); > > > } > > > } > > > > > > I then wrote a simple test harness to load the test class using > > > reflection and invoke any methods starting with "test". > > > > > > public static void main(String[] args) throws Exception { > > > Class testClass = Class.forName( "test.MyTest" ); > > > Object testObject = testClass.newInstance(); > > > Method method[] = testClass.getMethods(); > > > for (int i = 0; i < method.length; i++) { > > > if (method[i].getName().startsWith("test")) { > > > System.out.println("Running " + > method[i].getName()); > > > try { > > > method[i].invoke( testObject ); > > > } catch (Throwable th) { > > > th.printStackTrace(); > > > } > > > } > > > } > > > } > > > > > > This ran the above test method and caught the following exception: > > > > > > java.lang.AssertionError: numbers are same expected:<1> but > > > was:<2> > > > > > > For me, this seems to demonstrate that using junit 4.1 style tests > > > will allow people to call their tests from their own test > > > harnesses without requiring junit-users to have to go to any special effort. > The > > > > > junit > > > Assert.* methods simply check some criteria and then call fail() > > > if that criteria is false. The fail() method simply throws an > > AssertionError. > > > > > > Writing the CTS tests using junit with annotations (rather than > > > extending TestCase) does seem to provide everything required to > allow > > > users to run the tests outside of junit, albeit with a dependency > > > on > > > > junit.jar but I don't see why that would be a problem for anyone? > > > We > > > > could write our own versions of the assert* methods but they would > do > > > exactly the same thing as the junit implementation so this seems > > > rather pointless. > > > > > > My conclusion is that we should continue writing SDO CTS tests > > > using > > > > junit, but ensure we use the annotation pattern rather than > extending > > > TestCase. Is everyone happy with this? > > > > > > Thanks, > > > > > > Andy. > > > > > > > > > -----Original Message----- > > > From: kelvin goodson [mailto:[EMAIL PROTECTED] > > > Sent: 17 April 2007 14:53 > > > To: tuscany-dev@ws.apache.org > > > Subject: Re: [Java SDO CTS] Junit 4.1 pattern for calling setUp > > > when > > > > classes don't inherit from TestCase > > > > > > Yes, I was about to write to the list on this subject. I'd like to > > > understand more of how the test harness agnosticism was intended, > and > > > whether its really practical. As it stands there is still junit > > > through and through, in particular, each test method still > references > > > > > junit assertion calls. Even if we could do assertions from > > > annotations (as per JML pre and post conditions), we still have > > > all > > the junit imports. > > > If that were possible, then I guess each of the test methods could > be > > > invoked by something other than the junit test harness, but they > > > would have trouble asserting anything about the success of the > method > > > invocation, since there are no artifacts available before or > > > after the method invocation to make assertions about. > > > > > > Kelvin > > > > > > On 17/04/07, Simon Nash <[EMAIL PROTECTED]> wrote: > > > > > > > > If the goal is to make the tests "harness agnostic", then I > > > > don't see much difference between a JUnit-specific inheritance > dependency > > > > and a JUnit-specific annotation dependency. Is the annotation > > > > dependency less troublesome for some reason? > > > > > > > > Simon > > > > > > > > kelvin goodson wrote: > > > > > > > > > Thanks for this Andy, I'll play with it tomorrow. > > > > > > > > > > Regards, Kelvin. > > > > > > > > > > On 16/04/07, Andy Grove <[EMAIL PROTECTED]> wrote: > > > > > > > > > >> > > > > >> > > > > >> I believe you just need to annotate the setUp method with > > @Before. > > > > >> This is described in the junit cookbook, here: > > > > >> > > > > >> http://junit.sourceforge.net/doc/cookbook/cookbook.htm > > > > >> > > > > >> I'm currently working on submitting some more XSD test cases > > > > >> in > > > > > >> the > > > > > > > >> CTS so I'll try this method out. Hopefully I can then remove > the > > > > >> current dependency on TestCase in those tests. > > > > >> > > > > >> Thanks, > > > > >> > > > > >> Andy. > > > > >> > > > > >> > > > > >> -----Original Message----- > > > > >> From: [EMAIL PROTECTED] > > > > >> [mailto:[EMAIL PROTECTED] > On > > > > Behalf > > > > >> Of kelvin goodson > > > > >> Sent: 16 April 2007 14:42 > > > > >> To: tuscany-dev@ws.apache.org > > > > >> Cc: Robbie Minshall > > > > >> Subject: [Java SDO CTS] Junit 4.1 pattern for calling setUp > when > > > > classes > > > > >> don't inherit from TestCase > > > > >> > > > > >> Hi, > > > > >> I'm looking at doing some work in the CTS. I was looking back > at > > > > >> Robbie's attached description about how to keep the tests > > > > >> "harness agnostic". I'm assuming that this is still a goal > > > > >> of the CTS although > > > > I > > > > >> may have missed something in my catching up. In my quest to > make > > > > >> the > > > > CTS > > > > >> better I note that a number of the test case classes still > extend > > > > > > >> the junit TestCase class. > > > > >> This is true for all test classes that have a setUp() method. > One > > > > > > >> that doesn't inherit from TestCase is XSDSerializationTest, > and > > > > >> adding a setUp method to this class doesn't cause junit to > invoke > > > > > > >> it in my eclipse environment. I'm trying to work out whether > > > > >> I should I expect a 4.1environment to discover and execute > > > > >> the setUp method when junit is used in this way. I seem to > > > > >> have Eclipse junit > > > > > > > >> plugins for 3.8.1 and > > > > >> 4.1.0.1 and the preferences tab for Junit doesn't seem to > > > > >> offer > > > > > >> much in the way of configuration, so I can't be sure I'm > > > > >> using > > > > >> 4.1 > > > behaviour. > > > > >> > > > > >> I really would like to be XSDSerializationTests to execute > setUp > > > > >> so > > > > that > > > > >> we can have a fresh HelperContext per test, and I guess the > easy > > > > > > >> way out is to make the test class inherit from TestCase like > the > > > > >> others, but I'd prefer not to introduce the explicit > > > > >> dependency > > > > > >> on Junit if I can avoid it. > > > > >> > > > > >> Regards Kelvin. > > > > >> > > > > >> > > > > >> On 07/12/06, Robbie Minshall <[EMAIL PROTECTED]> wrote: > > > > >> > > > > > >> > This sounds quite good. > > > > >> > > > > > >> > I have written some test cases with Brian Murray which I > would > > > > >> > be > > > > > > > >> > happy to contribute to tuscany. Identifying duplication > > > > >> > and differences in similar tests would probably be an > > > > >> > intersting > > > > excercise > > > > >> right off the bat. > > > > >> > > > > > >> > One decision that we spent a little time mulling over was > > > > >> > the > > > > > >> > framework to use for our test suite. Originally we used > > > > >> > the much > > > > > > > >> > loved junit harness which worked well. Different runtimes > > > > >> > ( command line, J2EE Application Server, a Service > > > > >> > Container ) have > > > > > > > >> > different > > > > >> classloader hierarchies etc. > > > > >> > Without many modifications to the junit code it was > > > > >> > difficult > > > > > >> > and > > > > > > > >> > quite ugly testing SDO within the context of a variety of > > > > >> > runtimes which the SDO APIs will be used. > > > > >> > > > > > >> > We took the approach of writing general test libraries > > > > >> > which can then simply be called from a variety of test > > > > >> > frameworks such as junit or a simple J2EE or SCA > > > > >> > Application test > harness. > > > > > > >> > I like this approach > > > > for > > > > >> > > > > >> > keeping the actual test code very simple, allowing for > > > > >> > integration a variety of test frameworks, and providing > ability > > > > > > >> > to test directly within the different runtimes people care > > about. > > > > >> > > > > > >> > Any thoughts on this ? > > > > >> > > > > > >> > Robbie > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > On 12/1/06, kelvin goodson <[EMAIL PROTECTED] > wrote: > > > > >> > > > > > > >> > > Andy, > > > > >> > > please attach them to the JIRA for this work and one of > us > > > > >> > > can pick them up, thanks. > > > > >> > > Best Regards, Kelvin. > > > > >> > > > > > > >> > > On 01/12/06, Andy Grove (Contractor) > > > > >> > > <[EMAIL PROTECTED]> > > > wrote: > > > > >> > > > > > > > >> > > > > > > > >> > > > Hi Dan, > > > > >> > > > > > > > >> > > > I was previously working with Kelvin Goodson to donate > some > > > > > > >> > > > junit > > > > >> > > tests > > > > >> > > > on behalf of Rogue Wave Software. > > > > >> > > > > > > > >> > > > These tests are written purely to the SDO API and I > > > > >> > > > have > > > > validated > > > > >> > > that > > > > >> > > > the tests do run against Tuscany as well as Rogue > > > > >> > > > Wave's > > > > >> > > implementation. > > > > >> > > > > > > > >> > > > > > > > >> > > > Should I send the tests to Kelvin? > > > > >> > > > > > > > >> > > > Thanks, > > > > >> > > > > > > > >> > > > Andy. > > > > >> > > > > > > > >> > > > -----Original Message----- > > > > >> > > > From: Dan Murphy [mailto:[EMAIL PROTECTED] ] > > > > >> > > > Sent: 30 November 2006 17:44 > > > > >> > > > To: Tuscany Developers; Tuscany Users > > > > >> > > > Subject: Proposal for a (Java) community test suite for > SDO > > > > >> > > > > > > > >> > > > I would like to propose starting a community test suite > for > > > > > > >> > > > service > > > > >> > > data > > > > >> > > > objects (SDO CTS) implementations written in Java. > > > > >> > > > Based > on > > > > > > >> > > > feedback from an earlier post this seems to be the > > > > >> > > > first logical step in getting interoperable SDO > > > > >> > > > implementations > > > > > >> > > > in all languages. I can see this leading to an > > > > >> > > > interoperability test suite to check serialisation > between > > > > >> > > > implementations also works (across languages and > > implementations). > > > > >> > > > > > > > >> > > > Proposal for Community Test Suite (CTS) for SDO Develop > > > > >> > > > a > > > > > >> > > > test suite to validate an SDO implementation behaves as > > > > >> > > > expected, according to the community's understanding of > the > > > > > > >> > > > SDO > > > > >> specification. > > > > >> > > > Should > > > > >> > > > the specification appear ambiguous or unclear then the > > > > >> > > > community will decide what to do; it may decide to test > the > > > > > > >> > > > area with an agreed expected behaviour, or decide not > > > > >> > > > to test > > > this area. > > > > >> > > > Ambiguities will be fed > > > > >> > > back > > > > >> > > > to > > > > >> > > > the specification group for clarification. Although we > will > > > > > > >> > > > run this against Tuscany, the test suite will only test > > > > >> > > > things that > > > > we > > > > >> > > > > >> > > > think any implementation should support. > > > > >> > > > > > > > >> > > > The SDO CTS will enable developers to choose or switch > SDO > > > > >> > > > implementations without the concern of having to > > > > >> > > > re-code > a > > > > >> > > > significant proportion of their application due to > > > > >> > > > differences between implementations. This community > > > > >> > > > test suite will first focus on areas identified > > > > >> > > > important to developers of > > > > >> > > > > > > >> > > > SDO > > > > >> > > > applications. SDO users feedback and involvement will > > > > >> > > > be crucial to > > > > >> > > the > > > > >> > > > success of this effort. Over time this may grow to > include > > > > >> > > > a > > > > large > > > > >> > > > > >> > > > proportion of the SDO specification, however the suite > > > > >> > > > should > > > > grow > > > > >> > > > > >> > > > according to the community's desire, rather than > attempting > > > > > > >> > > > to be a validation > > > > >> > > or > > > > >> > > > compliancy suite. > > > > >> > > > > > > > >> > > > To encourage everyone with an interest in SDO to > contribute > > > > > > >> > > > and use > > > > >> > > the > > > > >> > > > suite, I propose we : > > > > >> > > > > > > > >> > > > 1. Create a separate module in SVN to separate this > from > > > > >> Tuscany > > > > >> > > > components and testcases. > > > > >> > > > 2. Make use of a java package namespace that is not > > > > >> attributable to > > > > >> > > > either Tuscany or any other SDO implementation: > test.sdo > > > > >> > > > 3. Refactor some of the existing Tuscany SDO Java > > > > >> > > > test > > > > > >> > > > cases > > > > to > > > > >> > > > > >> > > > remove > > > > >> > > > any Tuscany specific coding and re-package these to > the > > > > >> test.sdo > > > > >> > > > namespace. > > > > >> > > > 4. Accept tests from anyone who wishes to contribute > > > > >> > > > them > > > > under > > > > >> > > > > >> > > > normal > > > > >> > > > Apache contribution conditions. > > > > >> > > > > > > > >> > > > > > > > >> > > > SDO users involvement will be crucial to this effort, > > > > >> > > > developers of > > > > >> > > SDO > > > > >> > > > implementations will benefit by contributing to and > > > > >> > > > consuming > > > > > > > >> > > > a community test suite, rather than working on their own. > > > > >> > > > > > > > >> > > > Who's up for working on this with me ? > > > > >> > > > > > > > >> > > > If you are interested in joining this effort; have any > > > > >> > > > concerns, comments or suggestions please append them... > > > > >> > > > > > > > >> > > > Thanks in advance to all those who volunteer :) Dan > > > > >> > > > > > > > >> > > > > > > > ---------------------------------------------------------------- > > > > -- > > > > >> > > > --- To unsubscribe, e-mail: > > > > >> > > > [EMAIL PROTECTED] > > > > >> > > > For additional commands, e-mail: > > > > >> > > > [EMAIL PROTECTED] > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > -- > > > > >> > * * * Charlie * * * > > > > >> > Check out some pics of little Charlie at > > > > >> > http://www.flickr.com/photos/[EMAIL PROTECTED]/sets/ > > > > >> > > > > > >> > Check out Charlie's al crapo blog at > > > > >> > http://robbieminshall.blogspot.com > > > > >> > > > > > >> > * * * Addresss * * * > > > > >> > 1914 Overland Drive > > > > >> > Chapel Hill > > > > >> > NC 27517 > > > > >> > > > > > >> > * * * Number * * * > > > > >> > 919-225-1553 > > > > >> > > > > >> > ----------------------------------------------------------------- > > > > >> -- > > > > >> -- To unsubscribe, e-mail: > [EMAIL PROTECTED] > > > > >> For additional commands, e-mail: > > > > >> [EMAIL PROTECTED] > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > > - To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
