On 1/16/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote: > Of course, although I'd be interested to see how you do fine-grain > testing of the internals, which in my opinion is very useful, as the > promises made by private and package access code to other class-level > and package-level code is just as important to the promises made by > public APIs to users.
I fully agree with this note. In addition, I also think that the distinction should be made between the conformance tests those purpose is to exercise the publicly visible API, and the unit tests, those purpose is to exercise "all functions and methods so that whenever a change causes a regression, it can be quickly identified and fixed" (definition taken from http://en.wikipedia.org/wiki/Unit_testing). By their definition, unit tests are aimed to check every piece or module of the code, including the private one. It is the purpose of the unit tests to keep the control over the package or class internals such that other pieces of the same package or class are not unintentionally broken during a change. This really helps when the multiple people or teams are working on the same package or class. For Harmony, the need of having unit tests for the package private functionality can be driven by the fact that the multiple classes may interact with each other through the package private API's. There is no way to guard against changes in the package private API's except that (a) use reflection/JNI or (b) have tests in the same package. I believe option (a) is just extremely inconvenient. Thank you, Andrey Chernyshev Intel Middleware Products Division > > > > > In the interests of avoiding any religious war can I just say that there > > is no religious war here : both unit test approaches work but neither > > should we look on one as being "the only way". So let's not argue about it > > on the list. Instead, let's see how the unit test code evolves over time > > before drawing any conclusions. > > Right - feel free to comment on the other message, titled "Test > Framework". The initial discussion surrounding same-package unit tests > stemmed from having to run on the boot classpath, IMO an important > consideration. However... > > geir > > > > Best regards, > > George > > ________________________________________ > > George C. Harley > > > > > > > > > > Geir Magnusson Jr <[EMAIL PROTECTED]> > > 16/01/2006 12:54 > > Please respond to > > harmony-dev@incubator.apache.org > > > > > > To > > harmony-dev@incubator.apache.org > > cc > > > > Subject > > Re: Test suite layout > > > > > > > > > > > > > > > > > > George Harley1 wrote: > >> Hi, > >> > >>> This is not the goal of unit tests. They do not test API, they test the > > > >> code, > >>> including package-access members etc. That is why unit tests must > > reside > >>> in the same package as the classes. > >> This is radical stuff. I have never heard the argument before that unit > >> tests *must* share the same package name as the type under test. One > >> important reason why many think it is a bad idea to take this approach > > is > >> that the resulting unit test code becomes too closely coupled with the > >> code under test. > > > > Huh? It's a common practice - it gives the unit test the ability to > > really test the implementation because it has package level priv. > > > >> Think about it ... > >> > >> If unit test code can access every package-access member and method in a > > > >> given type then the test code will eventually reach a point where it > >> contains a lot of brittle assumptions about internal implementation. > > > > It can, yes. OTOH, it is testing internal private methods, so yes, > > there's going to be those assumptions built in. > > > >> Consequently, when implementation changes occur, the corresponding test > >> code will break. Note : I'm not talking about API changes breaking the > >> test code here, I'm talking about changes to protected and > > package-access > >> stuff which - by definition - the developer has chosen to keep > > non-public. > > > > Right - and the unit tests test that code. How else do you test it in a > > fine-grained way? Only via public methods? > > > > > >> So as the internals change the unit tests break. And further internal > >> changes make the unit tests break. And ultimately maintaining the tests > >> with close-coupling to the internals of the type under test becomes a > > bit > >> of a pain and their overall value in the test harness diminishes because > > > >> minor refactorings to the type-under-test now take so darned long to > > carry > >> out. > > > > But it gives you very fine-grained coverage, if you want that. > > > > geir > > > >> > >> Best regards, > >> George > >> ________________________________________ > >> George C. Harley > >> > >> > >> > >> > >> > >> Mikhail Loenko <[EMAIL PROTECTED]> > >> 16/01/2006 11:58 > >> Please respond to > >> harmony-dev@incubator.apache.org > >> > >> > >> To > >> harmony-dev@incubator.apache.org > >> cc > >> > >> Subject > >> Re: Test suite layout > >> > >> > >> > >> > >> > >> > >> On 1/16/06, Tim Ellison <[EMAIL PROTECTED]> wrote: > >> ... > >>> If your unit tests are intended to test API, then they should be > > calling > >>> the API in the same manner that an application will call the API. Just > >> This is not the goal of unit tests. They do not test API, they test the > >> code, > >> including package-access members etc. That is why unit tests must reside > >> in the same package as the classes. > >> > >> Other kinds of tests like compatibility or functional tests call just > >> public and protected methods so they can be in different packages. > >> > >> Thanks, > >> Mikhail Loenko > >> Intel Middleware Products Division > >> > >> > >> > >> > >> > >>> look at the number of places where we check to see if the classloader > > == > >>> null to see where this matters. > >>> > >>>> Some of them may depend on security manager component if > >>>> you like to run a test in some security restricted environment. But it > >>>> is not a big issue: the security manager component can be configured > >>>> dynamically and you should install your custom security manager before > >>>> running a test. > >>> Yes, we should do that on the application classloader to test security > >>> manager functionality. > >>> > >>> Regards, > >>> Tim > >>> > >>>> What do you think? > >>>> > >>>> Thanks, > >>>> Stepan Mishura > >>>> Intel Middleware Products Division > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: Tim Ellison [mailto:[EMAIL PROTECTED] > >>>>> Sent: Thursday, January 12, 2006 8:04 PM > >>>>> To: harmony-dev@incubator.apache.org > >>>>> Subject: Re: Test suite layout > >>>>> > >>>>> Geir Magnusson Jr wrote: > >>>>> > >>>>>> Tim Ellison wrote: > >>>>> <snip> > >>>>> > >>>>>>> We would have written it as java.io.tests, but the java.<whatever> > >>>>>>> namespace is reserved, so the formula is simply > >>>>>>> > >>>>>>> <package>.<type> -> org.apache.harmony.tests.<package>.<type>Test > >>>>>> Ug - then you have the problem of not being in the namespace as what > >>>> you > >>>> > >>>>>> are testing. > >>>>>> > >>>>>> THat's why people use parallel trees - so your test code is > >>>> physically > >>>> > >>>>>> separate but you have freedom of package access. > >>>>> For 'normal' application code then you can do this, but since we are > >>>>> writing the java packages themselves then you come unstuck because > > the > >>>>> java packages have to run on the bootclasspath, and the tests on the > >>>>> application classpath. > >>>>> > >>>>> You don't want to run all your tests on the bootclasspath (because > >> then > >>>>> it would not be subject to the same security sandboxing as > >> applications > >>>>> using the APIs you are testing); and you cannot put java.<whatever> > > on > >>>>> the application classpath because the VM will catch you out (you'd > > get > >>>> a > >>>> > >>>>> linkage error IIRC). > >>>>> > >>>>> > >>>>>>> This makes it clear what is being tested, and where to add new > > tests > >>>>> etc. > >>>>> > >>>>>> So would > >>>>>> > >>>>>> test/org/apache/harmony/io/TestFoo.java > >>>>>> > >>>>>> (to test something in org.apache.harmony.io, and arguable to test > > the > >>>>>> Foo.java class in there. (or TestFoo.java - it's early - no coffee > >>>> yet) > >>>> > >>>>> Not sure what you are saying here... For java.<whatever> packages we > >>>>> need a prefix on the test packages to keep the VM happy, but for > >>>>> org.apache.harmony packages we can have either pre- or post-. > >>>>> > >>>>> I'd actually prefer a postfix of .tests for non-API packages, though > > I > >>>>> can understand if people object to the inconsistency; so > >>>>> > >>>>> org.apache.harmony.tests.java.io.FileTest.java <- test API > >>>>> org.apache.harmony.io.tests.FileImpltest.java <- test public methods > >>>>> in our IO impl' > >>>>> > >>>>> > >>>>>> Simiarly > >>>>>> > >>>>>> test/java/util/TestMap.java > >>>>>> > >>>>>> > >>>>>>> Then within the test class itself the methods are named after the > >>>> method > >>>> > >>>>>>> under test, with a familar JNI-style encoding, so we have things > >>>> like: > >>>> > >>>>>>> org.apache.harmony.tests.java.io.FileTest contains > >>>>>>> public void test_ConstructorLjava_io_FileLjava_lang_String() { > >>>>>>> ... > >>>>>>> } > >>>>>>> > >>>>>>> and > >>>>>>> > >>>>>>> org.apache.harmony.tests.java.lang.FloatTest contains > >>>>>>> public void test_compareToLjava_lang_Float() { > >>>>>>> ... > >>>>>>> } > >>>>>> ...or whatever the convention is for JUnit. I think that's one of > >>>> the > >>>> > >>>>>> nice things about TestNG, is that it's annotated, so you have the > >>>>>> freedom there. > >>>>>> > >>>>>> > >>>>>>> If the test is added due to a regression, then it is put into the > >>>> right > >>>> > >>>>>>> place in the test suite, and flagged with a comment (i.e. a > >>>> reference to > >>>> > >>>>>>> the Harmony JIRA number). > >>>>>> Yes - and I'd even advocate a parallel directory there too so that > >>>> it's > >>>> > >>>>>> clear that the regressions are different, but whatever. The only > >>>> snag > >>>> > >>>>>> there is name collision with the classes. > >>>>> I thought we'd agreed that 'regression' was not a useful > >> classification > >>>>> within the test suite layout ... > >>>>> > >>>>> > >>>>>> I think that a simple comment is enough. If we want to get cute, > >>>> maybe > >>>> > >>>>>> a javadoc tag so we can manage mechanically in the future. > >>>>> ok -- do you have a usecase in mind? > >>>>> > >>>>> Regards, > >>>>> Tim > >>>>> > >>>>> > >>>>> > >>>>>>> George Harley1 wrote: > >>>>>>> > >>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> I think that regression tests should be marked in some way. > >>>>>>>> > >>>>>>>> Agreed. But can we please *resist* the temptation to do this by > >>>>>>>> incorporating JIRA issue numbers into test case names (e.g. > > calling > >>>>>>>> unit test methods test_26() or test_JIRA_26()). I've seen this > > kind > >>>>>>>> of approach adopted in a couple of projects and, in my experience, > >>>> it > >>>> > >>>>>>>> often leads to the scattering of duplicated test code around the > >>>> test > >>>> > >>>>>>>> harness. > >>>>>>>> > >>>>>>>> Better, methinks, to either create a new test method with a > >>>>>>>> meaningful name or else augment an existing method - whatever > > makes > >>>>>>>> more sense for the particular issue. Then marking certain code as > >>>>>>>> being for regression test purposes could be done in comments that > >>>>>>>> include the URL of the JIRA issue. Perhaps an agreed tag like > >>>> "JIRA" > >>>> > >>>>>>>> or "BUG" etc could be used as an eye-catcher as well ? > >>>>>>>> e.g. > >>>>>>>> // BUG http://issues.apache.org/jira/browse/HARMONY-26 > >>>>>>>> > >>>>>>>> > >>>>>>>> My 2 Euro Cents. > >>>>>>>> Best regards, George > >>>>>>>> ________________________________________ > >>>>>>>> George C. Harley > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> "Mishura, Stepan M" <[EMAIL PROTECTED]> 12/01/2006 04:56 > >>>>>>>> Please respond to > >>>>>>>> harmony-dev@incubator.apache.org > >>>>>>>> > >>>>>>>> > >>>>>>>> To > >>>>>>>> <harmony-dev@incubator.apache.org> > >>>>>>>> cc > >>>>>>>> > >>>>>>>> Subject > >>>>>>>> RE: regression test suite > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Hello, > >>>>>>>> Tim Ellison wrote: > >>>>>>>> [snip] > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> What is the useful distinction for regression tests being kept > >>>>>>>> separate? > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> I can see that you may distinguish unit and 'system-level' tests > >>>> just > >>>> > >>>>>>>>> because of the difference in frameworks etc. required, but why do > >>>> I > >>>> > >>>>>>>> care > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> if the test was written due to a JIRA issue or test-based > >>>> development > >>>> > >>>>>>>> or > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> someone who get's kicks out of writing tests to break the code? > >>>>>>>>> > >>>>>>>> I agree that separating regression tests doesn't make sense. > >>>>>>>> However I think that regression tests should be marked in some > > way. > >>>>>>>> This will signal a developer that a test was created to track > >>>> already > >>>> > >>>>>>>> known issue. IMHO, a regression test should point out to a bug > >>>> report > >>>> > >>>>>>>> and a bug report (after resolving a bug) should contain a > > reference > >>>> to > >>>> > >>>>>>>> corresponding regression test in repository. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Stepan Mishura > >>>>>>>> Intel Middleware Products Division > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>> -- > >>>>> > >>>>> Tim Ellison ([EMAIL PROTECTED]) > >>>>> IBM Java technology centre, UK. > >>> -- > >>> > >>> Tim Ellison ([EMAIL PROTECTED]) > >>> IBM Java technology centre, UK. > >>> > >> > >> > > > > > > >
