Re: status of the build

[Andrew Eberbach]

Hello,

There are three broad categories of
tests: unit tests, functional tests, and conformance tests. I'll quickly
describe what I mean by all three so that we're all clear (in case my terminology
might be off).

Unit tests: per-class tests that check
against boundary conditions in the implementation of classes. These tests
are as localized and self-contained as possible in one class, so as to
localize any failures quickly. These tests run locally (i.e. not over web
services, just simple java method invovation).

Functional tests: per-capability tests
that exercise higher level functionality. These tests verify that the behaviour
of the built-in Capability functions, as they are described in the standards.
Here, the tests can be run locally or "remotely" (ie. putting
the serialization + transport framework through its paces, while not necessarily
leaving the physical machine). Tests in this category touch several classes
if invoked on local instantiations of the capabilities and a whole slew
of classes, if invoked using webservices.

Conformance tests: These tests are low-level
protocol tests that are run against endpoints to make sure all encodings
and spec-level transport details are correct. While they share similarities
with functional tests, I'd like to keep them separate since they are much
more prone to spec changes, as they deal with very precise details of encoding
and interoperability between WSDM, SOAP, WSA, etc. implementations. 

I'd like to concentrate on the unit
and functional tests at first. 

A suitable framework for writing these
tests is junit. It is standard, has a lot of tooling, and integration (ant,
eclipse, etc) around it and is fairly generic. That said, some work must
be done in order to set up the scaffolding for invoking webservice calls
locally. This is not a huge issue as it can be handled with either embedded
tomcat or osgi as the transport stack. The idea being that simple endpoints
would be deployed by the scaffolding code providing an EPR to tests that
run in the same suite. Those tests could then take the EPR and create proxies
against it.

Here are some ideas for the layout of
the tests in the repository:

For unit tests:

Each top-level muse component (such
as muse-core, muse-tools, etc.) would have a corresponding "-test"
directory (muse-core-test, muse-tools-test, etc). This  directory
would have a structure mirroring the component it is testing (as is common
practice with junit). Each non-trivial class would have a corresponding
test suite and the test suites would be self-contained within the project.
This is so that users don't have to check out several "-test"
directories in order to run one  (dependencies become very tedious,
very quickly). 

For functional tests:

This would loosely follow Dan's current
approach of top-level directories that are related to specific capabilties
(wsrf, wsn, etc). The structure here isn't as easy to generalize, but some
directory/heirarchical organization that seems sane would suffice. Again,
the goal here is self-containment as to avoid an overly complex test environment.


From what I understand, maven provides
the mechanisms to support junit and interaction with a set of build libraries
natively. This seems ideally suited for the purpose.  However, since
junit is a standard, the test suites could easily be run using either ant
integration (junit task) or eclipse integration to run the test suites
with minimal set up and very nice reporting capbilities. 

Finally, since modifying something simple
is usually easier than generating something simple from scratch, I'd suggest
we create two "read only" top-level directories which would lay
out a reasonable skeleton for users contributing their own junit and functional
tests for top-level directories. 

This is a first pass and any input would
be appreciated, especially if it seems that the approach might be leaving
a platform, or set of platforms, in a difficult situation.


Thanks,
Andrew

Andrew Eberbach
Autonomic Computing
(919) 254-2645
T/L: 444-2645
[EMAIL PROTECTED]

> Joel! Welcome back to the program!
> 
> I wonder if this is a JDK issue - I thoug that the basic XPath APIs
were 
> part of the core JDK and that Xalan was not required, but I suppose
that, 
> like JAXP/DOM, there may be some discrepancies. Or I may just be wrong.
My 
> dev JDK doesn't have Xalan in /lib/endorsed, but I have five JDKs
on my 
> system, so I will look into that some more. In the meantime, I will
update 
> the build with your fix.
> 
> The test cases (unit and other) bring up an important topic. In the
past, 
> I have kept all tests/test projects as separate projects or modules.
They 
> had a copy of the latest jars from the build, but there were no 
> dependencies between the test projects and the source tree. I did
this to 
> make sure that new people could come in and run the tests (and tweak
the 
> tests) without setting up the full build.
> 
> I would like to consider this as we decide how to set up and record
the 
> tests in /trunk. Our current test/sample projects are stored in a
way that 
> is easy to ship but may not be optimal for daily tests. I'm open to

> suggestions. Andrew, I know you've thought a lot about the test harness
in 
> general - would you like to make a proposal before I make my own?
Joel, 
> you may have (will have) OSGi-specific things that need to be considered

> when setting up the test harness, so be sure to educate us if we're

> oversimplifying for the Axis2/POJO scenario.
> 
> Dan
> 
> 
> 
> 
> "Hawkins, Joel" <[EMAIL PROTECTED]> wrote
on 06/27/2006 02:40:45 
> PM:
> 
> > Dan,
> > 
> > I hit a snag when trying to build xml-utils. I found that the
following
> > dependency was needed in the pom in order to build the XPath
utility
> > classes.
> > 
> >     <dependency>
> >       <groupId>xalan</groupId>
> >       <artifactId>xalan</artifactId>
> >       <version>2.7.0</version>
> >       <scope>compile</scope>
> >     </dependency>
> > 
> > Once I tossed that in, everything went smooth. Am I missing something,
> > or do you have more mojo in your environment than I do? :-)
> > 
> > Starting to work on the OSGi build stuff now. Any ideas on how
you'd
> > like to handle unit test cases, etc.? 
> > 
> > Thanks,
> > Joel
> > 
> > 
> > 
> > -----Original Message-----
> > From: Daniel Jemiolo [mailto:[EMAIL PROTECTED] 
> > Sent: Thursday, June 15, 2006 5:43 PM
> > To: muse-dev@ws.apache.org
> > Subject: status of the build
> > 
> > I have created a Maven-based build for each of the source code
modules
> > in 
> > /trunk/modules. There is a build-all.bat file that ties these
together 
> > because it does not appear that Maven 2.0 allows users to tie
together 
> > multiple project files into one aggregate project. If I'm wrong
about 
> > this, I'd be happy to make the change necessary to create a "pure"
Maven
> > 
> > build.
> > 
> > Because of Maven's "one jar file per project" rule,
I have split some of
> > 
> > the WS-* modules into two projects - one for API, one for
> > implementation. 
> > This required a change to the projects' directory structure,
so the code
> > 
> > for those projects will have to be re-commited (under the new

> > directories). I apologize ahead of time for the extra noise on

> > muse-commits. I originally thought that Maven would allow me
to generate
> > 
> > multiple jars from a single source tree, but answers to past
queries on 
> > users@maven.apache.org indicate that "one jar file per project"
is the 
> > only way to go.
> > 
> > My next steps are:
> > 
> > 1) add the .properties files containing error/log messages to
the build 
> > jars 
> > 
> > 2) figure out how to get Maven to stop including its metadata
files in
> > the 
> > build jars 
> > 
> > 2a) if #2 is not possible, update the build-all.bat file to strip
them
> > out 
> > manually
> > 
> > 3) add the test projects to the build - I'll do this in a way
that is 
> > similar to other ws.apache.org projects
> > 
> > 
> > Dan
> > 
> > 
> > 
> > Dan Jemiolo
> > IBM Corporation
> > Research Triangle Park, NC
> > 
> > 
> > +++ I'm an engineer. I make slides that people can't read. Sometimes
I
> > eat 
> > donuts. +++
> > 
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> > 
> > The contents of this e-mail are intended for the named addressee
only. 
> It 
> > contains information that may be confidential. Unless you are
the named 
> > addressee or an authorized designee, you may not copy or use
it, or 
> disclose 
> > it to anyone else. If you received it in error please notify
us 
> immediately 
> > and then destroy it. 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> > 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>