Wow ... I was just nagging BJ Hargrave about exactly this subject when
he told me you had written this mail. This topic is very close to my
heart. The OSGi metadata is so rich that it could make the build
process a lot easier if it could take advantage of it. Unfortunately,
maven's model is much more simplistic and can not take advantage of
this rich model. On the other hand of the spectrum we have Eclipse
which is a fantastic tool, but is not very suited for continuous
builds.
For the OSGi Alliance we have setup a build system that runs
continuous. For statistics, we have 114 projects but we generate a
total of more than 400 bundles. This large number is caused by the
test suite that has a lot of embedded bundles for testing. The build
takes a couple of minutes on a laptop (6 minutes for a clean build).
We build 4x a day: an incremental build and a full clean build. The
full clean build automatically runs the test harness on all bundles.
We had the requirement that it should be easy for people to use
Eclipse, but other tools were allowed as well. We use ant as the
"shell". However, the heavy lifting is done by btool, our own bundle
tool. Btool is the predecessor of bnd.
Paramount for us was to centralize our build rules. We therefore have
a build.xml that only includes our centralized build.xml. There are
only two projects that add extra rules, all other projects describe
the target bundles with a property file. This property file submits
the following primitives: include packages (merged or specific) from
the classpath or include resources from the file system (with renaming
and preprocessing). Notice that packages can come from other projects
as well as any JAR/directory on the classpath.
We use the Eclipse .classpath file for project dependencies classpath
settings. The ant task reads this file and ensure projects are build
in the right order, as defined in the .classpath file. The btool also
reads this file and use the classpath to find the sources for the
packages. btool also discovers version information in manifest of jars
on the classpath or packaginfo files in the package directories. This
means you do not have to maintain the versions of the jar by hand.
The central build.xml is quite small: 288 lines with extensive
comments. I think this was achieved by being fanatic about not having
rules in the project directories and parameterizing everything with a
set of properties in the project.
We could automatically discover new projects, but we found out that it
is nice to be able to discard a project when the authors are
struggling with some issues (translation: they screwed up the build).
We therefore have a central file with all the active projects. This
file is also used to create larger distros.
We are currently thinking what we should do for the next release. We
went reluctantly to Ant for R4. The only reason we left make for Ant
was that Ant was "portable"; looking back I think a lot of things were
a lot harder with ant than with make. I am pretty sure we will make
the move to maven due to its complexity and because it is not flexible
enough.
As you might have read in the previous paragraphs, I am a sucker for a
standardized project directory layout. However, trying to standardize
this over the industry is a bit too ambitious imho. Needs differ. For
some the maven structure is overkill, for others it is just not
enough. The impedance mismatch between Eclipse and maven is for
example awkward because neither tools is flexible enough to adjust.
So I have been looking around as well for improvements in the build
structure. For one, it will likely center around the bnd tool. This
tool is already part of the osgi plugin for maven; it calculates the
manifest and it uses a pull model instead of the traditional push
model. Instead of creating a directory structure and jarring it, the
bnd tool reads a recipe and collects the files from this recipe and
puts them in the bundle. This saves a lot of disk space and processing
time.
So , lets imagine.
I have been thinking for a long time that ant is overkill and actually
quite bad in what it is supposed to do. At least make had dependency
management, in ant each task must do that for itself. As a script
language, well lets not go there. As a file manipulation tool it had
some interesting ideas but, again, the tasks have to support all the
options that a simple shell could standardize. That is redirecting,
filtering, etc. Ant has become an integration tool but it is a very
bad one.
So I think it is time to look for a good bundle build tool that
combines the good features of Eclipse, maven, ant, and make without
having the flaws that each brings to the table. A short list of my
requirements would be:
- Centralized rules. Default project files should have an empty build
file. There are many projects, and only one central place. Complexity
should be concentrated centrally. No Redundancy.
- Full script language with lots of primitives to work with the file
system.
- Features must be orthogonal. For example, in the shell you have
piping that allows you to combine small functions that do one thing. A
build tool should not require the plugins to create the same
functionality over and over again like ant or maven. In maven, almost
any Apache project has its own plugin and plugins are often not used
between projects. This is a bad sign.
- Support for pluggable repositories. The concept of a repository is
cool, it is the killer app of maven. However, the maven repository is
too limited. Bundles contain a lot of metadata and a repository
should be able to leverage this. I'd love to have OBR as a repo.
- A full blown programming language. I want the projects to be clean
and declarative. However, when I need to do something special I want
to be able to use all the facilities of a programming language and
not have to create special tasks to implement loops, conditionals, or
functions.
- Clean syntax, especially for property handling. <property name="x"
value="123"> is, well silly, when you could write x = 123. I think
syntax is important for this. If you read a build file, the
consequences should be crystal clear and not be cluttered by cruft.
- The repo should not require additional metadata and checksums.
This is all already handled by the OSGi meta data.
- It should be easy to add exceptional code to a build file shared by
many projects, as well as for a single project
- Fast
- Be able to create many bundles per project, as well as distros
containing other artifacts.
- Minimize recompiles and jar building by checking dependencies and
looking at the time stamps (remember make?)
- Minimal output during run (just to see progress), but maximal
output when there is a failure. Errors should be easy to diagnose
- Easy to extend with processor that create one type of resource to
another without reverting to a mile long phase list like maven. For
example, use javacc to create the grammar classes, or other
preprocessor.
- Automatic generation of manifest headers. With the option to have
minimal redundancy
- Compiling against different versions of the same packages
- Running test cases
- Deploying to a repository
- Supporting multiple languages
- Integration with SVN, CVS, Clearcase ( if necessary)
- Runs flawlessly on Windows, Linux, Mac, and others
- Remain lean and mean! No bloat!
Hmm. I wish there was a customer that hired me to develop this tool
:-( A build tool is so incredibly important, but so few companies are
willing to spend money on it ...
Anyway. I think this an interesting subject for all of us. If you are
interested we could create an RFP in the OSGi and all contribute with
requirements as I have given here. Maybe we can then find a number of
companies that are willing to spend resources on such a project?
I talk to lots of different companies and their major problems is
surprisingly often the build environment. When I do consultancy on
projects, it is often the first aspect I investigate. It is most of
the time a complete disaster.
So who would be interested in working on creating an RFP on this
subject?
Kind regards,
Peter Kriens
JV> Greetings,
JV> We're looking to re-implement/replace our current continuous
JV> integration system, which has been building our OSGi project for the
JV> past 10 months or so. The current builder was quickly hacked together
JV> over a period of a week, and we're starting to run into its
JV> limitations now that our project has surpassed 150 bundles.
JV> Some of the big requirements that we set for our current system were:
JV> 1. Automatic discovery of new bundle projects in the source
JV> repository. We didn't want to configure the builder each time we
JV> added a new bundle to the project.
JV> 2. Build as much as needed, but no more. Based on what was changed
JV> since the previous build, we wanted to use OSGi dependency analysis to
JV> figure out what needed to be built (i.e., changed bundles and bundles
JV> depending upon them). Because of this need, the builder must be able
JV> to fetch dependencies required for build from our bundle repository.
JV> 3. Resolve build dependencies based upon OSGi metadata (not, for
JV> instance, Maven dependencies).
JV> 4. Build Eclipse PDE bundle projects, since that's what we use as our
JV> development environment.
JV> To cover these requirements, we wrote a custom continuous integration
JV> system consisting of:
JV> 1. A repository scanner (written using JavaSVN) able to detect new
JV> projects in our repository; and
JV> 2. A bundle builder based around Eclipse PDE export operations.
JV> We did not use the PDE build system but rather do the equivalent of
JV> launching an Eclipse instance, selecting bundles and invoking "Export
JV> Deployable plug-ins and fragments".
JV> This system has worked fairly well for us. However, it's becoming
JV> slower and slower, and it is not easily extensible. We'd like to be
JV> able to make use of some existing builder for integrated
JV> testing/reporting/etc.
JV> I've noticed that a lot of people are using Maven to build OSGi
JV> bundles. However, as far as I can tell, Maven requires its own
JV> dependency format, and cannot be extended resolve based upon OSGi
JV> dependency metadata. Please correct me if this is incorrect.
JV> Does anyone have any systems similar to this, or any pointers to build
JV> systems that can be easily extended to support these types of
JV> features?
JV> Thanks,
JV> Jeremy Volkman
JV> _______________________________________________
JV> OSGi Developer Mail List
JV> [email protected]
JV> http://www2.osgi.org/mailman/listinfo/osgi-dev
--
Peter Kriens Tel +33467542167
9C, Avenue St. Drézéry AOL,Yahoo: pkriens
34160 Beaulieu, France ICQ 255570717
Skype pkriens Fax +1 8153772599
_______________________________________________
OSGi Developer Mail List
[email protected]
http://www2.osgi.org/mailman/listinfo/osgi-dev
