A few weeks ago I mentioned that I was going to write some email about a build system that I had been working on. Unfortunately due to some of the entity locking & jbossmq problems I was unable to focus much time on actually writing that email. Since then I have found that there is always something else todo besides write that mail, so I figure I just have to sit down and do it. A build system should be easy to use. It should also be configurable on a per-user and per-project basis and have a mechanism to group source modules together, produce release archives, make coffee in the morning and take care of you when you get sick. Well, I think I have a suitable solution to *most* of this; I still haven't got the coffee part done yet, and if you are sick enough you might think the build system is taking care of you. Over the years I have been working with build systems, first based on clumsy usage of make, to more advanced usage, then on the ant and now here I am. My first build system (build-foo) was based on something that we used at Mpath Interactive, called build-glue. When I moved over to Ant I renamed it to BuildMagic. Over the last six months I have been learning to adapt to some of the more ridiculous problems created by Ant as well as embracing the elegance that it provides. The drive to redesign BuildMagic was based on its lack of ability to aggregate modules together in a unified, manageable fashion (which I need for open source libraries that I produce, the increasingly complex software modules which I have to deal with at work). Ok enough babel, here is a brief overview: BuildMagic is a set of extension tasks for Ant 1.3 as well as a common configuration and build output structure. There are two major "things" in BuildMagic: Projects and Modules. A project is an aggregation of modules. It controls the configuration and execution of the modules build system. It provides a mechanism to invoke a group of modules and to collect there output to produce a project release. If anyone as played with NetBeans, some of the module configuration inspiration came from looking at there build system in depth. A module is something that contains source files to compile, documents to perform filter substitution on and other such things. The distinction between the two is important. One key feature of BuildMagic is how configuration properties are processed. Ant 1.3 does not currently have any native late binding property; BuildMagic *fixes* this problem. It also provides some logic and looping tasks, but I just implemented those for fun really. * * * I will go in more detail shortly, but for those of you who would like to see it in action you can look at Bliss: http://sourceforge.net/projects/bliss/ cvs -d:pserver:[EMAIL PROTECTED]:/cvsroot/bliss login cvs -z3 -d:pserver:[EMAIL PROTECTED]:/cvsroot/bliss get -r bliss_0_9_1 bliss You can check out the main branch, but it is not guaranteed to compile, due to some re-factoring. export JAVA_HOME=/where/the/jdk/lives cd bliss/build ./build release If all goes well you will end up with a output/bliss-0.9.1-*.jar file under build, which contains the modules api and jar file for the common module. * * * Each project is based in a single root directory (you could probably modify the build files to get around this, but I would not recommend it). Each project looks something like this: project_root/ build/ source1/ source2/ tools/ thirdparty/ project_root, source1, source2 and build are arbitrary. tools contains libraries, scripts and whatever is used by the build system (and only the build system) and thirdparty contains libraries scripts and whatever that is used commonly between all of the source modules. The distinction between tools and thirdparty is subtle, but necessary. Consider a project that used ant 1.3 to build, but requires ant 1.1 to function (for whatever reason that might be). Both the tools and thirdparty directories are laid out like this: <package_name>/<package_files> package_name contains no version numbers or vendor names, though I suppose you could encode vendor names if you liked. So for example if a project depends on JMX v1.0ri you would want something like this: thirdparty/jmx/lib thirdparty/jmx/lib/jmxgrinder.jar thirdparty/jmx/lib/jmxri.jar thirdparty/jmx/lib/jmxtools.jar With Bliss I have chosen to exploit the CVSROOT/modules file to build the tools and thirdparty structure. Check out the modules file for a better idea: cvs -d:pserver:[EMAIL PROTECTED]:/cvsroot/bliss get -c There are ups and downs to this, one of the downs is that it makes it hard to manage additions and removals of thirdparty modules with branch specs. Though and advantage is that if you have more than one project that depends on a version of a thirdparty library, you only have to update the library once (instead of for each project). Any ways, as long at the name space ends up the same it does not really matter how the files get there. Tools is a bit different, only that you will want to leave the ant and buildmagic directory structure as is, otherwise you will need to update all of the scripts. The build directory above is a special module; called the project module. It contains the build logic to call source modules and such. Each project will probably have one of these directories... I thought about what I should name if for a while and ended up calling it 'build'. The basic anatomy of a project or source module looks like this: build.xml config.xml etc/local.properties-example src/main src/whatever output build.xml is the main ant build file which you are familiar with. At the top there is a comment on the configuration files, which I will copy for you here. For a project module: Configuration Files (in order of precedence): ${project.config}/build.xml* (this file) ${project.config}/override.properties ${user.home}/.buildmagic.properties ${user.home}/.ant.properties ${project.config}/local.properties ${project.config}/config.properties ${project.config}/config.xml* * Required For a source module: Configuration Files (in order of precedence): ${module.root}/build.xml* (this file) ${project.config}/override.properties ${module.root}/override.properties ${user.home}/.buildmagic.properties ${user.home}/.ant.properties ${project.config}/local.properties ${module.root}/local.properties ${project.config}/config.properties ${module.root}/config.properties ${module.root}/config.xml* * Required config.xml is required in both cases (as it is included via the DOCTYPE). It is meant to hold most of the configuration properties for the project/module as well as some hooks to configure and initialize the build system. The point was here to separate the actions for the configuration. local.properties (which will get copied and filtered from etc/local.properties-example on the first run or when the configure target is run) is meant to allow users to override any of the properties set in build.xml or config.xml. One very common thing that I find users like to override is whether or not to use Jikes, so I generally include an example file with something like this: #build.compiler=jikes #build.warnings=true #build.pedantic=true #javac.depend=on So that users can just uncomment to enable Jikes. Because BuildMagic enables late property binding, you can define a default in config.xml and users can override that in local.properties without having to modify config.xml. One of the reason for this was that I found that many engineers were checking in there modified build files, which would break the build for other users. Generally config.xml sets up path's and properties, which are used by targets in build.xml. At this point if you have not looked at bliss, you might want to, since I think it might be a little silly to copy all of the config over. If you will notice the order in the above list, you will notice that properties set in build/local.properties will override those in a those of a source module. This makes it really convenient to enable jikes in one location (or javac.debug) and have it propagate to all source modules. The main targets for both the project and source modules are: all Builds everything. clean Cleans up most generated files. clobber Cleans up all generated files. compile Compile all source files. configure Configure the module build system. docs Builds all documentation. install Installs released files into project name space. jars Builds all jar files. main Executes the default target (jars). release Builds a release distribution. tests Execute all tests. The project module does contain targets like compile, but is configured to pass that target name on to all of the modules which it is configured to execute. One important rule of the current system is that all modules are peers (at a directory level... even the project module). This simplifies the configuration for source modules to locate the thirdparty, tools and build directories with out having todo anything fancy. All output from a module goes into a sub-directory called 'output'. So when <javac> compiles stuff it usually will go into output/classes and <javadoc> into output/api. When a module is released the files generated from the compile target are copied into output/release. When building a release from the project module, files are compiled, then they are installed into the output/release directory of the project module. This way the module is in control of what goes into the release, or more so, the project does not need to keep track of what is being released (as that might change from version to version of the source module). When generating releases, a build number and id is constructed to allow for each archiving of builds. For development this id can be overridden in local.properties (to avoid creating new jars each time). * * * If any of this sounds interesting, and you have not looked over the files from bliss, you really should take a look. There is some other "magic", but I will not get into that right now. A small note about .bat/.cmd files. I have not even tried to construct these and probably will not, since it just complicates things. Win32 users should use Cygwin to gain unix compatibility. If there is a huge problem with that let me know and we can discus it further. Ok, my hands are getting tired from all this non-code typing, so I am going to finish this up with a brief rundown for how I think we could migrate jboss to this system. But before that another brief note that this system is not perfect, so if you find any problems please be patient and let me know. I will get it fixed, but it will never be perfect... not until I re-write an ant-like thing that doesn't anger me so much. If you look at the cvs organization of bliss, you will notice that I have nested things one level deeper than JBoss. This was just so I could have a name space separator for more than one project. I would recommend the same, but I understand that it is difficult to change at the moment. In either case I would recommend a minor name space change (either at the cvsroot level or by using the modules files with -d. Lets say you wanted to check out everything needed to build/run jboss, you might do something like this: cvs get jboss and the result might look like this: jboss/ jboss/build jboss/server jboss/cx jboss/sx jboss/mq jboss/pool jboss/test jboss/naming jboss/admin jboss/varia jboss/tools jboss/thirdparty It might be missing a few. I think that the website related stuff might be better suited for a separate project, if there was a cvsroot re-org then top level would look like this: jboss/ website/ Any ways. In the first one, the current jboss would map to server, since that is what it is (or should end up, the core bits that make up the server), then each of the current jbossXX or XX would get a directory under them. The modules would not have to change much, just updating the build control files (build.xml, config.xml, etc/local.properties-example) to conform to the requirements. Another benefit of this cvs modules organization is that it will make tagging much easier (we can still do that now, but we would need to setup a module that would map all other modules together). * * * So I propose that we move JBoss to this system. I think that there is nothing to loose and a lot to gain. That said, I think that the best way to move forward with this would be if I set up a newjboss (or uniquely named) module in CVSROOT/modules that will create the correct environment, then branch that to a pre-buildmagic branch. I will then update the build systems on each module, create the project module and otherwise get the system to a state where anyone can check-out and build the *entire* system by (cd jboss/build; ./build). I will keep this branch uptodate from main, so that when I am done (and it is proven to work), we can merge from this branch back to main and get on with our collective open source lives. Questions? Comments? Please let me know. --jason _______________________________________________ Jboss-development mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-development
