On Apr 25, 2009, at 8:33 AM, Ralf Wildenhues wrote:
I hardly know scons. What's cool about it that autotools don't have?
This is worthy of its own thread, so I re-subjected the mail.
I worked up a prototype usage of scons in OMPI a few years ago. It
was actually fully functional -- it built (and installed? I don't
remember, but I assume so) a usable OMPI. The work is still available
here:
https://svn.open-mpi.org/trac/ompi/browser/tmp-public/scons
The OMPI source tree was quite different back then; that work would
need to be significantly updated for it to be relevant to today's OMPI
SVN trunk.
Here's what I remember liking (and not liking) about scons. Take this
with a big grain of salt -- this was a few years ago; both my memory
may be bad and things may (likely) have changed in scons since then (I
have not kept up with scons since). Note that *all* of this is IMHO.
Likes:
+++ Treating the whole tree as a single entity: with a traditional
"make" setup, you have a standalone Makefile in each directory in your
tree, and upper-level Makefiles effectively "cd subdir; $(MAKE)".
scons treats your entire source tree as one entity (even though you
can have multiple scons data files scattered throughout the tree --
you don't have to have everything in a single, top-level file).
Specifically: scons will analyze the entire tree at once, decide what
to build, and build it. There's no notion of recursive traversal.
Sidenote: of course, you can implement such a system with Make as
well, but, IMHO, make is not well setup to handle that (especially for
very large source trees). Automake does it a bit better, but it would
still be cumbersome to have (effectively) one Makefile.am for the
entire OMPI tree (meaning: no recursive traversal, even if we had a
single top-level Makefile.am + lots of Makefile.include's scattered
around the tree).
+++ I was writing [python] code to build OMPI. This was incredibly
natural; significantly easier than writing rules with embeded bourne
shell scripts (sometimes with extremely awkward quoting). Some of my
friends / colleagues disagreed with me -- they worried that you would
then need to QA the build system (because it's developer-written
code). But I contend that we have to do that today already (look how
much m4 code is in contrib/*m4 and elsewhere in the OMPI tree!).
+++ I had subroutines -- subroutines! -- for common (to OMPI) build
actions. This was *fantastic* -- I could customize the build process
by writing real code, and having the full power of python behind it.
For example, building a component was something trivial like this:
Import('ompi_build_component')
ompi_build_component(sources = Split("""
ptl_tcp.c
ptl_tcp_component.c
ptl_tcp_peer.c
ptl_tcp_proc.c
ptl_tcp_recvfrag.c
ptl_tcp_sendfrag.c
ptl_tcp_sendreq.c"""))
The ompi_build_component() python subroutine (that I obviously wrote)
that did the Right Thing to build the OMPI components, regardless of
whether we were slurping components up into libmpi (etc.) or building
as standalone DSOs.
Having the ability to effectively extend scons and have my own
subroutines for project-specific functionality was simply
fantabulous. This is probably one of the two biggest things that I
liked about scons -- I had subroutines (and the power of a real/full
scripting language) that I could invoke anywhere in the project's
source tree.
--> From the above example, I don't see/remember offhand how the TCP
PTL's .h files got slurped into the distribution tarball; it's
possible that I stopped doing the scons work before I got to the part
of making the tarball. Or perhaps scons did source code analysis and
just figured it out...? I honestly don't remember.
+++ No need for a fixed, pre-determined directory layout. A decision
that we made way back in the beginning of Open MPI was that we wanted
to make it [relatively] easy for developers to add new frameworks and
components -- essentially, make a new directory, put in a few specific
files, and then re-run autogen.sh. Specifically: normal developers
shouldn't have to edit the configure/build system to add new
frameworks and components. I still stand by this decision; it would
have been a Big Mess if we had to educate all of our developers in the
intricacies of the autotools about how to add new frameworks/
components (in short: the vast majority of developers don't know squat
about configure/build systems, so we wanted to make the bar as low as
possible to add new frameworks/components without needing to touch the
configure/build system at all since it's a relatively common OMPI
developer action).
This decision ended up generating *significantly complicated*
autogen.sh and various m4 scripts in OMPI's pre-configure/configure/
build system. Early in 2009, Brian Barrett and I sat down with
another OMPI developer and tried to explain OMPI's configure/build
system. Needless to say, it was a many-hour conversation, and the
other developer was reeling when we were all done. :-)
With scons, I had subroutines to -- at build time -- discover OMPI's
frameworks and components, and then build them. Specifically: with
scons, it was trivial to have a pre-determined directory layout (just
like the autotools). But it was also fairly straightforward to have a
dynamic directory layout: I wrote real python logic to go look at a
directory, decide if it was a valid framework/component, and if so, go
build it.
This was the 2nd of the "two biggest things" that I liked about
scons. It's likely that OMPI is fairly unique in its "we want to
discover frameworks / components at run time" requirement, but I would
imagine that other plugin-based projects might use such functionality
if it were easy to do (it is *not* easy to do with the current
autotools!).
Disclaimer: like I mentioned above, when I did this, the OMPI tree was
significantly different than it is today. I'm don't remember how I
handled the configuration vs. building of conditional components
(e.g., only build the openib component if you have OpenFabrics devel
support installed on your system). I.e.: just "discovering" a
component at build time is not enough -- it also has to have been
configured properly first. I don't remember the details of what I did
there.
+++ As mentioned above, you could split the build specification in
multiple files throughout the tree (similar to how we have
Makefile.am's throughout the tree). A nice feature is that each
separate scons file (named SConscript) was in its own python
namespace. This gave a nice, well-defined mechanism for sharing data
without fear of accidentally tromping on variables in another file
elsewhere in the tree.
+++ There were pre-defined constructs (subroutines, IIRC?) that were
Automake-like in that you would just say "build a library, here's the
sources" -- and the right Magic happened behind the scenes. Such a
feature is pretty much a given for a build system, of course -- but it
did make the learning curve for "write code to build your code" much
less steep.
+++ Scons would build a distribution tarball that would include enough
scons bootstrapping such that the end-user did not need to have scons
pre-installed just to build Open MPI. This was(is) a huge requirement
-- it was a deal-breaker if a build system didn't support this (I
believe that, at least at the time, only the autotools and scons did
this).
+++ Support for Windows native Microsoft build system stuff. I didn't
get to investigate this at all, but supposedly scons would generate
project files that could be slurped into MS Developer Studio (or
whatever it's called). That would have been nice.
Dislikes:
--- Scons also tried to replace Autoconf by allowing you to embed
tests in the build process (e.g., built-in subroutines the equivalent
of AC_CHECK_HEADER, etc.). This has the side effect of running the
tests *every* time you build. Even if the answers are cached and the
test is therefore really fast, that just seems weird to me.
Configuration is different than building. Indeed, look at the length
of OMPI's configure script -- there's no way I'd want to run that
every time I do a build (even if the answers are cached). As a
developer, I build OMPI many, many more times than I configure it. In
the OMPI scons solution that I coded up, I still used Autoconf/
configure; it output a file with all the results of its tests that was
then slurped up into scons for building.
--- Scons support of various compilers was not nearly as complete /
comprehensive as Libtool's.
--- I had to learn Python (at least, learn enough to get by). Python
seemed like a neat language and it wasn't *that* hard to learn, but I
started from zero and had to do a bunch of learning before I could
really use scons. There were definitely times where I had to go spend
an hour and figure something out in Python before I could continue
what I wanted to do in scons. This is probably true of most tools if
you're not already familiar with the back-end language, so it may not
be a fair "dislike", but I mention it anyway. Did I mention that this
is all IMHO? :-)
Hope this helps!
--
Jeff Squyres
Cisco Systems
