Hi With the recent buildbot saga and Patricia's Windows building nightmare, a number of serious issues were highlighted which got me thinking about how our build system could be improved.
The problems with building AOO were researched and documented years ago ( https://wiki.openoffice.org/wiki/Build_Environment_Effort) and a very good analysis was done in https://wiki.openoffice.org/wiki/Build_System_Analysis which is worth a read. Their solution was gbuild: a build system using GNU make's new "eval" feature to rewrite the makefile from within, which would generate one giant dependency graph of every file in AOO and produce reliable, stable, deterministic, and maximally parallelized builds, with minimal rebuilding on changes. The problem is, the full benefits of gbuild can only be seen once every module is converted to gbuild, and with only about 10% having been converted so far, it doesn't even try to build a complete project-wide dependency graph, but rather gets invoked per-module just like dmake. Converting dmake modules to gbuild is very difficult: so far I only managed to convert main/formula, but main/animations was mysteriously segfaulting AOO... Besides, LO has completed converting every module to gbuild, guinea pigging gbuild perfectly for us ;-). How has it gone for them? One of our committers, jani, evaluated LO's gbuild in April 2013 in a thread here called "Make x Dmake x Build", where he said how it's "not particulary fast either", and called mkdir 431 times on a directory that already existed. And worst of all, gbuild uses its own language with horrible syntax (eg. LISP-like syntax, spaces around commas not allowed, blank lines sometimes mandatory and at other times forbidden), is almost completely undocumented, and very difficult to debug - a maintenance nightmare. So what other alternatives are there? There was a comment in https://wiki.openoffice.org/wiki/Build_Environment_Effort/New_Build_System_Requirements that got my attention: "One might consider using Python as a tool for _all_ non-standard build tasks. Python is easy to distribute and install on all platforms. However, it is also a very fat dependency and thus it is questionable, if it would be worth the effort just to get rid of cygwin (or another Unix environment on windows)." We already build and use Python, and there is a good build system written in Python: SCons (http://www.scons.org). It's modern yet well established, having started in the year 2000, and is used by a number of open source and commercial projects. It's MIT licensed so we can include it as a dependency. The build scripts are simple Python scripts and can use the Python language and libraries, though you don't need to know (much) Python to use it. It's well documented. It's debuggable and has options to print dependencies and explain actions taken. It supports all the operating systems and architectures we support (even unofficially), including Cygwin, *BSD, Solaris, OS/2. It's "local" package is designed for easy embedding into a pre-existing project so it runs straight after extraction with no need to install it system-wide, and is only 445 kB. It has many features we need or would benefit from: * It's able to parse included headers from C/C++ files and use them as dependencies. * It can generate Visual Studio project files. * Supports many languages including C, C++, D, Java, Fortran, Yacc, Lex. * Builds zip and tar archives. * Just like gbuild, it has a global dependency view and can fully parallelize building files across module boundaries while maintaining stability. * Using Python APIs instead of *nix CLI tools might eventually result in us being able to ditch Cygwin entirely, not to mention better performance due to less subprocess spawning. * Unlike classical make tools which only use mtimes, it determines what to rebuild using several configurable options, including MD5 checksums of file contents, meaning changes which don't result in different binaries (eg. formatting or comment changes) will only recompile changed files and stop there, because it knows the object files are the same as before, so no linking or downstream rebuilding needs to be performed. In other words SCons does most of what gbuild is supposed to be good for, and more. I see it outperforming LO's pure gbuild, while being more readable, maintainable, debuggable, and a sight for sore eyes after the triple whammy of dmake, gbuild and build.pl we've endured for so long. If people are happy, I'd suggest a migration plan where first scons is added as a dependency, then the new version of serf is built with it (see i126312), then gbuild modules are ported to scons until gbuild is eliminated, then dmake modules are ported to scons until dmake is eliminated, and finally scons replaces build.pl. Thoughts? Damjan
