Hi Summary notes from the trenches, for porting to gbuild. This is meant to be read in addition to https://wiki.openoffice.org/wiki/Build_Environment_Effort/Module_Migration
You are not only welcome but encouraged to help, and I am happy to review patches and answer questions, but please let's first coordinate porting on the dev mailing list so we don't duplicate work. ---------------------------- build.pl and dmake ---------------------------- 1. We still build with build.pl, which means execution always starts in main/<module>/prj/build.lst. That file should have only 2 lines, the first exactly as it is already, and the second running "nmake" in that module's prj directory. See any gbuild module for an example - you can find them by running "ls */prj/makefile.mk" from main/. 2. Using that second line in build.pl, dmake runs on main/<module>/prj/makefile.mk, which is exactly the same in all gbuild modules, and be copied from any. All that this file does is get dmake to run GNU make, which we will examine just now. 3. After everything is done - and only if done successfully - build.pl will run "deliver" which runs the commands in main/<module>/prj/d.lst. This file will be obsolete with gbuild, and need to be an empty, 0 byte long file, in all gbuild modules for now. What it does in dmake modules, needs to be done by GNU make in its makefiles instead, when converted to gbuild. Your first step isn't writing build scripts, it's intelligence gathering. Thoroughly read build.lst and see what directories it goes into and what action it runs in them (the "usr1" action seems to be a no-op, "nmake" runs dmake, the "get" action is unclear to me but also seems to be a no-op). Thoroughly read d.lst and understand what is being copied to solver and from where. Thoroughly read all of dmake's makefile.mks in that module. Read the .map files to see which symbols binary deliverables are exporting. Understand all the dmake variables used. Build a complete picture of what is being done. Ideally draw a picture of all files and actions. Dmake's makefile.mk files are full of gotchas, obscure settings, and strange behaviour. It seems like in AOO building with dmake is only done on 1 directory of files at a time, and when files from multiple directories need to be compiled into one deliverable, each directory is built into its own static library, and a separate directory links these static libraries together into the final deliverable :-D. We can do better. See my main/package patch(es?) for an example that eliminated these intermediate static libraries. ----------------- GNU make ----------------- 1. In step 2 above, GNU make is invoked from dmake, and it runs the main/<module>/Makefile which is the same in all gbuild modules. 2. That sets up gbuild and runs main/<module>/Module_<module>.mk which you need to write. Modules can (and usually do) contain multiple deliverables. By convention, each deliverable is in a separate file, named <DeliverableType>_<DeliverableName>.mk (eg. Library_fileaccess.mk, Package_inc.mk), and these are included into the module's Module_<module>.mk using gb_Module_add_targets and the like. Inclusion of these deliverable makefiles can be conditional, happening only on a certain operating system or only if certain dependencies are present. This is accomplished in the Module_<module>.mk, using GNU make's if conditions and only calling gb_Module_add_targets if those conditions are met (note that in general the *add* functions like *_add_targets can be called multiple times, but the *set* ones usually can't). See module/x11_extensions/Module_x11_extensions.mk for a simple example of how to only run actions on *nix. 3. Deliverables are the sole reason modules exist, and all the actual action happens inside the makefiles for those deliverables. So what types of deliverables do we have, how are they used, and what do they do? The easiest to understand is the "Package" deliverable, which simply copies files to the solver. The easiest modules to understand are those that do nothing but package files, such as main/MathMLDTD and main/x11_extensions. The Package action pretty much does what the old main/<module>/prj/d.lst file did. Note that gbuild will automatically package other types of deliverables such as libraries and executables, and only files that gbuild doesn't have specific deliverable actions for have to be packaged explicitly. These are typically: * Header files, for modules that provide them for use by other modules, are typically in a Package_inc.mk. Header files don't have to be packaged for the module to be able to use them to compile, they only have to be packaged so other modules can use them, which isn't always necessary. * DTD files are typically in Package_dtd.mk * XML files are typically in Package_xml.mk * Any other custom files you wish to copy. The "Library" deliverable compiles a shared library, on Window it's a DLL. There's also a "StaticLibrary" for static libraries. It is mandatory to also register every library in main/Repository.mk. Libraries come in several "groups" or "layers", listed in main/Repository.mk, and the group affects the way the library is named and its RPATH. Look at the openoffice/program directory of a *nix AOO installation. The lib<name>.so are usually in the OOOLIBS group, while the <name>.uno.so are usually in the UNOLIBS_OOO group, and the lib<name>gcc3.so are usually in the RTLIBS group. Note that this is just a naming convention. UNO libraries can be in the OOOLIBS group, and some are. Also main/RepositoryFixes.mk hacks some names... UNO libraries have a .component file which you specify using gb_Library_set_componentfile. Dmake did it in a much uglier way, with a long make rule. Note that for such libraries, you have to update main/postprocess/packcomponents/makefile.mk and change the component name in the my_components variable from eg. binaryurp to component/binaryurp/source/binaryurp (this comes from the path within the module). For some components, this has to be done in the undocumented main/urp/source/makefile.mk too. gb_*_add_exception_objects is for C++ files that are to be compiled with exception handling enabled, and gb_*_add_noexception_objects is for those to be compiled with exception handling disabled. How do you know which to use? ENABLE_EXCEPTIONS in makefile.mk, which is off by default. Other supported languages for libraries and executables include C and Objective C which have their own actions, while Java has a separate "Jar" deliverable type which is currently only used in main/wizards and relatively easy to understand compared to the C++ "Library". Listing a source filename that doesn't exist currently results in an infinite loop when building, so be careful! When using add_linked_libs, you have the use gbuild's naming (eg. sal) instead of dmake's (eg. $(SALLIB)). It's not always that obvious. Precompiled headers must either be empty, or 1 per deliverable, instead of 1 per module. It is a build-breaking error - on Windows only (since other platforms don't use precompiled headers) - to use the same non-empty precompiled header in 2 or more deliverables. Gbuild modules have to link with "stl" on Windows, or you get mysterious link errors. Gbuild changed from dmake's exporting all symbols by default and having to hide unwanted ones using a .map file, to hiding all symbols by default and having to export the wanted ones with SAL_DLLPUBLIC_EXPORT or the like. Yes, you are editing C++ code for this, not just makefiles. If you forget to export, the build will usually break on Windows but succeed on *nix and presumably fail at runtime. "Executable" is for applications. These also have groups and also need to be registered in main/Repository.mk. "GoogleTest" is for C/C++ unit tests that run during the build. It's very similar to Executable, because it builds executables that run the tests. (After porting our unit tests from cppunit to Google Test, I discovered it's possible to do it through libraries too lol.) "JunitTest" is for "subsequent checks" that can be run after AOO is built with "make subsequentcheck" in the module directory. "AllLangResTarget" is for translations. These come from dmake's .src files. See main/formula, main/uui and others for examples. When a module is ported to gbuild, you must add it to main/Module_ooo.mk, I think this lets it participate in global actions like "make clean". As you can see, AOO uses an advanced, complex build system with many programming languages, much custom infrastructure, custom compilers, custom source documentation generation, custom translations, etc. The gbuild makefiles in solenv/gbuild are currently around 8000 lines and still incomplete. Using another build system instead of gbuild would first require porting all this build logic to that build system. And as should be clear this far, the dmake->gbuild conversion involves multiple changes: changes to symbol visibility, elimination of build.pl, elimination of dmake, elimination of deliver, changes to syntax, changes to the naming of dependencies, changes to include paths, etc. Any gbuild->other migration will be a lot easier by comparison! Syntax is a bit of a pain. You can't put spaces around commas, have to end non-final lines in a definition with a backslash, etc. On the whole though, gbuild makefiles for modules are fewer in number, clearer and more readable than dmake's, and only found in the module's root directory instead of hidden in subdirectories. The high level declarative syntax is also a pleasure to work with compared to dmake's. When in doubt, read other modules' makefiles, not the almost indecipherable gbuild code. --------------- The future --------------- 52 of 182 modules (29%) are currently using gbuild, up from 40 (22%) before my recent conversion burst began. I am hacking and porting as many modules to gbuild as possible, as soon as possible. The most important thing right now is to build up experience and document as much of gbuild as possible (and sadly dmake/build.pl/deliver, since understanding their files comes first), to prepare for tackling the large modules and making changes to the gbuild makefiles themselves. The UNO modules seem easier to convert and many are really small. I am currently converting the Java modules, and javaunohelper should be the next to get committed. I think another group needs to be added for libraries like libstore.so.3 whose naming format isn't provided by any library group. Yacc support also needs adding to gbuild for several modules that use it to parse languages (IDL compilers, main/connectivity for parsing SQL queries, etc.), and other gbuild infrastructure probably needs developing further. It seems Sun was trying to implement a "split build" before all the modules were ported, where modules in first part are built with build.pl, and those in the second part are built purely with a single GNU make instance that can build multiple modules concurrently at a file level, with maximum parallelization to give the highest performance ;-), and would no longer need the prj/ subdirectories. It's not clear to me how they were planning to do this or on exactly which modules (the "late" modules would have been the first to go into this gbuild-only part, of which Calc is the last of the "big 6" that aren't in gbuild yet), but I can't wait to start telling dmake and build.pl goodbye! Happy hacking Damjan --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org