Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Am Fri, 19 Mar 2010 17:09:50 +0100 schrieb Thorsten Behrens : > too long a silence around this important topic for my taste You know: If the kids are quiet, something is up ... ;) Mba and I continued to work/evaluate on cmake and GNU/make. I did work mostly on windows and it works basically, but there are still things to decide regarding dllexport/visibility. Windows is especially brain-dead in some regards[1]. There are a few solutions to consider for this problems, but I certainly want to get rid of cumbersome hacks like the "filter-lists" we are currently using here and there (which make us link every such dll twice -- once to get the symbols and once to really link). In addition to being a performance-hog and being simply ugly, this is also causing a lot of really bad reexports (like fwi and fwe re-exporting most of cppuhelper). It also leads to lots of "interesting" differences depending on inlining, debug-builds etc. > - so let me tease you folks with this here new kid on the block: > > http://aruiz.synaptia.net/siliconisland/2010/03/buildj-build-configuration-for-the-mases.html Meh. Waay to young for my taste, although replacing autoconf is a noble effort. I also liked the idea behind http://code.google.com/p/quagmire/ but it seems to be struck in a quagmire (pun intended), since there is little work going on. Lets see where BuildJ is in one or two years (or when it is adopted by a major project (gnome)). However, using BuildJ on top of waf on top of Python isnt exactly lean (how does that go through a python 2.x -> 3.x -> 4.x migration?) You where right to complain about the silence on the build env topic. Blog posts by Mathias (I guess) and me are coming your way RSN. Best Regards, Bjoern [1] for example: http://www.codesynthesis.com/~boris/blog/2010/01/18/dll-export-cxx-templates/ - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Hi, too long a silence around this important topic for my taste - so let me tease you folks with this here new kid on the block: http://aruiz.synaptia.net/siliconisland/2010/03/buildj-build-configuration-for-the-mases.html ;) Cheers, -- Thorsten pgp1VOCzhBbNV.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote:
On Wed, Feb 17, 2010 at 11:57 AM, Martin Hollmichel
wrote:
usually I analyse the build logs for understanding these kind of mechanisms,
here's my extract for sal module on Linux:
...
tr -d "\015" < sal.map | awk -f ./solenv/bin/addsym.awk >
../unxlngi6.pro/misc/sal_uno_sal.map
Making: ../unxlngi6.pro/lib/libuno_sal.so.3
g++ -Wl,-z,noexecstack -Wl,-z,combreloc -Wl,-z,defs -Wl,-Bsymbolic-functions
-Wl,--dynamic-list-cpp-new -Wl,--dynamic-list-cpp-typeinfo
-Wl,--hash-style=both -Wl,-rpath,'$ORIGIN' "-Wl,-hlibuno_sal.so.3" -shared
-Wl,-O1 -Wl,--version-script ../unxlngi6.pro/misc/sal_uno_sal.map ...
There seem to be two phases in this. First you generate the target
file with awk and then pass it to the linker. The first one is
straightforward.
The latter is a bit trickier but not very hard. First you set a
dependency between the shared library and the generated version file.
Then you need to pass the file to the linker. CMake allows you to
define custom linker flags per target. To get the above you would add
something like this:
set_target_properties(sal PROPERTIES LINK_FLAGS "-Wl,--version-script
${GENERATED_SAL_MAP_FILE}")
the process for Windows (MSVC) is very similar, I already invented a
function add_mapfile( ) to do the job. I guess this
approach will work for all other platforms as well.
thanks,
Martin
-
To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org
For additional commands, e-mail: dev-h...@tools.openoffice.org
-
To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org
For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Wed, Feb 17, 2010 at 11:57 AM, Martin Hollmichel
wrote:
> usually I analyse the build logs for understanding these kind of mechanisms,
> here's my extract for sal module on Linux:
> ...
> tr -d "\015" < sal.map | awk -f ./solenv/bin/addsym.awk >
> ../unxlngi6.pro/misc/sal_uno_sal.map
>
> Making: ../unxlngi6.pro/lib/libuno_sal.so.3
> g++ -Wl,-z,noexecstack -Wl,-z,combreloc -Wl,-z,defs -Wl,-Bsymbolic-functions
> -Wl,--dynamic-list-cpp-new -Wl,--dynamic-list-cpp-typeinfo
> -Wl,--hash-style=both -Wl,-rpath,'$ORIGIN' "-Wl,-hlibuno_sal.so.3" -shared
> -Wl,-O1 -Wl,--version-script ../unxlngi6.pro/misc/sal_uno_sal.map ...
There seem to be two phases in this. First you generate the target
file with awk and then pass it to the linker. The first one is
straightforward.
The latter is a bit trickier but not very hard. First you set a
dependency between the shared library and the generated version file.
Then you need to pass the file to the linker. CMake allows you to
define custom linker flags per target. To get the above you would add
something like this:
set_target_properties(sal PROPERTIES LINK_FLAGS "-Wl,--version-script
${GENERATED_SAL_MAP_FILE}")
-
To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org
For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote:
On Thu, Feb 11, 2010 at 2:01 PM, Martin Hollmichel
wrote:
one problem right now is that the usage of linker mapfiles (e.g.
sal/util/sal.map) are not straight supported by cmake, unfortunately all
the
creation of several win32 import libraries are dependent to such
mechanism,
only some of the OOo Libs support the declspec(dllexport) or the Unix
visibility mechanisms. so here some work is required.
I'll try to look into this.
thanks,
I looked into it and got lost in a twisty passage of makefiles, Perl
scripts and lib/a/so files. Could someone give a brief description on
how the system works. Specifically how the different files are
processed and what is finally passed to the linker.
usually I analyse the build logs for understanding these kind of
mechanisms, here's my extract for sal module on Linux:
...
tr -d "\015" < sal.map | awk -f ./solenv/bin/addsym.awk >
../unxlngi6.pro/misc/sal_uno_sal.map
Making: ../unxlngi6.pro/lib/libuno_sal.so.3
g++ -Wl,-z,noexecstack -Wl,-z,combreloc -Wl,-z,defs
-Wl,-Bsymbolic-functions -Wl,--dynamic-list-cpp-new
-Wl,--dynamic-list-cpp-typeinfo -Wl,--hash-style=both
-Wl,-rpath,'$ORIGIN' "-Wl,-hlibuno_sal.so.3" -shared -Wl,-O1
-Wl,--version-script ../unxlngi6.pro/misc/sal_uno_sal.map ...
Another question is whether the map files currently solve any problem
that cannot be dealt with native symbol visibility settings?
as the linker script above shows, the option "-version script" is used,
the reason for this is explained in the ld man page:
"Specify the name of a version script to the linker. This is typically
used when creating shared libraries to specifc additional information
about the version hierarchy for the library being created. This option
is only fully supported on platforms which support shared libraries; see
VERSION. It is partially supported on PE platforms, which can use
version scripts to filter symbol visibility in auto-export mode: any
symbols marked local in the version script will not be exported."
e.g. an exerpt from the sal-map file:
"...
UDK_3.7 { # OOo 2.4
global:
osl_loadModuleRelative;
} UDK_3.6;
UDK_3.8 { # OOo 3.0
global:
rtl_bootstrap_encode;
rtl_convertStringToUString;
rtl_math_approxValue;
} UDK_3.7;
UDK_3.9 { # OOo 3.1
global:
osl_mapFile;
osl_unmapFile;
osl_readFileAt;
osl_writeFileAt;
rtl_math_expm1;
rtl_math_log1p;
rtl_math_atanh;
} UDK_3.8;
UDK_3.10 { # OOo 3.2
global:
rtl_logfile_hasLogFile;
rtl_math_erf;
rtl_math_erfc;
rtl_math_asinh;
rtl_math_acosh;
} UDK_3.9;"
But you're also right, there are also still libraries around where the
native symbol visibilty could (and IMHO should) be used, but this would
require quite a lot manual code changes (many already have been done,
but not all yet),
hth,
Martin
-
To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org
For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Thu, Feb 11, 2010 at 2:01 PM, Martin Hollmichel wrote: >>> one problem right now is that the usage of linker mapfiles (e.g. >>> sal/util/sal.map) are not straight supported by cmake, unfortunately all >>> the >>> creation of several win32 import libraries are dependent to such >>> mechanism, >>> only some of the OOo Libs support the declspec(dllexport) or the Unix >>> visibility mechanisms. so here some work is required. >> >> I'll try to look into this. >> > > thanks, I looked into it and got lost in a twisty passage of makefiles, Perl scripts and lib/a/so files. Could someone give a brief description on how the system works. Specifically how the different files are processed and what is finally passed to the linker. Another question is whether the map files currently solve any problem that cannot be dealt with native symbol visibility settings? - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Martin Hollmichel wrote: There's also the dmake -> cmake converter script so you don't have to keep writing the files by hand. from what I've learned from cmake so far, it seems the best idea to do an 1:1 conversion since this might lead to too much targets (and thus dependencies) and writing CMakeLists.txt seem in some cases lead to more efficient makefiles. of course this should read as "it seems _not_ the best idea, to do an 1:1 conversion", Martin - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote: On Wed, Feb 10, 2010 at 9:00 PM, Martin Hollmichel wrote: yes, thank you, I used and modified some of your work (add_idl_db and build_rdb_from db functions) and added a more modules (up to rsc now). A word of warning: the functions I have written to deal with javamaker etc are broken. They work when run the first time but not the second time. I did not have the time and energy to debug them, I just wanted to get some sorts of results. as I said, I modified them to make them finally work :-) one problem right now is that the usage of linker mapfiles (e.g. sal/util/sal.map) are not straight supported by cmake, unfortunately all the creation of several win32 import libraries are dependent to such mechanism, only some of the OOo Libs support the declspec(dllexport) or the Unix visibility mechanisms. so here some work is required. I'll try to look into this. thanks, Martin - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Wed, Feb 10, 2010 at 9:00 PM, Martin Hollmichel wrote: > yes, thank you, I used and modified some of your work (add_idl_db and > build_rdb_from db functions) and added a more modules (up to rsc now). A word of warning: the functions I have written to deal with javamaker etc are broken. They work when run the first time but not the second time. I did not have the time and energy to debug them, I just wanted to get some sorts of results. > one problem right now is that the usage of linker mapfiles (e.g. > sal/util/sal.map) are not straight supported by cmake, unfortunately all the > creation of several win32 import libraries are dependent to such mechanism, > only some of the OOo Libs support the declspec(dllexport) or the Unix > visibility mechanisms. so here some work is required. I'll try to look into this. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote: On Mon, Jan 11, 2010 at 8:11 PM, Martin Hollmichel wrote: I started some time ago a cmake prototype for OOo in my spare time (http://hg.services.openoffice.org/hg/cws/mh6bc/) for the latest status please see the latest ReadMe.txt in the Source root for the most recent status). You're invited to join this prototype, but be warned: it does not work and kills your cat when try to run that :-). You might want to look look at my attempt: http://lists.freedesktop.org/archives/ooo-build/2009-August/000181.html yes, thank you, I used and modified some of your work (add_idl_db and build_rdb_from db functions) and added a more modules (up to rsc now). It goes quite a lot further and solves some of the issues listed in your readme (detecting STLPort and Boost, etc). for the moment the external libraries are not in my main focus, There's also the dmake -> cmake converter script so you don't have to keep writing the files by hand. from what I've learned from cmake so far, it seems the best idea to do an 1:1 conversion since this might lead to too much targets (and thus dependencies) and writing CMakeLists.txt seem in some cases lead to more efficient makefiles. one problem right now is that the usage of linker mapfiles (e.g. sal/util/sal.map) are not straight supported by cmake, unfortunately all the creation of several win32 import libraries are dependent to such mechanism, only some of the OOo Libs support the declspec(dllexport) or the Unix visibility mechanisms. so here some work is required. Martin - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, Feb 1, 2010 at 1:44 PM, bjoern michaelsen - Sun Microsystems -
Hamburg Germany wrote:
> With full deps in one process, bar{1-99}.cxx can be compiled right from
> the start, because only the linking of libbar has to be done after the
> linking of libfoo. A recursive makefile (one make per lib) makes the
> deps a lot rougher: It will only start to compile bar{1-99}.cxx and
> friends after it compiled all foo{1-99}.cxx and linked libfoo.
This is a fair point. I don't know whether CMake does this as it is an
implementation detail of its makefile generator. But I still say that
the only usable performance results are obtained through testing and
measuring.
-
To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org
For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Wed, 27 Jan 2010 13:57:22 +0200
Jussi Pakkanen wrote:
> I think I see your concern now. When CMake does the library
> resolvation part it does it itself (i.e. not with Make) and this is
> always done with a single process and looking at the global state. The
> makefiles it produces are recursive, but they are generated with full
> info similar to the way the single process GNU Make thing works.
Not quite. Consider the following scenario:
Library libfoo linked from objects compiled from foo1.cxx, foo2.cxx,
foo3.cxx ...
Library libbar linked from objects compiled from bar1.cxx, bar2.cxx, ...
and is linking against libfoo.
With full deps in one process, bar{1-99}.cxx can be compiled right from
the start, because only the linking of libbar has to be done after the
linking of libfoo. A recursive makefile (one make per lib) makes the
deps a lot rougher: It will only start to compile bar{1-99}.cxx and
friends after it compiled all foo{1-99}.cxx and linked libfoo.
This can be prevented by reordering the makefiles (one for compiling,
one for linking), but thats getting ugly fast if done manually.
Best Regards,
Bjoern
--
===
Sitz der Gesellschaft:
Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten
Amtsgericht München: HRB 161028
Geschäftsführer: Thomas Schröder, Wolfgang Engels
Vorsitzender des Aufsichtsrates: Martin Häring
===
-
To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org
For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Stephan Bergmann wrote: > On 01/27/10 09:07, Mathias Bauer wrote: >> Jussi Pakkanen wrote: >>> Yes, sure. If and only if you have target_link_libraries(some_exe >>> some_library) then the linker invocation for some_exe will have >>> '-lsome_library'. Whether or not it was built accidentally, CMake will >>> not link the exe against the library. >> >> Cool. > > Mathias, I'm not sure I understand your excitement here. ;) Ehm, yes. :-) Reading it a second time I must have misunderstood what Jussi wrote. But this wasn't the first misunderstanding in this thread. :-) I thought that he wanted to point out that CMake will quit working in that case, even before the build starts. Though in fact that seems to happen as he explained in his last mail, the quoted text above indeed was something different. Regards, Mathias -- Mathias Bauer (mba) - Project Lead OpenOffice.org Writer OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS Please don't reply to "nospamfor...@gmx.de". I use it for the OOo lists and only rarely read other mails sent to it. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Wed, Jan 27, 2010 at 10:07 AM, Mathias Bauer wrote: > The parallelization comes from the build.pl. Let's go back to our simple > example with the 4 libraries/modules. When build.pl is called to build > D, it discovers it has to build B and C before (as the build.lst of D > tells so). And as B depends on A, build.pl will start either witch > building C (and will fail as C also depends on A but that dependency was > forgotten in C's build.lst) or with building A (as B's build.lst tells > that A needs to be built before B). In the latter case the missing > dependency stays unnoticed because A already has been built when finally > C is built. I think I see your concern now. When CMake does the library resolvation part it does it itself (i.e. not with Make) and this is always done with a single process and looking at the global state. The makefiles it produces are recursive, but they are generated with full info similar to the way the single process GNU Make thing works. Your goal has been "build everything from a single process", but what you actually want is "resolve dependency graphs with global scope". The latter is what CMake does by default. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On 01/27/10 09:07, Mathias Bauer wrote: Jussi Pakkanen wrote: Yes, sure. If and only if you have target_link_libraries(some_exe some_library) then the linker invocation for some_exe will have '-lsome_library'. Whether or not it was built accidentally, CMake will not link the exe against the library. Cool. Mathias, I'm not sure I understand your excitement here. ;) What ultimately has to happen if lib2 depends on lib1 is that the command line used to link lib2 mentions lib1 to link against (e.g., something like "-llib1" on Unix). With our current approach, this is a two-level mechanism, where some makefile.mk contains a line SHL2STDLIBS+=$(LIB1LIB) and additionally some build.lst contains the corresponding module dependency. Every sane build system that has a global view of things would certainly only require this to be a one-level mechanism, where the user-provided information that lib2 needs to link against lib1 (the "SHL1STDLIBS+=$(LIB1LIB)" part in our current system) is sufficient for the build system to automatically derive the corresponding dependencies (that lib1 needs to be built before lib2 can be linked). Or what am I missing? -Stephan - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote: > BTW, what does the GNU Make solution use for platform configuration? > Autoconf? Some in-house tool? We didn't want to change that - so far it's planned to be the same stuff as today (calling our configure script). >> Of course, TANSTAAFL: I'm sure we will be the first to explore this >> (native Windows builds with CMake) for a project of such a big size and >> diversity. Surely we will find many "interesting" problems. But that >> shouldn't stop us from discussing it now. :-) > > KDE builds with CMake on Linux, OSX, and Windows. That's pretty big. We'll see, from our own experience with anything we touched (be it gcc, dbx, the Sun Solaris Compiler, etc.) we get new and interesting problems. That isn't a deal breaker, but it's worth mentioning. > Yes, sure. If and only if you have target_link_libraries(some_exe > some_library) then the linker invocation for some_exe will have > '-lsome_library'. Whether or not it was built accidentally, CMake will > not link the exe against the library. Cool. > I don't know about CMake's internals enough to tell you how it divides > the work between itself and make. But what I can tell you is that > CMake+make combo does handle the entire dependency chain from > executables to source, headers, generated headers etc. It has a > built-in dependency scanner for #includes. Sounds promising. >> I would appreciate to be able to get my hands on it. :-) Trying is so >> much better than talking. Until now I only found rather trivial examples >> in the web (even more trivial like my examples with 4 libraries). > > Attached is a sample with 3 shared libraries and one executable. To > run on Unix, extract it, cd into it, then: > > mkdir builddir > cd builddir > cmake -DCMAKE_BUILD_TYPE=debug .. > make > > It should build without any changes on Linux, OSX, Visual C, MinGW > etc. I always recommend creating a separate build directory so as not > to clutter your source tree with generated files. Thanks, I will try it out. >> BTW: those CMakeList.txt files are a PITA. We are currently moving away >> from cluttering our source tree with files that are not under control of >> the SCM. Is there a way to avoid the placement of these files into the >> source tree? Can it be put somewhere else? > > CMakeLists.txt will be under SCM. They are the master build > information. The generated Makefiles will not be placed under revision > control. They and everything else that is generated can be put into a > separate build directory (this is in fact the recommended way to use > CMake). Of course, sorry, I wanted to mention the generated makefiles, not the sources. :-) Good to hear that this can be moved into a working directory. >> Again this is the problem I want to avoid: errors caused by missing >> dependencies/prerequisites don't appear before you actually execute the >> build. This leaves too much room for random behavior caused by parallel >> builds. > > I don't see how the parallel build thing can happen. Does the current > system have some kind of a wildcard linking rule? Something like "link > against everything in such_and_such directory"? The parallelization comes from the build.pl. Let's go back to our simple example with the 4 libraries/modules. When build.pl is called to build D, it discovers it has to build B and C before (as the build.lst of D tells so). And as B depends on A, build.pl will start either witch building C (and will fail as C also depends on A but that dependency was forgotten in C's build.lst) or with building A (as B's build.lst tells that A needs to be built before B). In the latter case the missing dependency stays unnoticed because A already has been built when finally C is built. The same would happen in a recursive GNU make approach (where GNU make not only would replace dmake but also build.pl): the makefile for D will detect that it needs B and C. It knows by a rule that getting these prerequisites requires invoking the make files for them. So it will execute a make for C or for B, the latter will itself execute make for A (as the makefile of B contains a rule telling so), the former wouldn't do so as a developer forgot to add this rule. Thus if the make for C is called first, the build will fail. If the make for B is called first, the build will not fail as this automatically will call make for A and so will "accidently" provide the missing prerequisite for C and the missing rule will cause no harm: the makfile for C will see the existing library A and everything is fine. In a process that first checks all dependencies in one step before it builds anything this won't happen. The rule to make A is not specified (and evaluated) in the makefile of B and C but in the makefile of D by including the makefile of A directly. >> BTW: how does CMake organize the parallel build? The "all in one >> process" approach has another major advantage: as it is one process, it >> can find the optimum distribution for the available proc
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Hi Jussi, before we continue let me add some words to avoid misunderstandings. In no way I'm trying to cast a bad light on CMake or want to defend our trial approach "just because". But before I consider to support a different approach I want to get the essential problems discussed in the same depth as we did it ourselves when evaluating the GNU Make approach. If CMake solved our essential problems it could be a nice alternative as it gives us the additional benefit of possible "native" builds on Windows (I assume we could still choose to use GNU Make also on Windows/Cygwin for now). Of course, TANSTAAFL: I'm sure we will be the first to explore this (native Windows builds with CMake) for a project of such a big size and diversity. Surely we will find many "interesting" problems. But that shouldn't stop us from discussing it now. :-) Jussi Pakkanen wrote: > On Thu, Jan 21, 2010 at 10:50 AM, Mathias Bauer wrote: > >> So, how can we implement "include, not execute" with CMake? > > You can do this, if it is absolutely necessary. I'll describe that at > the end of the message. > >> Consider that you have the modules A, B, C and D. D depends on B and C, >> that both depend on A. This is reflected by corresponding entries in the >> build.lst files of these modules. Let's assume what happens if I forget >> to add the dependency on A in the build.lst from C. >> >> When I do a complete build from scratch for module D, it will work if B >> is built before C as building B will also build A, so C got its >> precondition met and will build fine also. If C is built before B, the >> build will break as it will miss A. > > CMake has no concept of "modules" as used in OOo. My understanding is > that a "module" is roughly one subdirectory (or as defined in > build.lst) that produces one or more libraries, executables and so on. As I wrote, it doesn't matter if we are looking at "modules" or single targets (libraries, jar files, executables etc.), the problem is the same. I just started with "modules" as this is what we have now. > In CMake all dependencies are between deliverables (i.e. libraries, > binaries, etc). Subdirectories are just a convenience. So in your > example we would have in subdirectory A something like this: > > add_library(A SHARED sourceA.cxx) > > Then in B you would have > > add_library(B SHARED sourceB.cxx) > target_link_libraries(B A) > > Similarly in C you would have > > add_library(C SHARED sourceC.cxx) > target_link_library(C A) > > And finally in D: > > add_executable(foo source.cxx) > target_link_library(D C B) > > What happens if we change C to accidentally remove the dependency to A? > > When make is run, it will produce a link error for C. It will not use > a stale library file one you may have in your build tree. Thus the > error will be immediately apparent to the person doing the change so > he will not check in code that will break on other people. Sure? What if you do a parallel build and "by accident" the library A is built before C is linked? That's exactly what happens in our current build system. For this trivial example it is close to impossible that a bug like this one stays unnoticed. But the multitude of targets and dependencies we have in OOo makes it much more probable to happen. And experience proves that: it happens at times and is a major PITA. The step forward in the "all in one process" solution is that the possible link error is detected in the process of checking the dependency tree, regardless how much stuff is built afterwards and how many parallel processes you will use then. I still can't see how this is done with CMake. It's totally possible that this is my fault. ;-) The CMake documentation isn't very helpful here. I can mainly see very trivial examples, nothing that comes even close to the complexity we have in OOo. Beside that I wouldn't be astonished if CMake couldn't do that - in my current (admittedly limited) understanding CMake never is meant to explore the whole dependency tree from top level binaries down to the included header files (or even further in case they are generated from other sources). That's what happens in the created "native" makefiles. Or does CMake double that effort and evaluates all dependencies by itself before the "native" makefile does the same again? > I tested the behavior just now with a sample to make sure it actually > does this. I can send a sample project if someone wants to try it > themselves. I would appreciate to be able to get my hands on it. :-) Trying is so much better than talking. Until now I only found rather trivial examples in the web (even more trivial like my examples with 4 libraries). I also will have a look on what Martin Hollmichel has created until now, maybe we already have something to try and see. >> Wouldn't it be better if the error detection is "built in"? > > Yes, and with CMake, it already is. :) So CMake *does* check all source and header files, libraries etc.? How does
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Thu, Jan 21, 2010 at 10:50 AM, Mathias Bauer wrote: > So, how can we implement "include, not execute" with CMake? You can do this, if it is absolutely necessary. I'll describe that at the end of the message. > Consider that you have the modules A, B, C and D. D depends on B and C, > that both depend on A. This is reflected by corresponding entries in the > build.lst files of these modules. Let's assume what happens if I forget > to add the dependency on A in the build.lst from C. > > When I do a complete build from scratch for module D, it will work if B > is built before C as building B will also build A, so C got its > precondition met and will build fine also. If C is built before B, the > build will break as it will miss A. CMake has no concept of "modules" as used in OOo. My understanding is that a "module" is roughly one subdirectory (or as defined in build.lst) that produces one or more libraries, executables and so on. In CMake all dependencies are between deliverables (i.e. libraries, binaries, etc). Subdirectories are just a convenience. So in your example we would have in subdirectory A something like this: add_library(A SHARED sourceA.cxx) Then in B you would have add_library(B SHARED sourceB.cxx) target_link_libraries(B A) Similarly in C you would have add_library(C SHARED sourceC.cxx) target_link_library(C A) And finally in D: add_executable(foo source.cxx) target_link_library(D C B) What happens if we change C to accidentally remove the dependency to A? When make is run, it will produce a link error for C. It will not use a stale library file one you may have in your build tree. Thus the error will be immediately apparent to the person doing the change so he will not check in code that will break on other people. I tested the behavior just now with a sample to make sure it actually does this. I can send a sample project if someone wants to try it themselves. > Wouldn't it be better if the error detection is "built in"? Yes, and with CMake, it already is. :) > A missing dependency declaration now means that the "super makefile" > does not include the makefile that explains how a particular > precondition can be built in case it isn't there (usually meaning: in > the solver). In fact a missing dependency now becomes a missing rule. What CMake does is roughly the same. Whenever you run make, it checks whether the build setup has changed (in practice, whether CMakeLists.txt's have been touched) in any way. If yes, then it will automatically recheck the build system's validity and regenerate the necessary makefiles. Then it runs make. The single process GNU Make thing loads all makefiles to validate them (which is roughly the same as CMake checking itself) and then uses that to build. In both cases build stability is assured. > there. Moreover, a missing rule will be detected even before the build > starts, so mistakes are found faster: it is sufficient to call "make" > with an empty solver to detect the missing rules. If you do something like this in CMake: target_link_libraries(some_exe misspelled_or_missing_library_name) it will not detect it before build starts (because the latter may be a system library such as xlib) but it will give an error when linking some_exe. It will not allow it to pass. > So unless you can't bear the overhead of evaluating all dependencies all > the time (what hopefully will take less time than running through all > modules today), you can always just call "make" after your changes > anywhere in the code and the result will be OK. This is precisely what CMake aims to do. Just type "make" or click build on your IDE of choice and you will get a correctly built product. Not doing this is _always_ a bug. And from what I know of CMake's development, a rather serious bug at that. > What does that tell us about CMake? If CMake does not allow us to use a > "super makefile" including all other makefiles, we don't get the build > stability improvements that the GNU Make approach can give us. As far as I can see, CMake does provide all the stability improvements you require. If there is anything more, just ask. > So again: how can we implement "include, not execute" with CMake? CMake allows CMake script files to include other files. You could write a subdir.cmake file for each module/subdir and then just include them in your master CMakeLists.txt. The end result is one makefile (or, I guess, O(1) makefiles due to implementation issues). Since CMake solves the stability issue differently, I do not think you should do this. But if you really, really need to, you can. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote: > On Wed, Jan 13, 2010 at 5:21 PM, bjoern michaelsen - Sun Microsystems > - Hamburg Germany wrote: > >> Mu. As long as you have a recursive build process, you have a lot of >> implicit dependencies and those are hurting parallelization. On top of >> that, process instantiation and file-I/O is very, very expensive on >> Windows, the slowest and most problematic platform of all (both are used >> extensively when doing recursion). This might not matter too much for a >> full build that package maintainers usually do. But it matters very >> much for the change->rebuild->change-cycle that devs usually do (Both >> implicit deps and recursion over mostly noop build tasks really hurt >> here). > > There has been much speculation on the performances but very little > hard facts. It's not only about performance, I still wonder if and how CMake (or better: the CMake+GNU Make+MSBuild combo) is able to support the "all in one process" approach. So far I can only see that with CMake we either can copy the current iterative approach (build.pl) or use a recursive one. So, how can we implement "include, not execute" with CMake? But IMHO we are talking too much about performance. Let me recall the even more important requirement that caused me to think about changes in the first place: consistency and reliability. Building a single target currently requires maintaining several files, at least a build.lst, a d.lst and several makefile.mk files (others like makefile.pmk etc. put aside for the moment). It's easy to forget something, but even if you don't: it's a PITA how many files you have to edit if you just move some files around. I'm doing some refactoring ATM and it really sucks. So it doesn't surprise that the current system is very susceptible to errors. Let me explain this in a simple example. Consider that you have the modules A, B, C and D. D depends on B and C, that both depend on A. This is reflected by corresponding entries in the build.lst files of these modules. Let's assume what happens if I forget to add the dependency on A in the build.lst from C. When I do a complete build from scratch for module D, it will work if B is built before C as building B will also build A, so C got its precondition met and will build fine also. If C is built before B, the build will break as it will miss A. Now consider the ~200 modules we have and the multitude of dependencies that they have. It's easy to understand that in case a particular dependency is missing, it depends on the number of modules that are built, the number of processes used (-Pn) and maybe other circumstances if your build breaks or runs and if the result is correct. So a build might work well for those who made some changes, but can break for others later on after the changes have been integrated. How can one detect such a missing dependency? The only way I see is call "build --all" with an empty solver for each and every module to get a consistent list in the first place and in future repeat this for every module that got its dependencies changed. This is not sufficient to support all flavors of build.pl, but at least it is sufficient for "build" and "build --all". Is it realistic to expect that everybody will do that? Wouldn't it be better if the error detection is "built in"? Let's consider the "single process" make. Now A,B,C and D are single targets (e.g. libraries), not modules, but that doesn't make a difference. The idea is to have one makefile for each target that can be used to build this target in case all preconditions are met (files are present in solver), exactly like you do a "build $(MODULE)" currently. (Sidestep: in fact we will plan to have a makefile that includes the makefiles of all targets in one module, to support the current working style of our developers. But that isn't important for now.) For a complete build there will be another makefile that includes the makefiles of all targets. Using this "super makefile" makes a big difference for the build stability. Regardless if you do a full or a partial build (like e.g. an incompatible build from vcl onwards), the same makefile is used always, containing all dependencies. A missing dependency declaration now means that the "super makefile" does not include the makefile that explains how a particular precondition can be built in case it isn't there (usually meaning: in the solver). In fact a missing dependency now becomes a missing rule. Different to the build.pl approach a missing rule/dependency means that it is not present anywhere - it does not happen that a build for "B" knows how to build "A", but a build for "C" doesn't know it. Thus the masking of missing dependencies that I have described above is not there. Moreover, a missing rule will be detected even before the build starts, so mistakes are found faster: it is sufficient to call "make" with an empty solver to detect the missing rules. So unless you can't bear the overhead of evaluating
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Wed, Jan 13, 2010 at 5:21 PM, bjoern michaelsen - Sun Microsystems - Hamburg Germany wrote: > Mu. As long as you have a recursive build process, you have a lot of > implicit dependencies and those are hurting parallelization. On top of > that, process instantiation and file-I/O is very, very expensive on > Windows, the slowest and most problematic platform of all (both are used > extensively when doing recursion). This might not matter too much for a > full build that package maintainers usually do. But it matters very > much for the change->rebuild->change-cycle that devs usually do (Both > implicit deps and recursion over mostly noop build tasks really hurt > here). There has been much speculation on the performances but very little hard facts. It seems to me that the most sensible way to proceed is for someone familiar with the Windows side of things (i.e. not me) to actually measure the build times. Since my CMake test only builds a subset of OOo, the other systems need to be limited to that as well. Of the top of my head, I can think of the following tests to compare: Existing build.pl/dmake The Gnu Make thing CMake with MSVC (MSBuild) generator CMake with MinGW makefile generator The code for all these exist, though CMake may need some platform adaptation. Without measurements any further discussion on performance only makes innocent electrons die in vain. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Wed, 13 Jan 2010 15:12:40 +0100 Martin Hollmichel wrote: > Mathias Bauer wrote: > > So the only way to reuse CMake makefiles for a complete build is > > recursively calling them or - as we do today in OOo - serialize the > > process. I don't think that this is a matter of performance per se, > > it's just that the benefit is missing we wanted to get from the new > > "single make process" approach. > > > no need to have this if you have unique targets, I guess ? Mu. As long as you have a recursive build process, you have a lot of implicit dependencies and those are hurting parallelization. On top of that, process instantiation and file-I/O is very, very expensive on Windows, the slowest and most problematic platform of all (both are used extensively when doing recursion). This might not matter too much for a full build that package maintainers usually do. But it matters very much for the change->rebuild->change-cycle that devs usually do (Both implicit deps and recursion over mostly noop build tasks really hurt here). BR, Bjoern -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
[tools-dev] Re: Building OpenOffice.org with GNU make
On 13/01/2010 15:12, Martin Hollmichel wrote: > Mathias Bauer wrote: >> Jussi Pakkanen wrote: >> >> >>> On Mon, Jan 11, 2010 at 7:57 PM, Thorsten Behrens >>> wrote: >>> >>> > functionality? Even if CMake eventually turns out to be too slow, > would it not make more sense to write your own custom CMake back > end rather than the configuration/generation front end? > > I guess it's now my turn to ask for sample code here. ;) >>> For a backend? No, sorry. I have never looked into that. >>> >>> But the issue raised earlier was that because CMake's Makefiles are >>> recursive (or something) they are too slow, probably because automake >>> does it this way and is slow. I personally do not think this will be >>> an issue. When running on Windows, the time taken by makefiles when >>> changing directories is insignificant compared to the time taken by >>> the compiler. But I have only tried it under Virtualbox and not at all >>> thoroughly. >>> >> The problem is not because the makefiles "are" recursive. The problem is >> that it looks if CMake does not offer a way to include all makefiles of >> the whole project (or at least larger parts of it if you think about a >> split build) into a single process without clashing of target names. >> > I can imagine that there might be a clashing of target names in a 1:1 > transistion from makefile,mk to other makefiles, but I general I would > consider the existance of target name clashes as a brain node (aka bug) > which should be solved easy. >> So the only way to reuse CMake makefiles for a complete build is >> recursively calling them or - as we do today in OOo - serialize the >> process. I don't think that this is a matter of performance per se, it's >> just that the benefit is missing we wanted to get from the new "single >> make process" approach. >> > no need to have this if you have unique targets, I guess ? i guess that if CMake does not _support_ this kind of splitting the definition of a single build definition over multiple CMakefiles, then i would expect it to not work in practice. because even if it happens to work today with some hacks and workarounds, if it is not an explicitly supported feature, some future change to CMake may break it. what we want is to split the definition of the build over several makefiles in multiple directories (so changes can be made "locally"), but then invoke some command that is able to read all of these "fragment" makefiles and check the timestamps of all the given dependencies only _once_. but of course i've never used CMake; how does CMake support this? >> Regards, >> Mathias >> >> > Martin -- I'm in search of myself. If you found me before I arrive, please have me wait. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Mathias Bauer wrote: Jussi Pakkanen wrote: On Mon, Jan 11, 2010 at 7:57 PM, Thorsten Behrens wrote: functionality? Even if CMake eventually turns out to be too slow, would it not make more sense to write your own custom CMake back end rather than the configuration/generation front end? I guess it's now my turn to ask for sample code here. ;) For a backend? No, sorry. I have never looked into that. But the issue raised earlier was that because CMake's Makefiles are recursive (or something) they are too slow, probably because automake does it this way and is slow. I personally do not think this will be an issue. When running on Windows, the time taken by makefiles when changing directories is insignificant compared to the time taken by the compiler. But I have only tried it under Virtualbox and not at all thoroughly. The problem is not because the makefiles "are" recursive. The problem is that it looks if CMake does not offer a way to include all makefiles of the whole project (or at least larger parts of it if you think about a split build) into a single process without clashing of target names. I can imagine that there might be a clashing of target names in a 1:1 transistion from makefile,mk to other makefiles, but I general I would consider the existance of target name clashes as a brain node (aka bug) which should be solved easy. So the only way to reuse CMake makefiles for a complete build is recursively calling them or - as we do today in OOo - serialize the process. I don't think that this is a matter of performance per se, it's just that the benefit is missing we wanted to get from the new "single make process" approach. no need to have this if you have unique targets, I guess ? Regards, Mathias Martin - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Thorsten Behrens wrote: >> Given that the syntax of the "build task description language" should >> be simple (because, if one needs it to be complex, one is likely Doing >> It Wrong(tm)) I wonder, if something that can be processed by the >> POSIX-shell or the C-Preprocessor would not be possible too(*). >> Actually, I am rather confident that would be possible. >> > Likely. There are two reasons I decided against using those: > * ugliness (I have yet to see the cpp macro program that does not >hurt my eyes) > * too much expressiveness (you'll get full turing completeness >easily, when you process stuff with the shell. I explicitely >wanted to keep things declarative, and prohibit those small local >if-then-else workarounds) "Ugliness" is in the eye of the beholder, but simple declarations instead of more or less complex expressions are indeed valuable. Let's see what we can do in the way Björn suggested. Regards, Mathias -- Mathias Bauer (mba) - Project Lead OpenOffice.org Writer OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS Please don't reply to "nospamfor...@gmx.de". I use it for the OOo lists and only rarely read other mails sent to it. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Thorsten Behrens wrote: > bjoern michaelsen - Sun Microsystems - Hamburg Germany wrote: >> > But the actual information contained in the above lines is actually >> > this: >> > >> > rscdep source files: tools/bootstrp/* >> > rscdep link libs: tl >> Quoting a band from Hamburg: "Jein" (=Yes and No). >> Of course, there is a lot of superfluous syntax in the current >> description, but you are dropping some information too: >> - Where is the information which naming convention the tl-lib is >> following? >> > That information is implicitely already contained in the name 'tl'. > As you notice further down, the gmakegen script has a static map for > those lib names. > >> - What about the files in tools/bootstrp, that are not generating >> object files that should be liked into rscdep (sspretty.cxx for >> example)? >> > Yeah, badly chosen example. The absolute majority of OOo code _is_ > organized in a fashion that allotting sources files to libs works on > a per-directory basis. At any rate, the generator script can do > both. I don't think that we must keep the bad habit of having source files of different libs in one directory. We can easily fix that on demand, so this shouldn't be a roadblock for anything. Regards, Mathias -- Mathias Bauer (mba) - Project Lead OpenOffice.org Writer OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS Please don't reply to "nospamfor...@gmx.de". I use it for the OOo lists and only rarely read other mails sent to it. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote: > On Mon, Jan 11, 2010 at 7:57 PM, Thorsten Behrens wrote: > >>> functionality? Even if CMake eventually turns out to be too slow, >>> would it not make more sense to write your own custom CMake back >>> end rather than the configuration/generation front end? >>> >> I guess it's now my turn to ask for sample code here. ;) > > For a backend? No, sorry. I have never looked into that. > > But the issue raised earlier was that because CMake's Makefiles are > recursive (or something) they are too slow, probably because automake > does it this way and is slow. I personally do not think this will be > an issue. When running on Windows, the time taken by makefiles when > changing directories is insignificant compared to the time taken by > the compiler. But I have only tried it under Virtualbox and not at all > thoroughly. The problem is not because the makefiles "are" recursive. The problem is that it looks if CMake does not offer a way to include all makefiles of the whole project (or at least larger parts of it if you think about a split build) into a single process without clashing of target names. So the only way to reuse CMake makefiles for a complete build is recursively calling them or - as we do today in OOo - serialize the process. I don't think that this is a matter of performance per se, it's just that the benefit is missing we wanted to get from the new "single make process" approach. Regards, Mathias -- Mathias Bauer (mba) - Project Lead OpenOffice.org Writer OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS Please don't reply to "nospamfor...@gmx.de". I use it for the OOo lists and only rarely read other mails sent to it. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Tue, 12 Jan 2010 17:37:59 +0200 Jussi Pakkanen wrote: > On Tue, Jan 12, 2010 at 12:35 PM, bjoern michaelsen - Sun Microsystems > - Hamburg Germany wrote: > > One dependency less (because you will need an additional "native" > > build system like GNU make or MS Build anyway). > > Well yes, but you add a build-dependency to Python (if there isn't one > already, I don't know). In a way there is, because of Mercurial. But I am also uncomfortable to depend on Python for the build. > Even if you ignore that, there is a dependency to either an external > build tool or to a custom build script thing. Choosing which way to go > should be naturally done by the people who have to use/maintain them. See my reply to Thorsten. I think the translation from a simple syntax to the makefile syntax is possible with quite a few tools we will already have as a dep, for example: - bash or even POSIX sh - the C preprocessor - GNU make itself > No, CMake is a replacement for Autoconf + Automake. Neither "builds > anything by themselves" but rather produce output for a lower level > tool (e.g. Make). A modern Make does not have to be low level at all. > Using CMake only for configuration is doing things very wrong. I guess > you could do it if you really, really wanted to but it is going to be > hard and counterproductive. "One job, one tool" has been proven time and again to be a Good Thing(tm). But that is a philosophical -- or even sophistic -- discussion we should postpone until all editor flamewars have been resolved. ;) Best Regards, Bjoern -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Tue, Jan 12, 2010 at 12:35 PM, bjoern michaelsen - Sun Microsystems - Hamburg Germany wrote: >> What is the benefit you get from this instead of using something like >> CMake that already has a mature implementation of this functionality? > One dependency less (because you will need an additional "native" > build system like GNU make or MS Build anyway). Well yes, but you add a build-dependency to Python (if there isn't one already, I don't know). Even if you ignore that, there is a dependency to either an external build tool or to a custom build script thing. Choosing which way to go should be naturally done by the people who have to use/maintain them. > I would consider that to be an option. Actually I think it is wrong to > describe CMake as a build system as it does not build anything itself. > CMake is a replacement for autoconf and should be considered as such in > first place. No, CMake is a replacement for Autoconf + Automake. Neither "builds anything by themselves" but rather produce output for a lower level tool (e.g. Make). Using CMake only for configuration is doing things very wrong. I guess you could do it if you really, really wanted to but it is going to be hard and counterproductive. > That being said, I have seen the GNU makefiles that are > being generated by CMake and their readability is horrible. If > something goes wrong there, it is really hard not only to debug the > makefile (for bugs in cmake itself), but it is also hard to just "trace > the error" from the errormessage of the native build system to the > sources (for bugs in the description given to cmake). This is a fair point. Anecdotally, I have used CMake on several different projects and have never had the need to look into the generated files, they have just worked. Then again, OOo has the tendency to break tools just by its sheer size. :) - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, 11 Jan 2010 16:05:57 +0200 Jussi Pakkanen wrote: > I would like to point out that what you are doing is generating your > own language and a build tool/generator based on that. There's nothing > wrong with it as such, but this is reinventing the wheel again (just > like Google's GYP). Instead of custom dmake/build.pl you would have > custom gnumakegen/gnumake_or_something. There is a huge difference between dmake/build.pl and the GNU make prototype. The current GNU make prototype (although incomplete) is a clean <1KLOC implementation including thb's stuff. The current dmake/build.pl toolchain is >38KLOC and pretty hard to maintain. > What is the benefit you get from this instead of using something like > CMake that already has a mature implementation of this functionality? One dependency less (because you will need an additional "native" build system like GNU make or MS Build anyway). > Even if CMake eventually turns out to be too slow, would it not make > more sense to write your own custom CMake back end rather than the > configuration/generation front end? I would consider that to be an option. Actually I think it is wrong to describe CMake as a build system as it does not build anything itself. CMake is a replacement for autoconf and should be considered as such in first place. That being said, I have seen the GNU makefiles that are being generated by CMake and their readability is horrible. If something goes wrong there, it is really hard not only to debug the makefile (for bugs in cmake itself), but it is also hard to just "trace the error" from the errormessage of the native build system to the sources (for bugs in the description given to cmake). Best Regards, Bjoern -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
bjoern michaelsen - Sun Microsystems - Hamburg Germany wrote: > > But the actual information contained in the above lines is actually > > this: > > > > rscdep source files: tools/bootstrp/* > > rscdep link libs: tl > Quoting a band from Hamburg: "Jein" (=Yes and No). > Of course, there is a lot of superfluous syntax in the current > description, but you are dropping some information too: > - Where is the information which naming convention the tl-lib is > following? > That information is implicitely already contained in the name 'tl'. As you notice further down, the gmakegen script has a static map for those lib names. > - What about the files in tools/bootstrp, that are not generating > object files that should be liked into rscdep (sspretty.cxx for > example)? > Yeah, badly chosen example. The absolute majority of OOo code _is_ organized in a fashion that allotting sources files to libs works on a per-directory basis. At any rate, the generator script can do both. Regarding your point of _where_ the globbing should take place: currently it's run from the generated makefile, so your thoughts on when to re-run the generator don't apply. ;) > > [makefile generator input] > > > > What do you think? > Looking a lot better that the GNU make input files, that are currently > used in the GNU make prototype for sure. However, I wonder how the > deeply nested lists and maps of python are looking to the > (python)-untrained eye. Subjecting an unsuspecting colleague to the > syntax will tell, I guess. > Thanks. :) So, unsuspecting colleagues with the python-untrained eyes: what would be your take on this? > Another thing to be considered with this is the introduction of python > as a dependency that early in the toolchain. AFAIK currently we have > these dependencies for the build system: > > [snip] > Generally a worthy goal to reduce dependencies; though of little practical relevance here IMHO. FWICT python has a broader supported base than java, and I think we're past this phase of having _that_ being an optional build requirement, do we? ;) Additionally, hg needs python anyway. > Given that the syntax of the "build task description language" should > be simple (because, if one needs it to be complex, one is likely Doing > It Wrong(tm)) I wonder, if something that can be processed by the > POSIX-shell or the C-Preprocessor would not be possible too(*). > Actually, I am rather confident that would be possible. > Likely. There are two reasons I decided against using those: * ugliness (I have yet to see the cpp macro program that does not hurt my eyes) * too much expressiveness (you'll get full turing completeness easily, when you process stuff with the shell. I explicitely wanted to keep things declarative, and prohibit those small local if-then-else workarounds) > Still, there remains the problem with makefile generation (when? what > are the dependencies? does the generator glob for files or does the GNU > make build system do this?), which I guess we should discuss first, > because "All problems in computer science can be solved by another > level of indirection except for the problem of too many layers of > indirection." > Sure. :) So, for my current draft, the gnu make system does it. For a future visual studio project generator, I guess the generator script would have to do that. Both approaches are valid in their respective area, I think. Whether the former might pose performance issues remains to be seen, but would have an easy fix (just fix the generator. having gnu makefiles as the authoritative input would make that much harder). Cheers, -- Thorsten pgpWb5TN0x6CQ.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On 01/11/10 23:44, bjoern michaelsen - Sun Microsystems - Hamburg Germany wrote: Am Mon, 11 Jan 2010 18:04:21 +0100 schrieb Stephan Bergmann : Which, at least in its simplest form, would imply that it necessarily modifies the source tree by adding files to it, something I personally would prefer moving away from. No. Generated makefiles are _generated_ and thus not source files. They shouldnt be generated to the read-only source tree, but reside in the workdir (those dirs with are located at $MODULE/$PLATFORM in the current build system, but they would not be in the source tree in a new build system). Good to hear you have that on your agenda. -Stephan - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Am Mon, 11 Jan 2010 18:04:21 +0100 schrieb Stephan Bergmann : > Which, at least in its simplest form, would imply that it necessarily > modifies the source tree by adding files to it, something I > personally would prefer moving away from. No. Generated makefiles are _generated_ and thus not source files. They shouldnt be generated to the read-only source tree, but reside in the workdir (those dirs with are located at $MODULE/$PLATFORM in the current build system, but they would not be in the source tree in a new build system). BR, Bjoern -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Am Mon, 11 Jan 2010 17:48:18 +0100 schrieb Thorsten Behrens : > I'd personally use it to dynamically generate the makefiles (i.e. > from configure, or triggered from a smallish global makefile). With GNU make there would not be a need for an "extra" makefile. You can include makefiles that arent generated yet, if there are rules available for their generation: make will then generate the included files and restart itself including the files. From the POV of the user there is no extra step needed to generate the makefiles. BR, Bjoern -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Am Mon, 11 Jan 2010 14:07:23 +0100
schrieb Thorsten Behrens :
> Ok, took longer than expected to find enough time for this, but here
> we go:
Hi Thorsten,
great to hear from you about this.
> After my initial performance worries had been sorted out, there are
> basically two points left I'd require from an all-new build system:
> - declarative syntax, as little duplication as possible
> - ability to do cross-platform builds
Valid point, but lets leave cross-platform builds out of the discussion
of a simple DSL for the build tasks as they have little to do with each
other, ok?
> For the former, look at this gnu makefile snippet (taken from
> gnu_make cws):
>
> [...]
>
> $(eval $(call gb_Executable_Executable,rscdep))
>
> $(eval $(call gb_Executable_set_linked_libs,rscdep,\
> $(call gb_OOoLibrary_get_linknames,\
> tl \
> ) \
> ))
>
> $(eval $(call gb_Executable_add_exception_objects,rscdep,\
> tools/bootstrp/appdef \
> tools/bootstrp/command \
>
> [etc.]
>
> Call me purist, but there's a lot of noise in these lines, that
> convey little, if any information. Plus, there's redundancy, in the
> form of name prefixes, gnu make plumbing to call, etc., and also the
> need to exhaustively list all object files (by listing paths names
> of cxx files, stripped from the .cxx extension).
>
> But the actual information contained in the above lines is actually
> this:
>
> rscdep source files: tools/bootstrp/*
> rscdep link libs: tl
Quoting a band from Hamburg: "Jein" (=Yes and No).
Of course, there is a lot of superfluous syntax in the current
description, but you are dropping some information too:
- Where is the information which naming convention the tl-lib is
following?
- What about the files in tools/bootstrp, that are not generating
object files that should be liked into rscdep (sspretty.cxx for
example)?
The first information is hidden in you translation-implementation,
which I think is not better that having it explicit in the makefile.
While "Beautiful is better than ugly.", there is also "Explicit is
better than implicit.".
The cleanest way (which is still dirty) IMHO would be to have the
mapping of libs and naming conventions defined in one global file (in
solenv/inc).
The root problem of course is the use of pretty silly naming conventions
(platform in the filename, wtf?) and the use of different
contradicting naming conventions.
The second information is of course a problem with our source tree, but
unfortunately it seems not at all uncommon to have *.cxx files for
totally different libs in the same dir in the OOo source tree (sw is an
example of a total mess in that regard). If we had "clean
dirs" (only sources for one binary in one dir), it would be rather easy
to implement this in GNU make itself (but it will cost a bit of
performance, as make would need to collect all source files on every
run).
Using a "gnu make generator" for globbing the list of source files
generates a new set of problems when one thinks about how the generator
is run. The generator might actually be run:
a) manually or
b) always (on every make call)
c) only when it needs to be run by checking dependencies
b) likely is a performance hog (needs investigation covering all the
cornercases like idl-compilation etc.).
a) is evil, as it introduces shaky dependencies and little is gained by
it. Lemme explain: We are introducing dynamically collected source
files so that a dev does not need to manually declare the additional
file to the build system. With b) however, we traded "manually
declaring the source file" against "manually needing to regenerate the
makefiles", which I feel to be not much gain as both will be forgotten
by the careless dev.
c) would be nice. Unfortunately, it is not cheap to find out if a
makefile needs to be regenerated as it depends on the dirstate of the
source tree (or to be more specific: the parts of the source tree that
are referenced in it). Finding out _if_ a makefile needs to be
regenerated is likely just as or more expensive that just generating
it, thowing us back to solution b) with its drawbacks.
> So I hacked up some python to parse a sort of declarative input
> file, and generate gnu makefiles with Björn's eval framework out of
> that. Corresponding to the above lines, this would be the input
> snippet:
>
> targets: [
> {
> executable: {
> name: 'rscdep'
> arch: host
> }
> sources: {
> glob: '*.cxx'
> paths: ['tools/bootstrp']
> }
> linking: ['tl']
> } ]
>
> I think it's cleaner, and there's definitely not much (should I say
> any?) redundancy left. Additionally, one can enhance the script to
> generate makefiles for pretty much every make tool of this world,
> including eclipse/netbeans/visual studio project files.
> ...
>
> What do you think?
Looking a lot better that the GNU make input files, that are currently
used in the GNU make prototype for sure. However, I wonder how the
deeply nested list
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, Jan 11, 2010 at 7:57 PM, Thorsten Behrens wrote: >> functionality? Even if CMake eventually turns out to be too slow, >> would it not make more sense to write your own custom CMake back >> end rather than the configuration/generation front end? >> > I guess it's now my turn to ask for sample code here. ;) For a backend? No, sorry. I have never looked into that. But the issue raised earlier was that because CMake's Makefiles are recursive (or something) they are too slow, probably because automake does it this way and is slow. I personally do not think this will be an issue. When running on Windows, the time taken by makefiles when changing directories is insignificant compared to the time taken by the compiler. But I have only tried it under Virtualbox and not at all thoroughly. If, however, both the Makefile and MSBuild generators turn out to be too slow, the CMake developers are very responsive and would most likely work with you to improve the performance. Only after this fails would you need to look into writing your own backend. > Mostly marketing. I'm not too interested who else is using it, > whether it has reached a tipping point etc. etc., but rather how an > actual cmake solution to the requirements mentioned before may look > like (scaffolding, drafting, etc. totally acceptable - just need the > gist of it). :) Well, my CMake build env experiment was posted here again just now. Is that sufficient for evaluation or do you need something more? - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, Jan 11, 2010 at 8:11 PM, Martin Hollmichel wrote: > I started some time ago a cmake prototype for OOo in my spare time > (http://hg.services.openoffice.org/hg/cws/mh6bc/) for the latest status > please see the latest ReadMe.txt in the Source root for the most recent > status). You're invited to join this prototype, but be warned: it does not > work and kills your cat when try to run that :-). You might want to look look at my attempt: http://lists.freedesktop.org/archives/ooo-build/2009-August/000181.html It goes quite a lot further and solves some of the issues listed in your readme (detecting STLPort and Boost, etc). There's also the dmake -> cmake converter script so you don't have to keep writing the files by hand. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Hi, I think it's cleaner, and there's definitely not much (should I say any?) redundancy left. Additionally, one can enhance the script to generate makefiles for pretty much every make tool of this world, including eclipse/netbeans/visual studio project files. I would like to point out that what you are doing is generating your own language and a build tool/generator based on that. There's nothing wrong with it as such, but this is reinventing the wheel again (just like Google's GYP). Instead of custom dmake/build.pl you would have custom gnumakegen/gnumake_or_something. What is the benefit you get from this instead of using something like CMake that already has a mature implementation of this functionality? Even if CMake eventually turns out to be too slow, would it not make more sense to write your own custom CMake back end rather than the configuration/generation front end? I second the demand for simple, readable description files. ath the first glance, this seem very easy with cmake, it was fun prototyping this on Linux for the OOo tool-chain up to the idlc. regmerge level. A lot of writing could be saved in comparisons to Ooo current build system. during the prototyping I also was surprise about all the stuff we're doing in Ooo's makefile's (some superfluous, some really needed) and I was surprised that much of the stuff already is covered by cmake (just proved by reading the documentation, not all tested in reality). Things are getting a bit more complicated, if you're looking on some old grown specialties of OOo, e.g the generations of import libraries on Win32. At some stage it is useful, if not even required to have an cmake expert with the OOo project who can determine with some experience how and where to implement such one-off's of OOo. so we still have three valid alternatives: * renew and modernize our old dmake environment * re-write the build environment with a more modern tool and use some more modern patterns * reuse and enhance already abstract buildenv like cmake. I'm not yet sure what will be the best way to go. For further information here is a Google Tech Talk about CMake and all related things (testing, code coverity, packaging, etc, etc) by one of the creators. If the build tool decision is not yet final, it is worth watching. The CPack and CTest enhancements of cmake are indeed quite impressive, a transition from scp to CPack would be very interesting (and kicking build.pl / deliver.pl and solver at the same time). I started some time ago a cmake prototype for OOo in my spare time (http://hg.services.openoffice.org/hg/cws/mh6bc/) for the latest status please see the latest ReadMe.txt in the Source root for the most recent status). You're invited to join this prototype, but be warned: it does not work and kills your cat when try to run that :-). Martin - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Jussi Pakkanen wrote: > > I think it's cleaner, and there's definitely not much (should I say > > any?) redundancy left. Additionally, one can enhance the script to > > generate makefiles for pretty much every make tool of this world, > > including eclipse/netbeans/visual studio project files. > > I would like to point out that what you are doing is generating your > own language and a build tool/generator based on that. > Yes. I'm piggy-backing on Björn's own rewrite, adding the (IMHO) crucial features that a new build system *should* have, once we're going for this endeavour. > There's nothing wrong with it as such, but this is reinventing the > wheel again (just like Google's GYP). Instead of custom dmake/ > build.pl you would have custom gnumakegen/gnumake_or_something. > What is the benefit you get from this instead of using something > like CMake that already has a mature implementation of this > functionality? Even if CMake eventually turns out to be too slow, > would it not make more sense to write your own custom CMake back > end rather than the configuration/generation front end? > I guess it's now my turn to ask for sample code here. ;) I've no strong opinion on cmake, except for the fact that it sucks at cross-building; the input syntax is ~ok, though not really good on enforcing structure; no idea how much effort there is writing a custom backend vs. having a dedicated approach in the first place (i.e. how much code could we share, e.g. from the eclipse/visual studio output targets in cmake, vs. having it all custom anyways?) > For further information here is a Google Tech Talk about CMake and all > related things (testing, code coverity, packaging, etc, etc) by one of > the creators. If the build tool decision is not yet final, it is worth > watching. > > http://www.youtube.com/watch?v=8Ut9o4OdSC0 > Mostly marketing. I'm not too interested who else is using it, whether it has reached a tipping point etc. etc., but rather how an actual cmake solution to the requirements mentioned before may look like (scaffolding, drafting, etc. totally acceptable - just need the gist of it). :) Cheers, -- Thorsten pgpBJfu60iKAW.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On 01/11/10 17:48, Thorsten Behrens wrote: Stephan Bergmann wrote: So I hacked up some python to parse a sort of declarative input file, and generate gnu makefiles with Björn's eval framework out of that. At what stage of the development/build process would that script be called? I'd personally use it to dynamically generate the makefiles (i.e. from configure, or triggered from a smallish global makefile). Which, at least in its simplest form, would imply that it necessarily modifies the source tree by adding files to it, something I personally would prefer moving away from. -Stephan - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Stephan Bergmann wrote: > >So I hacked up some python to parse a sort of declarative input > >file, and generate gnu makefiles with Björn's eval framework out of > >that. > > At what stage of the development/build process would that script be called? > I'd personally use it to dynamically generate the makefiles (i.e. from configure, or triggered from a smallish global makefile). -- Thorsten pgprzdC1BzTxq.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On 01/11/10 14:07, Thorsten Behrens wrote: So I hacked up some python to parse a sort of declarative input file, and generate gnu makefiles with Björn's eval framework out of that. At what stage of the development/build process would that script be called? -Stephan - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, Jan 11, 2010 at 3:07 PM, Thorsten Behrens wrote: > I think it's cleaner, and there's definitely not much (should I say > any?) redundancy left. Additionally, one can enhance the script to > generate makefiles for pretty much every make tool of this world, > including eclipse/netbeans/visual studio project files. I would like to point out that what you are doing is generating your own language and a build tool/generator based on that. There's nothing wrong with it as such, but this is reinventing the wheel again (just like Google's GYP). Instead of custom dmake/build.pl you would have custom gnumakegen/gnumake_or_something. What is the benefit you get from this instead of using something like CMake that already has a mature implementation of this functionality? Even if CMake eventually turns out to be too slow, would it not make more sense to write your own custom CMake back end rather than the configuration/generation front end? For further information here is a Google Tech Talk about CMake and all related things (testing, code coverity, packaging, etc, etc) by one of the creators. If the build tool decision is not yet final, it is worth watching. http://www.youtube.com/watch?v=8Ut9o4OdSC0 - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Mathias Bauer wrote:
> > Understood. I pretty much agree with the goals, assume you did your
> > due diligence on verifying that gnu make does not trip over on the
> > full input set (that was my point of cautioning you, with the bjam
> > story), and am now trying to explore ideas on how to make the actual
> > makefiles appealing - the current state is not convincing, just
> > plainly because they're not substantially different from the ideal
> > dmake makefile in the existing system - and with their redundancies,
> > will suffer the same bitrot as the old system.
> Sounds good. :-)
> Any improvement would be welcome.
>
Ok, took longer than expected to find enough time for this, but here
we go:
After my initial performance worries had been sorted out, there are
basically two points left I'd require from an all-new build system:
- declarative syntax, as little duplication as possible
- ability to do cross-platform builds
For the former, look at this gnu makefile snippet (taken from
gnu_make cws):
[...]
$(eval $(call gb_Executable_Executable,rscdep))
$(eval $(call gb_Executable_set_linked_libs,rscdep,\
$(call gb_OOoLibrary_get_linknames,\
tl \
) \
))
$(eval $(call gb_Executable_add_exception_objects,rscdep,\
tools/bootstrp/appdef \
tools/bootstrp/command \
[etc.]
Call me purist, but there's a lot of noise in these lines, that
convey little, if any information. Plus, there's redundancy, in the
form of name prefixes, gnu make plumbing to call, etc., and also the
need to exhaustively list all object files (by listing paths names
of cxx files, stripped from the .cxx extension).
But the actual information contained in the above lines is actually
this:
rscdep source files: tools/bootstrp/*
rscdep link libs: tl
So I hacked up some python to parse a sort of declarative input
file, and generate gnu makefiles with Björn's eval framework out of
that. Corresponding to the above lines, this would be the input
snippet:
targets: [
{
executable: {
name: 'rscdep'
arch: host
}
sources: {
glob: '*.cxx'
paths: ['tools/bootstrp']
}
linking: ['tl']
} ]
I think it's cleaner, and there's definitely not much (should I say
any?) redundancy left. Additionally, one can enhance the script to
generate makefiles for pretty much every make tool of this world,
including eclipse/netbeans/visual studio project files.
The "arch: host" line is not yet fleshed out in the script, nor in
the gnu_make cws, but is indented for doing cross-builds: just
annotate every target with one out of "arch: [host:target:both]" to
later advise the build system which output architecture to compile
for.
Prototypical script, input files & patch against CWS gnu_make
attached (to handle input file globs). For testing, call it like
this: ./gmakegen.py tools_rscdep.in
What do you think?
-- Thorsten
gnumakegen.tgz
Description: application/compressed-tar
pgpJpcsblWlGK.pgp
Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Thorsten Behrens wrote: > Mathias Bauer wrote: >> So if you could explain how bjam (or any other make system that >> someone wants to suggest here) solves our problems or why the >> problems that require bjam to be resolved are even bigger than >> those we try to fix, we might be able to get somewhere. >> > I did that, if you could check my initial posting. Bjoern said he > profiled gnu make with large enough synthetic rule sets, point is > settled for the while, I'd say. OK, so I misunderstood your intention for mentioning bjam. >> You can find most of our findings, discussions etc. in the wiki (I have >> posted the link several times) or in Björn's and my blogs. >> > Nothing we're discussing here except Bjoern's initial blog from > Friday, sorry Mathias. This is not to criticize anybody, just > pointing out that, when such things are announced after-the-fact, > many of the stupid ideas & questions that have probably been > resolved internally before, will crop up in the public discussion > again. Do you really believe that posting the "stupid ideas and questions" would have prevented that they crop up in the public discussion? :-) >> Björn also will post some more data about GNU make investigations >> soon. They never were planned to be "internal findings", but >> posting them without prior explanation wouldn't have made sense >> and even now they don't add anything to the discussion we wanted >> to start. The topic is not "can GNU make do that", but "do we have >> the right goals" and "GNU make can reach the goals, which other >> tools can do it also or perhaps even better". >> > Understood. I pretty much agree with the goals, assume you did your > due diligence on verifying that gnu make does not trip over on the > full input set (that was my point of cautioning you, with the bjam > story), and am now trying to explore ideas on how to make the actual > makefiles appealing - the current state is not convincing, just > plainly because they're not substantially different from the ideal > dmake makefile in the existing system - and with their redundancies, > will suffer the same bitrot as the old system. Sounds good. :-) Any improvement would be welcome. Regards, Mathias -- Mathias Bauer (mba) - Project Lead OpenOffice.org Writer OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS Please don't reply to "nospamfor...@gmx.de". I use it for the OOo lists and only rarely read other mails sent to it. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Mathias Bauer wrote: > So if you could explain how bjam (or any other make system that > someone wants to suggest here) solves our problems or why the > problems that require bjam to be resolved are even bigger than > those we try to fix, we might be able to get somewhere. > I did that, if you could check my initial posting. Bjoern said he profiled gnu make with large enough synthetic rule sets, point is settled for the while, I'd say. > You can find most of our findings, discussions etc. in the wiki (I have > posted the link several times) or in Björn's and my blogs. > Nothing we're discussing here except Bjoern's initial blog from Friday, sorry Mathias. This is not to criticize anybody, just pointing out that, when such things are announced after-the-fact, many of the stupid ideas & questions that have probably been resolved internally before, will crop up in the public discussion again. > Björn also will post some more data about GNU make investigations > soon. They never were planned to be "internal findings", but > posting them without prior explanation wouldn't have made sense > and even now they don't add anything to the discussion we wanted > to start. The topic is not "can GNU make do that", but "do we have > the right goals" and "GNU make can reach the goals, which other > tools can do it also or perhaps even better". > Understood. I pretty much agree with the goals, assume you did your due diligence on verifying that gnu make does not trip over on the full input set (that was my point of cautioning you, with the bjam story), and am now trying to explore ideas on how to make the actual makefiles appealing - the current state is not convincing, just plainly because they're not substantially different from the ideal dmake makefile in the existing system - and with their redundancies, will suffer the same bitrot as the old system. Cheers, -- Thorsten pgpzK3a9qWqvu.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Hi Thorsten, Björns approach is the first one in the last 10 years I can see has potential, to simplify the build, to improve scalability, to improve dependencies, to improve build performance, to align our makefiles, to reduce code. A build system, in particular one based on (GNU) make, is a query system. I as a user ask it for a target / file, the system figures out how to best provide it. If the system is good, it utilizes my resources properly. As outlined in a blog posting a while ago, we shouldn't need to write any makefiles anyway, agreed. This is some way to go. Compiler / Linker flags for a particular source file should certainly be located in or close to the source file. Same holds true for linking libraries. If one follows some conventions, a build system can easily derive what needs to be linked. Ideally the build system would even be something I call "stable", after a build (cold or hot) the output directory would never hold a file not needed for the target, directly or indirectly. But whatever we can think of, it needs to be implemented iteratively, I am sure you agree. Björns prototype really is a good step forward, all the other nice things can be implemented on top. So let's go Kay Thorsten Behrens wrote: Hi Mathias, most of the points you've raised I already replied to in my followup to Bjoern (including my ideal msword lib makefile) - Mathias Bauer wrote: build.pl uses module dependencies, not target dependencies, so it has an inherent susceptibility to bottlenecks. Basically all of our c++ sources could be built in parallel (as they don't depend on each other), but with build.pl we always have to wait until header files are "delivered" or created. And because the dependency granularity level is the "module" (not a real target like e.g. a library), we can't use as much parallelization as possible. This becomes even more painful if you don't build the whole office, but only some parts of it, e.g. in a split build or if you rebuild several "modules". Ah interesting - so we're then moving away from the solver concept? No, really, there's nothing nailed until now. If you or anybody else knew a better way and(!) offered help and cooperation, there's nothing that would hold us back from doing it differently. I find this "and(!)" slightly worrying - not that I would not lend a helping hand; but why are you refusing advise from people just because they may not have time helping you with coding? That's why he went for bjam ... Can you explain if bjam is able to fulfill the requirements Björn and I have mentioned and what else it can do better than GNU make? Or can you at least explain why you perhaps prefer it? I didn't say I'd prefer it. I was just pointing to that project, and the experiences. Maybe it would help the discussion if more of the internal findings you mentioned would be public - come to think of it, maybe it would have been nice to have the whole discussion that led to this prototype public, in the first place ... ;) Cheers, -- Thorsten - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Thorsten Behrens wrote: > Hi Mathias, > > most of the points you've raised I already replied to in my followup > to Bjoern (including my ideal msword lib makefile) - > > Mathias Bauer wrote: >> build.pl uses module dependencies, not target dependencies, so it has an >> inherent susceptibility to bottlenecks. Basically all of our c++ sources >> could be built in parallel (as they don't depend on each other), but >> with build.pl we always have to wait until header files are "delivered" >> or created. And because the dependency granularity level is the "module" >> (not a real target like e.g. a library), we can't use as much >> parallelization as possible. This becomes even more painful if you don't >> build the whole office, but only some parts of it, e.g. in a split build >> or if you rebuild several "modules". >> > Ah interesting - so we're then moving away from the solver concept? This is irrelevant for the system in question. With or without solver, you will have the same problems with our current build system. The only thing that matters is how many modules you want to build in parallel. > >> No, really, there's nothing nailed until now. If you or anybody else >> knew a better way and(!) offered help and cooperation, there's nothing >> that would hold us back from doing it differently. >> > I find this "and(!)" slightly worrying - not that I would not lend a > helping hand; but why are you refusing advise from people just > because they may not have time helping you with coding? Did I say that we would refuse advise if it clearly showed improvements? Of course not, why should we? But I wouldn't call "don't use X, Z is much better" an advise. Or, to say it with Monty Python: "This is not an argument, this is contradiction." "No, it isn't!" "Yes it is!" etc. etc. Perhaps I should rephrase my statement: the best I can see is that people show us that what we want to achieve by using GNU make can be done even better, simpler, faster etc. in a way we don't know until now and help us to get that implemented. As far as I'm concerned, the second part is optional, but of course much desired. >> > That's why he went for bjam ... >> >> Can you explain if bjam is able to fulfill the requirements Björn and I >> have mentioned and what else it can do better than GNU make? Or can you >> at least explain why you perhaps prefer it? >> > I didn't say I'd prefer it. I was just pointing to that project, and > the experiences. Just pointing to a project without giving a hint how that could solve the problems isn't very helpful ATM (though I appreciate the hint and surely will have a look when I have some time). So if you could explain how bjam (or any other make system that someone wants to suggest here) solves our problems or why the problems that require bjam to be resolved are even bigger than those we try to fix, we might be able to get somewhere. > Maybe it would help the discussion if more of the > internal findings you mentioned would be public - come to think of > it, maybe it would have been nice to have the whole discussion that > led to this prototype public, in the first place ... ;) You can find most of our findings, discussions etc. in the wiki (I have posted the link several times) or in Björn's and my blogs. Björn also will post some more data about GNU make investigations soon. They never were planned to be "internal findings", but posting them without prior explanation wouldn't have made sense and even now they don't add anything to the discussion we wanted to start. The topic is not "can GNU make do that", but "do we have the right goals" and "GNU make can reach the goals, which other tools can do it also or perhaps even better". Regards, Mathias -- Mathias Bauer (mba) - Project Lead OpenOffice.org Writer OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS Please don't reply to "nospamfor...@gmx.de". I use it for the OOo lists and only rarely read other mails sent to it. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, 07 Dec 2009 13:46:19 +0100 Thorsten Behrens wrote: > That's exactly why I wonder if we should have a level of abstraction > above naked gnu makefiles - i.e. a DSL that gets translated into > makefiles. Actually, the files describing the actual stuff to build are a kind of DSL. If you look at target_lib_msword.mk there is no makefile syntax or logic beyond the $(eval $(call)) statements. What is called by the eval/call statements is our own high level DSL/API. You could rather trivially parse and translate that to MSBuild-XML, CMake-Files, Jam-Files or anything else, should the need arise someday. The important stuff and the hard part will be to: a) translate all files into clean declarative statements. b) keep those files clean and not letting them bit rot. Best Regards, Bjoern -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
bjoern michaelsen - Sun Microsystems - Hamburg Germany wrote: > > # loads cxxflags, weird sw include setup etc. > > include sw_defaults > Here you will either need to use dirty tricks or you will clash in a > global namespace, because you would like these defaults to be applied > to msword and others in sw, but not the others. > That's exactly why I wonder if we should have a level of abstraction above naked gnu makefiles - i.e. a DSL that gets translated into makefiles. > > And in fact inescapable, ~every implementation will leak under its > > abstractions. So I guess you're rather arguing for having only _one_ > > implementation - which is not obtainable in a cross-platform, > > opensource project. ;) > No, I am all for (according to the Zen of Python): > "There should be one-- and preferably only one --obvious way to do it." > Zen is nice, Python even more so, but let's face reality - the world out here is not homogeneous (and not a single version for every used tool) at all. Cheers, -- Thorsten pgppjb4Is4gry.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, 07 Dec 2009 11:01:02 +0100 Thorsten Behrens wrote: > This is a circular argument. Correct dependencies and the underlying > build/make system used are orthogonal. As an aside, I guess I'm only > questioning the rationale here, not the goal - getting rid of the > build.pl perl hell would be quite appreciated. ;) This is not an issue of correctness of dependencies, its about granularity of build.pl's dependencies. Build.pl dependencies are no finer than a module -- and the smallest build unit it knows about is a directory. > Granted, quite some redundancy results out of OOo's inconsistency on > many levels; Jep. In the long run it would be a good idea to get rid of those and not to work around them in build system. > still, why not encoding that knowledge in one place, > instead of spreading it over umpteen makefiles again? Actually the > current libs.mk does a better job of encapsulating things, compared > to the manual fetch of lib names via OOoLibrary_get*, RTLibrary_get* > etc. That is open to discussion. However, having a centralized library name registry is feeling just as wrong. (I remember saying that I wanted ugly things to be ugly in the build system and not do any cosmetics on them, so that they might be fixed at the root -- once you do cosmetics, you quickly end up depending on the ugliness and thus will never get rid of it.) > # loads cxxflags, weird sw include setup etc. > include sw_defaults Here you will either need to use dirty tricks or you will clash in a global namespace, because you would like these defaults to be applied to msword and others in sw, but not the others. What you could do is something like this: -- file inc_sw_defaults.mk #include guards here define set_sw_defaults $$(eval $$(call gb_OOoLibrary_set_include,$(1),...)) ... endef -- end file sw_defaults -- file target_lib_msword.mk include inc_sw_defaults.mk $(eval $(call set_sw_defaults,msword)) end file > msword_linklibs=$(extractdeps $(msword_SOURCES)) > > The last line may well be a pipe dream for the while, but at least > in theory, one should be able to glean the needed libs from the > #include statements ... You share that pipe dream with Kay Ramme (and me too). But lets be realistic -- we will have to learn to crawl before we try running. > > So a generated project file might introduce subtle different builds, > > is dependent on the version of the generator and is read-only. > > Since you can "generate a project from existing code" trivially in > > NetBeans, Visual Studio and others and just as well kick of a > > makefile-based build in all modern IDEs, what exactly is gained by > > generated project files? > > > If that gives the same developer experience - sure. I'd then be > interested in some pointers inside the wiki's build guide. ;) Will be considered. However, for the bigger modules like sw having just one module (and headers from solver) in the IDE can be really slow. Still, for stuff like renaming a method with a hard-to-grep/sed name (hmmm, like "Update" or "Modify") it might be worth it. > Would then be interesting to know how implicit rules are treated > inside gnu make (i.e. whether each instantiation counts) - we'd be > in that ballpark easily, with some 11k cxx files & what else comes > on top. Pattern rules definitely scale different than explicit rules for each target. One thing I will have to watch out for are variable scopes. However, we do not need a variable scope per cxx file -- in most cases only the linking targets (the libraries) have a variable scope and those are not so many that it would scare me. Still, I will do some (synthetic) test on those too. > Prior art: > http://www2.parc.com/csl/groups/sda/publications/papers/Kiczales-IMSA92/for-web.pdf Pfft, thats easy. There is always prior art at Xeroc PARC. ;-) > And in fact inescapable, ~every implementation will leak under its > abstractions. So I guess you're rather arguing for having only _one_ > implementation - which is not obtainable in a cross-platform, > opensource project. ;) No, I am all for (according to the Zen of Python): "There should be one-- and preferably only one --obvious way to do it." Best Regards, Bjoern Michaelsen -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Thorsten Behrens wrote: No, really, there's nothing nailed until now. If you or anybody else knew a better way and(!) offered help and cooperation, there's nothing that would hold us back from doing it differently. I find this "and(!)" slightly worrying - not that I would not lend a helping hand; but why are you refusing advise from people just because they may not have time helping you with coding? Because talk is cheap and if we have one thing in much more abundance than we need it is people with an opinion what others should do ? Just my 2 cents, pl -- "One SVN to rule them all, One SVN to check out from, One SVN to commit them all and on the harddisks bind them In Las Vegas where the server lies." - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
[tools-dev] Re: Building OpenOffice.org with GNU make
Hi Mathias, most of the points you've raised I already replied to in my followup to Bjoern (including my ideal msword lib makefile) - Mathias Bauer wrote: > build.pl uses module dependencies, not target dependencies, so it has an > inherent susceptibility to bottlenecks. Basically all of our c++ sources > could be built in parallel (as they don't depend on each other), but > with build.pl we always have to wait until header files are "delivered" > or created. And because the dependency granularity level is the "module" > (not a real target like e.g. a library), we can't use as much > parallelization as possible. This becomes even more painful if you don't > build the whole office, but only some parts of it, e.g. in a split build > or if you rebuild several "modules". > Ah interesting - so we're then moving away from the solver concept? > No, really, there's nothing nailed until now. If you or anybody else > knew a better way and(!) offered help and cooperation, there's nothing > that would hold us back from doing it differently. > I find this "and(!)" slightly worrying - not that I would not lend a helping hand; but why are you refusing advise from people just because they may not have time helping you with coding? > > That's why he went for bjam ... > > Can you explain if bjam is able to fulfill the requirements Björn and I > have mentioned and what else it can do better than GNU make? Or can you > at least explain why you perhaps prefer it? > I didn't say I'd prefer it. I was just pointing to that project, and the experiences. Maybe it would help the discussion if more of the internal findings you mentioned would be public - come to think of it, maybe it would have been nice to have the whole discussion that led to this prototype public, in the first place ... ;) Cheers, -- Thorsten pgpKhF392crUR.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Mon, 07 Dec 2009 11:17:30 +0100 Jan Holesovsky wrote: > Any chance to profile GNU make in this scenario, please? May be > there is some stupid mistake there, similar to the one in mkdepend > (http://www.openoffice.org/issues/show_bug.cgi?id=73081 - the > complexity of it grew exponentially)? I have not yet considered profiling GNU make, because even with OOo this will not be an issue when using pattern rules for the most common targets (C++ sources etc.). In a first attempt, I generated an explicit (non-pattern) rule for each target, and there it might have come up as an issue. But using pattern rules (which is WIP and almost finished), we should rarely end up with more than maybe 30 rules -- everything else is done via variables. Best Regards, Bjoern Michaelsen -- === Sitz der Gesellschaft: Sun Microsystems GmbH, Sonnenallee 1, D-85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Haering === - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Hi Björn, On Monday 07 of December 2009, Björn Michaelsen wrote: > > Additionally, and since you mentioned the desire to have only one > > make instance - last time someone tried to have gnu make hold all of > > OOo's dependency tree in one process, that guy (Kai Backman) ended > > up with absolutely pathetic performance & ridiculous mem usage. > > I did more than a few scalability tests with GNU make. There is a magic > barrier on the number of rules it can handle (above ~1 > scalability grinds to a halt), Any chance to profile GNU make in this scenario, please? May be there is some stupid mistake there, similar to the one in mkdepend (http://www.openoffice.org/issues/show_bug.cgi?id=73081 - the complexity of it grew exponentially)? Thank you, Kendy - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
[tools-dev] Re: Building OpenOffice.org with GNU make
On 04/12/2009 23:45, Rene Engelhard wrote: > Did anyone in this discussion propose libtool? I didn't see it. > (You can use automake without libtool) > > Grüße/Regards, > > Rene >> On 04/12/2009 18:41, Thorsten Behrens wrote: >>> Hi Bjoern, *, >>> >>> In line with that, why not using autotools? It's ugly, it has many >>> drawbacks, but it's _the_ standard for FLOSS. Plus, it has excellent >>> cross-build capabilities, something I consider increasingly >>> important. so, i always thought that "GNU autotools" is short for "GNU autoconf + GNU automake + GNU libtool". am i wrong here? is there an official definition somewhere? wikipedia says (GNU_build_system): The GNU build system comprises the GNU utility programs Autoconf, Automake, and Libtool. regards, michael -- "That's right!" shouted Vroomfondel, "we demand rigidly defined areas of doubt and uncertainty!" - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Björn Michaelsen wrote: > > That's not what I meant. The reason to also hand over a specific > > dmake option for multi-processor builds is that sometimes only one > > or two directories have their actual dependencies satisfied & are > > thus buildable. So it would be ~trivial to only hand "-P16" over to > > build.pl, and let it distribute that across the active dmake > > instances. > No, thats not trivial: the current bottlenecks cant really be noticed > by build.pl until its to late (unless build.pl kills runnning > dmake-processes for a directories and restarts them again, which will > hardly improve performance and might trigger additional subtile and hard > to debug bugs in the build system). > This is a circular argument. Correct dependencies and the underlying build/make system used are orthogonal. As an aside, I guess I'm only questioning the rationale here, not the goal - getting rid of the build.pl perl hell would be quite appreciated. ;) > > There's still loads of redundancy - > I cannot see a lot of redundancy there, unless one is willing to > reorganize quite a bit of the files in the source tree: > - clean up the weird include paths in sw > - for each shared library, keep all files in one directory > That would allow to collect the source files implicitly (still, > collecting source files dynamically might be a performance issue for > complete (re-)builds). > Granted, quite some redundancy results out of OOo's inconsistency on many levels; still, why not encoding that knowledge in one place, instead of spreading it over umpteen makefiles again? Actually the current libs.mk does a better job of encapsulating things, compared to the manual fetch of lib names via OOoLibrary_get*, RTLibrary_get* etc. - so what I'd ideally like to see is something along this lines (asking for a DSL again, I guess), instead of target_lib_msword.mk: # loads cxxflags, weird sw include setup etc. include sw_defaults libtarget=msword msword_SOURCES= \ $(listcxx sw/source/filter/rtf) \ $(listcxx sw/source/filter/ww8) msword_linklibs=$(extractdeps $(msword_SOURCES)) The last line may well be a pipe dream for the while, but at least in theory, one should be able to glean the needed libs from the #include statements ... > > Yes. And? It's still significantly lowering the barrier of entry. > > Loads of other projects do it (with the same limitation). > An digression beforehand: I guess we are talking about Windows here > mostly > Nope. I'm talking about IDE users in general, who are used to method/param completion, symbol database etc etc. > So a generated project file might introduce subtle different builds, > is dependent on the version of the generator and is read-only. Since you > can "generate a project from existing code" trivially in NetBeans, > Visual Studio and others and just as well kick of a makefile-based > build in all modern IDEs, what exactly is gained by generated project > files? > If that gives the same developer experience - sure. I'd then be interested in some pointers inside the wiki's build guide. ;) > > Additionally, and since you mentioned the desire to have only one > > make instance - last time someone tried to have gnu make hold all of > > OOo's dependency tree in one process, that guy (Kai Backman) ended > > up with absolutely pathetic performance & ridiculous mem usage. > I did more than a few scalability tests with GNU make. There is a magic > barrier on the number of rules it can handle (above ~1 > scalability grinds to a halt), but rarely one on targets. Unfortunately, > we will only really know in the end (In theory, theory and practice are > same. In practice -- not.). But the four modules in the prototype are > already a significant part of the source files and probably a major > part of the header targets (via #include). > Would then be interesting to know how implicit rules are treated inside gnu make (i.e. whether each instantiation counts) - we'd be in that ballpark easily, with some 11k cxx files & what else comes on top. > (1) http://www.joelonsoftware.com/articles/LeakyAbstractions.html > Prior art: http://www2.parc.com/csl/groups/sda/publications/papers/Kiczales-IMSA92/for-web.pdf And in fact inescapable, ~every implementation will leak under its abstractions. So I guess you're rather arguing for having only _one_ implementation - which is not obtainable in a cross-platform, opensource project. ;) Cheers, -- Thorsten pgp1ToyAHALCx.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Thorsten Behrens wrote: > Björn Michaelsen wrote: >> > - regarding parallelization, that's surely fixable with much less >> >effort in build.pl, no? >> Currently we are starting one dmake-process per directory and that >> dmake process does paralellization the directory. Implementing a >> recursive jobserver that communicates between dmake and build.pl would >> not only be ugly, it would also be a major effort. >> > That's not what I meant. The reason to also hand over a specific > dmake option for multi-processor builds is that sometimes only one > or two directories have their actual dependencies satisfied & are > thus buildable. So it would be ~trivial to only hand "-P16" over to > build.pl, and let it distribute that across the active dmake > instances. build.pl uses module dependencies, not target dependencies, so it has an inherent susceptibility to bottlenecks. Basically all of our c++ sources could be built in parallel (as they don't depend on each other), but with build.pl we always have to wait until header files are "delivered" or created. And because the dependency granularity level is the "module" (not a real target like e.g. a library), we can't use as much parallelization as possible. This becomes even more painful if you don't build the whole office, but only some parts of it, e.g. in a split build or if you rebuild several "modules". A make system with full dependencies and a single dependency tree only listing real targets, not virtual ones like "modules" does not have these defects. > >> > - what kind of dependency tracking is missing in the current system? >> Those that bite you on compatible builds. >> > Ah. That's what I thought. So nothing inherently missing in > dmake/build.pl, but "just" bugs in the makefiles. No, not in the make file. The bugs are in the notorious "build.lst" files. As you surely know, our makefiles don't have useful rependencies as this would require not only spcifying prerequisites but also rules. If you link a library against another one, there's nothing in our build system that tells how to create it in case it doesn't exist. Instead of that, you have to maintain another pseudo-makefile, the build.lst, that tells build.pl to "visit" other modules before, so that the case "prerequisite does not exist but no rule is defined" does not happen. This is very error prone, especially if the build.lst is changed later on. And at least on Windows this is quite time consuming. I was bitten by that quite often in my recent refactoring activities. The reason is that mistakes can remain unnoticed for a long time: if you miss a module dependency in a build.lst, the build might work despite that because the missing module was built "by luck", because another module, that either is built before or in parallel, has a dependency on it. In a single process make you would see the mistake as soon as you do the first build where the prerequite who's rule you forgot to specify is not present. To overcome this and to avoid to have information stored in different places (we have at least n makefile.mk, one build.lst, one d.lst and one makefile.pmk per "module") makefiles must be able to include others. Our current system is not able to do that because all makefiles use the same target names. This is the most remarkable difference of the makefiles in Björn's prototype. > Sorry, but > http://hg.services.openoffice.org/hg/cws/gnu_make/raw-file/2518db232510/sw/prj/target_lib_msword.mk > then leaves a lot to be desired. There's still loads of redundancy - > and actually, when compared to a dmake-style makefile without the > carried-over cargo-cult contents - quite similar. It seems that you still did not see the major difference that I have described above. So then please tell us the "lot to be desired". That's exactly what this discussion is about. We hope to get input and facts. I also fail to see redundancy. Due to its nature of a prototype it doesn't factor out commonly used defines as much as possible (like e.g. compiler flags), but I think that you know that. So where else do you see redundancies? Can you do less in a makefile for a library than specifying the name of the library, the libraries it links against and the object files it is thought the be built from? The list of include pathes is just caused by our current structure (and in a final version surely would be done by a macro that factors out all commonly used patterns). >> see my other replies here and on the blog. As a sidenote: project files >> for common IDEs only give you more trouble, if they are one way (and >> they currently all are): They are a just minor simplification for >> newcomers for a simple build without changing anything. But I leave it >> to you to explain to release engineers, why it is their job to translate >> the changes made by a new dev in his IDE-project back to the cmake >> source. They will rightfully refuse that. Thus the newcomer will have >> to fiddle with the makefiles again
[tools-dev] Re: Building OpenOffice.org with GNU make
Am Sun, 6 Dec 2009 22:13:47 +0100 schrieb Thorsten Behrens : > That's not what I meant. The reason to also hand over a specific > dmake option for multi-processor builds is that sometimes only one > or two directories have their actual dependencies satisfied & are > thus buildable. So it would be ~trivial to only hand "-P16" over to > build.pl, and let it distribute that across the active dmake > instances. No, thats not trivial: the current bottlenecks cant really be noticed by build.pl until its to late (unless build.pl kills runnning dmake-processes for a directories and restarts them again, which will hardly improve performance and might trigger additional subtile and hard to debug bugs in the build system). This is because bottlenecks happen, when a big module with many dependencies -- like svx -- is still building and all unbuild modules depend on it directly or indirectly. However, once build.pl notices that it cant start another dmake because of deps on svx, the dmakes of svx are already long running (with little paralellization) and the only way to allow those more paralelization is to kill and restart those. Its an inherent problem of the current architecture. > > > - what kind of dependency tracking is missing in the current > > > system? > > Those that bite you on compatible builds. > > > Ah. That's what I thought. So nothing inherently missing in > dmake/build.pl, but "just" bugs in the makefiles. Yes, and going to the moon is "just" going up long enough ... ;-) And its not only the makefiles, the build.lst and d.lst files are far more evil. > Sorry, but > http://hg.services.openoffice.org/hg/cws/gnu_make/raw-file/2518db232510/sw/prj/target_lib_msword.mk > then leaves a lot to be desired. Like, what? > There's still loads of redundancy - I cannot see a lot of redundancy there, unless one is willing to reorganize quite a bit of the files in the source tree: - clean up the weird include paths in sw - for each shared library, keep all files in one directory That would allow to collect the source files implicitly (still, collecting source files dynamically might be a performance issue for complete (re-)builds). I wanted the prototype to be able to work without major refactorings on the source itself, which is actually _harder_ to archive than when one allows the new build system to completely reorder files in the source tree. That does not mean we should not do such bigger changes per se, just that we do not _have_ to. > Yes. And? It's still significantly lowering the barrier of entry. > Loads of other projects do it (with the same limitation). An digression beforehand: I guess we are talking about Windows here mostly, makefiles are the "native build system" on pretty much all *nix-systems. On Windows, anyone getting all the prerequisites setup right will not even notice any difficulty living without read-only project files. Other than that, I really like some release and QA engineers opinions on this, because I am only guessing on their opinion. We already have more than enough issues with (non-)reproducible builds (weather by rebuilds, that slightly differ from a complete build, platform specific stuff or other reasons). When I talked with release engineers and QA engineers, I got the feeling they where not interested in asking back on every issue, which build environment the dev used to make a build (or worse: QA waves through a cws, because the tested builds look fine, but when building the "offical" way in releng, subtile isssues show up on the master). Another issue with using an external project generator is versioning. If for example we would use cmake to generate Visual Studio project files, we might find different versions creating different project files (maybe even in best faith by bugfixes) and thus causing subtle bugs by updating the project generator. Also an external project generator using a abstraction as high level as cmake does might develop in a way that diverges from what we would like to do. Of course, we could require only an old version to be used or fork -- but then we are no better of than we currently are with dmake. So a generated project file might introduce subtle different builds, is dependent on the version of the generator and is read-only. Since you can "generate a project from existing code" trivially in NetBeans, Visual Studio and others and just as well kick of a makefile-based build in all modern IDEs, what exactly is gained by generated project files? I dont at all see how this is lowering the barrier to entry as it just introduces another concept to learn for a newcomer, but it does not make life easier in any relevant way. Nothing more confusing to a newcomer than leaky abstractions(1). > So stepping back a bit, I should probably mention (again) that I > find the general idea really great - the make system is truly > arcane & leaves a lot to desire. But despite your initial request for > input, plans & thoughts, I cannot but have the impression y
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
Björn Michaelsen wrote: > > - regarding parallelization, that's surely fixable with much less > >effort in build.pl, no? > Currently we are starting one dmake-process per directory and that > dmake process does paralellization the directory. Implementing a > recursive jobserver that communicates between dmake and build.pl would > not only be ugly, it would also be a major effort. > That's not what I meant. The reason to also hand over a specific dmake option for multi-processor builds is that sometimes only one or two directories have their actual dependencies satisfied & are thus buildable. So it would be ~trivial to only hand "-P16" over to build.pl, and let it distribute that across the active dmake instances. > > - what kind of dependency tracking is missing in the current system? > Those that bite you on compatible builds. > Ah. That's what I thought. So nothing inherently missing in dmake/build.pl, but "just" bugs in the makefiles. > > - the question of correct dependencies is probably rather > >orthogonal to the question of which build system to use. much of > >the problems here are in modules like scp2, helpcontent, > >writerfilter etc. where tons of stuff is built via ad-hoc rules > >that simply don't get dependencies right. Having those handled in > >a declarative way would convince me a bit better, that a change > >in the build system actually addresses these issues - > >(see also Kay's > >http://blogs.sun.com/GullFOSS/entry/and_what_about_make) > We are using a declarative language -- also Kay knows what we are doing > and has contributed valuable input. I guess he would tell me, if I > tried something obviously wrong. ;-) > And yes, the build system to use is only "the color of the bikeshed", > but this is a chance to get rid of quite a bit of self-maintained > dependencies. As said elsewhere, the effort never aimed to simple > replace build.pl/dmake with GNU make for its own sake. > Sorry, but http://hg.services.openoffice.org/hg/cws/gnu_make/raw-file/2518db232510/sw/prj/target_lib_msword.mk then leaves a lot to be desired. There's still loads of redundancy - and actually, when compared to a dmake-style makefile without the carried-over cargo-cult contents - quite similar. > see my other replies here and on the blog. As a sidenote: project files > for common IDEs only give you more trouble, if they are one way (and > they currently all are): They are a just minor simplification for > newcomers for a simple build without changing anything. But I leave it > to you to explain to release engineers, why it is their job to translate > the changes made by a new dev in his IDE-project back to the cmake > source. They will rightfully refuse that. Thus the newcomer will have > to fiddle with the makefiles again manually, winning nothing in the end > (other than additional confusion and probably some needless > communication on mailing lists). > Yes. And? It's still significantly lowering the barrier of entry. Loads of other projects do it (with the same limitation). And the need to add a new file to the project is usually not the first, not the second, but the nth thing you do when starting to hack ... So stepping back a bit, I should probably mention (again) that I find the general idea really great - the make system is truly arcane & leaves a lot to desire. But despite your initial request for input, plans & thoughts, I cannot but have the impression you're already quite determined to follow the outlined route - sadly the build system has seen quite a few attempts for change; and survived all of them ~unmodified, thus some extra thought & consultation is surely advisable ... From my humble point of view, what has usually worked best in OOo land is some iterative approach to change; which in this case & my opinion would mean cleaning up makefiles one by one, either using a declarative DSL directly that could later be mapped to gnu make or whatever tool we see fit, or - using a meta-language like automake cmake, or something homegrown, *still* use dmake for the while, and then, after some critical mass has been attained, switch the make tool wholesale (and adapt the metalang-to-makefile generator). Additionally, and since you mentioned the desire to have only one make instance - last time someone tried to have gnu make hold all of OOo's dependency tree in one process, that guy (Kai Backman) ended up with absolutely pathetic performance & ridiculous mem usage. That's why he went for bjam ... Cheers, -- Thorsten pgp2YkH7PTPIv.pgp Description: PGP signature
[tools-dev] Re: Building OpenOffice.org with GNU make
Am Fri, 4 Dec 2009 18:41:08 +0100 schrieb Thorsten Behrens : > - regarding parallelization, that's surely fixable with much less >effort in build.pl, no? Currently we are starting one dmake-process per directory and that dmake process does paralellization the directory. Implementing a recursive jobserver that communicates between dmake and build.pl would not only be ugly, it would also be a major effort. > - what kind of dependency tracking is missing in the current system? Those that bite you on compatible builds. The only way to work around those reliably is to throw away and rebuild all dependent modules. Also we currently have many, many intermodule dependencies that are implicit and thus very fragile. Just ask Mathias -- he had to deal with a lot of those when working on the split build stuff. > - the question of correct dependencies is probably rather >orthogonal to the question of which build system to use. much of >the problems here are in modules like scp2, helpcontent, >writerfilter etc. where tons of stuff is built via ad-hoc rules >that simply don't get dependencies right. Having those handled in >a declarative way would convince me a bit better, that a change >in the build system actually addresses these issues - >(see also Kay's >http://blogs.sun.com/GullFOSS/entry/and_what_about_make) We are using a declarative language -- also Kay knows what we are doing and has contributed valuable input. I guess he would tell me, if I tried something obviously wrong. ;-) And yes, the build system to use is only "the color of the bikeshed", but this is a chance to get rid of quite a bit of self-maintained dependencies. As said elsewhere, the effort never aimed to simple replace build.pl/dmake with GNU make for its own sake. > More generally, I'd guess the whole make system question, once > posed, has a much (if not more) potential for religious wars than > the dscm question. :) Yes, and it shows in the comments of the blog. > In line with that, why not using autotools? It's ugly, it has many > drawbacks, but it's _the_ standard for FLOSS. Plus, it has excellent > cross-build capabilities, something I consider increasingly > important. Or something like cmake, which could give you project > files for common IDEs (see > http://lists.freedesktop.org/archives/ooo-build/2009-August/000181.html > for some initial attempts)? see my other replies here and on the blog. As a sidenote: project files for common IDEs only give you more trouble, if they are one way (and they currently all are): They are a just minor simplification for newcomers for a simple build without changing anything. But I leave it to you to explain to release engineers, why it is their job to translate the changes made by a new dev in his IDE-project back to the cmake source. They will rightfully refuse that. Thus the newcomer will have to fiddle with the makefiles again manually, winning nothing in the end (other than additional confusion and probably some needless communication on mailing lists). Best Regards, Bjoern Michaelsen - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
[tools-dev] Re: Building OpenOffice.org with GNU make
Am Sat, 5 Dec 2009 00:08:45 +0100 schrieb Thorsten Behrens : > Michael wrote: > > [reasons against autotools] > > > Granted. Though re-working OOo's make system in and of itself will > consume substantial resources, so I guess the question whether to > code another make system that's only used inside OOo, or whether to > use/extend something that's in wide-spread use out there deserves > some thoughts ... Well, automake provides few additional features we need for building OOo and we have lots of requirements that wont be provided by automake because OOo is special (huge, cross platform and almost a distribution of its own). Also, automake, in addition to being ugly, is not without dependencies itself: amongst them Perl(*). Best Regards, Bjoern Michaelsen (*) Yes, I know, that should not matter as it is intended to be run _before_ configure by the "developer" and not by the "user" (in autotools terms: those who want to compile the product are called users, which might be a bit confusing at first), but still. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Fri, 2009-12-04 at 21:46 +0100, Michael Stahl wrote: > but regarding IDEs, Bjoern just yesterday tried loading the sw module > and > the inc directory from the solver into some IDEs... > iirc, visual studio parsed the headers for half an hour, and was then > unusably slow; netbeans (with some gigabytes of heap) was barely > usable. > > does anybody really use an IDE for developing OOo? > from what i know people use text editors, or they use an IDE basically > like a text editor. I use SlickEdit, which some people regard as IDE, though the SlickEdit developers themselves insist on calling it a text editor because it doesn't require you to create a project before you are able to edit files. I'm probably biased since I used to work for them, but from a variety of IDEs and IDE-like text editors I've tried, SlickEdit is the only one that can handle C++ projects the size of OOo. Eclipse comes close, but its handling of C++ is rather limited, and it takes a long time for it to start up. Oh BTW, SlickEdit can tag source files inside, say, sw in less than a minute or a few minutes depending on the speed of your machine. But there is still a reason why you may want to create a solution file for Visual Studio even if you don't use it for hacking. Many developers use whatever IDE's or text editors they have faith in, and use Visual Studio just for building the project on Windows. Kohei -- Kohei Yoshida - OpenOffice.org Engineer - Novell, Inc. - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
[tools-dev] Re: Building OpenOffice.org with GNU make
Michael wrote: > [reasons against autotools] > Granted. Though re-working OOo's make system in and of itself will consume substantial resources, so I guess the question whether to code another make system that's only used inside OOo, or whether to use/extend something that's in wide-spread use out there deserves some thoughts ... > [cmake] > > but regarding IDEs, Bjoern just yesterday tried loading the sw module and > the inc directory from the solver into some IDEs... > iirc, visual studio parsed the headers for half an hour, and was then > unusably slow; netbeans (with some gigabytes of heap) was barely usable. > Reportedly eclipse works rather well with a limited set of modules; which will be more or less attained via the split build endeavour at some point in time. > does anybody really use an IDE for developing OOo? > from what i know people use text editors, or they use an IDE basically > like a text editor. > I know a few cases where people have tried; ultimately though, if one becomes a core dev, folks work on OOo the way you describe it (which may be related to the fact that maintaining those project files manually is a PITA). But may considerably lower the barrier of entry for new devs. Cheers, -- Thorsten pgpRFrO0S5w04.pgp Description: PGP signature
Re: [tools-dev] Re: Building OpenOffice.org with GNU make
On Fri, Dec 04, 2009 at 09:46:56PM +0100, Michael Stahl wrote: > but the real problem comes with libtool... on windows. > when i last used autotools (which is... 4 years ago), libtool was a shell > script some 10.000 lines long. > starting a process on that so-called OS takes _ages_, so it is not > unsurprising that building C/C++ code on windows with a long shell script > is _very_ slow. > iirc, on the same hardware it was 5-10 times slower building on > windows(mingw/msys) than on linux. > oh, and what is libtool's support for MSVC like? does it exist at all? > so unless someone proves me wrong here it seems to me that libtool is not > really an option. Did anyone in this discussion propose libtool? I didn't see it. (You can use automake without libtool) Grüße/Regards, Rene - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
[tools-dev] Re: Building OpenOffice.org with GNU make
On 04/12/2009 18:41, Thorsten Behrens wrote: > Hi Bjoern, *, > > In line with that, why not using autotools? It's ugly, it has many > drawbacks, but it's _the_ standard for FLOSS. Plus, it has excellent > cross-build capabilities, something I consider increasingly > important. well, we already use autoconf, and imho there is nothing wrong with that. automake mainly lets you write portable makefiles, that work in any POSIX-compliant make. nowadays, that seems pretty useless to me: if you want "make" then just use GNU make, it's portable, and powerful. no need to care about whatever crappy make implementation ships with some ancient OS. also, when i last used it, it seemed to me that automake kind of encourages writing recursive makefiles (i.e. makefiles that invoke make for subdirectories), which is an awful idea (because make needs to stat everything all over again for every invocation); but i am not sure to what extent that is really automake's fault (could you just have a single Makefile.am, or at least one that includes the ones from subdirectories?). but the real problem comes with libtool... on windows. when i last used autotools (which is... 4 years ago), libtool was a shell script some 10.000 lines long. starting a process on that so-called OS takes _ages_, so it is not unsurprising that building C/C++ code on windows with a long shell script is _very_ slow. iirc, on the same hardware it was 5-10 times slower building on windows(mingw/msys) than on linux. oh, and what is libtool's support for MSVC like? does it exist at all? so unless someone proves me wrong here it seems to me that libtool is not really an option. oh, another reason to use automake would be that it nicely wraps libtool, which is then also irrelevant. but on the other hand, autotools indeed make cross compiling very easy. > Or something like cmake, which could give you project > files for common IDEs (see > http://lists.freedesktop.org/archives/ooo-build/2009-August/000181.html > for some initial attempts)? hmmm... cmake also seemed nice last i looked at it... but regarding IDEs, Bjoern just yesterday tried loading the sw module and the inc directory from the solver into some IDEs... iirc, visual studio parsed the headers for half an hour, and was then unusably slow; netbeans (with some gigabytes of heap) was barely usable. does anybody really use an IDE for developing OOo? from what i know people use text editors, or they use an IDE basically like a text editor. > Cheers, > > -- Thorsten regards, michael -- "Most programming environments are meta-engineered to make typical software easier to write. They should instead be meta-engineered to make incorrect software harder to write." -- Daniel J. Bernstein - To unsubscribe, e-mail: dev-unsubscr...@tools.openoffice.org For additional commands, e-mail: dev-h...@tools.openoffice.org
[tools-dev] Re: Building OpenOffice.org with GNU make
Hi Bjoern, *, so first off, really cool to see thoughts & effort spent on this: http://blogs.sun.com/GullFOSS/entry/building_openoffice_org_with_gnu I didn't really spent much time mulling over this, it's basically just my current random thoughts on it: - regarding parallelization, that's surely fixable with much less effort in build.pl, no? - what kind of dependency tracking is missing in the current system? - the question of correct dependencies is probably rather orthogonal to the question of which build system to use. much of the problems here are in modules like scp2, helpcontent, writerfilter etc. where tons of stuff is built via ad-hoc rules that simply don't get dependencies right. Having those handled in a declarative way would convince me a bit better, that a change in the build system actually addresses these issues - (see also Kay's http://blogs.sun.com/GullFOSS/entry/and_what_about_make) More generally, I'd guess the whole make system question, once posed, has a much (if not more) potential for religious wars than the dscm question. :) In line with that, why not using autotools? It's ugly, it has many drawbacks, but it's _the_ standard for FLOSS. Plus, it has excellent cross-build capabilities, something I consider increasingly important. Or something like cmake, which could give you project files for common IDEs (see http://lists.freedesktop.org/archives/ooo-build/2009-August/000181.html for some initial attempts)? Cheers, -- Thorsten pgpcxXUQkKnaI.pgp Description: PGP signature
