So, after having foolishly opened my mouth on IRC, I have officially
decided to learn more about the OO.o build system to see if compilation
times (for people who don't have access to Hamburg's compilation
cluster) could be speeded up a bit.
At this point my main goal is learning about the different options
we have. A primary target is getting some good profiles on the
existing system .. if nothing else than to find out that 99% of
the time is spent in helpcontent2 .. :-)
We'll start with transex3:
- Profiling the different build steps (C++, lex and the build tool itself)
- Batching of files to the compiler (done on jam)
- Header precompilation
- dmake and jam file structure (to see how large they tend to get)
Volker has been kind enough and volunteered to hack the dmake
files. I'm a total dmake noob, so I need to dive in and fully familiarize
myself with the strengths and weaknesses of it. The main reason I'm
using jam for the tests is because that's the best build tool I've found so
far. It would be delightful to find something better!
I' d also love to hear war stories from the seasoned devs, especially
those who have been through the last major rewrite of the build
system. There is about 200k lines of build scripts around,
which is considerably more than a lot of other projects have code..
Any experiences you can share will help a lot and hopefully save us
some sleepless nights ..
Again, don't hold your breath on the build times speeding up even
a single tenth of a second. At this stage the main purpose is learning
more, if that results in improvements that is a failure indeed! (and we
will naturally revert to the old non-improved version immediately!)
I've attached the IRC discussion log for posteriority.
Take care and looking forward to your feedback and insights,
Kai
--
Kai Backman, programmer, [EMAIL PROTECTED]
http://www.ShortHike.com - space station game
<KaiB> vq: So, do you have suggestions for what to hack on? If not, I'll
continue testing Jam as a build tool to see if I can speed up/profile the
builds. Ause gave me a good list of modules to work on, and I'll know a lot
more once I've completed transex3.
* yamal has quit IRC ("My damn controlling terminal disappeared!")
<KaiB> vq: My basic strategy is to add the build non-intrusively at this point.
Ie. you can either use the existing build system or the jam build and both
should produce the same target files in terms of solver.
* MenZa has quit IRC (Connection timed out)
<vq> KaiB: Everyting that speeds up the build is very appreciated.
<vq> KaiB: Is Jam multiplatform? Does it work with Linux, FreeBSD, MacOSX ?
<KaiB> vq: The only point where I need to change existing files (as opposed to
just adding new files) is when I'll start working on precompiled headers. Again
the changes don't break standard (or non precompiled builds) but it requires me
to touch almost all cxx and hxx files (to reorder the #include's)
<KaiB> vq: Yep. It's a small C program. I think it's been ported to most
existing platforms. Boost and Freetype are two large projects I know that use
it.
* vq googles ...
<KaiB> vq: Google for "Perforce Jam" to get the upstreams version.
* yalu has joined #openoffice.org
<KaiB> vq: The build language is a superset of dmake/build/deliver as far as I
can tell. transex3 is minimal, but copying basic functionality was pretty
straightforward.
<KaiB> vq: My initial tests was just batching together files into a single
compile instead of compiling them one by one. Even this small speedup (and
probably the speedup gained from using Jam) improved the compile time
remarkably. Larger C++ compiles should have even larger speed boosts.
<vq> jam - jet another make?
<yalu> Q: does openoffice.org offer builds of OOo with the installer included
(instead of rpms)?
<KaiB> vq: :-) Might be. No idea where it comes from. It's pretty ancient
actually.. I was using a 8 year old version with abandon until I realized there
was a never one.. Didn't matter, there was no new functionality I needed .. :-)
<vq> I don't find the switch for paralell builds
<KaiB> vq: -j n
<vq> KaiB: Like in make :)
* lgodard has joined #openoffice.org
<KaiB> vq: :-)
<lgodard> cbosdonnat: ping
<KaiB> vq: This is an old article but pretty interesting:
http://www.perforce.com/jam/doc/scm7.html
<KaiB> vq: Their situation is pretty similar to ours (in terms of code size).
C++ is harder than C, but almost ten years should have corrected the speed
difference ..
<vq> KaiB: Are you sure that we spend much time in build.pl and dmake? I was
thinking we are slow because we don't use precompiled headers and compile one
source file at a time.
<KaiB> vq: I honestly don't know. I started working on dmake profiling, but it
quickly became clear just rewriting the stuff in Jam was faster (and would
provide better profile information).
* vq knows that the dependency management in dmake is quite fast.
<KaiB> vq: Jam is slightly more expressive than dmake. You don't need
build/deliver as separate modules, you can do everything with one tool.
<KaiB> vq: Ie. it makes the build simpler. Even so, batching probably does
increase speed, but my simple measurements also indicated that jam had some
part in speeding up the build.
* louis_to has quit IRC
<KaiB> vq: I'm refering to my transex3 example here. The only way to realy
measure it is probably to run a no change build with both tools. I know running
a toplevel dmake no-change takes ages on my machine.
<vq> KaiB: The build deliver stuff is made from the Sun guys. You could use
dmake/make/whatever for this task. Unfortunately the local build guru (ause) is
not here but he knows the reason why build.pl and deliver.pl are used. Adding
makefile rules instead is not the problem
* SimonAW has joined #openoffice.org
<vq> KaiB: There are political reasons here that some stuff is not done.,
right? (stx12,mh_)
<vq> KaiB: There is nothing like a toplevel dmake. That "toplevel" dmake does:
cd instsetoo
<vq> _native && build --all
<KaiB> vq: btw. Does dmake instance itself into subdirectories or is it a
global (parse everything, then execute) tool? The main speedup of jam is
usually that it parses the whole project in one go.
<KaiB> vq: Ahaa. Ok. I could have guessed that. Well, a build --all with no
changes versus a similar version with dmake.
<KaiB> vq: jam that is .. :-)
<vq> dmake: Only follows the rules in the makefiles.
<blauwal> vq: Political reasons is a too strong word.
<blauwal> vq: There are reason, some are historical
<KaiB> blauwal: So why is the build/deliver system in place? If in the field
mesurements show jam to be faster (and perform exactly the same build) would
there be issues with switching?
* martink has joined #openoffice.org
<mh_> KaiB: yes, just doing it :)
<blauwal> vq: (noone want to change a thousand makefile etc), some are due to
the size of OOo (i.e. the module dependcy system), some things are the way they
are because it's convenient for the Hamburg site with it's prebuild binaries
scheme etc
* karunakar is now known as karunakarAWAY
<blauwal> vq: Nothing is set in stone ...
<KaiB> mh: :-) Well.. It's just 170k lines of make/build script .. (more than a
medium sized open source project ..)
<blauwal> vq: but any change will be weighted for effort and benefit
<mh_> what are the pros and cons of switching ?
<blauwal> vq: :-)
<KaiB> That was my idea in doind it one module at a time and keeping the
dependency system intact.
<KaiB> mh: Take a look at this Sybase article for a deeper analysis of their
situation: http://www.perforce.com/jam/doc/scm7.html
* trovao has joined #openoffice.org
<KaiB> mh: I think OO.o has pretty much the same issues. Pro: Faster (probably)
build system, single flexible build language, external tool (somone else
maintains it), larger userbase of tool.
* holycow has joined #openoffice.org
<paveljanik> KaiB: aren't these benefits of dmake? ;-)
<KaiB> mh: Cons: The work in getting it done .. At this point (with my limited
experience of OO build and larger experience with Jam) I can't see any
fundamental issues that can't be solved.
<blauwal> KaiB: The build process is probably not dominated in any way by dmake
itself. Even if jam is twice faster than dmake it doesn't mean a huge benefit
in building time
<paveljanik> KaiB: so what is The Benefit?
<KaiB> paveljanik: Simplicity, single tool used for the whole build process.
Speed usually results from that as well, but simplicity is more important imho.
* smoketester has joined #openoffice.org
<paveljanik> KaiB: dmake is not simple?
<blauwal> KaiB: Oh, dmake isn't what is complex, OOo itself is complex
<vq> blauwal,KaiB: That's why I asking. I don't see a problem with the
capabilities of dmake. And our thousands of makefiles are working.
<KaiB> paveljanik: Simple from the user perspective. Shorter buildscripts.
<paveljanik> I'd prefer to concentrate on optimizing other parts of the build
than dmake - repeated java build compiles, caching Java compile outputs so they
can be reused in next builds etc.
<paveljanik> KaiB: users do not care!
<blauwal> KaiB: well formed makefile.mk are short
<KaiB> Well, my question is then, why hasn't batching and precompiled header
been implemented earlier? :-)
<blauwal> KaiB: And for the complex rules we got our local guru
<vq> KaiB: Noone did it! The unix guys use ccache
<paveljanik> KaiB, vq: yes, and this is the reasoning for asking to implement
MS compilers support in ccache ;-)
<blauwal> KaiB: We've got a very different system here in Hamburg for batching
... and it works
<KaiB> blauwal: How do you do it then? Why isn't it part of the public OO? This
is very interesting ..
* pmladek is back
<blauwal> KaiB: Because we can't open source our Hardware farm :-)
<paveljanik> blauwal: it is not based on T1? ;-)
<blauwal> paveljanik: Huauuauu
* cloph has joined #OpenOffice.org
<blauwal> paveljanik: If I ever get such a thing ...
<KaiB> blauwal: Ahaa, so you solve the problem in hardware? Well, that's a
solution but it doesn't help developers outside hamburg .. :-)
<vq> KaiB: As for the makefile capabilities of dmake, see the man page, but on
a quick glance I didn't see what is missing. If you need extra diagnostic
output for dmake, tell me what you want. Maybe it's already included or I can
add it for you.
<blauwal> KaiB: Well of course. The depedency system allows a paralleization of
the build process. Thats works as well with OOo
<KaiB> vq: Ok. My first question is can you manipulate the dependency graph
directly? Ie. add scripting/conditionals etc. to create dependencies between
targets?
<KaiB> blauwal: But it kind of only hides the underlying problem of no
batching/precompiles ..
<vq> KaiB: Sure, as in every [d]make
<blauwal> KaiB: Batching works .. just different than in other projects.
Precompiled header are another thing
<blauwal> KaiB: We had them long ago and they were terrible ... we got a lot of
unrecognized bugs because of them
<blauwal> KaiB: There is nothing in our build environment which prvents the
revitalization ... of course
<KaiB> blauwal: That's interesting. I've always considered them stable (as long
as you don't do strange #define macro tricks)
* paveljanik changes topic to '[English] Will it work today? | FAQ
(http://openoffice.org/faq.html) or users@openoffice.org list (subscribe),
#users.openoffice.org or Google... | http://planet.go-ooo.org/ | *Unofficial*
development community builds (also debs)
http://ftp.linux.cz/pub/localization/OpenOffice.org/devel/680/ | SRC680_m151 is
ready | 348 issues to be fixed for 2.0.2'
<blauwal> KaiB: it needs top work on Linux, Solaris, Windows, FreeBSD, MacOSX
etc
<vq> 348 ?????
<paveljanik> yes
<vq> :-
<vq> Grr, I mean :-O
<blauwal> KaiB: We do any number of strange #define tricks :-)
<KaiB> vq: I can't really do that in make .. Not the kind of Dep changes I'm
talking about. At least not with Gnu make. Ie. change dependencies on a project
wide scale.
* SimonAW has quit IRC (Connection timed out)
<KaiB> blauwal: :-)
<vq> KaiB: You are taking of adding target dependencies if some prerequisits
are fulfilled?
<blauwal> KaiB: I don't want to discourage you. The main problem with any
change is the sheer size ... the last change in the build system took a man
year or so
<KaiB> vq: Well, I was using autotools for a number of years until I switched
to jam.
<vq> KaiB: Easiest case: mytarget : $(MYDEPENDENCIES)
* SimonAW has joined #openoffice.org
<KaiB> vq: I found certain dodgy deployment/dependency issues that were
difficult to handle in make.
<KaiB> vq: Like a conditional dependency.
<vq> So the dependencies are actually defined in a macro. You have all means
changing the content of that macro (during runtime) and so change the
dependencies.
<vq> KaiB: Yes, I forgot how bad make is compared with dmake :)
<KaiB> vq: You have a file and depending on some conditions set on the file it
could either depend on another file or not.
<KaiB> vq: I know what we'll do. I'll hack up transex3 with Jam and then you'll
do it with dmake. That way we can compare apples to apples?
<KaiB> blauwal: Well, with my slow machine I'll save years of my life hacking
OO if I'm only able to halve the compile time .. :-)
<vq> KaiB: Sounds like a plan :)
<KaiB> vq: Then we can move on to a slightly larger module and see how it works
out. I'm too unfamiliar with dmake to use it efficiently in cases like this. :-)
<KaiB> vq: I'll hack in the precompiled headers as well, so we get a bit more
data on it.
<vq> KaiB: Oranges and potatoes instead? ;)
* kart_ has joined #openoffice.org
<KaiB> blauwal: Again, I honestly don't know if this will improve the system
enough to be worthwhile. But with such a long compile it's sure worth a try..
* KaiB starts wondering which one is which .. :-)
* JoergB is away: for the night
<vq> KaiB: To get the syntax right: Jam variable == dmake macro
<blauwal> KaiB: But jam itself will not make a real change.. Precompiled header
will do. But that can be done with dmake, to
<blauwal> s/to/too/
<lgodard> i want to use %origin% in a .xcu file
<lgodard> is it valid for any .xcu file ?
<lgodard> or only for addons.xcu ?
<lgodard> oops, building an addon for deploying a template
<KaiB> blauwal: Very true, but I'm interested in the dependency calculations as
well. The sybase paper claimed that they were able to speed up them
significantly. dmake is better (probably) than what they used, and that's why
I'm interested in seeing how it performs.
<KaiB> blauwal: If it's faster than jam I'll switch all my own projects over to
it .. :-)
<KaiB> vq: Your @oo email address is working?
* Mechtilde has quit IRC (Read error: 113 (No route to host))
<vq> KaiB: I wouldn't bet that it's faster, but if it's only slightly slower
it's not worth the effort to change the makefile for a 1.5GB source project.
<blauwal> KaiB: If you look closer you'll notice that we do only dependencies
inside a module.
<KaiB> blauwal: I saw that. But some of the modules are large.
<vq> KaiB: Yes, but expect some CollabsNet delay. Otherwise use quetschke at
scytek dot de
<KaiB> vq: But you'll probably need to change the files anyway to get
precompiled to work..
<vq> KaiB: Well, at the moment all the rules and dependencies are put in
solenv/inc/ . You change it there, you fix it everywhere.
<blauwal> KaiB: It's possible to switch on full dependecies but no one can want
this ... :-)
<KaiB> blauwal: Is it too slow?
<blauwal> KaiB: If you keep your sources oon a remote volume ... yes !!!!!!!!
<vq> blauwal: full as in per module or full as in full? Noone would want the
latter :)
<blauwal> KaiB: Even on local disks it's dog slow.
<blauwal> vq: full as in full
<vq> :-O Urgs!
<KaiB> Well, if it was fast it would be great .. :-)
<blauwal> vq: Even the best tool would still have to stat each dependennt file.
That will always take time
<blauwal> KaiB: Even the best tool would still have to stat each dependennt
file. That will always take time
* blauwal is typing worse than ever
<KaiB> blauwal: As I said, if it was fast .. :-)
<blauwal> KaiB: It can be fast if you have to - say - stat files via samba or
nfs
<KaiB> Well, I think we have enough theorethical talking about this for now.
Yes, the build system can be improved. And yes, it might take ages to do so.
What we need next is some hard data on how the system performs when improved,
no?
* iznogoud has quit IRC (No route to host)
<blauwal> KaiB: Please go ahead ...
* Cyrille has quit IRC ("And just like that, pfft, he's gone.")
<blauwal> KaiB: but please remember the golden rule of OOo: it has to work on
all platforms ...
<cloph> lgodard: Works in other xcu files as well. At least I used it
successfully with a custom schema + xcu
<KaiB> blauwal: I already checked that. GCC supports precompiled and the
difference should be minimal on non-precompile supporting compilers.
<blauwal> KaiB: Yeah, but you need to check Solaris, too
<KaiB> blauwal: What is the official target list btw. Somewhere I saw Win32,
Linux, OSX, Solaris? Any forgotten?
<blauwal> KaiB: But it did, and yes, it has a similiar system, now
<blauwal> KaiB: FreeBSD.
<KaiB> blauwal: Ok, but it's GCC based, right?
<blauwal> KaiB: But that is a gcc platform, too
<blauwal> KaiB: Right
<stx12> paveljanik: do you mind if i change the time when the issue script
(feed) is run?
<paveljanik> stx12: no, of course. What do you want?
<stx12> 6:23 and 18:23
<paveljanik> stx12: UTC?
<blauwal> KaiB: If you search for a OOo project which is in dire need of
speeding up, well, there is always helpcontent2 :-)
<paveljanik> or your time?
<stx12> paveljanik: crontab time :-)
<lgodard> cloph: thx :)
<KaiB> blauwal, stx12: Take a look at tools btw. We had some word from the KDE
devs on the SVN migration.
<paveljanik> stx12: I'd prefer the IZ feed be ready at 6:15 my time :-))
<KaiB> blauwal: Yeah, paveljanik keeps reminding me about that .. :-)
<lgodard> cloph: so that's my template deployement that is wrong :-/
<lgodard> cloph: thanks for your aswer
<stx12> 5:45 and 17:45
<paveljanik> KaiB: yes, yes, yes. hc2, please :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
