Re: Using -MMD instead of -MD for depndency generation
On Thu, 2010-10-28 at 12:18 -0400, Paul Smith wrote: Is there anything I can help with? bisect GNU make 3.81 - 3.82 to see when it got slower? Or anything on our GNU make/automake usage? E.g. we use a non recursive makefile I'll try to look into this more this afternoon/tonight. Until then I wouldn't bother doing too much work on it. Let me try to reproduce your results then we can see where we are. I was able to install the proper packages and get a complete build of WebKit. After this I see that GNU make 3.82 is twice as slow as 3.81, but even 3.82 only takes about 30 seconds to run (a do-nothing build) on my system. Taking 5+ minutes is crazy. There must be something odd about your system or setup causing this. As for why 3.82 is slower, unfortunately I'm having problems figuring it out. I compiled with gprof but the cumulative profiled code in GNU make only took 6 seconds or so, so I suppose the other 24 seconds must be in libc or someplace... but trying to install a profile-enabled version of libc on my system did not succeed (or rather the package install worked but linking with the profiled libc did not work). All I can assume is there's some difference in heap management in 3.82 which is causing this, because that's the only difference I can think of that would not show up in the GNU make code itself. I'll try using valgrind and see what I get. If anyone has other suggestions for profiling tools let me know. Cheers! -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist
Re: Using -MMD instead of -MD for depndency generation
On 10/31/2010 06:24 PM, Paul Smith wrote: All I can assume is there's some difference in heap management in 3.82 which is causing this, because that's the only difference I can think of that would not show up in the GNU make code itself. I'll try using valgrind and see what I get. If anyone has other suggestions for profiling tools let me know. Hi, I can use oprofile on my system and send you the oparchive? Or if perf supports something like this use that. My guess is that limited RAM + dm-crypt on faster but still slow laptop disks made make a difference.
Re: Using -MMD instead of -MD for depndency generation
Paul Smith psm...@gnu.org writes: As for why 3.82 is slower, unfortunately I'm having problems figuring it out. I compiled with gprof but the cumulative profiled code in GNU make only took 6 seconds or so, so I suppose the other 24 seconds must be in libc or someplace... but trying to install a profile-enabled version of libc on my system did not succeed (or rather the package install worked but linking with the profiled libc did not work). The prof program distributed with recent linux kernels is extremely useful too -- it's much easier and less intrusive to use than gprof in many cases. [The annoyance is that it's still not packaged in debian for whatever reason... you can fetch the linux sources and just build in the prof directory, but you need to make sure the source versions match the your kernel you're using, since the kernel interfaces have changed around at various points.] -Miles -- Suppose He doesn't give a shit? Suppose there is a God but He just doesn't give a shit? [George Carlin]
Re: Using -MMD instead of -MD for depndency generation
On Wed, 2010-10-27 at 23:33 +0200, Holger Freyther wrote: On 10/27/2010 10:25 PM, Paul Smith wrote: On Wed, 2010-10-27 at 20:51 +0200, Ralf Wildenhues wrote: But I do have libgstreamer dev packages installed: webkit-1.3.5$ locate gst.h /usr/include/gstreamer-0.10/gst/gst.h hmm. Could you check the config.log if video was enabled? Maybe the configure decided that video was disabled but we still build that file... I checked and I have a number of gstreamer pkg-config files: /usr/lib/pkgconfig/gstreamer-0.10.pc /usr/lib/pkgconfig/gstreamer-base-0.10.pc /usr/lib/pkgconfig/gstreamer-check-0.10.pc /usr/lib/pkgconfig/gstreamer-controller-0.10.pc /usr/lib/pkgconfig/gstreamer-dataprotocol-0.10.pc /usr/lib/pkgconfig/gstreamer-net-0.10.pc but configure is looking for some I don't have: configure:20784: $PKG_CONFIG --exists --print-errors gstreamer-0.10 = $GSTREAMER_REQUIRED_VERSION gstreamer-app-0.10 gstreamer-base-0.10 gstreamer-interfaces-0.10 gstreamer-pbutils-0.10 gstreamer-plugins-base-0.10 = $GSTREAMER_PLUGINS_BASE_REQUIRED_VERSION gstreamer-video-0.10 As a result, GSTREAMER_CFLAGS etc. is empty. The default in configure.ac is that video is enabled, and it doesn't seem like it gets disabled if pkg-config cannot find all the right packages: # check if gstreamer is available if test $enable_video = yes; then PKG_CHECK_MODULES([GSTREAMER], [gstreamer-0.10 = $GSTREAMER_REQUIRED_VERSION gstreamer-app-0.10 gstreamer-base-0.10 gstreamer-interfaces-0.10 gstreamer-pbutils-0.10 gstreamer-plugins-base-0.10 = $GSTREAMER_PLUGINS_BASE_REQUIRED_VERSION gstreamer-video-0.10], [have_gstreamer=yes], [have_gstreamer=no]) AC_SUBST([GSTREAMER_CFLAGS]) AC_SUBST([GSTREAMER_LIBS]) fi FWIW, I have a Intel Core2 6600 at 2.4GHz with 2G RAM, running 64bit 2.6.35+ kernel on an ext4 partition. My desktop is still in a container on some boat and I am stuck with my lower end laptop. Is there anything I can help with? bisect GNU make 3.81 - 3.82 to see when it got slower? Or anything on our GNU make/automake usage? E.g. we use a non recursive makefile I'll try to look into this more this afternoon/tonight. Until then I wouldn't bother doing too much work on it. Let me try to reproduce your results then we can see where we are.
Re: Using -MMD instead of -MD for depndency generation
Hello Holger, * Holger Freyther wrote on Mon, Oct 25, 2010 at 09:39:29AM CEST: On 10/25/2010 08:09 AM, Ralf Wildenhues wrote: * Holger Freyther wrote on Sat, Oct 23, 2010 at 10:47:47PM CEST: when typing make in the build directory of WebKit/GTK+ it takes multiple minutes until the first file is compiled. Can you go into (a lot) more detail here? What happens during those minutes, which processes are running (look at ps output etc.)? I am talking about executing make after the source has been built once. E.g. there is only one file changed and I want to rebuild the library with this single new file. My benchmark is to type 'time make' on unchanged source. From my expectation it will parse the GNUmakefile and all the files (the .Plo, Po with the dependencies) included by it, then it needs to verify the dependencies of all these rules for the 'all' target and then exits. Right now this takes 5:42m usertime (+- 10 seconds). OK. That's a lot. My half-stable-half-testing system needs some love to build webkit OOTB so I can't provide comparison data yet. Is this cold-cache or hot-cache? Only the first time the code is compiled, or the build tree is used, i.e., does it happen again after 'make clean'? Or are you talking about configure (-q?) run time here? My system is a MSI X340 with a U7300 @ 1.30GHz and had about 2GB of RAM available, it is running 2.6.36rc8 (but all kernels have shown this behavior) on a Fedora F14 installation. GNU make is of version 3.82, automake is 1.11.1. This is with a 'hot-cache' but with only 2GB of RAM not every file will fit into the page cache. The disk speed is quite okay, slowed down a bit due dm-crypt. The system is not loaded at all, it runs a wm, some gnome-terminals with remote ssh sessions, when executing make it will be the process that takes 99% of the CPU. Have you tried moving the build tree to a ramdisk (/dev/shm)? Do you use the noatime or relatime mount option? If not, please seriously consider doing so on mount points where you keep temporary data (not your mail folders!). For most effective edit-compile-test cycle, I typically move to a big system with lots of RAM and use make -jN with the build tree (not necessarily the source tree) on ramdisk and optionally also use ccache and/or distcc (if the system is not that big). I've found the ramdisk to make file system accesses fairly irrelevant. Eventually, the right answer to lots of file accesses for builds this size or larger is to use a beta build system. I hope somebody ports tup to GNU make syntax, or extends GNU make to have tup-like functionality. If the above all doesn't help make your problem irrelevant, I might be willing to look into optionally going for -MMD instead of -MD, but I do think that from a correctness POV it is not the right thing. If your installed software changed, then you need to also rebuild. Cheers, Ralf
Re: Using -MMD instead of -MD for depndency generation
On Wed, 2010-10-27 at 20:51 +0200, Ralf Wildenhues wrote: I am talking about executing make after the source has been built once. E.g. there is only one file changed and I want to rebuild the library with this single new file. My benchmark is to type 'time make' on unchanged source. From my expectation it will parse the GNUmakefile and all the files (the .Plo, Po with the dependencies) included by it, then it needs to verify the dependencies of all these rules for the 'all' target and then exits. Right now this takes 5:42m usertime (+- 10 seconds). I saw this come across my inbox and thought I'd give it a tree; 5:42 seems like a very, very long time. I downloaded the tarball you linked to, to my system but wasn't able to build; it configured and built about half the code, then failed: CXX WebKit/gtk/WebCoreSupport/libwebkitgtk_1_0_la-FullscreenVideoController.lo WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:32: fatal error: gst/gst.h: No such file or directory But I do have libgstreamer dev packages installed: webkit-1.3.5$ locate gst.h /usr/include/gstreamer-0.10/gst/gst.h Do I need to do something else? Anyway, with half the tree built (I think): webkit-1.3.5$ find -name \*.o | wc -l 1437 make 3.81 takes about 8 seconds on my system to get back to trying to build the above file, and failing. I do notice that make 3.82 takes a lot longer (23s !!) so there's definitely something I need to check out in that respect. But 5m seems pretty crazy. FWIW, I have a Intel Core2 6600 at 2.4GHz with 2G RAM, running 64bit 2.6.35+ kernel on an ext4 partition.
Re: Using -MMD instead of -MD for depndency generation
Hello Paul, * Paul Smith wrote on Wed, Oct 27, 2010 at 10:25:03PM CEST: I downloaded the tarball you linked to, to my system but wasn't able to build; it configured and built about half the code, then failed: CXX WebKit/gtk/WebCoreSupport/libwebkitgtk_1_0_la-FullscreenVideoController.lo WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:32: fatal error: gst/gst.h: No such file or directory Rerun with 'make V=1' to see the command that causeed the failure. Anyway, with half the tree built (I think): webkit-1.3.5$ find -name \*.o | wc -l 1437 make 3.81 takes about 8 seconds on my system to get back to trying to build the above file, and failing. I do notice that make 3.82 takes a lot longer (23s !!) so there's definitely something I need to check out in that respect. But 5m seems pretty crazy. Agreed. Thanks for sharing this data! Ralf
Re: Using -MMD instead of -MD for depndency generation
On 10/27/2010 10:25 PM, Paul Smith wrote: On Wed, 2010-10-27 at 20:51 +0200, Ralf Wildenhues wrote: But I do have libgstreamer dev packages installed: webkit-1.3.5$ locate gst.h /usr/include/gstreamer-0.10/gst/gst.h hmm. Could you check the config.log if video was enabled? Maybe the configure decided that video was disabled but we still build that file... Do I need to do something else? Well, the problem is more with the .Po and .Plo files. If half the files contain '#dummy' it will probably still be a lot faster, if all of them contain the dependencies it will be a lot slower (at least for me). FWIW, I have a Intel Core2 6600 at 2.4GHz with 2G RAM, running 64bit 2.6.35+ kernel on an ext4 partition. My desktop is still in a container on some boat and I am stuck with my lower end laptop. Is there anything I can help with? bisect GNU make 3.81 - 3.82 to see when it got slower? Or anything on our GNU make/automake usage? E.g. we use a non recursive makefile
Re: Using -MMD instead of -MD for depndency generation
Hello Holger, thanks for the report. * Holger Freyther wrote on Sat, Oct 23, 2010 at 10:47:47PM CEST: when typing make in the build directory of WebKit/GTK+ it takes multiple minutes until the first file is compiled. Can you go into (a lot) more detail here? What happens during those minutes, which processes are running (look at ps output etc.)? Is this cold-cache or hot-cache? Only the first time the code is compiled, or the build tree is used, i.e., does it happen again after 'make clean'? Or are you talking about configure (-q?) run time here? What system is this on, is it very loaded with other processes? Can you please post a link to the tarball you're compiling from, so we can try to reproduce this? I would like to reduce the size of the .Plo (dep files) in my build tree and build them with -MMD instead of -MD. Is there a way to instruct automake to use -MMD for gcc3 like depmode? No, there isn't, but also, in my experience, that will not help all that much at all. All files listed as dependencies with -MD will be opened anyway by the compiler eventually, and a list of files usually won't amount to sizeable file size. I'm still guessing that it's something else that slows down your build by so much. Thanks, Ralf
Re: Using -MMD instead of -MD for depndency generation
On 10/25/2010 08:09 AM, Ralf Wildenhues wrote: Hello Holger, thanks for the report. * Holger Freyther wrote on Sat, Oct 23, 2010 at 10:47:47PM CEST: when typing make in the build directory of WebKit/GTK+ it takes multiple minutes until the first file is compiled. Can you go into (a lot) more detail here? What happens during those minutes, which processes are running (look at ps output etc.)? I am talking about executing make after the source has been built once. E.g. there is only one file changed and I want to rebuild the library with this single new file. My benchmark is to type 'time make' on unchanged source. From my expectation it will parse the GNUmakefile and all the files (the .Plo, Po with the dependencies) included by it, then it needs to verify the dependencies of all these rules for the 'all' target and then exits. Right now this takes 5:42m usertime (+- 10 seconds). Is this cold-cache or hot-cache? Only the first time the code is compiled, or the build tree is used, i.e., does it happen again after 'make clean'? Or are you talking about configure (-q?) run time here? My system is a MSI X340 with a U7300 @ 1.30GHz and had about 2GB of RAM available, it is running 2.6.36rc8 (but all kernels have shown this behavior) on a Fedora F14 installation. GNU make is of version 3.82, automake is 1.11.1. This is with a 'hot-cache' but with only 2GB of RAM not every file will fit into the page cache. The disk speed is quite okay, slowed down a bit due dm-crypt. The system is not loaded at all, it runs a wm, some gnome-terminals with remote ssh sessions, when executing make it will be the process that takes 99% of the CPU. What system is this on, is it very loaded with other processes? Can you please post a link to the tarball you're compiling from, so we can try to reproduce this? Just grab a recent WebKit/GTK+ tarball[1], and run configure on the sources, the build will take quite a bit of time. I would like to reduce the size of the .Plo (dep files) in my build tree and build them with -MMD instead of -MD. Is there a way to instruct automake to use -MMD for gcc3 like depmode? No, there isn't, but also, in my experience, that will not help all that much at all. All files listed as dependencies with -MD will be opened anyway by the compiler eventually, and a list of files usually won't amount to sizeable file size. I'm still guessing that it's something else that slows down your build by so much. Sure, it is not the only reason why it is slow. I have made a micro benchmark with -MD vs. -MMD and typing 'time make' on a changed source tree.. -MD: real5m37.810s user0m51.572s sys 4m45.353s (best measurement, worst is 5m42.810s) -MMD: real4m16.368s user0m41.604s sys 3m33.983s So it appears by reducing the number of rules and files things will be faster (no real surprise here). It would be nice if you could help us to reduce the complexity of the rules created, but on the other hand we really do have many many files. any help is more than appreciated holger [1] http://webkitgtk.org/webkit-1.3.5.tar.gz
