Re: make doc problem
- Original Message - From: "Phil Holmes" To: ; "Julien Rioux" Cc: Sent: Friday, January 27, 2012 1:38 PM Subject: Re: make doc problem - Original Message - From: "Julien Rioux" To: Cc: Sent: Thursday, January 26, 2012 11:00 PM Subject: Re: make doc problem On 26/01/2012 11:13 AM, Reinhold Kainhofer wrote: On 22/01/2012 20:58, Julien Rioux wrote: Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. The problem here is that lilypond builds up memory from 400MB to ~1GB without releasing... Most of these allocations don't seem to be memory leaks, but rather due to guile. Cheers, Reinhold Is it a bug? We're talking about lilypond running with the -dread-input-files flag here. Once a snippet has been processed and lilypond moves on to the next one, there is no reason to hold onto the memory used by the previous snippet, right? -- Julien I'm trying to look at memory usage on my Ubuntu box. If I watch the build using System Monitor, I rarely see more than a gig used. However, using "free" even my dormant machine is now showing 3 Gigs used. Is this because I have around 2 Gigs marked as "cache"? Is the actual in-use memory used-cache? Here's a graph of used-cache, which fits well with my observations of system monitor. My machine when lying dormant having finished make/make doc uses 970 MiB memory. The peak I saw during make doc was 2.4 Gig. So make doc takes around 1.4 Gigs when doing its greatest processing with 8 parallel processors. Is this too much for your machine, Julien? If it's a desktop, 4 Gigs of memory is about £30 now, which might be a simpler solution than messing with how make doc runs. -- Phil Holmes <>___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
- Original Message - From: "Julien Rioux" To: Cc: Sent: Thursday, January 26, 2012 11:00 PM Subject: Re: make doc problem On 26/01/2012 11:13 AM, Reinhold Kainhofer wrote: On 22/01/2012 20:58, Julien Rioux wrote: Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. The problem here is that lilypond builds up memory from 400MB to ~1GB without releasing... Most of these allocations don't seem to be memory leaks, but rather due to guile. Cheers, Reinhold Is it a bug? We're talking about lilypond running with the -dread-input-files flag here. Once a snippet has been processed and lilypond moves on to the next one, there is no reason to hold onto the memory used by the previous snippet, right? -- Julien I'm trying to look at memory usage on my Ubuntu box. If I watch the build using System Monitor, I rarely see more than a gig used. However, using "free" even my dormant machine is now showing 3 Gigs used. Is this because I have around 2 Gigs marked as "cache"? Is the actual in-use memory used-cache? -- Phil Holmes ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 26/01/2012 6:14 PM, Graham Percival wrote: On Thu, Jan 26, 2012 at 05:57:43PM -0500, Julien Rioux wrote: On 22/01/2012 2:58 PM, Julien Rioux wrote: Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. Could we redistribute the regression test input files into subfolders of say ~50 files each, possibly sorting them into categories if it makes sense? Wanted to do that for ages, but that's not something to get into now. But this shouldn't be an issue -- don't we split it into lists of IIRC 300 files? We had some kind of emergency bugfix like that. ... yeah, see make/lysdoc-rules.make You may want to look into that; perhaps changing the division from 300 to 50 would speed up the build? - Graham The `echo' command-line is broken down into 300-file chunks. They all still end up into one collated-files.tely Anyway, it kind of sounds like a workaround or an emergency action as you say. Were the memory released at each snippet then it shouldn't be an issue. Cheers, Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On Thu, Jan 26, 2012 at 05:57:43PM -0500, Julien Rioux wrote: > On 22/01/2012 2:58 PM, Julien Rioux wrote: > >Thanks, you're quite right CPU is not the limiting factor for the build. > >Disk access and usage of swap when compiling > >input/regression/collated-files slows down the build to a crawl for me. > > Could we redistribute the regression test input files into > subfolders of say ~50 files each, possibly sorting them into > categories if it makes sense? Wanted to do that for ages, but that's not something to get into now. But this shouldn't be an issue -- don't we split it into lists of IIRC 300 files? We had some kind of emergency bugfix like that. ... yeah, see make/lysdoc-rules.make You may want to look into that; perhaps changing the division from 300 to 50 would speed up the build? - Graham ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 26/01/2012 11:13 AM, Reinhold Kainhofer wrote: On 22/01/2012 20:58, Julien Rioux wrote: Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. The problem here is that lilypond builds up memory from 400MB to ~1GB without releasing... Most of these allocations don't seem to be memory leaks, but rather due to guile. Cheers, Reinhold Is it a bug? We're talking about lilypond running with the -dread-input-files flag here. Once a snippet has been processed and lilypond moves on to the next one, there is no reason to hold onto the memory used by the previous snippet, right? -- Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 2:58 PM, Julien Rioux wrote: Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. Could we redistribute the regression test input files into subfolders of say ~50 files each, possibly sorting them into categories if it makes sense? -- Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
- Original Message - From: "Reinhold Kainhofer" To: "Julien Rioux" Cc: ; Sent: Thursday, January 26, 2012 4:13 PM Subject: Re: make doc problem On 22/01/2012 20:58, Julien Rioux wrote: Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. The problem here is that lilypond builds up memory from 400MB to ~1GB without releasing... Most of these allocations don't seem to be memory leaks, but rather due to guile. Cheers, Reinhold It only takes that amount of memory for big builds. I've never seen much over 1 Gig total memory during a make with 8 CPUs all running lilypond. -- Phil Holmes ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 20:58, Julien Rioux wrote: > Thanks, you're quite right CPU is not the limiting factor for the > build. Disk access and usage of swap when compiling > input/regression/collated-files slows down the build to a crawl for me. The problem here is that lilypond builds up memory from 400MB to ~1GB without releasing... Most of these allocations don't seem to be memory leaks, but rather due to guile. Cheers, Reinhold -- -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial & Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
"Julien Rioux" wrote in message news:4f1c6a79.3040...@physics.utoronto.ca... On 22/01/2012 2:38 PM, David Kastrup wrote: Julien Rioux writes: I can't run -j, I have a single core. This is factually incorrect. You can run -j just fine, but you can't expect much of a speedup. On a single-core machine, make -j 2 typically gives you a speedup of maybe 15% (given sufficient memory) since the CPU can keep busy on processing a second job when the other job is waiting for the disk to provide new input. More importantly, you'll get to see the same kind of problems that the true multi-core people experience when using -j. So very much recommended for testing. Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. -- Julien As a general rule, CPU is very much the limiting factor on make and make doc. With my "single cpu" virtual machine on my Windows box, the CPU is stuck at 100%. With my multi-core Ubuntu box, most of the time all 8 CPUs run 100%, with memory never over 1.5 Gigs. This is using a fairly large SSD, but I'm 99% certain that, all other things being equal, CPU is king when building lily. -- Phil Holmes Bug Squad ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
Julien Rioux writes: > Thanks, you're quite right CPU is not the limiting factor for the > build. Disk access and usage of swap when compiling > input/regression/collated-files slows down the build to a crawl for > me. If it is really usage of swap: getting more memory will be by far the cheapest method of speeding up your computer much more than buying a faster CPU ever could. -- David Kastrup ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 2:38 PM, David Kastrup wrote: Julien Rioux writes: I can't run -j, I have a single core. This is factually incorrect. You can run -j just fine, but you can't expect much of a speedup. On a single-core machine, make -j 2 typically gives you a speedup of maybe 15% (given sufficient memory) since the CPU can keep busy on processing a second job when the other job is waiting for the disk to provide new input. More importantly, you'll get to see the same kind of problems that the true multi-core people experience when using -j. So very much recommended for testing. Thanks, you're quite right CPU is not the limiting factor for the build. Disk access and usage of swap when compiling input/regression/collated-files slows down the build to a crawl for me. -- Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
Le 22/01/2012 20:22, Julien Rioux disait : On 22/01/2012 2:15 PM, Julien Rioux wrote: On 22/01/2012 2:11 PM, Jean-Charles Malahieude wrote: What I've done to check is: open Documentation/fr/usage/running.itely add the five X at the beginning of the first text line XCe chapitre passe en revue ce qui se passe lorsque vous lancez LilyPond. save it and "make -j3 doc LANGS='fr'" File out-www/offline-root/Documentation/usage/running-lilypond.fr.html still shows "Ce chapitre passe en revue ce qui se passe lorsque vous lancez LilyPond." -- Jean-Charles You're right, I forgot that I caused that. Give me a moment I will push the fix to staging... Regards, Julien Done: 930dbeff8f5d31faa9654365c8ed84a30e489e83 Hope this fixes it for you as well. Yes, many thanks! -- Jean-Charles ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
Julien Rioux writes: > I can't run -j, I have a single core. This is factually incorrect. You can run -j just fine, but you can't expect much of a speedup. On a single-core machine, make -j 2 typically gives you a speedup of maybe 15% (given sufficient memory) since the CPU can keep busy on processing a second job when the other job is waiting for the disk to provide new input. More importantly, you'll get to see the same kind of problems that the true multi-core people experience when using -j. So very much recommended for testing. -- David Kastrup ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 2:15 PM, Julien Rioux wrote: On 22/01/2012 2:11 PM, Jean-Charles Malahieude wrote: What I've done to check is: open Documentation/fr/usage/running.itely add the five X at the beginning of the first text line XCe chapitre passe en revue ce qui se passe lorsque vous lancez LilyPond. save it and "make -j3 doc LANGS='fr'" File out-www/offline-root/Documentation/usage/running-lilypond.fr.html still shows "Ce chapitre passe en revue ce qui se passe lorsque vous lancez LilyPond." -- Jean-Charles You're right, I forgot that I caused that. Give me a moment I will push the fix to staging... Regards, Julien Done: 930dbeff8f5d31faa9654365c8ed84a30e489e83 Hope this fixes it for you as well. Thanks, Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 2:11 PM, Jean-Charles Malahieude wrote: What I've done to check is: open Documentation/fr/usage/running.itely add the five X at the beginning of the first text line XCe chapitre passe en revue ce qui se passe lorsque vous lancez LilyPond. save it and "make -j3 doc LANGS='fr'" File out-www/offline-root/Documentation/usage/running-lilypond.fr.html still shows "Ce chapitre passe en revue ce qui se passe lorsque vous lancez LilyPond." -- Jean-Charles You're right, I forgot that I caused that. Give me a moment I will push the fix to staging... Regards, Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
Le 22/01/2012 19:32, Julien Rioux disait : On 22/01/2012 1:19 PM, Jean-Charles Malahieude wrote: Hi all! Not only that I can no longer use "-j" on a first build (it is OK on the next builds), [...] I can't run -j, I have a single core. Can you please report more precisely why you can no longer use it? I make a clean and let you now... -- Jean-Charles ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 1:32 PM, Julien Rioux wrote: The second issue I have not seen. If I correct a typo in a file in Documentation then make doc will rebuild it. OK I should check Documentation/fr right now... When I edit Documentation/fr/essay/literature.itely and issue a make doc from within build/Documentation/fr then Documentation/fr/essay.tely is recompiled. I can see the result in Documentation/fr/out-www/essay/* and in Documentation/out-www/essay.fr.* [1] Please report in more details the problems that you encounter, e.g. which file are you modifying and how are you calling make. Thanks, Regards, Julien [1] Big side note: It doesn't work flawlessly: notation.tely and a bunch of other manuals are also recompiled when they shouldn't, since I didn't touch any of their source files. This problem I trace back to the CHAIN_RULE in make/ly-rules.make, which I didn't dare touch up to now. This rule states that web depends on usage depends on notation depends on... etc. so that only one instance of lilypond-book is running at once on any given manual. As a side-effect it means that manuals depend on each other when that isn't in fact the case. ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
- Original Message - From: "Julien Rioux" To: "Phil Holmes" Cc: "Jean-Charles Malahieude" ; "LilyPond Bugs" ; "LilyPond Devel" Sent: Sunday, January 22, 2012 6:35 PM Subject: Re: make doc problem On 22/01/2012 1:30 PM, Phil Holmes wrote: I've run a few with -j9 CPU_COUNT=9 this afternoon with no problems. The only oddity I've noticed is that make doc make doc now seems to rebuild a lot of files the second run. -- Phil Holmes I've hit a roadblock with issue 2125 which attempted to fix that make doc; make doc problem. With that patch, on my machine the second make doc reports `nothing to be done'. So it works on my machine but not yours and I haven't quite figured it out yet. Regards, Julien Please shout if you want logfiles or testing. I can run tests fairly quickly if I've got my lily machine up and running. -- Phil Holmes ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 1:30 PM, Phil Holmes wrote: I've run a few with -j9 CPU_COUNT=9 this afternoon with no problems. The only oddity I've noticed is that make doc make doc now seems to rebuild a lot of files the second run. -- Phil Holmes I've hit a roadblock with issue 2125 which attempted to fix that make doc; make doc problem. With that patch, on my machine the second make doc reports `nothing to be done'. So it works on my machine but not yours and I haven't quite figured it out yet. Regards, Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
On 22/01/2012 1:19 PM, Jean-Charles Malahieude wrote: Hi all! Not only that I can no longer use "-j" on a first build (it is OK on the next builds), which means 40' for a "make doc LANGS='fr' and about 2 hours for all languages, I now have to "make doc-clean" in order to view a corrected typo. I've tried a "touch masterfile.tely" before "make doc" but it doesn't change anything. Does anybody else experiment this as well? Cheers, Jean-Charles Hi Jean-Charles, I can't run -j, I have a single core. Can you please report more precisely why you can no longer use it? The second issue I have not seen. If I correct a typo in a file in Documentation then make doc will rebuild it. OK I should check Documentation/fr right now... Regards, Julien ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
Re: make doc problem
- Original Message - From: "Jean-Charles Malahieude" To: "Lily Bugs" ; "lilypond-devel" ; "Julien Rioux" Sent: Sunday, January 22, 2012 6:19 PM Subject: make doc problem Hi all! Not only that I can no longer use "-j" on a first build (it is OK on the next builds), which means 40' for a "make doc LANGS='fr' and about 2 hours for all languages, I now have to "make doc-clean" in order to view a corrected typo. I've tried a "touch masterfile.tely" before "make doc" but it doesn't change anything. Does anybody else experiment this as well? Cheers, Jean-Charles I've run a few with -j9 CPU_COUNT=9 this afternoon with no problems. The only oddity I've noticed is that make doc make doc now seems to rebuild a lot of files the second run. -- Phil Holmes ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
make doc problem
Hi all! Not only that I can no longer use "-j" on a first build (it is OK on the next builds), which means 40' for a "make doc LANGS='fr' and about 2 hours for all languages, I now have to "make doc-clean" in order to view a corrected typo. I've tried a "touch masterfile.tely" before "make doc" but it doesn't change anything. Does anybody else experiment this as well? Cheers, Jean-Charles ___ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
