Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
Hello GNU Make bug-list subscribers On http://www.gnu.org/software/make/manual/make.html#Automatic-Prerequisites The rule for automatically generating prerequisites is %.d: %.c @set -e; rm -f $@; \ $(CC) -M $(CPPFLAGS) $< > $...@.; \ sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $...@. > $@; \ rm -f $...@. but could be replaced with %.d : %.c @$(CC) -MT $@ -MT $*.o -MM $(CPPFLAGS) $< > $@ This is simpler. This fixes disagreements about object file names between the sed command and the cc command, should the c file not be found in the current directory. Perhaps it would be time to make this rule part of the default rules in GNU Make? In that case, the -MG option should also be considered for avoiding errors when auto generated headers are included, perhaps via an CCMFLAGS variable. Best regards Robert --- Robert Jørgensgaard Engdahl Cum Laude MSc. C.S. SW-Developer, R&D Video Services & Applications Bang & Olufsen A/S Peter Bangs Vej 15 DK-7600 Struer Denmark Phone: (+45) 96 84 44 05 Internal phone: 454405 e-mail: r...@bang-olufsen.dk Location: Factory 1 - West___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
On 4/28/10, Robert Jørgensgaard Engdahl wrote: > > Hello GNU Make bug-list subscribers > > On > > http://www.gnu.org/software/make/manual/make.html#Automatic-Prerequisites > > The rule for automatically generating prerequisites is > > %.d: %.c > @set -e; rm -f $@; \ > $(CC) -M $(CPPFLAGS) $< > $...@.; \ > sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $...@. > $@; \ > rm -f $...@. > > but could be replaced with > > %.d : %.c > @$(CC) -MT $@ -MT $*.o -MM $(CPPFLAGS) $< > $@ > > This is simpler. It's also unnecessary - you don't need a rule for %.d at all. You can just generate the dependencies as a side-effect of compilation using -MMD or similar. Then use '-include' to grab all the deps if they exist. By providing a rule for the .d files you'll cause make to re-execute itself, so you'll just end up parsing the Makefile twice. -Mike ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
> It's also unnecessary - you don't need a rule for %.d at all. You can > just generate the dependencies as a side-effect of compilation using > -MMD or similar. Well, if a .d file gets deleted while its .o file exists, you do need to regenerate it - or regenerate the .o (which may cause wasteful knock-on actions since it's now newer than it was, even if unchanged). This matters if, for example, you have a make clean-depend rule to purge dependency information. > Then use '-include' to grab all the deps if they exist. better: include only the .d files that correspond to .o files *that exist* - if the .o file doesn't exist, you don't need to know what it depends on, aside from the primary source from which it's compiled (which is indicated by the pattern rule that makes the .o file). > By providing a rule for the .d files you'll cause make to > re-execute itself, so you'll just end up parsing the Makefile twice. Only if you include .d files that aren't really needed (which is common practice). Amendments along these lines are part of the doc-update I keep not finding time to finish up ... Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
On Wed, Apr 28, 2010 at 7:12 AM, Edward Welbourne wrote: >> It's also unnecessary - you don't need a rule for %.d at all. You can >> just generate the dependencies as a side-effect of compilation using >> -MMD or similar. > > Well, if a .d file gets deleted while its .o file exists, you do need > to regenerate it - or regenerate the .o (which may cause wasteful > knock-on actions since it's now newer than it was, even if unchanged). > This matters if, for example, you have a make clean-depend rule to > purge dependency information. Delete a "clean-depend" rule on sight, or rename it to the more accurate "break-future-builds". I can think of absolutely no reason to delete dependency files without deleting the objects they describe. >> Then use '-include' to grab all the deps if they exist. > > better: include only the .d files that correspond to .o files *that > exist* - if the .o file doesn't exist, you don't need to know what it > depends on, aside from the primary source from which it's compiled > (which is indicated by the pattern rule that makes the .o file). Yeah, that's true. Nice. >> By providing a rule for the .d files you'll cause make to >> re-execute itself, so you'll just end up parsing the Makefile twice. > > Only if you include .d files that aren't really needed (which is > common practice). There's still the case of a .o and .d pair that both exist where the .o is out of date to one of its dependencies. If the .d file is a target, then make will first rebuild the .d, re-exec, then rebuild the .o. The re-exec is unnecessary there; building the .d as a side-effect of building the .o is sufficient. Philip Guenther ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
Surely the -MMD solution with -include $(objects:.o=.d) does the trick cleaner than the method mentioned in the manual. Edward, are you rewriting the Make manual? I hope that you do include an example, besides the explanation below. You guys have been most helpful. Cheers Edward Welbourne 28-04-2010 16:12 Please respond to e...@opera.com To Mike Shal cc r...@bang-olufsen.dk, bug-make@gnu.org Subject Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual > It's also unnecessary - you don't need a rule for %.d at all. You can > just generate the dependencies as a side-effect of compilation using > -MMD or similar. Well, if a .d file gets deleted while its .o file exists, you do need to regenerate it - or regenerate the .o (which may cause wasteful knock-on actions since it's now newer than it was, even if unchanged). This matters if, for example, you have a make clean-depend rule to purge dependency information. > Then use '-include' to grab all the deps if they exist. better: include only the .d files that correspond to .o files *that exist* - if the .o file doesn't exist, you don't need to know what it depends on, aside from the primary source from which it's compiled (which is indicated by the pattern rule that makes the .o file). > By providing a rule for the .d files you'll cause make to > re-execute itself, so you'll just end up parsing the Makefile twice. Only if you include .d files that aren't really needed (which is common practice). Amendments along these lines are part of the doc-update I keep not finding time to finish up ... Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
> Delete a "clean-depend" rule on sight, I cannot agree. If I write a rule to make something, I also write a rule to get rid of it. It's just basic hygiene ... > or rename it to the more accurate "break-future-builds". If you have a sensible rule to generate .d files when needed, you haven't broken your builds - you've just obliged yourself to regenerate .d files. Which may be wasteful, but see below. > I can think of absolutely no reason > to delete dependency files without deleting the objects they describe. Do an update from your version control system and watch the make files move to a new location in the directory tree. If your make files refer to source files via relative paths, your .d files are now broken, but the .o files are fine. (This is rare: I cite it because it happens to be the most recent reason I've actually needed it in practice.) Crucially, if you *don't* have a clean-depend rule, you can't move your make files without breaking version-control tools that do bisect [*] or obliging them to do full make clean between each test build. With a clean-depend rule, however, only the dependencies need regenerated - and this cuts out all but the preprocessing step of the compiler - except for the files that actually changed. [*] for those unfamiliar with bisect: modern version-control systems such as git include a command that, given two revisions between which a bug was introduced and a script that builds and tests for the bug, will automatically find the revision at which the bug was introduced. Back before disk got cheap, I needed a clean-depend rule for when the disk got full during the last build and a bunch of .d files got corrupted because they only got half-written. And, before I got all the subtleties ironed out from our dependency tracking, it was also useful when debugging your dependency generation. Speaking of the subtleties of dependency tracking: do an update in your version control system, watch some header go away - and all files that used to reference it drop those references. Your .d files claim a bunch of stuff depends on this missing file; but you have no rule to regenerate it. So make will not even try to compile anything (even though everything *would* compile fine) because your .d file say that all the .o files that need recompiled depend on a file that doesn't exist any more; make clean-depend fixes that. (I have a better fix for that, but I'll leave that for my manual update.) Without it, you have to make clean to make any progress, or fix every .d file that references the lost .h file. >>> By providing a rule for the .d files you'll cause make to >>> re-execute itself, so you'll just end up parsing the Makefile twice. >> >> Only if you include .d files that aren't really needed (which is >> common practice). > There's still the case of a .o and .d pair that both exist where the > .o is out of date to one of its dependencies. If the .d file is a > target, then make will first rebuild the .d, re-exec, then rebuild the > .o. The re-exec is unnecessary there; building the .d as a > side-effect of building the .o is sufficient. If generating .d as a side-effect, don't listen to the manual's advice that says you need to sed its content to claim that the .d depends on the same things the .o does. If those things have changed, the .o shall be regenerated and hence so shall the .d; and you don't need this updated version of the .d file to discover that the .o needs rebuilt. Changes to .h files consequently never trigger re-exec. That just leaves the case of where the .d and .o are both out of date relative to their primary source file, when both shall still, indeed, be rebuilt. I can't think of a clean way to get rid of this and, in practice, have never seen it as a big problem. A few source files have changed, they get preprocessed twice and make re-execs in between. It's a waste, but you can avoid it by deleting the .o file matching any source file you edit, so that its .d gets ignored - and I don't bother even with that. It's more of a cost when updating from version-control and lots of stuff has changed; my response to that is to kick off a build in background as soon as I update, so that it'll have sorted itself out by the time I actually care (i.e. I've made my edits and am ready to build what I want), at which point we're back to few affected compilation units. Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
Edward Welbourne wrote on 29-04-2010 12:59:13: > > Delete a "clean-depend" rule on sight, > > I cannot agree. > If I write a rule to make something, I also write a rule to get rid of > it. It's just basic hygiene ... > > > or rename it to the more accurate "break-future-builds". > > If you have a sensible rule to generate .d files when needed, you > haven't broken your builds - you've just obliged yourself to > regenerate .d files. Which may be wasteful, but see below. > > > I can think of absolutely no reason > > to delete dependency files without deleting the objects they describe. > > Do an update from your version control system and watch the make files > move to a new location in the directory tree. If your make files > refer to source files via relative paths, your .d files are now > broken, but the .o files are fine. (This is rare: I cite it because > it happens to be the most recent reason I've actually needed it in > practice.) Crucially, if you *don't* have a clean-depend rule, you > can't move your make files without breaking version-control tools that > do bisect [*] or obliging them to do full make clean between each test > build. With a clean-depend rule, however, only the dependencies need > regenerated - and this cuts out all but the preprocessing step of the > compiler - except for the files that actually changed. > > [*] for those unfamiliar with bisect: modern version-control systems > such as git include a command that, given two revisions between which > a bug was introduced and a script that builds and tests for the bug, > will automatically find the revision at which the bug was introduced. > > Back before disk got cheap, I needed a clean-depend rule for when the > disk got full during the last build and a bunch of .d files got > corrupted because they only got half-written. And, before I got all > the subtleties ironed out from our dependency tracking, it was also > useful when debugging your dependency generation. > > Speaking of the subtleties of dependency tracking: do an update in > your version control system, watch some header go away - and all files > that used to reference it drop those references. Your .d files claim > a bunch of stuff depends on this missing file; but you have no rule to > regenerate it. So make will not even try to compile anything (even > though everything *would* compile fine) because your .d file say that > all the .o files that need recompiled depend on a file that doesn't > exist any more; make clean-depend fixes that. (I have a better fix > for that, but I'll leave that for my manual update.) Without it, you > have to make clean to make any progress, or fix every .d file that > references the lost .h file. If an update to new source code, that would compile just fine in a clean checkout, breaks the incremental build, the build system is errornuous. Anything that seeks to fix such bugs by user intervention is not a real solution. At least that is my opinion. I just don't know how that could be solved nicely in Make. > > >>> By providing a rule for the .d files you'll cause make to > >>> re-execute itself, so you'll just end up parsing the Makefile twice. > >> > >> Only if you include .d files that aren't really needed (which is > >> common practice). > > > There's still the case of a .o and .d pair that both exist where the > > .o is out of date to one of its dependencies. If the .d file is a > > target, then make will first rebuild the .d, re-exec, then rebuild the > > .o. The re-exec is unnecessary there; building the .d as a > > side-effect of building the .o is sufficient. > > If generating .d as a side-effect, don't listen to the manual's advice > that says you need to sed its content to claim that the .d depends on > the same things the .o does. If those things have changed, the .o > shall be regenerated and hence so shall the .d; and you don't need > this updated version of the .d file to discover that the .o needs > rebuilt. Changes to .h files consequently never trigger re-exec. > > That just leaves the case of where the .d and .o are both out of date > relative to their primary source file, when both shall still, indeed, > be rebuilt. I can't think of a clean way to get rid of this and, in > practice, have never seen it as a big problem. A few source files > have changed, they get preprocessed twice and make re-execs in > between. It's a waste, but you can avoid it by deleting the .o file > matching any source file you edit, so that its .d gets ignored - and I > don't bother even with that. It's more of a cost when updating from > version-control and lots of stuff has changed; my response to that is > to kick off a build in background as soon as I update, so that it'll > have sorted itself out by the time I actually care (i.e. I've made my > edits and am ready to build what I want), at which point we're back to > few affected compilation units. > >Eddy. >
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
> If an update to new source code, that would compile just fine in a clean > checkout, breaks the incremental build, the build system is errornuous. I would like to agree with you, but this constraint is, in general, incompatible with incremental building, which is too good a benefit to throw away. > Anything that seeks to fix such bugs by user intervention is not a real > solution. At least that is my opinion. I just don't know how that could be > solved nicely in Make. Quite. In the mean time, we get as close to this ideal as possible and we have clean-rules to purge selective chunks of what incremental building normally caches for us, to deal with the messy cases. If people *routinely need* to use any clean rule in the course of their normal work-flows, I consider the build system inadequate: but I'll still include clean rules for use when the exceptional situations do come up (and as an aid to debugging). Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
RE: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
>> If an update to new source code, that would compile just fine in a clean >> checkout, breaks the incremental build, the build system is errornuous. > I would like to agree with you, but this constraint is, in general, > incompatible with incremental building That's a entertainingly provocative claim. I'm not convinced that it's been demonstrated and, further, I don't believe it to be true. Where we parted company was here: >>> Speaking of the subtleties of dependency tracking: do an update in >>> your version control system, watch some header go away - and all files >>> that used to reference it drop those references. Your .d files claim >>> a bunch of stuff depends on this missing file; but you have no rule to >>> regenerate it. You don't need to regenerate it. In a make-using system that I maintain, but am not allowed to share with you (which was why I bit my tongue earlier - not very public-spirited of me), there's a .DEFAULT rule, which essentially does: echo Pretending to have generated $@ 1>&2 Now, lying to the computer, like I am there, often doesn't end well. Having a default target that claims to be able to build anything, if that's what I had, would cause pain when debugging the makefiles. There may be other aspects of the system I'm using, which I haven't shared and perhaps am not even aware of, which are saving me from falling into trouble due to these caveats. However, in my experience and opinion, these issues are not as serious as losing incremental building. -Original Message- From: bug-make-bounces+mdorey=bluearc@gnu.org [mailto:bug-make-bounces+mdorey=bluearc@gnu.org] On Behalf Of Edward Welbourne Sent: Thursday, April 29, 2010 08:00 To: Robert Jørgensgaard Engdahl Cc: bug-make@gnu.org Subject: Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual > If an update to new source code, that would compile just fine in a clean > checkout, breaks the incremental build, the build system is errornuous. I would like to agree with you, but this constraint is, in general, incompatible with incremental building, which is too good a benefit to throw away. > Anything that seeks to fix such bugs by user intervention is not a real > solution. At least that is my opinion. I just don't know how that could be > solved nicely in Make. Quite. In the mean time, we get as close to this ideal as possible and we have clean-rules to purge selective chunks of what incremental building normally caches for us, to deal with the messy cases. If people *routinely need* to use any clean rule in the course of their normal work-flows, I consider the build system inadequate: but I'll still include clean rules for use when the exceptional situations do come up (and as an aid to debugging). Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
On 4/29/10, Edward Welbourne wrote: > > If an update to new source code, that would compile just fine in a clean > > checkout, breaks the incremental build, the build system is errornuous. > > > I would like to agree with you, but this constraint is, in general, > incompatible with incremental building, which is too good a benefit to > throw away. This is true for make, but not true for build systems in general. It is entirely possible to have minimal incremental builds and avoid the use of a manual "clean" to support cases where files have been moved or deleted. I agree with Robert's position that the build system is broken if an incremental build fails where a full build would succeed after an update. -Mike ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
On Thu, Apr 29, 2010 at 3:59 AM, Edward Welbourne wrote: >> Delete a "clean-depend" rule on sight, > > I cannot agree. > If I write a rule to make something, I also write a rule to get rid of > it. It's just basic hygiene ... I propose the following guideline: If you have a target that generates A (and B as a side-effect), then a 'clean*' rule that deletes B should also delete A (and vice versa). So, I do have a rule to delete *.d files, it's called "clean". Since I don't have rules for building .d files other than with .o files, it Just Works. >> or rename it to the more accurate "break-future-builds". > > If you have a sensible rule to generate .d files when needed, you > haven't broken your builds - you've just obliged yourself to > regenerate .d files. Which may be wasteful, but see below. Okay, so if you have a rule to delete .d files without deleting .o files, you need rules to build .d files. In my experience, the only reason to have *either* of those is "because that's how the Makefile was originally written and the current behavior doesn't hurt enough for me to spend the time to fix it". ... > Speaking of the subtleties of dependency tracking: do an update in > your version control system, watch some header go away - and all files > that used to reference it drop those references. Your .d files claim > a bunch of stuff depends on this missing file; but you have no rule to > regenerate it. So make will not even try to compile anything (even > though everything *would* compile fine) because your .d file say that > all the .o files that need recompiled depend on a file that doesn't > exist any more; make clean-depend fixes that. The fix for that has been documented for years on Paul's webpage, and is most easily done now with gcc's -MP option. > If generating .d as a side-effect, don't listen to the manual's advice > that says you need to sed its content to claim that the .d depends on > the same things the .o does. If those things have changed, the .o > shall be regenerated and hence so shall the .d; and you don't need > this updated version of the .d file to discover that the .o needs > rebuilt. Changes to .h files consequently never trigger re-exec. Ah, it looks like your comments are addressed at just what's in the GNU make info pages and not the advanced method on Paul's webpage. I agree that what's in the info pages has many of the problems you mention...which is why this thread is about updating what's there. Philip Guenther ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
On Fri, Apr 30, 2010 at 5:46 AM, Edward Welbourne wrote: >> The fix for that has been documented for years on Paul's webpage, and >> is most easily done now with gcc's -MP option. > > URL ? http://make.paulandlesley.org/autodep.html#norule (That explains the problem and what needs to be put in the dependency files to solve it, and then gives an implementation that works with all 'makedepend' methods; gcc's -MP option came later is is just an optimized method for those using -MD) Philip Guenther ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
>>> The fix for that has been documented for years on Paul's webpage, and >>> is most easily done now with gcc's -MP option. >> >> URL ? > http://make.paulandlesley.org/autodep.html#norule Thanks. > (That explains the problem and what needs to be put in the dependency > files to solve it, and then gives an implementation that works with > all 'makedepend' methods; gcc's -MP option came later is is just an > optimized method for those using -MD) Looks like I'm going to need to read that log generally to fold ideas from it into my work on the manual - I use a quite different solution to this problem ! (This one, however, is easier to explain than mine, which argues in its favour.) Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual
--- Robert Jørgensgaard Engdahl Cum Laude MSc. C.S. SW-Developer, R&D Video Services & Applications Bang & Olufsen A/S Peter Bangs Vej 15 DK-7600 Struer Denmark Phone: (+45) 96 84 44 05 Internal phone: 454405 e-mail: r...@bang-olufsen.dk Location: Factory 1 - West Philip Guenther wrote on 29-04-2010 19:26:40: > On Thu, Apr 29, 2010 at 3:59 AM, Edward Welbourne wrote: > >> Delete a "clean-depend" rule on sight, > > > > I cannot agree. > > If I write a rule to make something, I also write a rule to get rid of > > it. It's just basic hygiene ... > > I propose the following guideline: If you have a target that generates > A (and B as a side-effect), then a 'clean*' rule that deletes B should > also delete A (and vice versa). > > So, I do have a rule to delete *.d files, it's called "clean". Since > I don't have rules for building .d files other than with .o files, it > Just Works. > > > >> or rename it to the more accurate "break-future-builds". > > > > If you have a sensible rule to generate .d files when needed, you > > haven't broken your builds - you've just obliged yourself to > > regenerate .d files. Which may be wasteful, but see below. > > Okay, so if you have a rule to delete .d files without deleting .o > files, you need rules to build .d files. In my experience, the only > reason to have *either* of those is "because that's how the Makefile > was originally written and the current behavior doesn't hurt enough > for me to spend the time to fix it". > > > ... > > Speaking of the subtleties of dependency tracking: do an update in > > your version control system, watch some header go away - and all files > > that used to reference it drop those references. Your .d files claim > > a bunch of stuff depends on this missing file; but you have no rule to > > regenerate it. So make will not even try to compile anything (even > > though everything *would* compile fine) because your .d file say that > > all the .o files that need recompiled depend on a file that doesn't > > exist any more; make clean-depend fixes that. > > The fix for that has been documented for years on Paul's webpage, and > is most easily done now with gcc's -MP option. > This is also kind of the solution I would have used (kinda, as I would just have made a %.h rule), but then the gcc manual is a bit confusing, as it mentions phony targets (which would really break the idea of incremental builds), but I see that the targets are only phony in the English sense, and not in the GNU Make sense (.PHONY). > > > If generating .d as a side-effect, don't listen to the manual's advice > > that says you need to sed its content to claim that the .d depends on > > the same things the .o does. If those things have changed, the .o > > shall be regenerated and hence so shall the .d; and you don't need > > this updated version of the .d file to discover that the .o needs > > rebuilt. Changes to .h files consequently never trigger re-exec. > > Ah, it looks like your comments are addressed at just what's in the > GNU make info pages and not the advanced method on Paul's webpage. I > agree that what's in the info pages has many of the problems you > mention...which is why this thread is about updating what's there. > > > Philip Guenther > ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
