Re: Making makefiles with primarily phony targets more friendly

[Britton Kerin]

On Wed, Aug 10, 2022 at 12:33 PM Jean-Baptiste Poittevin
 wrote:
>
> By using a lot of PHONY targets, I think you're closing a door to one make 
> greatest feature : not redoing those things that are already up to date.
>
> If you consider that every rule can generate a useful file (file containing 
> container image for example) or a dummy file, you can benefit from the date 
> comparison between it and its pre-requisites and diminish greatly the need 
> for PHONY targets. In my opinion, PHONY targets should just be used for 
> comfort of the user launching make, and those targets have simply a 
> dependency to a real file.

It's true that some phony targets can ultimately be integrated into
the DAG, but it's safe to assume most users are aware of this and have
some reason for preferring to keep small scripts in the Makefile
itself (where they have easy access to make variables).

Britton

Fwd: Making makefiles with primarily phony targets more friendly

[Britton Kerin]

-- Forwarded message -
From: Britton Kerin 
Date: Wed, Aug 10, 2022, 11:58 AM
Subject: Re: Making makefiles with primarily phony targets more friendly
To: Katherine Pata 




On Wed, Aug 10, 2022, 11:44 AM Katherine Pata  wrote:

> I often find myself using makefiles to handle things like project linting,
> container images, various scripts, initiating tests, and other misc tasks.
> Sometimes these include tasks that have real dependencies that make should
> keep track of, but often the vast majority of targets are phony.
>
> Something like --reverse-phony or a variable like .GENUINE or .REAL that
> is mutually exclusive with .PHONY would be really helpful.
>

I'm not sure I like that exact solution since I don't have mostly phony
targets. But I have enough that some more compact and less repetitive
syntax would be really convenient.

Britton



>
>

Re: Goodbye to GNU make's "build.sh" ... ?

[Britton Kerin]

On Sat, Jun 25, 2022, 1:48 PM Paul Smith  wrote:

> I'm trying to decide what the future is for GNU make's "build.sh"
> bootstrapping script. As you may recall, this script is provided to
> allow GNU make to build on systems which don't already have an instance
> of make installed. Its goal is to build the first make binary, without
> of course all the fancy parts of avoiding rebuilds, generating
> dependency files, etc.
>
> Unfortunately, this really cannot work if GNU make is going to rely on
> gnulib, because a MANY gnulib modules require not only autoconf to
> work, but also automake. That is, it's not enough to just run configure
> and then you get a set of source files and header files that you can
> compile by hand by just invoking the compiler. You must ALSO run make,
> because the modules provide makefile recipes that invoke sed, etc. to
> convert files into their final form.
>
> I had a discussion about this with the Gnulib maintainers a while ago:
>
> https://lists.gnu.org/archive/html/bug-gnulib/2019-09/msg00041.html
>
> However the gnulib maintainers were disinclined to modify the practices
> of the gnulib modules.
>
> This leaves me with two options:
>
>1. Stop using gnulib, or at least sharply limit the modules we will
>   include to those with trivial-enough configurations.
>2. Abandon the build.sh script and require an existing make program
>   in order to build a new version of GNU make.
>
> #1 is what I followed for GNU make 4.3, which has a limited subset of
> carefully-chosen modules. However this becomes harder over time. For
> example any module that needs unistd.h requires a very complex automake
> rule.
>
> If #2 is chosen, then a bootstrap process would involve first obtaining
> an older version of make, such as GNU make 4.3 or lower, and building
> that with its build.sh, then using the resulting make to build the
> newer version.
>

I realize it's easy to say and pretty obvious but option 2 from a user
perspective is sort of horrific.


> I'm interested in thoughts about these options.
>
>

Re: Bug in $(shell ...) I can't understand

[Britton Kerin]

On Sun, Feb 6, 2022 at 8:19 AM Dmitry V. Levin  wrote:
>
> On Sun, Feb 06, 2022 at 11:23:03AM -0500, Paul Smith wrote:
> > OK, someone posted a question to SO and that led me to an hour or more
> > of banging my head against a wall trying to understand what's
> > happening... and I can't.
> >
> > The problem is that the user would like to invoke $(shell ...) and
> > capture errors, even errors that the program being run doesn't exist.
> > The shell function only captures stdout, not stderr.  This simple idea
> > won't work of course:
> >
> >   out := $(shell bad-program 2>&1)
> >   $(info out = $(out))
> >
> >   $ make
> >   /bin/sh: bad-program: not found
> >   out =
> >
> > because this redirects the output of the bad-program, but it's the
> > shell printing the error not the program.
>
> What's actually happening is this:
>
> strace-buildroot/usr/bin/strace -f -y -eexecve,/pipe,/dup,read,write -o'|cat 
> >&3' --signal=none make 3>&2 &>/dev/null
> 4175640 execve("/usr/bin/make", ["make"], 0x7ffdad18ea88 /* 17 vars */) = 0
> 4175640 read(4, 
> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 
> 832
> 4175640 read(4, 
> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) 
> = 832
> 4175640 read(4, "out := $(shell bad-program 2>&1)"..., 4096) = 
> 54
> 4175640 pipe2([5, 6], 0) = 0
> 4175642 dup2(6, 1) = 1
> 4175642 execve("/bin/sh", ["/bin/sh", "-c", "bad-program 2>&1"], 
> 0x7ffd69c8a5a8 /* 17 vars */) = 0
> 4175640 read(5,  
> 4175642 read(4, 
> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\20\0\0\0\0\0\0"..., 832) = 
> 832
> 4175642 read(4, 
> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\343\2\0\0\0\0\0"..., 832) 
> = 832
> 4175642 read(4, "#\n# Please refer to nsswitch.con"..., 
> 4096) = 1808
> 4175642 read(4, "", 4096) = 0
> 4175642 read(4, "root:x:0:0:System Administrator:"..., 4096) = 
> 1326
> 4175643 dup2(1, 2) = 2
> 4175643 write(2, "/bin/sh: bad-program: command no"..., 
> 40) = 40
> 4175640 <... read resumed>"/bin/sh: bad-program: command no"..., 200) = 40
> 4175640 read(5,  
> 4175643 +++ exited with 127 +++
> 4175640 <... read resumed>"", 160)  = 0
> 4175642 +++ exited with 127 +++
> 4175640 write(2, "/bin/sh: bad-program: command no"..., 40) = 40
> 4175640 write(1, "out = ", 6) = 6
> 4175640 write(1, "\n", 1)= 1
> 4175640 read(4, "", 4096) = 0
> 4175640 write(2, "make: *** No targets.  Stop.\n", 29) = 29
> 4175640 +++ exited with 2 +++
>
> Try the following instead:
>
> out := $(shell bad-program 2>&1 ||:)
> $(info out = $(out))

or

 out := $(shell bad-program 2>&1 || true)
 $(info out = $(out))

which makes it more obvious what's going on.

Britton

Re: .SILENT: clobbered by .SILENT: with_target

[Britton Kerin]

On Wed, Jan 12, 2022 at 6:17 AM Dmitry Goncharov
 wrote:
>
> On Tue, Jan 11, 2022 at 3:50 PM Britton Kerin  wrote:
> > It looks like the .SILENT: with a recipe clobbers the .SILENT:
> > without
>
> Consider the following
>
> hello.tsk:
> hello.tsk: hello.h
> hello.tsk: hello.o; $(CC) -o $@ $<
>
>
> Here, hello.tsk is the default goal and it depends on hello.o and hello.h.
> Multiple rules can provide different prerequisites to the same target.
> The above is equivalent to
> hello.tsk: hello.o hello.h
> You can see that this example mimics your .silent example.
> Your makefile provided a prerequisite to .SILENT. Make then knows that
> .SILENT has a prerequisite.

I agree that it's consistent syntax, but semantically it's bad.  The
expectation and the manual both imply that .SILENT: will have a global
effect, and a  real union-of-effects would have .SILENT: meaning
everything and .SILENT: some_target redundant.  The root of the
problem is that the rule syntax has been recycled for an unrelated
purpose, especially so for directives without prerequisites.

Britton

.ONESHELL causes quoting in .SHELLFLAGS to not work as expected

[Britton Kerin]

$ cat Makefile
.ONESHELL:
SHELL = /usr/bin/perl
.SHELLFLAGS = -w -E 'use warnings FATAL => "all";' -E
all:
print 'it works'
$ make all
print 'it works'
Can't find string terminator "'" anywhere before EOF at -e line 1.
make: *** [Makefile:5: all] Error 255
$

Without .ONESHELL: the above works as expected.

Britton

.SILENT: clobbered by .SILENT: with_target

[Britton Kerin]

.SILENT: (without prerequisites) doesn't work as expected when
.SILENT: some_target (with prerequisites) is present:

 $ cat Makefile
 .SILENT:

 .SILENT: target_a
 target_a:
 echo target_a_recipe

 target_b:
 echo target_b_recipe
 $ make target_b
 echo target_b_recipe
 target_b_recipe
 $

It looks like the .SILENT: with a recipe clobbers the .SILENT:
without, which is at best undocumented spooky action-at-a-distance wrt
target_b behavior.  Reversing the order of the two .SILENT:
declarations gives the same behavior.  I think what should happen is
the global .SILENT: (with no prerequisites) wins.  Whether other
.SILENT: with_targets in the same run should then result in an error
or warning is debatable.  I guess this could be a breaking change but
the present behavior seems clearly buggy.  Is there a strategy for
incrementally changing this sort of thing?

Britton

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Britton Kerin]

On Sun, Oct 17, 2021 at 4:44 PM Dmitry Goncharov
 wrote:
>
> On Sun, Oct 17, 2021 at 3:05 PM Britton Kerin  wrote:
>
> > If I understand right that the idea is a special case for .d files
>
> The question being discussed is what to do when make cannot include a 
> makefile.

I got that but it sounds like the proposal is for it to do one thing
if the file is a .d and otherwise something else?

Britton

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Britton Kerin]

On Sun, Oct 17, 2021 at 8:33 AM Dmitry Goncharov
 wrote:
>
> Follow-up Comment #8, bug #61226 (project make):
>
> > > Switching to -include robs the user of a useful message, should there be a
> real issue.
>
> > I'm not sure what this means: in what situation do we lose a useful
> message?
>
> -include robs the user of a not readable or corrupted .d file, even though
> user's intervention is required.
>
> i think, hand written included makefiles require different error handling than
> generated .d files.
>
> A missing hand written makefile is an error. The user has the choice of
> 'include' vs '-include' as you described above. I agree with that logic.
>
> On the other hand, a missing .d file (or any other included makefile generated
> by a rule) is not an error. It is simply a clean build.
>
> i think, make should not print a warning when a .d file is missing. make
> should proceed and create the missing file. However, when .d is present, but
> make cannot include it, then make should print an error and stop.
>
>
> The traditional make behavior, which is still in 4.3 is exactly this.
> i think, we should retain this behavior.
>
>
> in regards to the change in main.c see the following possibilities
>
> 1. Revert the change in main.c.
> 2. Modify the change in main.c to allow for missing .d files, but print an
> error and stop in the other cases.

If I understand right that the idea is a special case for .d files I
dislike this sort of thing in make.  All the various C-centric rules
etc. I tend to just disable even when building C, and this sounds
worse because it's not just a disableable library.

Admittedly I haven't used the generated dependency files in a while
but if some additional semantics are needed to support them well I'd
prefer it be exposed as a general feature rather than hidden.  This
might add a bit of interface  complexity but that's still better
hidden special case behavior IMO.

Britton

[bug #60297] optimize autodeps

[Britton Kerin]

Follow-up Comment #18, bug #60297 (project make):

I strongly second Dmitry's proposal that something like .NOTINTERMEDIATE:
(preferably with a no-dependencies-means-all interpretation) should be added. 
The IMO weird behavior of .SECONDARY where it effectively creates strange weak
dependencies just bit me for the second time. This time I just said a short
prayer and removed .SECONDARY:, but what I really want is to disable the
eccentric handling of intermediate files entirely.

This might add more implementation complexity and one more feature to
consider, but it gives a road to simpler behavior.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

.SECONDARY: unwanted effect of not causing rebuilds when intermediate files missing

[Britton Kerin]

I don't like how .SECONDARY: (without prereqs) causes make to not
rebuild things wen intermediate files are missing, e.g. uncommenting
.SECONDARY in this make file causes make to not rebuild even when
foo.c.validation_stamp is removed:

 OBJS = foo.o

 #.SECONDARY:

 OBJS = foo.o bar.o

 $(OBJS): %.o: %.c
 cp $< $@   # Fake compilation

 # Extra validation requirements for foo.c in particular:
 foo.o: foo.c.validation_stamp
 foo.c.validation_stamp: foo.c
 validation_script.perl $<
 touch $@

 $(OBJS): %.o: %.c
 cp $< $@   # Fake compilation

 clean:
 rm -rf *.o *.c.validation_stamp

This behavior seems just wrong to me.  If we consider intermediate
targets to be significant enough that we don't want them to be
automagically deleted, surely they're important enough that their
absence should trigger a rebuild?

The --debug output for the above Make with .SECONDARY: is unhelpful:

 $ make foo.o --debug
 GNU Make 4.2
 Built for x86_64-pc-linux-gnu
 Copyright (C) 1988-2016 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later

 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.
 Reading makefiles...
 Updating makefiles
 Updating goal targets
  Prerequisite 'foo.c.validation_stamp' of target 'foo.o' does not exist.
 make: 'foo.o' is up to date.

It would be nice if it said why it doesn't rebuild something that it
just said is a prerequisite.

Even if this behavior of .SECONDARY: is considered desirable, it would
be nice to have some way of telling make what I want to tell it, which
is simply that I want all special handling of intermediate files
disabled (i.e. no automatic removal or strangely weakened dependency
handling).

Britton

error checking version of $(shell)

[Britton Kerin]

On Mon, May 18, 2020 at 11:24 AM Pete Dietl  wrote:
>
> Speaking of
>
> > return value of a call to $(shell) is available in $(.SHELLSTATUS).
>
> I think it would be a nice addition to have some global setting where
> any failed $(shell )
> command automatically fails Make.

Agreed.  I have a couple different approaches for this lying around:

# This function works almost exactly like the builtin shell command, except it
# stops everything with an error if the shell command given as its argument
# returns non-zero when executed.  The other difference is that the output
# is passed through the strip make function (the shell function strips
# only the last trailing newline).  In practice this doesn't matter much
# since the output is usually collapsed by the surroundeing make context
# to the same result produced by strip.  WARNING: don't try to nest calls
# to this function.
SHELL_CHECKED = \
  $(strip \
$(if $(shell (($1) 1>/tmp/SC_so) || echo 'non-empty'), \
  $(error shell command '$1' failed.  Its stderr should be above \
  somewhere.  Its stdout is available for review in '/tmp/SC_so'), \
  $(shell cat /tmp/SC_so)))

# FIXME: SHELL_CHECKED should probably be replaced with this better version,
# which should work ok with both parallel make and nested calls.  I'm hoping
# for a little more feedback from paul declaring that the parser is indeed
# sequential and .SHELLSTATUS is indeed per-process, so that this is ok.
# This currently loses the /tmp/SC_so so it migth be harder to get a look
# at the output of a failing command.  That could be restored by making a
# random-named temp file, and putting the command and it's output in there,
# if that ever seems worth it.
SHELL_CK = \
  $(shell $1) $(if $(filter $(.SHELLSTATUS),0),,$(error command '$1' failed))

Britton

Re: Idea: Add .COMMANDCHANGE and .CACHE

[Britton Kerin]

> I think .COMMANDCHANGE is not complex.
> Since it caches the expanded command, at worst it will re-run a script
> when before it would not have done so.
> So it's relatively low risk; at worst, it'll run a command more often than 
> before.
> It won't rerun if only an environment variable changes, but that is *already* 
> true.

True but it's a sort of misleading way to look at things.  With .COMMANDCHANGE
you're effectively saying "add this thing and you can efficiently intermingle
build system changes with other development without worry".  Fail to deliver on
that promise and the feature will likely cause net pain.

> I'm hoping that there's a better name than .COMMANDCHANGE.
>
> I *agree* that .CACHE is more complex to use.
> Its implementation is NOT complex - indeed, it can be
> implemented in relatively few lines of code.
> The challenge is that while .COMMANDCHANGE causes execution
> to happen more than now, .CACHE can cause an execution to be
> *completely* skipped and its cache results would be used instead.
>
> The risk of .CACHE is that it can produce "wrong results", and
> that is a valid concern.  In any cache system (including this one)
> the cache key must be calculated from an absolutely complete
> set of all relevant inputs, and the output (target) list *must*
> be a complete statement of the resulting targets.
> If there are multiple targets, using "&:" (not ":") may be critically
> important, not just something that can be caught on the next

I'm unlikely to ever use &, because I predate it by decades and  stamps/proxies
have always worked fine for me (I've never been tempted to edit or delete a
proxied file and been burned as a result).  So it would be nice if
.COMMANDCHANGE/.CACHE still worked without &, and I think to be really correct
it must since nothing in existing make forbids these common arrangements.

> run of "make".  If environment variables' settings matter, they
> need to be included in the inputs.  For example, the environment variables
> could be expanded somewhere in the command (e..g, @printf '' "$MYVAR").
> That means that .CACHE makes the system even more dependent
> on accurate statements about targets & prerequisites.
> For example: I didn't include read/write/execute permissions of prerequisites
> in the inputs to the cache keys on purpose.  However, technically their
> permission values can have an effect, and the current .CACHE doesn't handle 
> that.
>
> I'd love to hear ways of reducing the risks of .CACHE.  Some ideas:
> 1. We could define a "CACHEENV" variable, a list of all environment variables
>whose expansions would be included in the cache key.
>Adding a few environment variables to this list would mean that changing 
> any
>of them would cause current cached values to be skipped.

To be conservative and automatic I would probably include the entire
environment and then explicitly subtract out e.g. X/DBUS garbage if feature was
provided to do so.  I guess this could be done with text functions easily enough
too though.

 > 2. When multiple targets are listed in a traditional rule (":" not "&:"),
>after running a rule in non-parallel mode, check to see if any of the
>other targets were modified.  If they were, warn or error out & explain 
> the problem
>("use &: instead of :, or create an intermediary value on line XYZ").
>This might be a good idea anyway, even if .CACHE isn't added...
>I just got bit by this mistake yesterday!  Automatically detecting common
>mistakes, and warning about them, is often a great idea.

Again, I think what you really want is a key derived from all the recipes in
the entire dependency DAG of the target.  I think this picks up the case you're
considering here, plus others where significant side effects exist that aren't
explicitly mentioned in the DAG at all.

Britton

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Idea: Add .COMMANDCHANGE and .CACHE

[Britton Kerin]

On Mon, Jun 10, 2019 at 5:14 PM David A. Wheeler  wrote:
>
> On Mon, 10 Jun 2019 15:40:53 -0800, Britton Kerin  
> wrote:
> > No, just the rules that :Makefile, which you can easily tune if it matters.
> > Heck, you can include some_fragment.mk that has the recipes that
> > are a concern and depend on that if you really need that granularity,
> > and then the dependency that you want is explicit.
>
> Using a lot of some_fragment.mk files gets you *closer*, but it's not
> the same thing.  My proposed .COMMANDCHANGE depends on
> the *expanded* set of commands, not the original commands.
> That way, if you change a value (say CCFLAGS) the set of commands
> is considered *different* and will be re-run.

I know, but you can put whatever you want in included files, including
variables.  You can't capture eg vars from the environment but if you're doing
much of that you're missing half the  point of make anyway (recipe capture).
You want make to add implicit dependencies on recipes, which is both a
different granularity than everything else in make (sub-file) and implicit
rather than explicit.  Seems somewhat weird.

And I think cache is harder that you're imagining to get right.  Plenty of make
files use some explicitly mentioned target as a proxy for other unmentioned
files which may be created in dependencies.  So you'd probably want to have all
the recipes in dependencies of a target represented in it's key.

All that said I have to admit I don't mind the idea as much as I did when I
first heard it.  Who knows by tomorrow I may like it :)

Britton

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Idea: Add .COMMANDCHANGE and .CACHE

[Britton Kerin]

On Mon, Jun 10, 2019 at 2:18 PM David A. Wheeler  wrote:
>
> On Mon, 10 Jun 2019 12:10:26 -0800, Britton Kerin  
> wrote:
> > The trickery required is not too fancy:
> >
> >  foo.o: foo.c Makefile
> >  ...
> > or
> >
> >  foo.o: foo.c $(PARANOID_REBUILD)
> >  ...
> >
> > Then use when desired:
> >  (export PARANOID_REBUILD=Makefile && Make)
>
> No.
>
> That's not at *ALL* the same thing.  With those uses, *all* rules
> are rerun if *ANY* change happens in the Makefile, even if almost

No, just the rules that :Makefile, which you can easily tune if it matters.
Heck, you can include some_fragment.mk that has the recipes that
are a concern and depend on that if you really need that granularity,
and then the dependency that you want is explicit.

> all the rules and parameters do not change at all.
> Since the whole point of make is to "only run what's needed",
> that doesn't resolve the problem at all.
>
> In contrast, ".COMMANDCHANGE" would only run a rule
> if that PARTICULAR rule changed (once expanded) in the Makefile.
> In many cases that would skip much of the work.
> Which is the point.

I get it, I just doubt it's worth the extra feature for something that
can be expressed in terms of existing features.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Idea: Add .COMMANDCHANGE and .CACHE

[Britton Kerin]

On Mon, Jun 10, 2019 at 9:20 AM David A. Wheeler  wrote:
>
> Idea: Add .COMMANDCHANGE and .CACHE
>
> Problem:
>
> Paul Smith noted on Sun, 09 Jun 2019 22:11:32 -0400:
> > Most of the requests I see these days that would require a "last state
> > database" wouldn't be helped by md5 checks: mainly they're asking for
> > things like rebuilding targets automatically when compiler options have
> > changed etc. Things like that can be done today with a lot of fancy
> > trickery but people would rather it "just worked".
>
> As noted above, "make"
> detects when *external* files change, but if a command *inside* a Makefile
> changes (including its expanded parameters) then make doesn't notice the 
> change.
> This could be resolved if make could automatically notice a change in
> expanded command. Doing this will require the ability to store state somehow.

The trickery required is not too fancy:

 foo.o: foo.c Makefile
 ...
or

 foo.o: foo.c $(PARANOID_REBUILD)
 ...

Then use when desired:
 (export PARANOID_REBUILD=Makefile && Make)

I do something like one or other (depending on project size) of these
most everywhere.

> In addition, sometimes recipes take a while, and it'd be nice to be able to
> cache old results. If we have to store state anyway,
> we may as well support caching previous results.

This is IMO an entirely new and mostly orthogonal project from Make,
and there is
already ccache (and presumably similar things for other langs).

Britton

>
>
> Proposed solution:
>
> I propose a new special target .COMMANDCHANGE.
> Its dependencies (or all targets if no target is listed) add an implied 
> dependency
> on the expanded commands of the rule.  Thus, if the commands (once expanded)
> change, the rule will be considered obsolete & be rerun.
>
> This requires a way to notice a change.  Instead of a conventional database
> (which is hard to share across instances & can be complicated to implement),
> I propose using a directory specified in MAKESTATE
> (by default ".makestate") for storing shared state.
>
> Every time a non-.PHONY expanded target with .COMMANDCHANGE and a non-empty 
> rule
> is being considered for execution, the
> change state file $(MAKESTATE)/$(FIRST TARGET).sha256 is first examined.
> If the file doesn't exist, or its contents do not match the SHA-256 hash
> of the expanded command for that target, then the target is considered
> out-of-date and the rule is re-run.
> If the set of commands executes successfully, then its SHA-256 hash
> is written to $(MAKESTATE)/$(FIRST TARGET).sha256
> (after creating its directory if necessary).
> (Note: this could be stored in a directory name, instead of file contents,
> so file opening wouldn't be necessary.  However, that would make it
> easy to have a weird inconsistent state where there are multiple entries,
> and I think it's worth avoiding that problem.)
>
> If first_target is an absolute path the state path
> will still be in $(MAKESTATE) by treating the target name as beginning as 
> ".TREEROOT".
> E.g., if the target is /usr/share/mystuff/stuff,
> then the change state file is .makestate//usr/share/mystuff/stuff.sha256.
> (This might require some nonsense on Windows, but it's doable.)
>
> In addition, if an executed rule is the target of .CACHE, has 1+ commands,
> and MAKESTATECACHENUM is defined as a positive integer, then that rule's
> execution results are cached:
>
> * After after executing
>   all of the generated targets are copied (preserving all metadata possible)
>   into $(MAKESTATE)/$(FIRST_TARGET).cache/$(SHA256 of expanded 
> command)/$(TARGET_NAME).
>   It then looks at "$(MAKESTATE)/$(FIRST_TARGET).cache" to see if there are
>   more than MAKESTATECACHENUM entries, and if so, recursively deletes
>   the oldest entries to get it down to that number.
>   My guess is MAKESTATECACHENUM should default to 3.
> * Before executing a rule that's outdated, check for
>$(MAKESTATE)/$(FIRST_TARGET).cache/$(SHA256 of expanded 
> command)/$(TARGET_NAME).
>If it exists, copy its contents to the appropriate targets and *delete* 
> targets
>of the rule if there's no corresponding entry.
>Note that .TREEROOT still applies.
>
> I suspect .COMMANDCHANGE is a terrible name; better ones welcome.
>
> --- David A. Wheeler
>
>
> ___
> Bug-make mailing list
> Bug-make@gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-make

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Idea: Enable .ONESHELL to be per-target

[Britton Kerin]

On Mon, Jun 10, 2019 at 4:56 AM David A. Wheeler  wrote:
>
> Another idea: Enable .ONESHELL to be per-target.
>
> Problem: According to the docs .ONESHELL is global in effect,
> so using it in an existing makefile requires a lot of rework.

Strongly seconded, I would definitely use this.  I've been
attracted to it many times but never dared try globally.
Would make use of shell vars much less ugly and full of \
lines etc.

I do think some thought on the syntax might be in order still
though.  With some of these it's not obvious which default
sense would be more useful, e.g.

 # GLobal setting
 .ONESHELL:

 tricky_target: .NOTONESHELL:

might be what is really wanted in many cases.

I guess these would taint prereqs, like local vars do.

> In many cases .ONESHELL is primarily only useful only
> for specific rules, and there's no need to rewrite the rest.
>
> The current rules are also oddly inconsistent with many
> of the other special targets;
> most other special targets allow a list of dependencies where
> it makes sense to support them.
>
>
> Solution:
> Let's expand .ONESHELL so that
> if specific prerequisites are identified for it,
> it applies only to those prerequisites.
>
> If no prerequisites are listed, it would work just like it does now.
> As a result this change is completely backwards compatible.
>
> Note that this works especially well with target-specific variables,
> so you can even set SHELL and .SHELLFLAGS for a specific target:
>
> 
> .ONESHELL: foo
> foo: SHELL = /usr/bin/perl
> foo: .SHELLFLAGS = -e
> foo: ...
> commands that will run as a single shell.
> 
>
> Note the ".SHELLFLAGS =", which is odd syntactically but should be allowed.
>
> This also works nicely with "Idea: Allow certain special targets as 
> dependencies":

Yes this would be nice.  And it's consistent with Make philosophy of
trying to be
very terse.  btw I assume these special prereqs would not be visible to $< etc

Britton

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Idea: Allow certain special targets as dependencies

[Britton Kerin]

On Sun, Jun 9, 2019 at 7:46 PM Masahiro Yamada
 wrote:
>
> On Mon, Jun 10, 2019 at 10:38 AM Paul Smith  wrote:
> >
> > On Sun, 2019-06-09 at 18:46 -0400, David A. Wheeler wrote:
> > > As syntactic sugar,
> > > I'd like to see selected special targets allowed as dependencies.
> > > When this happens, it's the same thing as if the target was listed
> > > as a dependency of the special target. E.g., you could express
> > > the same thing above as:
> > >
> > > all: .PHONY do-this do-that
> >
> > That's a cool idea.  I can't think of any issues with it.
>
>
> Just talking about Linux Makefiles,
> we need to duplicate phony targets anyway
> since phony targets are assigned to a variable 'PHONY'.
>
> For example, like this
> https://github.com/torvalds/linux/blob/v5.2-rc4/Makefile#L501
>
> (BTW, I think this idea was provided by Paul.)
>
>
>
> Besides, this might be an unrelated topic,
> is it a good idea to support .PHONY for pattern rules?

I definitely recall wanting this at  some point.  After a few
minutes search I can't find when, where, or why though :)

Britton

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Idea: Allow certain special targets as dependencies

[Britton Kerin]

On Sun, Jun 9, 2019 at 2:47 PM David A. Wheeler  wrote:
>
> Idea: Allow certain special targets as dependencies
>
> Problem:
> It's often the case that a target also needs to be a special target. E.g.:
> .PHONY: all
> all: do-this do-that
>
> Obviously this *works*, but it consumes many extra lines &
> requires duplication of target names.  For example,
> if you misspell the "duplicated" names bad things happen.
> For short makefiles this isn't a big deal, but with lots of rules
> it gets annoying.
>
> Solution:
> As syntactic sugar,
> I'd like to see selected special targets allowed as dependencies.
> When this happens, it's the same thing as if the target was listed
> as a dependency of the special target. E.g., you could express
> the same thing above as:
>
> all: .PHONY do-this do-that
>
> This doesn't add new *functionality* to a makefile, but if you're
> creating a GNU make specific makefile anyway the savings of
> lines can be significant.  It also eliminates a source of errors.

I also find having to write .PHONY: all over annoying.  Not sure
the proposed syntax is the best (not that I have any particular
problem with it, just wonder if there are any good alternatives).

> I propose that this abbreviation with the following special targets:
> .PHONY
> .PRECIOUS
> .INTERMEDIATE
> .SECONDARY
> .IGNORE
> .SILENT
> .NOTPARALLEL
> .ONESHELL

Aren't e.g. NOTPARALLEL and ONESHELL global in effect at
the moment?  Is the proposal to allow making them target-local
or does make already allow that somehow and I'm not aware?

The manual says for ONESHELL:

ONESHELL
If .ONESHELL is mentioned as a target, then when a target is built all
lines of the recipe will be given to a single invocation of the shell
rather than each line being invoked separately (see Recipe Execution).

The documentation for other special built-in targets mentions their
sensitivity to supplied prereqs (e.g. .PHONY).  Maybe the docs
need fixed here.


Britton

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #55137] $(file …) is executed too early when used in recipe

[Britton Kerin]

On Thu, May 16, 2019 at 7:37 AM Paul Smith  wrote:
>
> I'm publishing this back to the list as I don't like to have private
> conversations on these subjects; hopefully you don't mind.
>
> On Wed, 2019-05-15 at 11:19 -0800, Britton Kerin wrote:
> > I agree that having file behave differently would be weird, but it
> > does seem like it would be nice if make functions were expanded a
> > command at a time (rather than a recipe at a time).  It would be too
> > breaking as a general change but perhaps as a .DIRECTIVE: ?
>
> There's no question whatsoever that this behavior has bitten many
> people, many times.  It's one of the more common questions on
> StackOverflow, etc.
>
> I'm honestly not sure why expansion is done that way (it predates my
> involvement in GNU make maintenance).  Offhand it seems like an
> arbitrary choice: I can't think of a good reason for it.  Maybe someone
> else can point out a use for it.
>
> I wonder how much code would actually break by just changing it.

I don't know for sure, but I can testify that I've used make pretty
extensively for many years, including many of the less usual features
(static pattern, order-only rules etc.) and I only recently learned
about this peculiarity.  Not sure if that means my code is insensitive
to it or unconsciously dependent in some way.

> Certainly, any such code would have to be written in a weird way
> although obviously "weird code" does exist.
>
> For example someone could write something like:
>
>foo:
>@cat somefile
>$(shell echo foo > somefile)
>
> which would break if we started delaying expansion until the recipe
> line was being invoked.  But who's going to write that?
>
> The only idea I have for "legitimate real-world use" is something like:
>
> foo:
> do some things
> $(DO-MORE)
>
> where DO-MORE is some boilerplate macro defined elsewhere, that wants
> to perform some checking and invoke $(error ...) if badness is
> detected.  It might be relying on the entire recipe being expanded up-
> front to do that error checking before any part of the recipe is
> invoked.
>
> I just really hate lots of tweakable options: you get an exponential
> increase in testing requirements etc.

Sadly true.  I'll test rat on my stuff if you want to try it as a
flat-out change, but tbh it strikes me as a bit risky that way.

Britton

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

thoughts and questions on order-independent Makefile method

[Britton Kerin]

Hi guys, I tried this on help-make  but was didn't get any response,
so I though I'd try again here.  I'm interested in submitting patches to
implement something like this, but I don't want to waste my time if it's
a total non-starter.  I'd like a --late-parse-rules (to parse all of rules
after variables)  and maybe --order-independent to verify no order-dependent
constructs in Makefile.  Or something like that.

Please let me know if it's a possibility.  Here's the original post again:

After a pretty large amount of Make I've concluded that one of the things that
tends to make make confusing is the fact that it uses linear order-dependent
code to express a DAG which isn't linear at all.  A Makefile is just be a
database of facts about a DAG, so order should be strictly a matter of style,
not semantics.

One example problem area is with include, where it frequently turns out to be
the case that a given include of e.g. generic_build_rules.mk has to be included
somewhere in the middle of a Makefile, since some code needs to go before the
include and other code after.

What I would like to do is achieve a style in which the order of variable and
rule declarations isn't relevant at all.  Rules for coming close as far as I
can see:

1. Never use simply expanded vars
2. Never use if etc, instead use $(if
3. Never use ?= (corollary to 2)
4. Enable .SECONDEXPANSION, and always use $$(VAR) in depends to get lazy eval

Are the any non-obvious bad consequences to these (particularly 4)?

The last part of the puzzle is what to do about targets themselves.  So far as
I can see there's no way to achieve late expansion of vars in targets.  So if
you have generic_build_rules.mk, and you want to introduce a target-specific
variable for a rule with a variable in it's target in a Makefile that includes
that file, you have an unavoidable order-dependency between the include and the
target-specific variable declaration.  Is there a way to avoid this that I'm
missing?

Static pattern rules are another sad case of inevitable order-dependence.
They're more robust and yield must less confusing error messages than implicit
rules, so it's sad no to be able to use them in an order-independent Makefile.

Would it possible to have something like .SECONDEXPANSION that applied to
targets and static pattern lists as well (or instead)?

This is an area I'd be willing to try to help with if it's deemed possible.

Thanks,
Britton

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

.SECONDARY documentation is slightly misleading

[Britton Kerin]

The documentation for the special .SECONDARY: target looks like this:

  .SECONDARY
  The targets which .SECONDARY depends on are treated as
  intermediate
  files, except that they are never automatically deleted. See
  Chains of
  Implicit Rules.

  .SECONDARY with no prerequisites causes all targets to be treated
  as
  secondary (i.e., no target is removed because it is considered
  intermediate).

The problem is the parenthesize comment i.e., no target   Normally
such a
comment is expected to be an overall summary of the preceding material
being
summarized, not a summary of a partial effect.  But in this case the
summary
ignores an important effect: the fact that prerequisites of .SECONDARY
are
treated as intermediate.  As the Chains of Implicit Rules section
describes,
intermediatenes has other significant effects besides eventual removal
of any
file associated with a target.  Things get especially entertaining when
your
phony targets are affected by .SECONDARY:.  In that case I believe I ran
into
places where use of .PHONY on the phony targets made a difference, but
I've
had enough make for this morning to try reproducing it: perhaps it will
be
obvious to an expert that that is expected, or impossible. 

I think the .SECONDARY target might be better if it didn't twiddle the
intermediateness of its prerequisites at all, but simply prevented them
from
being automagically removed if intermediate.  Note that no new
capabilities are
added by having .SECONDARY affect intermediateness: .INTERMEDIATE: can 
intermediateize a file, and .SECONDARY: can't unintermediatize one.

Its probably too late now to change .SECONDARY: but perhaps a new target
which only prevents deletion could be added.

Britton




___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: wish: multiple target patterns in a static pattern rule

[Britton Kerin]

   bk Implicit rules can handle multiple target patterns, is there any
   bk reason static pattern rules couldn't also?
 
 The reason is that static pattern rules are shorthand for writing
 explicit rules, not longhand for writing implicit rules.
 
 That is, a static pattern rule is turned into a long list of explicit
 rules as the makefile is read in, and so it has all the behaviors of an
 explicit rule instead of the behaviors of an implicit rule.
 
 Changing that would be a major change to the way make works.

I guess I don't understand the implementation, but wouldn't you just need 
to do exactly that, only once for each target pattern?  With a requirement 
that each element of TARGETS map to only one target pattern, it should be 
able to be expanded into a list of explicit rules.

   bk There could be a simple requirement that each element of the
   bk TARGETS match exactly one of the target patterns.  The workaroud
   bk requires me to split up the GOB_GENERATED_FILES variable into
   bk three and write three static pattern rules to get the effect I
   bk want, which is a bit clunky,
 
 You don't have to split it into three from a user's perspective: you
 could split it into three in the static pattern rules themselves,
 something like:
 
   $(filter %.h,$($(MN)_GOB_GENERATED_FILES)): %.h: %.gob
 
 etc.
 
 Slightly less clunky.

Yes.  Also since I read about canned commands I don't mind repeating as 
much.  Incidentally, muchos kudos on the order-only prereqs, you have 
though of (and implemented) just about everything.

   bk especially since (with automatic dependency tracking) I don't have
   bk to care about header files much, and don't have any reason to put
   bk them in seperate variables.
 
 I'm assuming there's some reason you can't just use an implicit rule
 here in the first place.  If you don't use implicit rules you'll have to
 be careful to use a dummy target if you want to have a single command
 line generate all those files: explicit (and static pattern) rules don't
 work like that.

I just like static pattern rules a lot better, implicit rules are
relatively error prone, especially when doing things as described in
Recursive Make Considered Harmful, where you end up with one big
Makefile.  Incidently, some kind of mechanism to automaticly add a
namespace prefix to all variables and targets introduced in an included
makefile fragment could be nice and save a ton of computed variable syntax
in the included fragment, though I certainly don't claim to have any good
idea how it might work in detail.

gob always generates all three files every time it runs, and I can't
(easily) change that.  I just don't know or want to know which
gob-produced file will be needed first.

Britton



___
Bug-make mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-make

wish: multiple target patterns in a static pattern rule

[Britton Kerin]

I have a program gob2 which does this:

 some_file.gob - {some_file.c some_file.h some_file-private.h}

So I would like to be able to write this static pattern rule:

 # This rule says how to generate C files from GOB files.
 $($(MN)_GOB_GENERATED_FILES): %.c %.h %-private.h: %.gob
 $($(MN)_GOB2) $

Implicit rules can handle multiple target patterns, is there any reason
static pattern rules couldn't also?  There could be a simple requirement
that each element of the TARGETS match exactly one of the target patterns.
The workaroud requires me to split up the GOB_GENERATED_FILES variable
into three and write three static pattern rules to get the effect I want,
which is a bit clunky, especially since (with automatic dependency
tracking) I don't have to care about header files much, and don't have any
reason to put them in seperate variables.

Britton Kerin







___
Bug-make mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-make