[bug #60595] make doesn't always restart when a makefile is rebuilt

[Paul D. Smith]

Update of bug #60595 (project make):

  Status:None => Fixed  
 Assigned to:None => psmith 
 Open/Closed:Open => Closed 
   Fixed Release:None => SCM
   Triage Status:None => Medium Effort  

___

Follow-up Comment #8:

This fix has been pushed to Git.  Thanks!

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[André Chalella]

Follow-up Comment #7, bug #60595 (project make):

I agree that the following scenario should throw an error, if that's what you
meant:

- a makefile _should_ have been read in AND
- such makefile _could not_ be read in (e.g. does not exist) AND
- make has finished rebuilding all makefiles AND
- make has determined that it _does not need to re-exec_.

I don't understand why make doesn't do that today, as shown by your simple
repro case (and by my real makefile when "include" is in the wrong order).
Maybe it's a separate bug?

Just clear up on something: in my opinion fixing _this_ bug alone wouldn't
increase occurrences of this error, because it will only trigger more
re-execs, never fewer. I say this because in your line below I got the
impression that you meant that fixing this bug alone will automatically start
triggering the error:

> As a note when we do fix this problem the above makefile will fail on the
"include" line after the restart because the rule that says it built m2.d
actually didn't.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[Paul D. Smith]

Follow-up Comment #6, bug #60595 (project make):

I understand that today, make won't throw an error.

I'm saying that _after_ we fix this issue so that make recognizes that a file
needed to be rebuilt, then make could throw an error.

Put another way, in my mind the fact that an included file can't be read is
separate from the process make may use to rebuild the included file.  Or to be
specific if, after make has tried to rebuild all the makefiles and determines
that it doesn't need to re-exec itself, it's still the case that some makefile
which _should_ have been read in _could not_ be read in, then that seems like
an error to me regardless of whether make invoked some rule to build the
makefile or not.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[André Chalella]

Follow-up Comment #5, bug #60595 (project make):

Thank you for all the information and for considering the issue in such
careful manner!

> I assume in your real makefile you've accounted for this and something does
build the target makefile (or maybe you include it with "-include").

No, when building the included makefile is skipped, touching the file is
skipped entirely (i.e this part of the recipe is basically "echo Skipping;
exit"). I haven't encountered any problems, that is, with regular "include"
(not "-include" or "sinclude"), it works fine without a single error or even a
warning: building the dependency makefile is skipped with just a custom
message, then make restarts, then it correctly builds said makefile, then make
restarts again, then it builds my project correctly.

I know the manual describes a failure mode like you said, but I haven't seen
it happen even once. Let me explain in more detail, quoting the manual:

> If an included makefile cannot be found in any of these directories, a
warning message is generated, but it is not an immediately fatal error;

I don't get any warning messages when the include files don't exist yet. Make
just carries on and tries to build them before restarting. It might be worth
mentioning that, in the "include" line, I specify the relative location of
each included file, like: "include dep/mod/helper.d dep/program.d".

> Once it has finished reading makefiles, make will try to remake any that are
out of date or don’t exist.

Yes, that happens just fine.

> Only after it has tried to find a way to remake a makefile and failed, will
make diagnose the missing makefile as a fatal error.

When making the makefile is skipped, there is no fatal error, like I said.
Maybe that's because the recipe terminates successfully ("exit") instead of
abnormally or with an error value?

With "make -d", when a makefile is skipped as described above, make outputs
"Reaping winning child" and "Successfully remade target file 'dep/program.d'",
even though the file still doesn't exist and will be created only in the next
restart (which occurs flawlessly). I thought this was simply because the
recipe exits without error, like I said above.

For all that, it never struck me as necessary preparing for the scenario you
described. Does it look like I'm doing something wrong?

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[Paul D. Smith]

Follow-up Comment #4, bug #60595 (project make):

Thanks for the description.  Here's a much simpler repro case:


all: ; @echo RESTARTS=$(MAKE_RESTARTS)

m1.d: ; touch $@

m2.d: m1.d ; test -f $< || touch $@

include m2.d m1.d


If you run this makefile with:


rm -f m?.d && make


it will correctly restart make and print RESTARTS=1.

If you change the last line to reverse the order of include:


include m1.d m2.d


then it will not restart make and print RESTARTS=.

As a note when we do fix this problem the above makefile will fail on the
"include" line after the restart because the rule that says it built m2.d
actually didn't.  I assume in your real makefile you've accounted for this and
something does build the target makefile (or maybe you include it with
"-include").

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[Paul D. Smith]

Follow-up Comment #3, bug #60595 (project make):

I agree that this is a bug: the behavior should match the manual.

Just to note that in the next version of GNU make the order of rebuilding
included makefiles will be reversed so that it happens in the "expected" way
(first file is checked first).  There was an issue with portability that
necessitated it.  The NEWS file for the next release says:


* WARNING: Backward-incompatibility!
  Previously the order in which makefiles were remade was not explicitly
  stated, but it was (roughly) the inverse of the order in which they were
  processed by make.  In this release, the order in which makefiles are
  rebuilt is the same order in which make processed them, and this is defined
  to be true in the GNU make manual.


Seems like we should address this bug in that release as well to avoid
problems.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[André Chalella]

Follow-up Comment #2, bug #60595 (project make):

I have attached an example Makefile (with comments) that illustrates my use
case. It models a project where 'program1' uses 'module1' which in turn uses
'module2'.

Dependency resolution is managed through .d files, which in the real system
are autogenerated from the real source files.

The module*.d recipe creates a 'guard' file as side-effect, whose purpose is
to signal to the program*.d recipe that it must wait the next restart to do
real work. That is because the program*.d recipe needs the rules in module*.d
correctly loaded in make. When make restarts, it removes the guard file.

In the example, 'make' makes the fake program 'program1' correctly, while
'make wrong' triggers the buggy behavior. The only difference is the include
order.

I suggest following the sequence below:

$ make
$ touch program1.d && make
$ touch module1.d && make
$ touch module2.d && make

You shall see make correctly updating and rebuilding only the needed files.
However, run:

$ touch module2.d && make wrong

You will see that the build occurs before 'program1.d' is properly rebuilt,
i.e make doesn't restart after updating 'module1.d', contrary to the manual.
Run 'make wrong' again and you will see that only now 'program1.d' gets
updated, but 'program1' has already been built, so it is too late.

(file #51463)
___

Additional Item Attachment:

File name: Makefile   Size:1 KB




___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[Dmitry Goncharov]

Follow-up Comment #1, bug #60595 (project make):

Can you please attach a makefile which reproduces the issue?

___

Reply to this item at:

  

___
  Сообщение отправлено по Savannah
  https://savannah.gnu.org/

[bug #60595] make doesn't always restart when a makefile is rebuilt

[André Chalella]

URL:
  

 Summary: make doesn't always restart when a makefile is
rebuilt
 Project: make
Submitted by: andrechalella
Submitted on: Thu 13 May 2021 07:32:32 AM UTC
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 4.3
Operating System: Any
   Fixed Release: None
   Triage Status: None

___

Details:

GNU make manual, in section 3.5 How Makefiles Are Remade
,
says:

> After all makefiles have been checked, if any have actually been changed,
make starts with a clean slate and reads all the makefiles over again.

I found the hard way this is not _quite_ true. If, during makefile rebuilding
("Updating makefiles"), the following conditions are met, then make won't
restart, carrying on as if nothing was updated.

0 One makefile is rebuilt as a *prerequisite* of another makefile;
0 The *goal* makefile of #1 is not rebuilt (e.g. recipe executes but exits
without modifying the file, so its mtime doesn't change).

I know this may seem an obscure corner case and maybe it is, and I apologize
if that's the case. I don't have much experience with "real world" makefiles;
I write mine based on the manual (which is excellent) and the rebuild-restart
functionality has made possible for me to write a simple yet powerful
dependency resolver for Fortran .

The thing is, it stopped working when I incidentally changed "include
$(files2) $(files1)" to "include $(files1) $(files2)" somewhere in it, and it
took me a couple days, many rounds of "make -d" and reading the source to
understand why.

(in the example line above, $(files1) are distant prerequisites to $(files2))

I can and will (at least for now) keep the original, working order, but it is
kind of the "wrong" order of files logic-wise, plus it only works because make
rebuilds makefiles in the reversed order in which they were included. I know
why make rebuilds in reverse and it's a valid reason, but as Paul said in
another answer it is an implementation detail and I wouldn't like to rely on
it.

I can provide a minimal working example if the issue is not clear enough.

I see two possible solutions:

0 In main.c:2272 (case us_none) check current mtimes against makefile_mtimes[]
and jump to re_exec if any is newer.
0 If it must not be fixed in code, a small update on the docs to specify that
makefiles rebuilt as prerequisites do not count for restarting make.

Thanks,
André




___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/