Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
> From: Paul Smith > Cc: stefano.lattar...@gmail.com, bug-make@gnu.org > Date: Fri, 03 May 2013 17:17:44 -0400 > > -O in no way changes that behavior, all it does is ensure that output > from any individual line or target of the recipe will not interfere with > any other individual line or target. At the Make's internals level, -O indeed doesn't change that behavior. But, whether we want it or not, Make users perceive the order in which they see the output as the order in which things got executed. So any significant changes in the order we show the results of remaking a target will, or at least might, be interpreted as changes in behavior. While it is OK for a non-default switch to change behavior, the modified behavior must make sense, or users will complain. For the default behavior, it must make a lot of sense. Without -O, output might be messy at times, but there's never any doubt in user's mind what happened: she understands that the jobs whose output appears on the screen at the same time run in parallel. Add -O (in its current implementation), and the user now stares at a puzzle: output appears serialized, but the order of that serialization is hard to make sense of. I'm not even sure I understand completely what confuses me when I look at such output. Perhaps my mind tries to interpret that as a non-parallel run, where a prerequisite target is always remade before the target that's dependent on that prerequisite. Or maybe some other mental mechanism is at work here. All I can say is that the order in which -O shows output confuses the heck out of me, and I need to look at it for a long time, reading and re-reading its portions, before I can convince myself that everything that needed to be remade was indeed remade, and in correct order. I guess that means I will seldom if ever use -O. And if that doesn't bother anyone, then let's just leave it at that. Either I'm an odd one out, or you will hear from others who might explain this better than I could. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
> Date: Sat, 04 May 2013 04:42:32 +0200 > Cc: e...@gnu.org, david.s.bo...@gmail.com, bug-make@gnu.org > From: Frank Heckenbach > > : /* This is needed to avoid the "label at end of compound statement" > : diagnostics on Posix platforms. */ > > I don't think that's a POSIX thing (which is a standard about > operating systems, not compiler diagnostics), is it? You quote this out of context, which is this: finished: #ifdef WINDOWS32 /* Switch to_fd back to its original mode, so that log messages by Make have the same EOL format as without --output-sync. */ _setmode (to_fd, prev_mode); #endif /* This is needed to avoid the "label at end of compound statement" diagnostics on Posix platforms. */ return; } I used "Posix platforms" here (not just POSIX the name of the standard) in contrast to WINDOWS32, where there's some code between the label and the closing brace, which prevents the (stupid, IMO) warning. > Also, I think in the Windows branch, the former condition also > should say just "output_sync" instead of > "output_sync == OUTPUT_SYNC_TARGET", a recent change during the > introduction of "-O job". You are right. This is a consequence of having 2 separate branches in the code. I fixed this one. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] cosmetics: fix few innocuous typos
On 04/30/2013 03:37 PM, Stefano Lattarini wrote: > Most of these were found using Lucas De Marchi's 'codespell' tool. > > * ChangeLog: Fix minor typos. > * ChangeLog.2: Likewise. > * README.Amiga: Likewise. > * TODO.private: Likewise. > * function.c: Likewise. > * glob/glob.h: Likewise. > * job.c: Likewise. > * main.c: Likewise. > * readme.vms: Likewise. > * remake.c: Likewise. > * tests/ChangeLog: Likewise. > * tests/NEWS: Likewise. > * tests/README: Likewise. > * tests/scripts/variables/private: Likewise. > * vmsdir.h: Likewise. > * signame.c: Likewise. While at it, improve line wrapping in the > touched comment. > > Copyright-paperwork-exempt: yes > Signed-off-by: Stefano Lattarini > Ping? Any reason not to apply this trivial patch? Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] cosmetics: fix few innocuous typos
On 05/04/2013 01:34 PM, Paul Smith wrote: > On Sat, 2013-05-04 at 11:52 +0200, Stefano Lattarini wrote: >> Ping? Any reason not to apply this trivial patch? > > You may have noticed, there's a lot going on right now... and I do have > an actually for-$$ job as well :-). Many of these kinds of things have > to wait until the weekends. > OK, no problem. I just wanted to ensure this didn't fall through the cracks. I will be more patient next time. > It's queued and I'll get to it, thanks! > Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
On Sat, 2013-05-04 at 09:57 +0300, Eli Zaretskii wrote: > > From: Paul Smith > > Cc: reinp...@win.tue.nl, bug-make@gnu.org > > Date: Fri, 03 May 2013 16:51:47 -0400 > > > > > I think enabling [-O] by default will be a no-brainer if/when we come up > > > with a way to get it to produce the same output as without -j. IOW, > > > run a parallel build, but output its results as if it were a > > > non-parallel build. > > > > If you mean literally exactly the same as a non-parallel build, that is > > enormously difficult. > > Yes, literally, but with one exception: the order of producing on the > screen output from targets that are remade in parallel (i.e. are > independent of each other), and are on the same level of recursion, > does not have to be preserved. Well that's not "literally exactly the same" then :-). But I won't agree to the caveats here, in particular the phrase "on the same level of recursion". Today with parallel builds and recursion, we don't guarantee anything about the order of execution between different sub-makes and as I mentioned, I see no justification for adding that new requirement to synchronized output. > Example: > > all: a b > a: xa > @echo $@ > xa xb xc: > @echo $@ > b: xb xc xd > @echo $@ > xd: > $(MAKE) foo > @echo $@ > By contrast, I _would_ mind to see this, for example: > > xa > xb > a > xc > xd > $(MAKE) > b I agree that is less than ideal and is one of the two issues I mention below. This will be fixed. However, you may see this: xa xb a $(MAKE) foo xc xd b There's nothing that can be done about that (and that's true of today's parallel build as well). > > My position is if we can get output sync to a level where it is no worse > > than today's parallel build behavior > > If we want it to be "no worse", then why do we need it at all, let > alone have it turned on by default? I thought -O should actually > improve something, so "no worse" is too weak to describe that, IMO. Obviously we gain synchronized output. I mean, "no worse" in terms of other behavior. > And what are our criteria for deciding whether it's no worse? The > current default behavior might be confusing and hard to interpret in > some cases, but at least it's familiar. -O changes that to a > different behavior which is confusing (at least to me) in different > situations, and is unfamiliar. I believe we can get to the point where anyone who can read and understand parallel output can even more easily read and understand the output from -O. Conceptually all we're doing is ensuring the output comes out at the same time, uninterrupted, instead of interleaved. It should not be hard to understand that. There are two known issues right now that are causing confusing output. Hopefully once I fix those the output will make more sense than normal parallel output. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
> From: Paul Smith > Cc: reinp...@win.tue.nl, bug-make@gnu.org > Date: Sat, 04 May 2013 09:04:24 -0400 > > you may see this: > > xa > xb > a > $(MAKE) foo > xc > xd > b If "a" appears before "xb", then that's all I ask for. > > If we want it to be "no worse", then why do we need it at all, let > > alone have it turned on by default? I thought -O should actually > > improve something, so "no worse" is too weak to describe that, IMO. > > Obviously we gain synchronized output. It's not synchronized. It's just has a coarser granularity than what we get without -O. IOW, the chunks are larger, but still interlaced in somewhat random order. > I believe we can get to the point where anyone who can read and > understand parallel output can even more easily read and understand the > output from -O. I'd surely like to get there ;-) ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Sat, 2013-05-04 at 08:57 +0200, Frank Heckenbach wrote:
> I shouldn't have written that. :-( Shortly afterwards, I found a bug
> or perhaps two:
>
> foo:
> @echo foo
> +@echo bar
>
> (a)
> % make -Ojob
> foo
> bar
> foo
>
> (b)
> % make -Otarget
> bar
> foo
>
> As you see, (a) "-Ojob" writes "foo" twice and (b) "-Otarget" writes
> the messages in the wrong order.
The second one is known and I mentioned it the other day (hard to keep
up with all the messages, I know). I'm working on a fix.
The first one I've seen but hadn't had time to debug. I'll look at your
patch. I left the truncate where it was rather than doing it after the
sync_output() because I was hoping to avoid truncating a file that we'll
never use again anyway, but I guess it isn't a big deal.
COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+'
prerequisite is to tell make that this line, even though it may not look
like it, will run a recursive make.
Since make doesn't parse the command line it can't know for sure which
ones actually recurse. It uses a heuristic, by looking for $(MAKE) or
${MAKE} in the unexpanded line. But this is easily defeated if your
sub-make invocation doesn't use that variable for some reason. Hence,
using "+" to force it.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
>> I think having this facility built into make is a win, especially as >> parallel builds become predominant. I would be even more happy about it >> if we can get it to the point where it can be enabled by default, and >> users don't even have to worry about it. > I agree with Paul. This is something that can't really be done right > by an external tool. While I agree that make is a better place to do this than an external tool, something in the back of my mind has been niggling about this still not really being the right answer. I suspect the root of that niggle is a desire to make the stdio FILE / descriptor implement this; there would be a way to "fork" a file descriptor (or a way to configure what fork() does with inherited file descriptors), such that each child forked off it gets to see something that looks and feels just like the original (e.g. isatty() and kindred functions give the same answers) but with some configuration options passed to the fdfork/ffork function that determine what happens to interleaved I/O among the children of a common parent. One option would then be to buffer all output from the child and deliver it when the child closes. (Another commonly desirable pattern would be to ensure that interleaving happens only at line boundaries, for example, to avoid the case some of us report with -j of one process's error coming in part way through a line of output from another.) When make fork()s its sub-processes, it could then (before exec()ing the sub-process) select which way to set the interleaf options on the file-handles it inherits from its parent. There would be nothing particularly special about make here: any program that invokes fork() would be able to do similar; and any program that fork()s many children is indeed apt to want the same choices you're considering for make (and maybe others). This is undoubtedly too big a thing to change in the standard library, and has been for decades, so we have to make do with doing it in make instead; but maybe it'll stir someone's brain to think of another way to achieve the same result, that's not so specific to make. Perhaps a "shim" round the standard library might implement ffork/fdfork, or something like them. I have no idea whether it's even possible to preserve descriptor attributes in the sense countenanced, using only the existing standard library facilities. Perhaps a new option to fcntl() could do the trick (but that'd be little use to make, today, as it'd take decades before all fcntl()s implement it). Anyway, enough rattling - I think what you've implemented sounds like a great feature and I await with glee its (suitably tried, tested and refined) release in the next version of GNU make, Eddy. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Instructions for building extensions
Should the Make manual include instructions, however short, about building extensions? Not writing the code (that is covered), but actually compiling the extensions so that Make will be able to load them. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote: > Suppose we do this: if we're about to invoke a line marked recursive > and we're in -Otarget mode, then before we run it we'll show the > current contents of the temp file (using the normal synchronized > output function). I've implemented this feature and it seems to work as expected. I also implemented the fix to the duplicate output being shown in some cases. I have two open issues I want to address before calling this feature done: first, fix make's writing of the command it's going to run (for rules that don't start with "@") as that's not working right. Second, fix the enter/leave issue. It turns out that these are currently somewhat bound together so I may have to solve the second one first. Oh, and a renaming as well :-) Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There will be more disruption I think. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
Hi Paul. On 05/05/2013 12:10 AM, Paul Smith wrote: > On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote: >> Suppose we do this: if we're about to invoke a line marked recursive >> and we're in -Otarget mode, then before we run it we'll show the >> current contents of the temp file (using the normal synchronized >> output function). > > I've implemented this feature and it seems to work as expected. I also > implemented the fix to the duplicate output being shown in some cases. > > I have two open issues I want to address before calling this feature > done: first, fix make's writing of the command it's going to run (for > rules that don't start with "@") as that's not working right. Second, > fix the enter/leave issue. It turns out that these are currently > somewhat bound together so I may have to solve the second one first. > The test 'features/output-sync' now fails for me: Test timed out after 6 seconds Error running /storage/home/stefano/src/gnu/make/tests/../make \ (expected 0; got 14): /storage/home/stefano/src/gnu/make/tests/../make \ -f work/features/output-sync.mk.1 -j --output-sync=target Caught signal 14! FAILED (4/6 passed) Can you reproduce the failure? If not, let me know which information you need to debug this, and I'll try to provide them. Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[PATCH] docs: port manual to Texinfo 5.x
* doc/make.texi: Here. It was sufficient to change an '@itemx'
into an '@item'.
Copyright-paperwork-exempt: yes
Signed-off-by: Stefano Lattarini
---
doc/make.texi | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/make.texi b/doc/make.texi
index ea58d6e..fa9b5d7 100644
--- a/doc/make.texi
+++ b/doc/make.texi
@@ -8812,7 +8812,7 @@ from complicated nests of recursive @code{make} commands.
rarely need to specify this option since @samp{make} does it for you;
see @ref{-w Option, ,The @samp{--print-directory} Option}.)
-@itemx --no-print-directory
+@item --no-print-directory
@cindex @code{--no-print-directory}
Disable printing of the working directory under @code{-w}.
This option is useful when @code{-w} is turned on automatically,
--
1.8.3.rc0.19.g7e6a0cc
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Paul Smith wrote:
> The first one I've seen but hadn't had time to debug. I'll look at your
> patch. I left the truncate where it was rather than doing it after the
> sync_output() because I was hoping to avoid truncating a file that we'll
> never use again anyway, but I guess it isn't a big deal.
One could probably optimize it to avoid truncating if it's not going
to be reused, but I'm not sure it's worth the effort (including
future maintenance).
> COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+'
> prerequisite is to tell make that this line, even though it may not look
> like it, will run a recursive make.
OK, let me just say that the meaning of "recursive" may not be
perfectly clear. Though the manual says: "The @samp{+} prefix marks
those recipe lines as ``recursive'' so that they will be executed
despite use of the @samp{-t} flag.", the example immediately
preceding this sentence has:
+touch archive.a
+ranlib -t archive.a
which are clearly not recursive make invocations.
I gather that make uses recursive in a wider sense as "anything to
be run regardless (because it probably arrages by itself not to do
anything serious in a dry run or so)", while the current
implementation of output-sync uses it in the more specific meaning
of a recursive invocation of GNU make (which will do its own
syncing).
Similar concerns seem to apply to other places where
COMMANDS_RECURSE is checked, e.g. (right after the output-sync code
in question):
/* If we aren't running a recursive command and we have a jobserver
pipe, close it before exec'ing. */
if (!(flags & COMMANDS_RECURSE) && job_fds[0] >= 0)
{
close (job_fds[0]);
close (job_fds[1]);
}
I don't think those touch and ranlib commands above need the
jobserver pipe (though, of course, this one doesn't hurt much, just
costs 2 available fds).
> Since make doesn't parse the command line it can't know for sure which
> ones actually recurse. It uses a heuristic, by looking for $(MAKE) or
> ${MAKE} in the unexpanded line. But this is easily defeated if your
> sub-make invocation doesn't use that variable for some reason. Hence,
> using "+" to force it.
If this is the actual intention, and the example is really a misuse
of the feature (maybe not ideal to put this in the manual then?),
this means indeed that "+" lines are exempt from output
synchronization and I see no easy way around it.
A (hypothetical?) use case would be another make-like tool which is
invoked from a Makefile. Unless it implements its own output-sync,
compatible with ours, its output remains unsynced.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Sun, 2013-05-05 at 04:37 +0200, Frank Heckenbach wrote:
> > COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+'
> > prerequisite is to tell make that this line, even though it may not look
> > like it, will run a recursive make.
>
> OK, let me just say that the meaning of "recursive" may not be
> perfectly clear. Though the manual says: "The @samp{+} prefix marks
> those recipe lines as ``recursive'' so that they will be executed
> despite use of the @samp{-t} flag.", the example immediately
> preceding this sentence has:
>
> +touch archive.a
> +ranlib -t archive.a
>
> which are clearly not recursive make invocations.
>
> I gather that make uses recursive in a wider sense as "anything to
> be run regardless (because it probably arrages by itself not to do
> anything serious in a dry run or so)", while the current
> implementation of output-sync uses it in the more specific meaning
> of a recursive invocation of GNU make (which will do its own
> syncing).
It's not just this new feature that relies on this meaning. The
jobserver feature, which also wants to know which commands are running
recursive make, also does.
If people misuse it then they'll get odd behavior. I don't see that
there's anything we can or should do about that.
You're right, though, that this example in the make manual might not be
the best.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
On Sun, 2013-05-05 at 00:44 +0200, Stefano Lattarini wrote: > The test 'features/output-sync' now fails for me: > > Test timed out after 6 seconds > Error running /storage/home/stefano/src/gnu/make/tests/../make \ > (expected 0; got 14): /storage/home/stefano/src/gnu/make/tests/../make \ > -f work/features/output-sync.mk.1 -j --output-sync=target > Caught signal 14! > FAILED (4/6 passed) > > Can you reproduce the failure? If not, let me know which information you > need to debug this, and I'll try to provide them. It didn't fail for me. However, it's possible that the 6 second timeout is just a little too short for reliability. Look at line 178 in the test/scripts/features/output-sync file. It will look like this: #MAKE#[1]: Leaving directory '#PWD#/bar'\n", 0, 6); The "6" at the end is the timeout. Try changing that to 10 just to see if it helps. If not then it's a real problem. I'll need details about your system. Also please send the contents of the work subdirectory after the failure. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
