Re: Another issue with -O?
From: Paul Smith psm...@gnu.org Cc: reinp...@win.tue.nl, bug-make@gnu.org Date: Fri, 03 May 2013 16:51:47 -0400 I think enabling [-O] by default will be a no-brainer if/when we come up with a way to get it to produce the same output as without -j. IOW, run a parallel build, but output its results as if it were a non-parallel build. If you mean literally exactly the same as a non-parallel build, that is enormously difficult. Yes, literally, but with one exception: the order of producing on the screen output from targets that are remade in parallel (i.e. are independent of each other), and are on the same level of recursion, does not have to be preserved. Doing that would require that instead of dumping output in the order that jobs finish, make would have to remember the order in which they were started and dump the output in that order. I meant targets, not jobs (the latter are single command lines, right?). Dump output of a whole target as a single unit. As for the order, as long as the targets are remade in parallel, it doesn't matter. Example: all: a b a: xa @echo $@ xa xb xc: @echo $@ b: xb xc xd @echo $@ xd: $(MAKE) foo @echo $@ In the above case, I'd like to see xa a xb xc $(MAKE) foo xd b but I wouldn't mind to see this instead: xa a xc xb $(MAKE) foo xd b or this: $(MAKE) foo xd xc xb b xa a By contrast, I _would_ mind to see this, for example: xa xb a xc xd $(MAKE) b My position is if we can get output sync to a level where it is no worse than today's parallel build behavior If we want it to be no worse, then why do we need it at all, let alone have it turned on by default? I thought -O should actually improve something, so no worse is too weak to describe that, IMO. And what are our criteria for deciding whether it's no worse? The current default behavior might be confusing and hard to interpret in some cases, but at least it's familiar. -O changes that to a different behavior which is confusing (at least to me) in different situations, and is unfamiliar. How do we decide which one is worse? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
: I've tested your changes and so far -O job works for my use cases.
I shouldn't have written that. :-( Shortly afterwards, I found a bug
or perhaps two:
foo:
@echo foo
+@echo bar
(a)
% make -Ojob
foo
bar
foo
(b)
% make -Otarget
bar
foo
As you see, (a) -Ojob writes foo twice and (b) -Otarget writes
the messages in the wrong order. -Omake is correct.
I debugged (a) and found that sync_output() is called twice without
an assign_child_tempfiles() in between which would ftruncate() the
temp files. The second one does not call it because
flags COMMANDS_RECURSE is set. This led me to this ChangeLog
entry:
: Fri Sep 17 00:37:18 1993 Roland McGrath (rol...@churchy.gnu.ai.mit.edu)
:
: * job.c (start_job_command): Set COMMANDS_RECURSE in FLAGS if we
: see a `+' at the beginning of the command line.
Well, I haven't often found an answer I was looking for in a
20 years old ChangeLog entry. :) However, that's kind of bad news
since I'd written my changes under the assumption that
COMMANDS_RECURSE means, well, to recurse.
(a), which is clearly a bug, can be fixed if we make sync_output()
ftruncate() the temp files (the original patch closed them at this
point, and let the next job open new ones). The patch below does
this. I think it also adds some clarity and resilience WRT future
changes, since it makes sure that the contents are purged as soon as
they were output.
In (b), since make thinks the second line is recursive, and the
first one not, it syncs the second one first, then outputs the
first one normally. In fact, you can see that + lines are not
synced at all with -Otarget:
test: foo bar
foo:
+@echo foo1
+sleep 1
+@echo foo2
bar:
+@echo bar1
+sleep 1
+@echo bar2
% make -Otarget -j
foo1
bar1
sleep 1
sleep 1
foo2
bar2
Maybe you consider this correct if + lines are indeed considered
recursive and this isn't just an implementation artifact. If so,
alright (it doesn't affect me personally, since I use -Ojob). But
if not, this might require slightly bigger changes such as splitting
COMMANDS_RECURSE into two flags. Paul?
PS: I wrote:
: The following patch makes these changes and fixes a precedence bug
: that had crept in during the introduction of child_out (although I
: can't actually test it on VMS :).
: - if (exit_code 1 != 0)
: + if ((exit_code 1) != 0)
Strictly speaking, it's not a bug, since exit_code 1 != 0 parses
as exit_code (1 != 0) which is the same as exit_code 1 and
thus (exit_code 1) != 0, but I doubt that was the intention. ;)
--- job.c.orig 2013-05-04 04:08:09.0 +0200
+++ job.c 2013-05-04 08:32:43.0 +0200
@@ -617,25 +617,7 @@
CLOSE_ON_EXEC (c-errfd);
}
}
-
- return 1;
}
-
- /* We already have a temp file. On per-job output, truncate and reset. */
- if (output_sync == OUTPUT_SYNC_JOB)
-{
- if (c-outfd = 0)
-{
- lseek(c-outfd, 0, SEEK_SET);
- ftruncate(c-outfd, 0);
-}
- if (c-errfd = 0 c-errfd != c-outfd)
-{
- lseek(c-errfd, 0, SEEK_SET);
- ftruncate(c-outfd, 0);
-}
-}
-
return 1;
}
@@ -679,6 +661,11 @@
}
}
+ /* Truncate and reset input file to avoid duplicating data
+ in case it will be used again. */
+ lseek (from_fd, 0, SEEK_SET);
+ ftruncate (from_fd, 0);
+
finished:
#ifdef WINDOWS32
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Fri, 03 May 2013 17:17:44 -0400 -O in no way changes that behavior, all it does is ensure that output from any individual line or target of the recipe will not interfere with any other individual line or target. At the Make's internals level, -O indeed doesn't change that behavior. But, whether we want it or not, Make users perceive the order in which they see the output as the order in which things got executed. So any significant changes in the order we show the results of remaking a target will, or at least might, be interpreted as changes in behavior. While it is OK for a non-default switch to change behavior, the modified behavior must make sense, or users will complain. For the default behavior, it must make a lot of sense. Without -O, output might be messy at times, but there's never any doubt in user's mind what happened: she understands that the jobs whose output appears on the screen at the same time run in parallel. Add -O (in its current implementation), and the user now stares at a puzzle: output appears serialized, but the order of that serialization is hard to make sense of. I'm not even sure I understand completely what confuses me when I look at such output. Perhaps my mind tries to interpret that as a non-parallel run, where a prerequisite target is always remade before the target that's dependent on that prerequisite. Or maybe some other mental mechanism is at work here. All I can say is that the order in which -O shows output confuses the heck out of me, and I need to look at it for a long time, reading and re-reading its portions, before I can convince myself that everything that needed to be remade was indeed remade, and in correct order. I guess that means I will seldom if ever use -O. And if that doesn't bother anyone, then let's just leave it at that. Either I'm an odd one out, or you will hear from others who might explain this better than I could. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Date: Sat, 04 May 2013 04:42:32 +0200 Cc: e...@gnu.org, david.s.bo...@gmail.com, bug-make@gnu.org From: Frank Heckenbach f.heckenb...@fh-soft.de : /* This is needed to avoid the label at end of compound statement : diagnostics on Posix platforms. */ I don't think that's a POSIX thing (which is a standard about operating systems, not compiler diagnostics), is it? You quote this out of context, which is this: finished: #ifdef WINDOWS32 /* Switch to_fd back to its original mode, so that log messages by Make have the same EOL format as without --output-sync. */ _setmode (to_fd, prev_mode); #endif /* This is needed to avoid the label at end of compound statement diagnostics on Posix platforms. */ return; } I used Posix platforms here (not just POSIX the name of the standard) in contrast to WINDOWS32, where there's some code between the label and the closing brace, which prevents the (stupid, IMO) warning. Also, I think in the Windows branch, the former condition also should say just output_sync instead of output_sync == OUTPUT_SYNC_TARGET, a recent change during the introduction of -O job. You are right. This is a consequence of having 2 separate branches in the code. I fixed this one. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] cosmetics: fix few innocuous typos
On 04/30/2013 03:37 PM, Stefano Lattarini wrote: Most of these were found using Lucas De Marchi's 'codespell' tool. * ChangeLog: Fix minor typos. * ChangeLog.2: Likewise. * README.Amiga: Likewise. * TODO.private: Likewise. * function.c: Likewise. * glob/glob.h: Likewise. * job.c: Likewise. * main.c: Likewise. * readme.vms: Likewise. * remake.c: Likewise. * tests/ChangeLog: Likewise. * tests/NEWS: Likewise. * tests/README: Likewise. * tests/scripts/variables/private: Likewise. * vmsdir.h: Likewise. * signame.c: Likewise. While at it, improve line wrapping in the touched comment. Copyright-paperwork-exempt: yes Signed-off-by: Stefano Lattarini stefano.lattar...@gmail.com Ping? Any reason not to apply this trivial patch? Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] cosmetics: fix few innocuous typos
On 05/04/2013 01:34 PM, Paul Smith wrote: On Sat, 2013-05-04 at 11:52 +0200, Stefano Lattarini wrote: Ping? Any reason not to apply this trivial patch? You may have noticed, there's a lot going on right now... and I do have an actually for-$$ job as well :-). Many of these kinds of things have to wait until the weekends. OK, no problem. I just wanted to ensure this didn't fall through the cracks. I will be more patient next time. It's queued and I'll get to it, thanks! Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
On Sat, 2013-05-04 at 09:57 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: reinp...@win.tue.nl, bug-make@gnu.org Date: Fri, 03 May 2013 16:51:47 -0400 I think enabling [-O] by default will be a no-brainer if/when we come up with a way to get it to produce the same output as without -j. IOW, run a parallel build, but output its results as if it were a non-parallel build. If you mean literally exactly the same as a non-parallel build, that is enormously difficult. Yes, literally, but with one exception: the order of producing on the screen output from targets that are remade in parallel (i.e. are independent of each other), and are on the same level of recursion, does not have to be preserved. Well that's not literally exactly the same then :-). But I won't agree to the caveats here, in particular the phrase on the same level of recursion. Today with parallel builds and recursion, we don't guarantee anything about the order of execution between different sub-makes and as I mentioned, I see no justification for adding that new requirement to synchronized output. Example: all: a b a: xa @echo $@ xa xb xc: @echo $@ b: xb xc xd @echo $@ xd: $(MAKE) foo @echo $@ By contrast, I _would_ mind to see this, for example: xa xb a xc xd $(MAKE) b I agree that is less than ideal and is one of the two issues I mention below. This will be fixed. However, you may see this: xa xb a $(MAKE) foo xc xd b There's nothing that can be done about that (and that's true of today's parallel build as well). My position is if we can get output sync to a level where it is no worse than today's parallel build behavior If we want it to be no worse, then why do we need it at all, let alone have it turned on by default? I thought -O should actually improve something, so no worse is too weak to describe that, IMO. Obviously we gain synchronized output. I mean, no worse in terms of other behavior. And what are our criteria for deciding whether it's no worse? The current default behavior might be confusing and hard to interpret in some cases, but at least it's familiar. -O changes that to a different behavior which is confusing (at least to me) in different situations, and is unfamiliar. I believe we can get to the point where anyone who can read and understand parallel output can even more easily read and understand the output from -O. Conceptually all we're doing is ensuring the output comes out at the same time, uninterrupted, instead of interleaved. It should not be hard to understand that. There are two known issues right now that are causing confusing output. Hopefully once I fix those the output will make more sense than normal parallel output. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
From: Paul Smith psm...@gnu.org Cc: reinp...@win.tue.nl, bug-make@gnu.org Date: Sat, 04 May 2013 09:04:24 -0400 you may see this: xa xb a $(MAKE) foo xc xd b If a appears before xb, then that's all I ask for. If we want it to be no worse, then why do we need it at all, let alone have it turned on by default? I thought -O should actually improve something, so no worse is too weak to describe that, IMO. Obviously we gain synchronized output. It's not synchronized. It's just has a coarser granularity than what we get without -O. IOW, the chunks are larger, but still interlaced in somewhat random order. I believe we can get to the point where anyone who can read and understand parallel output can even more easily read and understand the output from -O. I'd surely like to get there ;-) ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Sat, 2013-05-04 at 08:57 +0200, Frank Heckenbach wrote:
I shouldn't have written that. :-( Shortly afterwards, I found a bug
or perhaps two:
foo:
@echo foo
+@echo bar
(a)
% make -Ojob
foo
bar
foo
(b)
% make -Otarget
bar
foo
As you see, (a) -Ojob writes foo twice and (b) -Otarget writes
the messages in the wrong order.
The second one is known and I mentioned it the other day (hard to keep
up with all the messages, I know). I'm working on a fix.
The first one I've seen but hadn't had time to debug. I'll look at your
patch. I left the truncate where it was rather than doing it after the
sync_output() because I was hoping to avoid truncating a file that we'll
never use again anyway, but I guess it isn't a big deal.
COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+'
prerequisite is to tell make that this line, even though it may not look
like it, will run a recursive make.
Since make doesn't parse the command line it can't know for sure which
ones actually recurse. It uses a heuristic, by looking for $(MAKE) or
${MAKE} in the unexpanded line. But this is easily defeated if your
sub-make invocation doesn't use that variable for some reason. Hence,
using + to force it.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
I think having this facility built into make is a win, especially as parallel builds become predominant. I would be even more happy about it if we can get it to the point where it can be enabled by default, and users don't even have to worry about it. I agree with Paul. This is something that can't really be done right by an external tool. While I agree that make is a better place to do this than an external tool, something in the back of my mind has been niggling about this still not really being the right answer. I suspect the root of that niggle is a desire to make the stdio FILE / descriptor implement this; there would be a way to fork a file descriptor (or a way to configure what fork() does with inherited file descriptors), such that each child forked off it gets to see something that looks and feels just like the original (e.g. isatty() and kindred functions give the same answers) but with some configuration options passed to the fdfork/ffork function that determine what happens to interleaved I/O among the children of a common parent. One option would then be to buffer all output from the child and deliver it when the child closes. (Another commonly desirable pattern would be to ensure that interleaving happens only at line boundaries, for example, to avoid the case some of us report with -j of one process's error coming in part way through a line of output from another.) When make fork()s its sub-processes, it could then (before exec()ing the sub-process) select which way to set the interleaf options on the file-handles it inherits from its parent. There would be nothing particularly special about make here: any program that invokes fork() would be able to do similar; and any program that fork()s many children is indeed apt to want the same choices you're considering for make (and maybe others). This is undoubtedly too big a thing to change in the standard library, and has been for decades, so we have to make do with doing it in make instead; but maybe it'll stir someone's brain to think of another way to achieve the same result, that's not so specific to make. Perhaps a shim round the standard library might implement ffork/fdfork, or something like them. I have no idea whether it's even possible to preserve descriptor attributes in the sense countenanced, using only the existing standard library facilities. Perhaps a new option to fcntl() could do the trick (but that'd be little use to make, today, as it'd take decades before all fcntl()s implement it). Anyway, enough rattling - I think what you've implemented sounds like a great feature and I await with glee its (suitably tried, tested and refined) release in the next version of GNU make, Eddy. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Instructions for building extensions
Should the Make manual include instructions, however short, about building extensions? Not writing the code (that is covered), but actually compiling the extensions so that Make will be able to load them. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote: Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). I've implemented this feature and it seems to work as expected. I also implemented the fix to the duplicate output being shown in some cases. I have two open issues I want to address before calling this feature done: first, fix make's writing of the command it's going to run (for rules that don't start with @) as that's not working right. Second, fix the enter/leave issue. It turns out that these are currently somewhat bound together so I may have to solve the second one first. Oh, and a renaming as well :-) Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There will be more disruption I think. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
Hi Paul. On 05/05/2013 12:10 AM, Paul Smith wrote: On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote: Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). I've implemented this feature and it seems to work as expected. I also implemented the fix to the duplicate output being shown in some cases. I have two open issues I want to address before calling this feature done: first, fix make's writing of the command it's going to run (for rules that don't start with @) as that's not working right. Second, fix the enter/leave issue. It turns out that these are currently somewhat bound together so I may have to solve the second one first. The test 'features/output-sync' now fails for me: Test timed out after 6 seconds Error running /storage/home/stefano/src/gnu/make/tests/../make \ (expected 0; got 14): /storage/home/stefano/src/gnu/make/tests/../make \ -f work/features/output-sync.mk.1 -j --output-sync=target Caught signal 14! FAILED (4/6 passed) Can you reproduce the failure? If not, let me know which information you need to debug this, and I'll try to provide them. Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[PATCH] docs: port manual to Texinfo 5.x
* doc/make.texi: Here. It was sufficient to change an '@itemx'
into an '@item'.
Copyright-paperwork-exempt: yes
Signed-off-by: Stefano Lattarini stefano.lattar...@gmail.com
---
doc/make.texi | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/make.texi b/doc/make.texi
index ea58d6e..fa9b5d7 100644
--- a/doc/make.texi
+++ b/doc/make.texi
@@ -8812,7 +8812,7 @@ from complicated nests of recursive @code{make} commands.
rarely need to specify this option since @samp{make} does it for you;
see @ref{-w Option, ,The @samp{--print-directory} Option}.)
-@itemx --no-print-directory
+@item --no-print-directory
@cindex @code{--no-print-directory}
Disable printing of the working directory under @code{-w}.
This option is useful when @code{-w} is turned on automatically,
--
1.8.3.rc0.19.g7e6a0cc
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Paul Smith wrote:
The first one I've seen but hadn't had time to debug. I'll look at your
patch. I left the truncate where it was rather than doing it after the
sync_output() because I was hoping to avoid truncating a file that we'll
never use again anyway, but I guess it isn't a big deal.
One could probably optimize it to avoid truncating if it's not going
to be reused, but I'm not sure it's worth the effort (including
future maintenance).
COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+'
prerequisite is to tell make that this line, even though it may not look
like it, will run a recursive make.
OK, let me just say that the meaning of recursive may not be
perfectly clear. Though the manual says: The @samp{+} prefix marks
those recipe lines as ``recursive'' so that they will be executed
despite use of the @samp{-t} flag., the example immediately
preceding this sentence has:
+touch archive.a
+ranlib -t archive.a
which are clearly not recursive make invocations.
I gather that make uses recursive in a wider sense as anything to
be run regardless (because it probably arrages by itself not to do
anything serious in a dry run or so), while the current
implementation of output-sync uses it in the more specific meaning
of a recursive invocation of GNU make (which will do its own
syncing).
Similar concerns seem to apply to other places where
COMMANDS_RECURSE is checked, e.g. (right after the output-sync code
in question):
/* If we aren't running a recursive command and we have a jobserver
pipe, close it before exec'ing. */
if (!(flags COMMANDS_RECURSE) job_fds[0] = 0)
{
close (job_fds[0]);
close (job_fds[1]);
}
I don't think those touch and ranlib commands above need the
jobserver pipe (though, of course, this one doesn't hurt much, just
costs 2 available fds).
Since make doesn't parse the command line it can't know for sure which
ones actually recurse. It uses a heuristic, by looking for $(MAKE) or
${MAKE} in the unexpanded line. But this is easily defeated if your
sub-make invocation doesn't use that variable for some reason. Hence,
using + to force it.
If this is the actual intention, and the example is really a misuse
of the feature (maybe not ideal to put this in the manual then?),
this means indeed that + lines are exempt from output
synchronization and I see no easy way around it.
A (hypothetical?) use case would be another make-like tool which is
invoked from a Makefile. Unless it implements its own output-sync,
compatible with ours, its output remains unsynced.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
