Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
Date: Sun, 05 May 2013 20:17:50 +0300 From: Eli Zaretskii e...@gnu.org Cc: bug-make@gnu.org From: Paul Smith psm...@gnu.org Cc: bug-make@gnu.org Date: Sun, 05 May 2013 12:56:48 -0400 On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. That's true, but it's common to have the same header be included when building the application or library which provides the exported services. However, if you really want it back the way it was please do choose a more unique name than MAIN. Something prefixed with GMK_ at least. OK, will do. Done. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
From: Paul Smith p...@mad-scientist.net Cc: bug-make@gnu.org Date: Sat, 04 May 2013 17:51:05 -0400 Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There's nothing to fix, as things still seem to work. Thanks! However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. The way they were together before is how programmers are used to see this stuff on Windows; splitting them will not something people will expect. I'm sure you can see this in quite a few packages out there. Here's a random example from GMP's gmp.h, slightly edited for clarity: #define __GMP_DECLSPEC_EXPORT __declspec(__dllexport__) #define __GMP_DECLSPEC_IMPORT __declspec(__dllimport__) #if __GMP_LIBGMP_DLL #if __GMP_WITHIN_GMP /* compiling to go into a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_EXPORT #else /* compiling to go into an application which will link to a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_IMPORT #endif #else /* all other cases */ #define __GMP_DECLSPEC #endif If you didn't like the symbol MAIN, then we can use any other symbol, like BUILDING_GMAKE or whatever. But having this split in two is not something I'd recommend. I think it's bad for maintenance, if nothing else. There will be more disruption I think. Looking forward to it ;-) One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. That wouldn't be right. The issue here is that Make has no idea what mode will be used by its children. We redirect the children's standard streams to a file, but we cannot force them to use this or that mode (nor should we, because that is up to the child program). Since we have no idea, and we must copy to our stdout/stderr everything the child wrote, we must assume the worst. And the worst is that the child did use binary mode, and as result wrote there some bytes that cannot be read in text mode without corrupting child's output. Examples include the ^Z byte, which stops text-mode reads and writes, and lone CR characters that get stripped by text-mode reads to disappear without a trace. So we use binary mode to read from the temporary file, because that's the only way to guarantee that everything the child wrote will be read verbatim. We then need to write that stuff to our output in the same mode, to produce identical output, as if the intermediate temporary file never happened. Does this make sense? If you want a practical example to illustrate this conundrum, just ask. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. I wanted to separate that in-make variant out so that users would never see it or have the possibility of accidentally using it, so I moved it into our internal headers which are never installed anywhere outside our source tree and would not be included in a GNU make binary package. This is slightly more potential maintenance on our part internally but is much safer for the user which is a tradeoff I'll almost always choose. However, if you really want it back the way it was please do choose a more unique name than MAIN. Something prefixed with GMK_ at least. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. Well, I assumed that something that invoked make could set the mode and then make could inherit that mode. I don't know if that's how it works or not in Windows. And of course I doubt anyone does that. I understand your point. I just wonder if this difference might end up being visible to the user in some way. But, we'll leave it as-is. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
From: Paul Smith psm...@gnu.org Cc: bug-make@gnu.org Date: Sun, 05 May 2013 12:56:48 -0400 On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. That's true, but it's common to have the same header be included when building the application or library which provides the exported services. However, if you really want it back the way it was please do choose a more unique name than MAIN. Something prefixed with GMK_ at least. OK, will do. Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. Well, I assumed that something that invoked make could set the mode and then make could inherit that mode. I don't know if that's how it works or not in Windows. And of course I doubt anyone does that. The text/binary mode cannot be inherited the way file descriptors are. That mode is entirely private to each application, and the startup code provided by MS (to which MinGW is compatible) unconditionally initializes it to text mode. Each application decides on its own how it should treat its standard streams; most leave them at the default text mode, but some switch to binary. Examples of the latter are ports of 'cat' (cat foo bar should create a file that is identical to 'foo'), carefully done ports of 'tr' and Sed (otherwise you cannot edit in or out CR characters), etc. I understand your point. I just wonder if this difference might end up being visible to the user in some way. If the users will see any difference, it means (barring bugs in Make) the child program used binary I/O where it would use text I/O had its stdout/stderr not been redirected. But this belongs to the broader class of problems with -O, not unlike the colored screen output that was discussed here a few days ago. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Fri, 03 May 2013 17:17:44 -0400 -O in no way changes that behavior, all it does is ensure that output from any individual line or target of the recipe will not interfere with any other individual line or target. At the Make's internals level, -O indeed doesn't change that behavior. But, whether we want it or not, Make users perceive the order in which they see the output as the order in which things got executed. So any significant changes in the order we show the results of remaking a target will, or at least might, be interpreted as changes in behavior. While it is OK for a non-default switch to change behavior, the modified behavior must make sense, or users will complain. For the default behavior, it must make a lot of sense. Without -O, output might be messy at times, but there's never any doubt in user's mind what happened: she understands that the jobs whose output appears on the screen at the same time run in parallel. Add -O (in its current implementation), and the user now stares at a puzzle: output appears serialized, but the order of that serialization is hard to make sense of. I'm not even sure I understand completely what confuses me when I look at such output. Perhaps my mind tries to interpret that as a non-parallel run, where a prerequisite target is always remade before the target that's dependent on that prerequisite. Or maybe some other mental mechanism is at work here. All I can say is that the order in which -O shows output confuses the heck out of me, and I need to look at it for a long time, reading and re-reading its portions, before I can convince myself that everything that needed to be remade was indeed remade, and in correct order. I guess that means I will seldom if ever use -O. And if that doesn't bother anyone, then let's just leave it at that. Either I'm an odd one out, or you will hear from others who might explain this better than I could. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote: Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). I've implemented this feature and it seems to work as expected. I also implemented the fix to the duplicate output being shown in some cases. I have two open issues I want to address before calling this feature done: first, fix make's writing of the command it's going to run (for rules that don't start with @) as that's not working right. Second, fix the enter/leave issue. It turns out that these are currently somewhat bound together so I may have to solve the second one first. Oh, and a renaming as well :-) Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There will be more disruption I think. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Thu, 02 May 2013 16:21:36 -0400 The one and only difference between them is that when running a recursive make, -Otarget WILL NOT capture the output of the sub-make, leaving whatever it prints going to the same stdout as the parent make, and -Omake WILL capture the output of the sub-make in the temporary file, to be dumped after the recipe is complete. Thanks for explaining that. I will have to try a few things to make sure I really get it this time, but one thing I can already say is that 'target' and 'make' are not very good names for these modes, since their semantics is quite different from the literal meaning of these two words. That difference creates a semantic dissonance that we should try to avoid, I think. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On Fri, 2013-05-03 at 09:50 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Thu, 02 May 2013 16:21:36 -0400 The one and only difference between them is that when running a recursive make, -Otarget WILL NOT capture the output of the sub-make, leaving whatever it prints going to the same stdout as the parent make, and -Omake WILL capture the output of the sub-make in the temporary file, to be dumped after the recipe is complete. Thanks for explaining that. I will have to try a few things to make sure I really get it this time, but one thing I can already say is that 'target' and 'make' are not very good names for these modes, since their semantics is quite different from the literal meaning of these two words. That difference creates a semantic dissonance that we should try to avoid, I think. It's good that we're having this discussion: I want to use it to try to inform my editing of the GNU make manual to be sure it's as clear as possible. I don't think the names are so inaccurate. I don't want to name things based solely on the details of how they differ from one another, which is all I was describing above. I prefer to name them based on how their most salient behavior manifests to the user. The way the user experiences the -Ojob option's results is that the output of every line of each recipe is dumped as soon as that line is complete. The way the user experiences the -Otarget option's results is that the output of all the lines of a given target are dumped at the same time once the target is completely built. The way the user experiences the -Omake option's results is that the output of an entire recursive make invocation is dumped, together, once the recursive make has completed. The issue of how -Otarget handles recursive make is, IMO, a detail necessitated by the architecture of recursive make invocations. I don't know that it's feasible to reflect that detail in the name. On the other hand I'm certainly not married to the current terms and I'm quite happy to change them if better ones can be found. It has already been suggested that -Oline would be better than -Ojob, and -Orecipe would be better than -Otarget, and -Omakefile would be better than -Omake. The current names are based more around _actions_ while the new suggestions are based more around semantic elements of make. To me -Omake is the most problematic. -Omakefile is not much better; in fact it might be worse (after all you can and often do invoke a recursive make on the same makefile). It would be nice to be more clear about the fact it applies only to recursive make invocations. Something like -Osubmake might be more accurate, except that I don't think we use the term sub-make in the documentation: we use recursive make. Is -Orecursive better? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Fri, 03 May 2013 07:47:09 -0400 The way the user experiences the -Ojob option's results is that the output of every line of each recipe is dumped as soon as that line is complete. I would suggest -Oline or -Ocommand for this. Job is not necessarily recognizable by users of Make for what we mean when we talk about that. The issue of how -Otarget handles recursive make is, IMO, a detail necessitated by the architecture of recursive make invocations. I don't know that it's feasible to reflect that detail in the name. It is a detail that IMO significantly qualifies the target part. In particular, targets that include little or nothing except a recursive invocations will be entirely exempt from this target scope. To me -Omake is the most problematic. -Omakefile is not much better; in fact it might be worse (after all you can and often do invoke a recursive make on the same makefile). It would be nice to be more clear about the fact it applies only to recursive make invocations. Something like -Osubmake might be more accurate, except that I don't think we use the term sub-make in the documentation: we use recursive make. Is -Orecursive better? Yes, I think -Orecursive is better. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Fri, 03 May 2013 09:08:27 -0400 You're concentrating on the one recursive make target and saying this doesn't follow the rule, while I'm concentrating on all targets in the sub-make and saying let's make sure all of these follow the rule (that their output is shown as soon as that target is complete). Recursive make targets are merely artifacts of the build. Users don't care about them; they're just used by makefile authors to organize things. If the makefile author rewrote the makefiles to be non-recursive, users wouldn't notice at all. Sorry, I don't understand this way of reasoning at all. Maybe we are talking about different things. Let me try again. Consider a target like this: foo: prepare something $(MAKE) -C subdir doit touch foo-done Is this a legitimate and good use of recursive Make? Should -O cater to this use case? Or is this use case rare (or even bad practice), so much so that the _default_ mode of -O is allowed to confuse the heck out of the user, who will see the output of the subdir job _before_ she sees the output of preparations for that subdir job? If the reason for exempting recursive Make invocations from -Otarget is that you want it to cover a different use case, like this: foo: ... $(MAKE) -C subdir doit then how about if this exemption will only be applied if the recipe has a single command? If the single-command-in-recursive-invocation is _not_ the use case which -Otarget is optimized for, then what use case is? IOW, what is the ideal Makefile where -Otarget doesn't have any downsides whatsoever? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
I have a solution for this problem that I think will work well, and will be simple to implement. Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). This will ensure that output from lines before the recursive make will be shown before the targets in the recursive make. It's not 100% identical but I can't see any way to do better. Thoughts? On Fri, 2013-05-03 at 16:39 +0300, Eli Zaretskii wrote: then how about if this exemption will only be applied if the recipe has a single command? In this case, if a recipe consisted of only one line then every target in the submake will be output immediately when it's finished, but as soon as you add another line to the recipe, like @echo Build is done!, now all of a sudden you get no output from the entire sub-make until the end. That would be too confusing. If the single-command-in-recursive-invocation is _not_ the use case which -Otarget is optimized for, then what use case is? -Otarget is not really about recursive invocations at all. It's there for the non-recursive targets. It would be nice if it worked well for the recursive targets, too, obviously. Consider every target in the entirety of build, including all submakes and all their targets as well, as one long list of targets (for example the results of a serial build). The fraction of those targets that are invoking sub-makes will be, at most, very small. -Otarget wants to collect the output of each individual target, then show the output for each target as soon as that target finishes. That's what users (should) expect when using this option. In the case of recursive make targets, this presents an unsolveable contradiction: how can you both show the output of EVERY target (including the targets run by the submake) as soon as it completes, and still not show the output of any target (including the targets that invoke a submake) before it's complete? You can't do both! The -Omake option chooses the latter as more important: it will delay the output until the submake is complete. The -Otarget option chooses the former as more important: it wants to behave properly for the large majority of targets which are not invoking a recursive make. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
One optimisation I have thought of in the past for this situation would be to allow a single job to hold onto the lock when it obtained it. This way it could output directly to the console while all other jobs would have to buffer. When it released, the next job lucky enough to grab the lock might have a full buffer already. It might appear to be less choppy. Not sure how it would perform. Regards, Tim On 2 May 2013 03:53, Eli Zaretskii e...@gnu.org wrote: From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Wed, 01 May 2013 16:27:58 -0400 If your recipe normally runs for 5 seconds (say) and it continually generates output during that time, then yes, certainly the -O feature will result in choppiness because instead of a sequence of continuous output over 5 seconds you get 5 seconds of silence, followed by all the output. I think that's the reason, yes. Consider if you have a makefile, like every single automake makefile for example!, where the top-level target is nothing more than a recursive invocation of a sub-make with some extra arguments, and the sub-make actually does all the work: all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive Now, if you do nothing special for recursive make, you'll get no output from the entire build until it is completely done, because all the output from the recursive make command is going to the temporary file for that target, then it all gets dumped at the same time. Not every Makefile looks like that on its top level. I agree that we should cater to the above, but perhaps we could do that without punishing the other use cases. For example, perhaps we should have a -Osub-make option that will _not_ activate sync-output on the top level, only in the sub-make's. This should produce the desired results, I think. shouldn't we have an option _not_ to make such an exception, but without -Omake, i.e. without waiting for the whole session to end? Whenever any top-level recipe finishes, it is flushed to the screen as a single unit. Does this make sense? I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
Eli Zaretskii e...@gnu.org writes: If you want different behavior you can change your rule to use + on the two echo lines, so that they're also considered recursive and not saved up. If I do that, the echo from rec1 and rec2 mix up: D:\gnu\make-3.82.90_GIT_2013-05-01gnumake -f mkfsync3 -j -O start rec2start rec1 gnumake -f mkfsync simple gnumake -f mkfsync simple Is this also expected? You're right, I'm wrong. Using + is not really like -Ojob, because with -Ojob we still sync the output of each line. Using + the job just sends the output directly to the terminal with no sync. Thus this mixing up is what I'd expect (same as not using -O at all). I begin to wonder if this really requires a new per-line special prefix character like @, +, -, that controls the syncing of that particular line. I'm very reluctant to go there as it is a BIG change and a backward-compat issue. Also it seems that unlike the existing prefix characters, for this one we'd have as much need for a way to turn OFF sync as to turn it ON... bleah. On Thu, 2013-05-02 at 05:53 +0300, Eli Zaretskii wrote: Now, if you do nothing special for recursive make, you'll get no output from the entire build until it is completely done, because all the output from the recursive make command is going to the temporary file for that target, then it all gets dumped at the same time. Not every Makefile looks like that on its top level. I agree that we should cater to the above, but perhaps we could do that without punishing the other use cases. For example, perhaps we should have a -Osub-make option that will _not_ activate sync-output on the top level, only in the sub-make's. This should produce the desired results, I think. Can you clarify what the desired results are? I seem to have lost the thread. What is the behavior you see now that you are trying to avoid (or you don't see that you're trying to achieve)? Capture of the sub-make will mean that the entire output of that sub-make, and all of its recipes including ITS sub-sub-makes, will be sent to a temporary file and not displayed on the screen until the entire sub-make is completed. In what situation would we want to choose this, regardless of level of sub-make? In general I see no benefit in trying to special-case any particular level of make. For some builds the top level does all the work. For some the second level. For some the third. For many, different levels for different parts of the same build. I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. I think you and I said the same thing: the output from recursive makes is saved up and flushed all at once...? Tim Murphy tnmur...@gmail.com writes: One optimisation I have thought of in the past for this situation would be to allow a single job to hold onto the lock when it obtained it. This way it could output directly to the console while all other jobs would have to buffer. When it released, the next job lucky enough to grab the lock might have a full buffer already. It might appear to be less choppy. Not sure how it would perform. It might be less choppy (or it might not: it depends on your targets: are they all more-or-less equally chatty?) but we discussed this possibility and decided that it would be too costly in terms of performance. All you need is for one long-running job to get the terminal and pretty soon it's the only job running as all others have finished but can't continue with the next one until they can grab the terminal and dump their output. Personally I think this is a serious enough problem that it's probably not even worth doing the work in GNU make to provide this as an option. I suspect people would probably much rather just not use -O and live with interleaved output, or use -O and live with some choppy output, than suffer essentially random increases in build times. I could be wrong though. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Thu, 02 May 2013 08:21:23 -0400 Now, if you do nothing special for recursive make, you'll get no output from the entire build until it is completely done, because all the output from the recursive make command is going to the temporary file for that target, then it all gets dumped at the same time. Not every Makefile looks like that on its top level. I agree that we should cater to the above, but perhaps we could do that without punishing the other use cases. For example, perhaps we should have a -Osub-make option that will _not_ activate sync-output on the top level, only in the sub-make's. This should produce the desired results, I think. Can you clarify what the desired results are? I seem to have lost the thread. What is the behavior you see now that you are trying to avoid (or you don't see that you're trying to achieve)? The desired results in my original use case are that the output of remaking each target is shown as one chunk in the order in which it is expected, i.e. from the first command to the last. Remaking a target, a.k.a. recipe in this context are all the commands that rebuild a single target. E.g., in this snippet: foo: bar cmd1 cmd2 $(MAKE) -C foo something-else cmd3 all the 4 commands and whatever they emit should be output in one go, and in the order they are in the Makefile. The desired results in the example you showed, i.e. all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive are that -O is in effect only for the sub-make which runs all-recursive. Capture of the sub-make will mean that the entire output of that sub-make, and all of its recipes including ITS sub-sub-makes, will be sent to a temporary file and not displayed on the screen until the entire sub-make is completed. In what situation would we want to choose this, regardless of level of sub-make? Not regardless of level, of course. If there are many levels of recursion, then doing that is not what most people will want. But if make all-recursive does not recurse anymore, then yes, what I described will be good. In general I see no benefit in trying to special-case any particular level of make. For some builds the top level does all the work. For some the second level. For some the third. For many, different levels for different parts of the same build. The user always knows what she is going to run, or at least ought to know. I think we already established that blindly appending -O to the Make command might cause surprising and even disappointing or annoying results, if one does it for a use case that does not play well with this feature. So some degree of adaptation between -O and its sub-modes to the depth and breadth (in parallelism sense) of the build is necessary anyway. I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. I think you and I said the same thing: the output from recursive makes is saved up and flushed all at once...? No, that's not what I said. I said whenever a _recipe_ finishes, not whenever the entire Make run finishes. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On Thu, 2013-05-02 at 20:24 +0300, Eli Zaretskii wrote: The desired results in my original use case are that the output of remaking each target is shown as one chunk in the order in which it is expected, i.e. from the first command to the last. Remaking a target, a.k.a. recipe in this context are all the commands that rebuild a single target. E.g., in this snippet: foo: bar cmd1 cmd2 $(MAKE) -C foo something-else cmd3 all the 4 commands and whatever they emit should be output in one go, and in the order they are in the Makefile. This is the way -Omake works. If you want that, you should set -Omake. The desired results in the example you showed, i.e. all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive are that -O is in effect only for the sub-make which runs all-recursive. I'm not sure what you mean by ONLY for the sub-make but if I understand correctly this is the way -Otarget works. So, you can have either method. Maybe what you're suggesting is that you want BOTH methods, but for different targets, in the same build? Today we don't support that. This is what I was talking about when I referred to a new prefix character on a per line basis to turn on/off ordered mode. I think trying to control this through the command line, by choosing different MAKELEVEL values to behave differently, is not a good solution. However I don't want to go there yet. It's a big, disruptive change and I'm not convinced that (a) we can't do something to obviate it and (b) even if we can't that the use-case is important enough to justify it. The build still behaves just as it did before, it's just that in your original case the output is shown in a strange order. In general I see no benefit in trying to special-case any particular level of make. For some builds the top level does all the work. For some the second level. For some the third. For many, different levels for different parts of the same build. The user always knows what she is going to run, or at least ought to know. I think we already established that blindly appending -O to the Make command might cause surprising and even disappointing or annoying results, if one does it for a use case that does not play well with this feature. So some degree of adaptation between -O and its sub-modes to the depth and breadth (in parallelism sense) of the build is necessary anyway. I don't think I agree with much of the above. But it's a matter of opinion so we'll just have to wait and see. I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. I think you and I said the same thing: the output from recursive makes is saved up and flushed all at once...? No, that's not what I said. I said whenever a _recipe_ finishes, not whenever the entire Make run finishes. -Omake only has any relevance when doing recursive makes. If you run make -Otarget and make -Omake in a non-recursive make environment, you will get identical behavior. The one and only difference between them is that when running a recursive make, -Otarget WILL NOT capture the output of the sub-make, leaving whatever it prints going to the same stdout as the parent make, and -Omake WILL capture the output of the sub-make in the temporary file, to be dumped after the recipe is complete. There's no -O mode which will save up the entire output of a non-recursive make and dump it all at the end. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On Tue, 2013-04-30 at 10:39 -0400, Paul Smith wrote: On Tue, 2013-04-30 at 16:04 +0200, Stefano Lattarini wrote: On 04/30/2013 03:37 PM, Paul Smith wrote: Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? No; the testsuite runs as a recursive make invocation (yes, this is sadly truly needed in order to support all the features offered by the Automake parallel testsuite harness --- sorry), but each test script (and there can be hundreds of them, as is the case for GNU coreutils or GNU automake itself) is run as a separate target, explicit for tests which have no extension and pattern-based for tests that have an extension. This should work very well with -Otarget then, except for the colorization/highlighting issue... once it works as expected. I'll look into this issue later and I would be interested to see your experience with it once it's resolved. OK, I found this bug. Definitely make recursion was not being handled properly with -Otarget and -Ojob in some situations; this broke as a side effect of my cleanup to reuse the same temporary file for the entire target, regardless of the output mode. This should be fixed now. Those who use recursive makefiles and were seeing annoying delays in output with -O, please try again with the latest commit and see if it works any better for you now. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
Hi Paul. On 05/01/2013 02:04 PM, Paul Smith wrote: On Tue, 2013-04-30 at 10:39 -0400, Paul Smith wrote: On Tue, 2013-04-30 at 16:04 +0200, Stefano Lattarini wrote: On 04/30/2013 03:37 PM, Paul Smith wrote: Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? No; the testsuite runs as a recursive make invocation (yes, this is sadly truly needed in order to support all the features offered by the Automake parallel testsuite harness --- sorry), but each test script (and there can be hundreds of them, as is the case for GNU coreutils or GNU automake itself) is run as a separate target, explicit for tests which have no extension and pattern-based for tests that have an extension. This should work very well with -Otarget then, except for the colorization/highlighting issue... once it works as expected. I'll look into this issue later and I would be interested to see your experience with it once it's resolved. OK, I found this bug. Definitely make recursion was not being handled properly with -Otarget and -Ojob in some situations; this broke as a side effect of my cleanup to reuse the same temporary file for the entire target, regardless of the output mode. This should be fixed now. Those who use recursive makefiles and were seeing annoying delays in output with -O, please try again with the latest commit and see if it works any better for you now. I can confirm the problem has disappeared for me. Thanks! (The issue with extra Entering/Leaving directory messages is still present though; I hope it can be fixed before the release ...). Best regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Date: Wed, 01 May 2013 08:04:08 -0400 Cc: bug-make@gnu.org This should work very well with -Otarget then, except for the colorization/highlighting issue... once it works as expected. I'll look into this issue later and I would be interested to see your experience with it once it's resolved. OK, I found this bug. Definitely make recursion was not being handled properly with -Otarget and -Ojob in some situations; this broke as a side effect of my cleanup to reuse the same temporary file for the entire target, regardless of the output mode. You forgot to make the same change in the WINDOWS32 branch. I did that in commit a87ff20. This should be fixed now. Those who use recursive makefiles and were seeing annoying delays in output with -O, please try again with the latest commit and see if it works any better for you now. Unfortunately, the delays are still here. Moreover, it looks like this change introduced some kind of regression. With the following Makefile: all: simple recursive simple: one two one two: @echo start $@ @sleep 1 @echo stop $@ @-false recursive: rec1 rec2 rec1 rec2: @echo start $@ $(MAKE) -f mkfsync simple @echo stop $@ I see this output with the previous version: D:\gnu\make-3.82.90_GIT_2013-04-20gnumake -f mkfsync -j -O gnumake -f mkfsync simple gnumake -f mkfsync simple mkfsync:6: recipe for target 'one' failed gnumake: [one] Error 1 (ignored) start one stop one mkfsync:6: recipe for target 'two' failed gnumake: [two] Error 1 (ignored) start two stop two start rec2 gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'one' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [one] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start one stop one gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'two' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [two] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start two stop two gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' stop rec2 start rec1 gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'one' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [one] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start one stop one gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'two' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [two] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start two stop two gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' stop rec1 Notice in particular how start rec1..stop rec1 occludes its sub-targets, and the same for rec2. After the change I see this: D:\gnu\make-3.82.90_GIT_2013-05-01gnumake -f mkfsync -j -O gnumake -f mkfsync simple gnumake -f mkfsync simple gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start one stop one mkfsync:6: recipe for target 'one' failed gnumake: [one] Error 1 (ignored) start two stop two mkfsync:6: recipe for target 'two' failed gnumake: [two] Error 1 (ignored) gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start one stop one mkfsync:6: recipe for target 'one' failed gnumake[1]: [one] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start two stop two mkfsync:6:
Re: Some serious issues with the new -O option
On 05/01/2013 05:26 PM, Eli Zaretskii wrote: Unfortunately, the delays are still here. I can say they're no longer there *in my use case*; I haven't tested other use cases though. Hope this is sorted out soon... Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On Wed, 2013-05-01 at 18:26 +0300, Eli Zaretskii wrote: You forgot to make the same change in the WINDOWS32 branch. I did that in commit a87ff20. Sorry, I missed that. This should be fixed now. Those who use recursive makefiles and were seeing annoying delays in output with -O, please try again with the latest commit and see if it works any better for you now. Unfortunately, the delays are still here. Very odd. This is the test program I used; can you verify: recurse: ; $(MAKE) -f $(firstword $(MAKEFILE_LIST)) all all: 2 4 6 .RECIPEPREFIX := | 2 4 6 : |@ echo start $@ |@ sleep $@ |@ echo end $@ Now running: $ make -Omake --no-print-directory -j -f ... should wait for 6 seconds, then print everything at the end. Running this on the other hand: $ make -O --no-print-directory -j -f ... should show start 2/end 2 after 2 seconds, start 4/end 4 after 2 more seconds (4 total), etc. And, just for completeness, running this: $ make -Ojob --no-print-directory -j -f ... should show all the start 2, start 4, start 6 right away (possibly out of order), then after 2 seconds end 2, then 2 seconds later end 4, etc. Is that what you see? If so then the feature is working as expected. I'd be interested to know more details of the makefiles where you're still seeing stuttering behavior. Are the targets in this environment printing a lot of output per recipe? Moreover, it looks like this change introduced some kind of regression. With the following Makefile: I don't think this is a regression. It's unfortunate but I don't see any alternative. BTW, you might find it simpler to use --no-print-directory as above to get rid of the enter/leave stuff until I work out how to do it better. Notice in particular how start rec1..stop rec1 occludes its sub-targets, and the same for rec2. Occludes? I don't sense anything occluded here. Maybe a terminology error... did you mean something like brackets? Yes, that's the behavior we'd expect to see if make was not treating the sub-make properly: it saves up the sub-make output and prints it once the sub-make is completed. The goal of this fix is to change that... After the change I see this: gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start rec1 stop rec1 start rec2 stop rec2 And now rec1 and rec2 are announced only at the end. This isn't a bug, this is expected behavior. It's slightly unfortunate, I agree. Consider your rule: rec1 rec2: @echo start $@ $(MAKE) -f mkfsync simple @echo stop $@ Here we have a 3-line recipe: the first and third are not considered by make to be recursive, while the second one is (due to the presence of $(MAKE)) When we run with -O (-Otarget), make will save up the output from the entire recipe and print it at the end, EXCEPT that output from a recursive line is not saved: that's the entire point of this feature, to allow sub-makes to show their output as they go. So, the results of lines one and three (the echo's) are saved until the rec1 target is completely built, and printed at the end, while the results of the make invocation in between are not saved, which is just what you see. If you want different behavior you can change your rule to use + on the two echo lines, so that they're also considered recursive and not saved up. Now this recipe is basically run in -Ojob mode, BTW. + has other side-effects though (running even in -n mode) which might be unpleasant. The only alternative to this would be for make to try to figure out if ANY of the lines in the recipe will be recursive, and if so treat ALL the lines as if -Ojob was enabled (show output as soon as the line is complete), while not treating them as recursive. However since today we don't parse lines completely until we're about to execute them, this would be a not-insignificant change I think. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Wed, 01 May 2013 14:41:25 -0400 Unfortunately, the delays are still here. Very odd. This is the test program I used; can you verify: recurse: ; $(MAKE) -f $(firstword $(MAKEFILE_LIST)) all all: 2 4 6 .RECIPEPREFIX := | 2 4 6 : |@ echo start $@ |@ sleep $@ |@ echo end $@ Now running: $ make -Omake --no-print-directory -j -f ... should wait for 6 seconds, then print everything at the end. Running this on the other hand: $ make -O --no-print-directory -j -f ... should show start 2/end 2 after 2 seconds, start 4/end 4 after 2 more seconds (4 total), etc. And, just for completeness, running this: $ make -Ojob --no-print-directory -j -f ... should show all the start 2, start 4, start 6 right away (possibly out of order), then after 2 seconds end 2, then 2 seconds later end 4, etc. Is that what you see? Yes. But I thought the change was about -Otarget, not -Ojob. Stefano was complaining about a plain -O, so -Ojob is not what was his problem. I don't think this is a regression. It's unfortunate but I don't see any alternative. Too bad. rec1 rec2: @echo start $@ $(MAKE) -f mkfsync simple @echo stop $@ Here we have a 3-line recipe: the first and third are not considered by make to be recursive, while the second one is (due to the presence of $(MAKE)) When we run with -O (-Otarget), make will save up the output from the entire recipe and print it at the end, EXCEPT that output from a recursive line is not saved: that's the entire point of this feature, to allow sub-makes to show their output as they go. So, the results of lines one and three (the echo's) are saved until the rec1 target is completely built, and printed at the end, while the results of the make invocation in between are not saved, which is just what you see. If you want different behavior you can change your rule to use + on the two echo lines, so that they're also considered recursive and not saved up. Now this recipe is basically run in -Ojob mode, BTW. + has other side-effects though (running even in -n mode) which might be unpleasant. That is completely unexpected for me: I thought that -Otarget meant that all the output from making a target, e.g. rec1, will be output as a single unit. If that's not the intent, then why is it called target? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Wed, 01 May 2013 14:41:25 -0400 If you want different behavior you can change your rule to use + on the two echo lines, so that they're also considered recursive and not saved up. If I do that, the echo from rec1 and rec2 mix up: D:\gnu\make-3.82.90_GIT_2013-05-01gnumake -f mkfsync3 -j -O start rec2start rec1 gnumake -f mkfsync simple gnumake -f mkfsync simple Is this also expected? Anyway, my mental model of what's going on under -O is now completely shattered. I cannot understand what good is this behavior: entire recipe and print it at the end, EXCEPT that output from a recursive line is not saved: that's the entire point of this feature, to allow sub-makes to show their output as they go. Why is it important to make that exception? And shouldn't we have an option _not_ to make such an exception, but without -Omake, i.e. without waiting for the whole session to end? Whenever any top-level recipe finishes, it is flushed to the screen as a single unit. Does this make sense? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On Wed, 2013-05-01 at 22:08 +0300, Eli Zaretskii wrote: Yes. But I thought the change was about -Otarget, not -Ojob. Stefano was complaining about a plain -O, so -Ojob is not what was his problem. Yes, it is about -Otarget. As I said, I added -Ojob output just for completeness. The important distinctions (for this thread) are between -Otarget and -Omake. If you see the behavior I describe for those two flags then things are working as expected. If you still see choppiness in the output with -Otarget and you see the behavior I describe, I'd be interested to know more about the targets you're building. I'd also be interested to know if using -Otarget vs. -Ojob makes any difference in the choppiness. If your recipe normally runs for 5 seconds (say) and it continually generates output during that time, then yes, certainly the -O feature will result in choppiness because instead of a sequence of continuous output over 5 seconds you get 5 seconds of silence, followed by all the output. That is the nature of the beast... if this bothers you (more than having interleaved parallel output) best to not use -O. I suggest that for MOST makefile targets, this is not the normal behavior. Most makefile targets (building a .o for example) are built relatively quickly and, more importantly, they don't expect to see much output except on failure. I would also point out that the more output your targets generate the more likely you are to get output corruption when running with high -j, and the more you'd be likely to benefit from -O. That is completely unexpected for me: I thought that -Otarget meant that all the output from making a target, e.g. rec1, will be output as a single unit. If that's not the intent, then why is it called target? That IS the intent. However, in the presence of makefile recursion this model fails. Consider if you have a makefile, like every single automake makefile for example!, where the top-level target is nothing more than a recursive invocation of a sub-make with some extra arguments, and the sub-make actually does all the work: all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive Now, if you do nothing special for recursive make, you'll get no output from the entire build until it is completely done, because all the output from the recursive make command is going to the temporary file for that target, then it all gets dumped at the same time. I think this makes the output sync feature much less useful: it's only appropriate for building in the background where no one looks at the output until it's done. If you want that behavior, though, that's exactly what the -Omake option selects so it's available. For -Otarget we introduce an exception to deal with this problem: if we detect we're about to invoke a recursive make, we do NOT redirect the output to the temp file. We let each target of that make print its output immediately. This way you don't have to wait for an entire recursive make to finish before you see any of its output. Why is it important to make that exception? And shouldn't we have an option _not_ to make such an exception, but without -Omake, i.e. without waiting for the whole session to end? Whenever any top-level recipe finishes, it is flushed to the screen as a single unit. Does this make sense? I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. If we can improve on this I'm very interested to hear the details. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Wed, 01 May 2013 16:27:58 -0400 If your recipe normally runs for 5 seconds (say) and it continually generates output during that time, then yes, certainly the -O feature will result in choppiness because instead of a sequence of continuous output over 5 seconds you get 5 seconds of silence, followed by all the output. I think that's the reason, yes. Consider if you have a makefile, like every single automake makefile for example!, where the top-level target is nothing more than a recursive invocation of a sub-make with some extra arguments, and the sub-make actually does all the work: all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive Now, if you do nothing special for recursive make, you'll get no output from the entire build until it is completely done, because all the output from the recursive make command is going to the temporary file for that target, then it all gets dumped at the same time. Not every Makefile looks like that on its top level. I agree that we should cater to the above, but perhaps we could do that without punishing the other use cases. For example, perhaps we should have a -Osub-make option that will _not_ activate sync-output on the top level, only in the sub-make's. This should produce the desired results, I think. shouldn't we have an option _not_ to make such an exception, but without -Omake, i.e. without waiting for the whole session to end? Whenever any top-level recipe finishes, it is flushed to the screen as a single unit. Does this make sense? I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
I'm guessing here but I imagine the main problem comes with delaying the results of submakes? I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. Regards, Tim On 30 April 2013 10:48, Stefano Lattarini stefano.lattar...@gmail.comwrote: I'm sorry to say that the new -O option can interact *badly* with Automake-generated parallel testsuites (at least when they are being run interactively, that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with make -j2 -O on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to always). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most interactive usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. Colouring is another issue which I would imagine could be done another way to let us have the best of both worlds. Regards, Tim On 30 April 2013 10:55, Tim Murphy tnmur...@gmail.com wrote: I'm guessing here but I imagine the main problem comes with delaying the results of submakes? I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. Regards, Tim On 30 April 2013 10:48, Stefano Lattarini stefano.lattar...@gmail.comwrote: I'm sorry to say that the new -O option can interact *badly* with Automake-generated parallel testsuites (at least when they are being run interactively, that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with make -j2 -O on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to always). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most interactive usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On 04/30/2013 11:55 AM, Tim Murphy wrote: I'm guessing here but I imagine the main problem comes with delaying the results of submakes? I think so, yes. The Distributed Make from Sun implements a similar output synchronization feature, and doesn't have the output delay problems I have experienced with GNU make (while it shares the issues with automatic output colorization). I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. Indeed. Regards, Tim I leave my original message quoted here in tis entirety, for reference: On 30 April 2013 10:48, Stefano Lattarini stefano.lattar...@gmail.comwrote: I'm sorry to say that the new -O option can interact *badly* with Automake-generated parallel testsuites (at least when they are being run interactively, that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with make -j2 -O on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to always). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most interactive usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On 04/30/2013 12:01 PM, Tim Murphy wrote: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I wasn't even aware of those differences; as of latest Git commit 'moved-to-git-46-g19a69ba', I don't see them documented in either the help screen, the manpage, the texinfo manual, nor the NEWS file. Colouring is another issue which I would imagine could be done another way to let us have the best of both worlds. That is not trivial to do I think. For example, Automake-generated testsuites check whether the stdout is a tty to decide whether or not to automatically enable output colorization. And testsuites produced by Autotest do the same, AFAIK. If the make connects the stdout of those processes to non-tty files behind the scene, those checks are doomed to fail. Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
I know this isn't going to go down all that well, but I really think the output should be annotated in such a way that colourisation could be applied to the log file after a build has already finished. e..g you load a makefile into VIM - it can colourise it. Or a bit of C source code. Why not the log of a build you did yesteday? It's still very nice to be able to distinguish things by colour later on. Regards, Tim On 30 April 2013 11:16, Stefano Lattarini stefano.lattar...@gmail.comwrote: On 04/30/2013 12:01 PM, Tim Murphy wrote: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I wasn't even aware of those differences; as of latest Git commit 'moved-to-git-46-g19a69ba', I don't see them documented in either the help screen, the manpage, the texinfo manual, nor the NEWS file. Colouring is another issue which I would imagine could be done another way to let us have the best of both worlds. That is not trivial to do I think. For example, Automake-generated testsuites check whether the stdout is a tty to decide whether or not to automatically enable output colorization. And testsuites produced by Autotest do the same, AFAIK. If the make connects the stdout of those processes to non-tty files behind the scene, those checks are doomed to fail. Regards, Stefano -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On 04/30/2013 01:22 PM, Tim Murphy wrote: I know this isn't going to go down all that well, but I really think the output should be annotated in such a way that colourisation could be applied to the log file after a build has already finished. While this might (underline might ;-) be an interesting feature, it is orthogonal to the issue under question -- that is, the need to retain the ability to automatically determine whether the output is going to a tty or a log file (that is, any file that is not a tty). e..g you load a makefile into VIM - it can colourise it. Or a bit of C source code. Why not the log of a build you did yesteday? It's still very nice to be able to distinguish things by colour later on. And notice that, currently, such a post-run colorization is easy to obtain, if you are only interested in having it working 99% of the time: just color a leading PASS: string in green, a leading FAIL: in red, e leading SKIP: in blue, etc. Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? I can see how that environment might be problematic for this new feature. It works much better for lots of smaller targets. However, you could avoid this issue if you specify that the target to be built is a sub-make, by prefixing it with a + operator for example. If this is true and you use -Otarget or -Ojob then make will not capture the output from that job. Of course this also has the unfortunate side-effect that running make -n will still try to run the test scripts. Hm. On Tue, 2013-04-30 at 11:48 +0200, Stefano Lattarini wrote: So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). This is a possibility for sure. I have to say that my experience with parallel builds hasn't been as wonderful as others here. I often get output which is corrupted, and not just by intermixing whole lines but also by having individual lines intermixed (that is the beginning of the line is one job's output, then the end of the line is another job's output, etc.) This kind of corruption is often completely unrecoverable and I simply re-run the build without parallelism enabled. I think it depends on how much output jobs emit and how many you are running at the same time. It could also be that Windows is better about avoiding interrupted writes to the same device. Tim Murphy tnmur...@gmail.com writes: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I always intended that and never suggested -Omake would be the default. I think -Omake is only really useful for completely automated, background builds. It wouldn't ever be something someone would use if they wanted to watch the build interactively. I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. This is the default. If you use -Otarget or (-Ojob) then what is supposed to happen is that make uses the same sub-make detection algorithms used for jobserver, etc. and if it determines that a job it's going to run is a submake it does NOT collect its output. However, I have suspicions that this is not working properly. I have a make-based cross-compilation environment (building gcc + tools) I've been using for years, and I tried it with the latest make and -O and saw similar problems (sub-make output collected into one large log instead of per-target). Thinking about it right now I think I might know what the problem is, actually. I'll look at this later. Stefano Lattarini stefano.lattar...@gmail.com writes: I wasn't even aware of those differences; as of latest Git commit 'moved-to-git-46-g19a69ba', I don't see them documented in either the help screen, the manpage, the texinfo manual, nor the NEWS file. I don't see where that code comes from? There is no Git commit in the standard GNU make archive with a SHA g19a69ba. The current HEAD on master is: 19a69ba Support dynamic object loading on MS-Windows. At any rate, the new option and option arguments are documented in the manual and there is an overview including the option arguments in the man page. The NEWS file doesn't discuss the individual option arguments, only the option. Ditto the --help output. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
Hi Paul. On 04/30/2013 03:37 PM, Paul Smith wrote: Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? No; the testsuite runs as a recursive make invocation (yes, this is sadly truly needed in order to support all the features offered by the Automake parallel testsuite harness --- sorry), but each test script (and there can be hundreds of them, as is the case for GNU coreutils or GNU automake itself) is run as a separate target, explicit for tests which have no extension and pattern-based for tests that have an extension. I can see how that environment might be problematic for this new feature. It works much better for lots of smaller targets. See above: we are speaking of hundreds of little targets here ... However, you could avoid this issue if you specify that the target to be built is a sub-make, The problem here is exactly the fact that the output of the sub-make is not displayed until its execution has ended. by prefixing it with a + operator for example. BTW, the + operator is not portable soem other make implementations still targeted by Automake (e.g., Solaris make). If this is true and you use -Otarget or -Ojob then make will not capture the output from that job. Of course this also has the unfortunate side-effect that running make -n will still try to run the test scripts. This would be horrific of course ;-) Hm. On Tue, 2013-04-30 at 11:48 +0200, Stefano Lattarini wrote: So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). This is a possibility for sure. I have to say that my experience with parallel builds hasn't been as wonderful as others here. OTOH, mine has been very good, and I almost alway use make -j8 or higher when building GNU software (or other good software the relies on GNU make, like Git). The only real issues I've experienced so far have been when I used the '-O' option ... I often get output which is corrupted, and not just by intermixing whole lines but also by having individual lines intermixed (that is the beginning of the line is one job's output, then the end of the line is another job's output, etc.) Most of these issues are solved using silent-rules, as offered by Automake or implemented ad-hoc by build systems of other software (Git, Linux). I think the best ways to use parallel GNU make are: - with silent rules when the output goes to a tty (it makes the output visually scannable, and makes warnings and error messages form compilers and most other tools stand out much more clearly). - with verbose rules and possibly the -O option enabled when the output goes to a log, to simplify post-build debugging and analysis (especially useful with CI systems and the like). This kind of corruption is often completely unrecoverable and I simply re-run the build without parallelism enabled. I think it depends on how much output jobs emit and how many you are running at the same time. Indeed; this corresponds to my experiences so far. It could also be that Windows is better about avoiding interrupted writes to the same device. Not using windows, I cannot say anything here, sorry. Tim Murphy tnmur...@gmail.com writes: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I always intended that and never suggested -Omake would be the default. I think -Omake is only really useful for completely automated, background builds. It wouldn't ever be something someone would use if they wanted to watch the build interactively. I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. This is the default. If you use -Otarget or (-Ojob) then what is supposed to happen is that make uses the same sub-make detection algorithms used for jobserver, etc. and if it determines that a job it's going to run is a submake it does NOT collect its output. This appears not to be the case, actually ... However, I have suspicions that this is not working properly. I have a make-based cross-compilation environment (building gcc + tools) I've been using for years, and I tried it with the latest make and -O and saw similar problems (sub-make output collected into one large log instead of per-target). Glad to know you can reproduce this. Thinking about it right now I think I might know what the problem is, actually. I'll look at this later. Thanks! Stefano Lattarini stefano.lattar...@gmail.com writes: I wasn't even aware of those
Re: Some serious issues with the new -O option
On Tue, 2013-04-30 at 16:04 +0200, Stefano Lattarini wrote: On 04/30/2013 03:37 PM, Paul Smith wrote: Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? No; the testsuite runs as a recursive make invocation (yes, this is sadly truly needed in order to support all the features offered by the Automake parallel testsuite harness --- sorry), but each test script (and there can be hundreds of them, as is the case for GNU coreutils or GNU automake itself) is run as a separate target, explicit for tests which have no extension and pattern-based for tests that have an extension. This should work very well with -Otarget then, except for the colorization/highlighting issue... once it works as expected. I'll look into this issue later and I would be interested to see your experience with it once it's resolved. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
