[bug #22923] option to prevent "interspersed" output in parallel builds

[Brian Kustel]

Follow-up Comment #3, bug #22923 (project make):

Thanks ipjrest and psmith for both replies. They are very appreciated. 

Responding to ipjrest's comments:
=
I agree some commands are long-running and that if "watching" progress is
important, the proposed behavior may not be desireable. This is one reason I
suggested the proposed new behavior be optional.

1) Line-buffering: While it would help in some cases, not intermixing lines
from different processes is only partially helpful. The fact that a "single
line" from process A might be interrupted by a line from process B is my
concern here. For example, instead of seeing
  Process A Fragment 1 was emitted and then fragment 2 was emitted.
  Process B Message
one might see 
  A: Process A Fragment 1 was emitted 
  B: Process B Message
  A: and then fragment 2 was emitted.
if the line for process A is emitted in 2 separate write operations. 
2) The VS line prefixing would be OK. In fact I like this idea. But it still
has the problem that single lines can be broken if the tool being run
sometimes emits lines "in pieces". I have noticed that gcc has this problem in
some cases (at least the gcc version I am currently using, 4.1.0).

Responding to psmith's comments:

Thanks for pointing out that make is not multithreaded. I guess I was
assuming it was. I agree this complicates matters somewhat. Also, I see your
point about my original stdout/stderr "marking" idea. In both cases I assumed
make would have either an auxilliary thread or its main thread doing these
tasks but I now realize that this is not how make currently works. 

But I think something along the lines of your "special variable" idea could
be employed in a generic fashion. Perhaps this is what you intended your
second paragraph below to imply? Specifically, the proposed option could cause
all "commands" to have their output piped to a standard tool which stores
(its) stdin to a temp file and then writes it synchronously to stdout after
stdin is closed. Also, I think it could be reasonable to allow for an option
to either combine stdout/stderr here or to pipe them to separate processes (so
that stdout/stderr could be atomically emitted to stdout/stderr
respectively).

While I have nothing against the "special variable" idea, I'd very much like
to see a way to accomplish this via a make command-line option and the
variable idea sounded as if it might require a makefile modification.


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #22923] option to prevent "interspersed" output in parallel builds

[Brian Kustel]

URL:
  

 Summary: option to prevent "interspersed" output in parallel
builds
 Project: make
Submitted by: bkustel
Submitted on: Wednesday 04/16/2008 at 03:26
Severity: 3 - Normal
  Item Group: Enhancement
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 3.81
Operating System: Any
   Fixed Release: None

___

Details:

In the Parallel Execution section of the manual, it accurately states: 

   "One unpleasant consequence of running several commands
simultaneously is that output generated by the commands 
appears whenever each command sends it, so messages 
from different commands may be interspersed."

But it should be possible have the output of each parallel command/process
batch up its output and only write it to the primary stdout & stderr streams
when the process completes in a synchronized fashion. This seems like it
should be relatively simple to accomplish. The only downside I can see is that
for LONG processes with lots of output, the display would not be in real time
and there could potentially be some storage issues with holding onto that data
until the process completes. But I imagine one could spool the stdout and
stderr off to a temp file marking which blocks are stdout and which are stderr
and then this could be replayed back later to reproduce the proper ordering of
the stdout/stderr output. 

I suspect most users of the Parallel execution feature would prefer to have
the output of each individual command be contiguous, though perhaps some would
not. 

Thus, I propose that an option be added to force the output of each parallel
process to be made continguous in the primary stdout & stderr streams. 




___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make