On Tue, May 3, 2011 at 1:20 PM, Paul Smith wrote:
> In that way, SYNC is a feature that the makefile USER selects, or not,
> and not something the makefile AUTHOR would choose.
>
> Does that make sense?
It makes perfect sense when you put it that way and I agree wrt to
both .ONESHELL and .PARALLE
On Tue, 2011-05-03 at 13:00 -0400, David Boyce wrote:
> > The other thing I was thinking is that this feature might want to be
> > enabled via a command-line argument. All the complex makefiles
> > generated by automake, etc. for example cannot take advantage of this if
> > you have to modify ever
On 3 May 2011 17:39, Eli Zaretskii wrote:
> That was exactly the scenario I had in mind when I wrote my message.
> Recursive Makefiles are the rule nowadays, at least with GNU software,
> and the top-level Makefile does little more than launch a "make all"
> job in each subdirectory. GCC or GDB
On Tue, May 3, 2011 at 1:33 AM, Paul Smith wrote:
> I wonder if we can figure out a way to make this work better, as Eli
> asked. Can we work out a way to handle "normal" rules (rules with "+"
> or $(MAKE)) and "sub-make" rules differently, so that output from normal
> rules wasn't collected behi
> From: Paul Smith
> Cc: bug-make@gnu.org
> Date: Tue, 03 May 2011 01:33:38 -0400
>
> But, I've been playing with a makefile that I have, that builds a
> complete suite of GCC tools from scratch. I'm building them on larger
> systems with -j5 or -j10 (not much compared to what some are doing I
>
On Tuesday 03 May 2011, Paul Smith wrote:
> On Tue, 2011-05-03 at 09:48 +0200, Stefano Lattarini wrote:
> > > The other thing I was thinking is that this feature might want to be
> > > enabled via a command-line argument. All the complex makefiles
> > > generated by automake, etc. for example cann
On Tue, 2011-05-03 at 09:48 +0200, Stefano Lattarini wrote:
> > The other thing I was thinking is that this feature might want to be
> > enabled via a command-line argument. All the complex makefiles
> > generated by automake, etc. for example cannot take advantage of this
> > if you have to modif
Hello Paul. Just my 2 cents regarding Automake ...
On Tuesday 03 May 2011, Paul Smith wrote:
>
> The other thing I was thinking is that this feature might want to be
> enabled via a command-line argument. All the complex makefiles
> generated by automake, etc. for example cannot take advantage o
On Fri, 2011-04-15 at 16:57 +0300, Eli Zaretskii wrote:
> > Date: Fri, 15 Apr 2011 12:43:53 +0100
> > From: Tim Murphy
> >
> > I think it's an inevitable consequence that if you have a long-running
> > task then the output from it won't appear until it has completely
> > finished and you won't be
I've filed this as an enhancement request
(https://savannah.gnu.org/bugs/index.php?33138) for better tracking.
Unfortunately there's a typo in the headline which I can't fix.
David
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman
On Tue, Apr 19, 2011 at 11:06 AM, David Boyce wrote:
> Here's an updated version which is much closer to "done"
Ignore this version, I found a bug :-(
David
___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make
On Tue, Apr 19, 2011 at 10:13 AM, Edward Welbourne wrote:
>> How about if we introduce a variable that can be set for specific
>> targets (using the target-specific variable mechanism). Targets that
>> set this variable to some predefined value will not have their output
>> redirected, so their o
Here's an updated version which is much closer to "done" at least from
my POV. I believe most of the requested improvements are there, the
semaphore code is factored, there's even a start on documentation.
Despite what was said about ifdefs before, the entire patch is
currently enclosed in "#ifdef
> How about if we introduce a variable that can be set for specific
> targets (using the target-specific variable mechanism). Targets that
> set this variable to some predefined value will not have their output
> redirected, so their output will go to the screen.
IIUC, the proposal is to make the
> From: Paul Smith
> CC: David Boyce , bug-make@gnu.org
> Date: Fri, 15 Apr 2011 14:53:52 -0400
>
> On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote:
> > What about the other issue: with the fact that output from a recipe is
> > only shown when the entire recipe is done. I think this is a
I wish I could play with one of those Connection machines and find out
about the bottlenecks at j=0.25*cpus that I can see on some machines
:-(
Regards,
Tim
On 15 April 2011 19:32, Howard Chu wrote:
> Eli Zaretskii wrote:
>>>
>>> Date: Fri, 15 Apr 2011 10:37:13 -0400
>>> From: David Boyce
>>> i
On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote:
> What about the other issue: with the fact that output from a recipe is
> only shown when the entire recipe is done. I think this is a serious
> drawback, at least in some use cases. Can anything be done about
> that?
I don't see how. The
Eli Zaretskii wrote:
Date: Fri, 15 Apr 2011 10:37:13 -0400
From: David Boyce
it's more a question of how many parallel jobs a given make process is
managing since limits on file handles/descriptors are per process.
What about people who use "make -j" without limits?
They're idiots. I've never
On Fri, Apr 15, 2011 at 1:03 PM, Eli Zaretskii wrote:
> Can Make be invoked with its stdout closed by the parent process? If
> it is, will this still work?
I've tried to code it such that if anything goes wrong setting up the
sync, of which a closed stdout would be one example, it prints a
messa
> What about the other issue: with the fact that output from a recipe is
> only shown when the entire recipe is done. I think this is a serious
> drawback, at least in some use cases. Can anything be done about
> that?
The only thing that I can imagine is that one would allow one recipe
to have
> Date: Fri, 15 Apr 2011 10:12:19 -0400
> From: David Boyce
> Cc: Eli Zaretskii , bug-make@gnu.org
>
> Attached is an updated patch with some improvements:
Thanks.
> - It uses stdout as its semaphore.
Can Make be invoked with its stdout closed by the parent process? If
it is, will this still
> Date: Fri, 15 Apr 2011 12:39:56 -0400
> From: David Boyce
> Cc: psm...@gnu.org, bug-make@gnu.org
>
> On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii wrote:
> > But this new option uses up 2 additional files per job, doesn't it?
>
> One or two, as discussed elsewhere in this thread.
>
> > Eac
On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii wrote:
> But this new option uses up 2 additional files per job, doesn't it?
One or two, as discussed elsewhere in this thread.
> Each sub-Make inherits all the file descriptors of all its parents,
> grandparents, etc. If a sub-Make was launched w
On Fri, 2011-04-15 at 18:09 +0300, Eli Zaretskii wrote:
> > But yes, any feature
> > which consumes additional resources increases the risk of running out
> > of that resource. I don't know that recursion comes into it though,
>
> Each sub-Make inherits all the file descriptors of all its parents,
> Date: Fri, 15 Apr 2011 10:37:13 -0400
> From: David Boyce
> Cc: psm...@gnu.org, bug-make@gnu.org
>
> > Finally, wouldn't it be a potential problem top inherit so many
> > handles to subordinate processes (2 for each running job)? We could
> > run out of available handles in deeply recursive jo
On Fri, Apr 15, 2011 at 9:48 AM, Eli Zaretskii wrote:
> So effectively, whenever a job finishes, its parent Make takes the
> semaphore, outputs all of the output of that job to the screen, then
> releases the semaphore, is that right?
Yes, exactly.
> And how do you communicate the name of the se
Hello everybody.
On Friday 15 April 2011, Eli Zaretskii wrote:
> Btw, there will be other side effects, at least on non-Posix
> platforms, due to the fact that stuff that was supposed to go to the
> screen is redirected to a file instead. Some programs sense that and
> behave differently, e.g. wi
On Thu, Apr 14, 2011 at 4:30 PM, David Boyce wrote:
> I don't know why this hasn't occurred to me or the authors of similar
> programs before, but it appears to be possible to get a lock on any
> writable file descriptor - for instance stdout or stderr, or one of
> the jobserver-fds. I just change
> Date: Fri, 15 Apr 2011 12:43:53 +0100
> From: Tim Murphy
> Cc: David Boyce , bug-make@gnu.org
>
> I think it's an inevitable consequence that if you have a long-running
> task then the output from it won't appear until it has completely
> finished and you won't be able to watch the progress - i
> Date: Fri, 15 Apr 2011 08:31:43 -0400
> From: David Boyce
> Cc: psm...@gnu.org, bug-make@gnu.org
>
> and I've attached output logs "without.txt" and "with.txt", both using
> -j for unlimited jobs. You'll notice that in without.txt the output of
> recipes is separated; all the BEGINNING lines sh
On Fri, Apr 15, 2011 at 6:45 AM, Eli Zaretskii wrote:
> I lack the higher-level picture. Can you describe in a few words how
> will this work in a running Make? I don't mean the details of how
> files are locked and unlocked (I understand that part), I mean the
> larger picture, like how it will
On Fri, 2011-04-15 at 07:18 +0100, Tim Murphy wrote:
> Right, of course :-) That is the obvious reason for using files,
> thanks! I was biased against writing things to disc which is probably
> silly, because it's probably not slow when compared to the size of the
> job that's running.
I think y
I replied to this without using "reply all" and I apologise. :-)
The order of the output from separate recipes may be anything -
depends on how make schedules them but at least the output from any
particular recipe will be grouped together and not mixed with the
output from others, hence allowing
> Date: Thu, 14 Apr 2011 16:30:42 -0400
> From: David Boyce
> Cc: Eli Zaretskii , bug-make@gnu.org
>
> I don't know why this hasn't occurred to me or the authors of similar
> programs before, but it appears to be possible to get a lock on any
> writable file descriptor - for instance stdout or st
> Date: Thu, 14 Apr 2011 15:15:15 -0400
> From: David Boyce
> Cc: psm...@gnu.org, bug-make@gnu.org
>
> On Thu, Apr 14, 2011 at 3:00 PM, Eli Zaretskii wrote:
> > Yes, but a few words about how is this semaphore supposed to get job
> > done, and in fact what kind of "synchronization" will this bri
Right, of course :-) That is the obvious reason for using files,
thanks! I was biased against writing things to disc which is probably
silly, because it's probably not slow when compared to the size of the
job that's running.
Stdout and stderr will now all be stdout, BTW (no matter what method
o
On Thu, Apr 14, 2011 at 3:08 PM, David Boyce wrote:
> On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith wrote:
>> I'm not sure I like the idea of having to define a lockfile to enable
>> this feature though. It feels a little too much like exposing the
>> implementation details to the user.
>
> Agree.
On Thu, 2011-04-14 at 12:24 -0700, Howard Chu wrote:
> > However, you could do both with some heuristics. Hm. Maybe not. I was
> > going to say you could merge them if stdout and stderr were going to the
> > same tty or file, but I don't think there's any good way in UNIX to know
> > whether two
On Thu, 2011-04-14 at 20:59 +0100, Tim Murphy wrote:
> To prevent any kind of deadlock you sort of want to empty everything
> the stderr and stdout pipe buffers may contain.
>
> It is conceivable that the stdout pipe might contain two lines of text
> and stderr pipe might contain 1 by the time you
hy wrote:
>> -- Forwarded message --
>> From: Tim Murphy
>> Date: 14 April 2011 20:43
>> Subject: Re: patch to support output synchronization under -j
>> To: psm...@gnu.org
>>
>>
>> The reason for splitting stderr and stdout is to do wi
That was a confused email - apologies. I will rethink it and return. :-)
On 14 April 2011 20:43, Tim Murphy wrote:
> -- Forwarded message --
> From: Tim Murphy
> Date: 14 April 2011 20:43
> Subject: Re: patch to support output synchronization under -j
> To:
-- Forwarded message --
From: Tim Murphy
Date: 14 April 2011 20:43
Subject: Re: patch to support output synchronization under -j
To: psm...@gnu.org
The reason for splitting stderr and stdout is to do with deadlock and
reading pipes. IIRC. e.g. blocking on a read to stderr which
Paul Smith wrote:
On Thu, 2011-04-14 at 14:08 -0400, David Boyce wrote:
Of course, either way some context is lost. If you put both into one
temp file you lose track of which was which; if you keep them in
separate files you lose ordering instead. So it becomes a matter of
taste, or perhaps an o
On Thu, 2011-04-14 at 14:08 -0400, David Boyce wrote:
> On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith wrote:
> > One example: I think saving stdout and stderr to different files and
> > then printing them separately is problematic; consider if your recipe
> > prints lots of information lines, with e
On Thu, Apr 14, 2011 at 3:00 PM, Eli Zaretskii wrote:
> Yes, but a few words about how is this semaphore supposed to get job
> done, and in fact what kind of "synchronization" will this bring to
> Make, would be appreciated. I don't think you described the feature
> too much in your original post
On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith wrote:
> There is no specific region of the file that's locked: the lockfile is
> basically a file-based, system-wide semaphore.
Yes, it's conceptually a semaphore. In fact a Windows port might
prefer to use real semaphores. The reason I stayed away fro
Paul Smith writes:
> One example: I think saving stdout and stderr to different files and
> then printing them separately is problematic; consider if your recipe
> prints lots of information lines, with errors (to stdout) interspersed.
> If you throw all the errors to the end you lose a lot of con
> Date: Thu, 14 Apr 2011 14:12:16 -0400
> From: David Boyce
> Cc: psm...@gnu.org, bug-make@gnu.org
>
> On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii wrote:
> > David, can you explain why you needed to lock the files? Also, what
> > region(s) of the file you are locking? fcntl with F_WRLCK won
File locking would be the exclusion mechanism with a shell
implementation, right? With Talon we used a system semaphore with
timeout-wait semantics . I would assume that one doesn't need it if
the code is in make since make can choose when to start and stop
reading from any particular file handle
On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii wrote:
> David, can you explain why you needed to lock the files? Also, what
> region(s) of the file you are locking? fcntl with F_WRLCK won't work
> on Windows, so the question is how to emulate it.
I was about to write this up but I see Paul beat
On Thu, 2011-04-14 at 20:48 +0300, Eli Zaretskii wrote:
> > From: Paul Smith
> > Date: Thu, 14 Apr 2011 13:29:09 -0400
> >
> > On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote:
> > > On Tue, Apr 12, 2011 at 1:46 PM, David Boyce
> > > wrote:
> > > > So I've made a proof-of-concept patch
> >
On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith wrote:
> I've looked at it and as a concept I don't have too many issues with it
Thanks.
> One example: I think saving stdout and stderr to different files and
> then printing them separately is problematic; consider if your recipe
> prints lots of inf
> From: Paul Smith
> Date: Thu, 14 Apr 2011 13:29:09 -0400
>
> On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote:
> > On Tue, Apr 12, 2011 at 1:46 PM, David Boyce
> > wrote:
> > > So I've made a proof-of-concept patch
> > > against 3.82.90 which seems to work without that overhead and my
> >
On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote:
> On Tue, Apr 12, 2011 at 1:46 PM, David Boyce wrote:
> > So I've made a proof-of-concept patch
> > against 3.82.90 which seems to work without that overhead and my
> > question is, would this be of interest towards 3.83?
>
> Ping?
>
> The or
On Tue, Apr 12, 2011 at 1:46 PM, David Boyce wrote:
> So I've made a proof-of-concept patch
> against 3.82.90 which seems to work without that overhead and my
> question is, would this be of interest towards 3.83?
Ping?
The original patch attachment was made by hand using "diff -u" and I
had som
I recently wrote a little program called "syncsh"
(https://github.com/boyski/SYNCSH) to synchronize the output of make
-j such that the results of a given job stay together. It's based on
ideas from a few posts on this list and an old "Ask Mr Make" article
(http://www.cmcrossroads.com/ask-mr-make/1
56 matches
Mail list logo
