make -j and errors
I've been thinking some more about it. POSIX says very little about parallel makes. The more I think about it, the more I think gnu-make's approach on this is stupid: if a job errors out in a fatal way, what do we gain if we keep going ? Especially for high -j values, the quicker we die, the better, as far as the error is concerned. (note that, in sequential mode, the first fatal error will kill us, so there's no point in considering further stuff running). but what about commands that take a long time to run ? Well, make already has a standard mechanism to flag those, that's called .PRECIOUS So, instead of the -dq quick death debugging option, I suggest we move to the following semantics: in case of an error, send ^C to all jobs making targets that are not tagged .PRECIOUS, wait for everything to come back, and that's it...
Re: make -j and errors
On Wed, Sep 26, 2012 at 06:21:34PM +0200, Marc Espie wrote: I've been thinking some more about it. POSIX says very little about parallel makes. The more I think about it, the more I think gnu-make's approach on this is stupid: if a job errors out in a fatal way, what do we gain if we keep going ? Especially for high -j values, the quicker we die, the better, as far as the error is concerned. (note that, in sequential mode, the first fatal error will kill us, so there's no point in considering further stuff running). but what about commands that take a long time to run ? Well, make already has a standard mechanism to flag those, that's called .PRECIOUS So, instead of the -dq quick death debugging option, I suggest we move to the following semantics: in case of an error, send ^C to all jobs making targets that are not tagged .PRECIOUS, wait for everything to come back, and that's it... Sounds like a rational approach to me. Ken
Re: make -j and errors
On Wed, Sep 26, 2012 at 06:21:34PM +0200, Marc Espie wrote: but what about commands that take a long time to run ? Well, make already has a standard mechanism to flag those, that's called .PRECIOUS What if most everything takes a fairly long time to run? Say, largish C++ sources or whatever? Mark every target as .PRECIOUS? So, instead of the -dq quick death debugging option, I suggest we move to the following semantics: in case of an error, send ^C to all jobs making targets that are not tagged .PRECIOUS, wait for everything to come back, and that's it... I think currently running jobs should be allowed to finish, as the default behavior. Why would you want to stop (at least potentially) successful work in progress? Is it because some very prominent things are done using a build-from-scratch every time? For those things, I agree that forcibly dying ASAP would save time and frustration. But for the more correct use of being able to fix a problem and have make pick up where it left off on the next invocation, your idea does not save time and frustration. If that's the case, then things that need a full build from scratch should know to invoke make with the proper flags. Kanpai! Darrin
Re: make -j and errors
On Wed, Sep 26, 2012 at 18:21, Marc Espie wrote: I've been thinking some more about it. POSIX says very little about parallel makes. The more I think about it, the more I think gnu-make's approach on this is stupid: if a job errors out in a fatal way, what do we gain if we keep going ? Especially for high -j values, the quicker we die, the better, as far as the error is concerned. (note that, in sequential mode, the first fatal error will kill us, so there's no point in considering further stuff running). I don't see what we gain by killing jobs. If the scheduler dice had come down differently, maybe those jobs would finish. Here's a downside, albeit maybe a stretch. What if the job doesn't like being killed? You're changing behavior here. Previously, the only way a job was interrupted was if the operator did it. In that case, I will pick up the pieces. I think letting the running jobs finish is actually a better match to the sequential make's behavior.
Re: make -j and errors
On Wed, Sep 26, 2012 at 12:01 PM, Ted Unangst t...@tedunangst.com wrote: I don't see what we gain by killing jobs. If the scheduler dice had come down differently, maybe those jobs would finish. Here's a downside, albeit maybe a stretch. What if the job doesn't like being killed? You're changing behavior here. Previously, the only way a job was interrupted was if the operator did it. In that case, I will pick up the pieces. I think letting the running jobs finish is actually a better match to the sequential make's behavior. +1
Re: make -j and errors
On Wed, Sep 26, 2012 at 12:41:10PM -0700, Philip Guenther wrote: On Wed, Sep 26, 2012 at 12:01 PM, Ted Unangst t...@tedunangst.com wrote: I don't see what we gain by killing jobs. If the scheduler dice had come down differently, maybe those jobs would finish. Here's a downside, albeit maybe a stretch. What if the job doesn't like being killed? You're changing behavior here. Previously, the only way a job was interrupted was if the operator did it. In that case, I will pick up the pieces. I think letting the running jobs finish is actually a better match to the sequential make's behavior. +1 Actually, I do see what we gain by killing jobs. You probably don't see the difference, because you run short stuff with not enough jobs. But for long running stuff, you will sometimes have an error, and notice it only a few minutes afterwards, 5000 lines of scrollback later, when they other jobs that were running finally reach completion... I see this all the time when working on large stuff. Seriously, try both. You have access to the quick death behavior right now. Try it on your normal work. Try the normal behavior as well. Yep, there's the downside that you have to pick up the pieces... happens almost never to me. Not any more often than the normal pick up the pieces because the build crashed, the makefile is bad, and you have to erase lots of shit because you restart. The way I see it, there are actually arguments both ways. I'm not talking hard killing of job, just SIGINT... build programs usually react correctly to SIGINT (gcc does)... and we're also talking about error detection and fixing. Play with both, please. This is definitely not a theoretical argument I'm trying to make...
Re: make -j and errors
On Thu, Sep 27, 2012 at 00:05, Marc Espie wrote: You probably don't see the difference, because you run short stuff with not enough jobs. But for long running stuff, you will sometimes have an error, and notice it only a few minutes afterwards, 5000 lines of scrollback later, when they other jobs that were running finally reach completion... I see this all the time when working on large stuff. Yeah, and I'm ok with that. Then I run make again without -j, look at the error and fix it.
