Re: Regression caused by Commit: 5bd7ad2b225b ("Preserve the real value of -jN in MAKEFLAGS using jobserver.")
On 10/30/2017 02:22 PM, Paul Smith wrote: On Tue, 2017-10-31 at 07:39 +1100, NeilBrown wrote: On Mon, Oct 30 2017, Tim Murphy wrote: For my money -j is always a top-level setting. To have it per makefile and then let makefiles depend on that behaviour would open the door to lots of scary bugs. I don't see what scary bugs we might have; I've seen issues where makefiles not intended to be used with -j will not work properly with -j, but I'm not sure I've ever seen an issue with a makefile intended to work with -j that fails when run without it, or with a different value. Here's a trivial example that shows how a build could fail if -j is not specified, but work if -j is specified (with any value, in this case), assuming "output" does not exist to start: all: reader writer reader: sleep 2 cat output writer: echo PASS > output Of course "work" is used somewhat loosely here, as the build might appear to work but in fact pick up stale data if "output" happens to exist prior to the start of the run (eg, in an incremental build). It's not a common scenario, to be sure, but I have encountered this "in the wild". Just last week I worked with a large commercial build which exhibited this behavior, due to a missing dependency between the reader and writer jobs. Regards, Eric Melski Chief Architect Electric Cloud, Inc. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Using hash instead of timestamps to check for changes.
On 04/11/2015 08:38 AM, Enrico Weigelt, metux IT consult wrote: On 07.04.2015 00:17, Eric Melski wrote: Hi, This problem is relatively common when using an SCM system that preserves *checkin* time on files rather than *checkout* time. I'd consider that a misbehavious of the SCM (IMHO, that's the reason why Git does not track the mtime). From the filesystem perspective, the mtime represents the time when the actual file was changed in the filesystem. So, resetting the mtime from some SCM repo actually is tricking the filesystem - pretty obvious that the mtime then isn't reliable anymore. It doesn't necessarily require resetting or manipulating the mtime at all. ClearCase has its own filesystem, MVFS, which simply behaves the way I described. In any case, it's not an utterly irrational position to consider the last modification time of a file tracked by the version control system to be the time that a new revision of the file was created. That *is* the last modification time, after all. However, this is not the place for this type of philosophical debate. The question at hand is whether or not make would benefit from a non-timestamp-based notion of up-to-dateness, and the answer seems to be clearly yes, for this and other reasons mentioned elsewhere in this thread. Regards, Eric Melski Chief Architect Electric Cloud, Inc. http://blog.melski.net/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Using hash instead of timestamps to check for changes.
On 04/04/2015 11:38 AM, Tim Murphy wrote: My thinking is that the timestamp is in fact an overly conservative test. We never have the case that the timestamp indicates something *has not* been changed when in fact it has (i.e. we always build if something has changed), That's interesting, because in my experience the main reason people are upset about timestamps these days is the exact opposite: with the increase in capabilities of systems, in particular larger build servers, it is possible to have situations where targets are updated too quickly to reliably determine out-of-date-ness based solely on timestamps. That may be a use case but I've not experienced it more than once, IIRC, but I've often experienced things which are automatically generated or copied and which trigger huge rebuilds for no ultimate purpose. This problem is relatively common when using an SCM system that preserves *checkin* time on files rather than *checkout* time. ClearCase does this in various configurations, and Perforce will if your client spec has modtime set. I'm sure other SCM systems can be setup this way too. In such environments, the timestamp on a file may not change *enough* even when the content has changed. For example, suppose the following: 1. I have file foo.c with time A. 2. My teammate modifies foo.c and checks in at time B. 3. I build in my workspace, without sync'ing, generating foo.c with time C. 4. I sync my workspace, updating foo.c but leaving it with time B. 5. I try to rebuild but foo.o is not updated because C is later than B. ClearCase's clearmake utility handles this, of course. The problem occurs frequently enough that we added a feature to Electric Make called the ledger expressly to allow us to notice this type of out-of-dateness. Regards, Eric Melski Chief Architect Electric Cloud, Inc. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Patch to allow make to load plugins that add new functions.
On 04/05/2012 11:59 PM, Tim Murphy wrote: As for other tools like Electric Make, I want to be able to arm-twist them to support plugins (politely of course, Eric :-) ) without them being able to say it's impossible or unreliable or that it requires them to simulate the volatile internals of make to some infeasible degree. I realise that this is just my interest but it makes the difference between bothering to use a plugin mechanism and not bothering at all since it couldn't offer me a uniform makefile that I could use across systems - I'd solve all my worries on GNU make and still be busted on emake and since GNU make doesn't offer that amazing filesystem which tells you when you have parallelsim problems and corrects them. In other words I can't give up on emake without converting 100s of developers into experts in writing good parallel makefiles and inventing a way to get make to work on clusters of machines (without pvmgmake's horrendously complicated setup). People are so much more difficult than software. So my vote, FWIW, is to define what one does expect make to offer plugins (the nice bits of make) and define what it definitely doesn't and won't in the near future. Anyone who wants a lot more is probably going to have to customise make itself anyhow. I think the plugin should say what environment it expects and make should decide if it can offer that. If make fails on something it claims to support then it's a bug on make otherwise it's clearly a plugin problem etc. As you say with expectations, one has to set them at some point. Adding a plugin mechanism to ElectricMake has been on my wishlist for quite some time actually. I'd love to collaborate with Paul, et al, on the design so that we come up with something that works for both gmake and emake. In my wildest imaginings we could cook up something that would be binary-compatible, so we only have to compile a plugin once to use it with both tools, but I doubt that's really achievable. The next best thing would be source-compatibility, so we at least don't have to tweak the plugin implementation to work with each tool (although I suppose that too could be manageable, in the same way that you see plugins for scripting languages with different 'bindings' for different languages -- still, we would need to agree on some fundamentals for that to work I think). The single biggest issue in this regard is a difference in architecture between gmake and emake. Fundamentally, gmake is one-make-per-process, and there are a lot of assumptions and shortcuts baked into the code because of that. emake is many-makes-per-process. In practical terms that means that we have to pass around additional context when doing expansions (eg, you have to know which make instance the expansion is part of, so that you use the right variable table). I think to make something that would work for both gmake and emake we would have to abstract away this difference -- for example, rather than directly accessing data structures, we would define a set of API's for the various operations that are needed during expansion. So there's definitely interest in collaboration on our end; is there any interest from the gmake devs? Best regards, Eric Melski Architect Electric Cloud, Inc. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Prioritizing non-dependent targets in parallel make
Eric Melski wrote: ElectricAccelerator doesn't factor runtimes into scheduling decisions, although we have talked about doing so in the past. I spent some time investigating the possibility, most recently a couple years ago. ... I ran the simulation on a variety of real-world builds. ... I'll see if I can dig up the patch for ElectricInsight and maybe write it up in a blog. up that code, and maybe write it up in a blog. Ugh, it took me a lot longer to put this together than I expected, but if anybody is still interested, I did finally do a write-up on the subject of a longest-job-first scheduler for make and why it doesn't work in practice: http://blog.melski.net/2010/08/23/faster-builds-through-smarter-scheduling-longest-job-first/ Best regards, Eric Melski http://blog.melski.net/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: insufficient debug info from gnu-make
Peter Lawrence wrote: great if you want to see what was actually executed, not so great if you want to figure out where in the make sources you need to start looking, both are realistic wants. when things get as messy as gcc's makefiles, I'ld like to be able to see both the before and after macro substitutions versions of the commands lines .or, at least the line numbers (along with directory/filename) so I know where to look in the original source ElectricAccelerator, a gmake replacement, can produce an annotated build log that includes this information. I've written about that feature here: http://blog.electric-cloud.com/2009/11/10/how-to-quickly-navigate-an-unfamiliar-makefile/ SparkBuild (http://www.sparkbuild.com) is a free version of Accelerator that also includes this feature. It can be pretty handy in situations like the one you describe. Best regards, Eric Melski Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
Hambridge, Philip J (ODP) wrote: I've not been following this thread too closely, but this may be relevant: http://www.cmcrossroads.com/ask-mr-make/12909-descrambling-parallel-build-logs I wrote the linked article; for those not interested in reading it, the strategy described there is the same as that used by the talon shell wrapper described elsewhere in this thread. Best regards, Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: What I want is transparent parallel make. Make can issue multiple shells simultaneously, but print their outputs in the same order as in a serial make. ElectricAccelerator is a gmake replacement that does exactly this. I wrote about this feature a while back: http://blog.electric-cloud.com/2008/12/01/untangling-parallel-build-logs/ You can read more about Accelerator on the blog, or here: http://www.electric-cloud.com/products/electricaccelerator.php Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Prioritizing non-dependent targets in parallel make
Tim Murphy wrote: Hi, I'm nobody official. I have seen Electric Accelerator using history files which may be doing something a bit like this. ... So I, personally, want something that learns. I would like to be able to store the learned information into source control and use it as the basis for the next attempt. One would have to check for patents on this, I think. Hi Tim! ElectricAccelerator doesn't factor runtimes into scheduling decisions, although we have talked about doing so in the past. I spent some time investigating the possibility, most recently a couple years ago. What I did was tweak the Simulator report in ElectricInsight to simulate this kind of scheduling, so each time it chooses the next job to run, it picks the longest job of those that are available to start. I ran the simulation on a variety of real-world builds. Unfortunately, the results were disappointing. In my recollection, the best case gave a savings of about 5% of the total build time, and in general it was barely measurable -- maybe 1%. It seems that in practice, most of the jobs in a build are about the same length. The notable exception is the final link, which is of course much longer. But then -- of course -- that can't start until all the other jobs are done anyway, so the fancy scheduling doesn't help. I'll see if I can dig up the patch for ElectricInsight and maybe write it up in a blog. up that code, and maybe write it up in a blog. Cheers! Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Prioritizing non-dependent targets in parallel make
Tim Murphy wrote: Hi, I'm nobody official. I have seen Electric Accelerator using history files which may be doing something a bit like this. Personally I don't like the idea of priorities very much. Large builds that I have done do have big targets but since we are building 5000 of the same kind of target (using a macro to define a generic template and $eval to instantiate it for each specific target) it becomes impossible for us to provide hints. In the end we'd have to force the users of our build system say webkit.dll is pretty huge or euser.dll isn't. Since this is all relative information, how can we expect a build made up of components that various people produce to have the right relative weights? So I, personally, want something that learns. I would like to be able to store the learned information into source control and use it as the basis for the next attempt. One would have to check for patents on this, I think. Hi Tim! ElectricAccelerator doesn't factor runtimes into scheduling decisions, although we have talked about doing so in the past. I spent some time investigating the possibility, most recently a couple years ago. What I did was tweak the Simulator report in ElectricInsight to simulate this kind of scheduling, so each time it chooses the next job to run, it picks the longest job of those that are available to start. I ran the simulation on a variety of real-world builds. Unfortunately, the results were disappointing. In my recollection, the best case gave a savings of about 5% of the total build time, and in general it was barely measurable -- maybe 1%. It seems that in practice, most of the jobs in a build are about the same length. The notable exception is the final link, which is of course much longer. But then -- of course -- that can't start until all the other jobs are done anyway, so the fancy scheduling doesn't help. I'll see if I can dig up the patch for ElectricInsight and maybe write it up in a blog. up that code, and maybe write it up in a blog. Cheers! Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Prioritizing non-dependent targets in parallel make
Tim Murphy wrote: Hi, I'm nobody official. I have seen Electric Accelerator using history files which may be doing something a bit like this. ... So I, personally, want something that learns. I would like to be able to store the learned information into source control and use it as the basis for the next attempt. One would have to check for patents on this, I think. Hi Tim! ElectricAccelerator doesn't factor runtimes into scheduling decisions, although we have talked about doing so in the past. I spent some time investigating the possibility, most recently a couple years ago. What I did was tweak the Simulator report in ElectricInsight to simulate this kind of scheduling, so each time it chooses the next job to run, it picks the longest job of those that are available to start. I ran the simulation on a variety of real-world builds. Unfortunately, the results were disappointing. In my recollection, the best case gave a savings of about 5% of the total build time, and in general it was barely measurable -- maybe 1%. It seems that in practice, most of the jobs in a build are about the same length. The notable exception is the final link, which is of course much longer. But then -- of course -- that can't start until all the other jobs are done anyway, so the fancy scheduling doesn't help. I'll see if I can dig up the patch for ElectricInsight and maybe write it up in a blog. up that code, and maybe write it up in a blog. Cheers! Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
