Re: [linux] possible race condition in makefile with multiple dependencies?
On Thu, 3 Jun 2021 11:55:38 -0400 (EDT) "Robert P. J. Day" wrote: > i took a look at it and it was a holdover from (i believe) QNX where > their version of make did not support parallelism so that many of the > rules were in the form: > target: dep1 dep2 dep3 dep4 > $(MAKE) -C subdir1 > $(MAKE) -C subdir2 > ... > $(MAKE) -C subdirn Yeah, that's bad. And echoing your other points... https://accu.org/journals/overload/14/71/miller_2004/ Regards, Dianne. To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On Thu, 3 Jun 2021 10:19:22 -0400 "Michael P. Soulier" wrote: > On 2021-06-03 10:09 a.m., Dianne Skoll wrote: > > See the -j option > Seems like a race condition waiting to happen. Shouldn't be. My understanding (which may be wrong) is that make calculates the dependency graph first, not in parallel. Then it executes as many parallel tasks as the -j option asks for, or that make sense... whichever is smaller. Since the dependency graph is a DAG, make should, in theory, be smart enough to see that two dependencies depend on the same node and run only one job to build the depended-upon node. I routinely use "make -j `nproc`" and have never run into any issues. Regards, Dianne. To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On Thu, 3 Jun 2021 09:29:43 -0400 "Michael P. Soulier" wrote: > I am not aware of gnu make having any parallel processing > capabilities. See the -j option Regards, Dianne. To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On Thu, 3 Jun 2021, Dianne Skoll wrote: > On Thu, 3 Jun 2021 10:19:22 -0400 > "Michael P. Soulier" wrote: > > > On 2021-06-03 10:09 a.m., Dianne Skoll wrote: > > > See the -j option > > > Seems like a race condition waiting to happen. > > Shouldn't be. My understanding (which may be wrong) is that make > calculates the dependency graph first, not in parallel. Then it > executes as many parallel tasks as the -j option asks for, or that > make sense... whichever is smaller. Since the dependency graph is a > DAG, make should, in theory, be smart enough to see that two > dependencies depend on the same node and run only one job to build > the depended-upon node. > > I routinely use "make -j `nproc`" and have never run into any > issues. reason i was asking is that, on current contract, devs were complaining that a legacy makefile was taking ***forever*** to run, so i took a look at it and it was a holdover from (i believe) QNX where their version of make did not support parallelism so that many of the rules were in the form: target: dep1 dep2 dep3 dep4 $(MAKE) -C subdir1 $(MAKE) -C subdir2 ... $(MAKE) -C subdirn and AFAIK, even under gnu make parallelism, the commands in any rule are still run sequentially, so that the above would have to finish the first make before going to the second, and so on. so i suggested that refactoring the makefile to turn all those sub-makes into actual dependencies would probably speed things up, given that the build system is a 40-core beast. rday To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
> On Jun 3, 2021, at 10:27, Robert P. J. Day wrote: > >> >> Shouldn't be. My understanding (which may be wrong) is that make >> calculates the dependency graph first, not in parallel. Then it >> executes as many parallel tasks as the -j option asks for, or that >> make sense... whichever is smaller. Since the dependency graph is a >> DAG, make should, in theory, be smart enough to see that two >> dependencies depend on the same node and run only one job to build >> the depended-upon node. >> >> I routinely use "make -j `nproc`" and have never run into any >> issues. > > this was my understanding as well, i just wanted to hear some > confirmation from someone else. Diane is correct. As long as the dependency graph is specified properly in the Makefile, make -j should work fine. I have ran make -j for ffmpeg and x265 on an AMD EPYC server with 128 threads. Amazing how fast a build can be and how much RAM can be sucked up with a few hundred compiles stacked up waiting to run. /sc To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On 2021-06-03 10:27, Robert P. J. Day wrote: > On Thu, 3 Jun 2021, Dianne Skoll wrote: > > > On Thu, 3 Jun 2021 10:19:22 -0400 > > "Michael P. Soulier" wrote: > > > > > On 2021-06-03 10:09 a.m., Dianne Skoll wrote: > > > > See the -j option > > > > > Seems like a race condition waiting to happen. > > > > Shouldn't be. My understanding (which may be wrong) is that make > > calculates the dependency graph first, not in parallel. Then it > > executes as many parallel tasks as the -j option asks for, or that > > make sense... whichever is smaller. Since the dependency graph is a > > DAG, make should, in theory, be smart enough to see that two > > dependencies depend on the same node and run only one job to build > > the depended-upon node. > > > > I routinely use "make -j `nproc`" and have never run into any > > issues. > > this was my understanding as well, i just wanted to hear some > confirmation from someone else. I routinely use make -j 8 on an 8-core machine to build the linux kernel. I only ran into problems when I tried to include the targets "modules_install" and "install" in the same multithreaded make command. It even works over NFS as long as the two clocks are reasonably close. > rday slainte mhath, RGB -- Richard Guy Briggs -- ~\-- ~\ -- \___ o \@ @Ride yer bike! Ottawa, ON, CANADA -- Lo_>__M__\\/\%__\\/\% Vote! -- _GTVS6#790__(*)__(*)(*)(*)_ To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On 03/06/2021 10:38, Michael P. Soulier wrote: On 2021-06-03 10:22 a.m., Dianne Skoll wrote: Shouldn't be. My understanding (which may be wrong) is that make calculates the dependency graph first, not in parallel. Then it executes as many parallel tasks as the -j option asks for, or that make sense... whichever is smaller. Since the dependency graph is a DAG, make should, in theory, be smart enough to see that two dependencies depend on the same node and run only one job to build the depended-upon node. I routinely use "make -j `nproc`" and have never run into any issues. Good to know. Does anyone have a solution for make across multiple machines, perhaps using ssh? distccd On Gentoo you configure your preferred "-j" value in /etc/portage/make.conf. It will compile that number files in parallel. distcc will allow that work to be shared across multiple machines - and cross compile for different architectures if required. Some gentoo packages need to be forced to build with -j1 if a) it needs hideous amounts of RAM b) some just need it (I assume some of these race conditions) Tug Mike To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On 2021-06-03 10:22 a.m., Dianne Skoll wrote: Shouldn't be. My understanding (which may be wrong) is that make calculates the dependency graph first, not in parallel. Then it executes as many parallel tasks as the -j option asks for, or that make sense... whichever is smaller. Since the dependency graph is a DAG, make should, in theory, be smart enough to see that two dependencies depend on the same node and run only one job to build the depended-upon node. I routinely use "make -j `nproc`" and have never run into any issues. Good to know. Does anyone have a solution for make across multiple machines, perhaps using ssh? Mike To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On Thu, 3 Jun 2021, Dianne Skoll wrote: > On Thu, 3 Jun 2021 10:19:22 -0400 > "Michael P. Soulier" wrote: > > > On 2021-06-03 10:09 a.m., Dianne Skoll wrote: > > > See the -j option > > > Seems like a race condition waiting to happen. > > Shouldn't be. My understanding (which may be wrong) is that make > calculates the dependency graph first, not in parallel. Then it > executes as many parallel tasks as the -j option asks for, or that > make sense... whichever is smaller. Since the dependency graph is a > DAG, make should, in theory, be smart enough to see that two > dependencies depend on the same node and run only one job to build > the depended-upon node. > > I routinely use "make -j `nproc`" and have never run into any > issues. this was my understanding as well, i just wanted to hear some confirmation from someone else. rday To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On 2021-06-03 10:09 a.m., Dianne Skoll wrote: See the -j option Seems like a race condition waiting to happen. To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
Re: [linux] possible race condition in makefile with multiple dependencies?
On 2021-06-03 8:29 a.m., Robert P. J. Day wrote: i *vaguely* recall (probably incorrectly) a possible race condition in that both t1 and t2 will check their dependencies and, if they're out of date, both try to update them at the same time, possibly stepping on each other's work. i think that's unlikely, so am i misremembering? I am not aware of gnu make having any parallel processing capabilities. Mike To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org
[linux] possible race condition in makefile with multiple dependencies?
i'm probably overthinking this, but is there any problem with defining a makefile with numerous targets that can be processed in parallel if they all have the same set of dependencies (which can also be processed in parallel)? sample makefile i'm playing with: .PHONY: all alldeps t1 t2 d1 d2 / start all: t1 t2 alldeps: d1 d2 d1: @echo "start d1" sleep 5 @echo "end d1" d2: @echo "start d2" sleep 5 @echo "end d2" t1: alldeps @echo "start t1" sleep 5 @echo "end t1" t2: alldeps @echo "start t2" sleep 5 @echo "end t2" / end as you can see, if i run this makefile, it should process targets t1 and t2 in parallel, but since those targets both have dependencies of "alldeps", the goal is that *all* dependencies (d1, d2) should be processed and updated before *any* of the targets t1 or t2 are allowed to proceed, but when they do, they can also be processed in parallel. i *vaguely* recall (probably incorrectly) a possible race condition in that both t1 and t2 will check their dependencies and, if they're out of date, both try to update them at the same time, possibly stepping on each other's work. i think that's unlikely, so am i misremembering? in short, i will have a set of "dependencies" for a set of "targets": * dependencies can be processed (updated) in parallel * targets can be processed (updated) in parallel * all dependencies must be up to date before any targets can start to be processed is it this simple? or is there something i'm forgetting? rday To unsubscribe send a blank message to linux+unsubscr...@linux-ottawa.org To get help send a blank message to linux+h...@linux-ottawa.org To visit the archives: https://lists.linux-ottawa.org