Re: Inconsistent behaviour when building in parallel
From: Dirk Heinrichs [EMAIL PROTECTED] Date: Sat, 20 Jan 2007 22:22:04 +0100 So the left hand doesn't know what the right hand does? ??? Would you care to explain where do you see two hands that don't know each what the other one does? As far as I see from Paul's clear description is that Make does the same thing in both situations. What would you expect Make to do in each case? In the parallel case you can't tell exactly what will happen: it depends on your system; how busy it is, what order the various jobs are scheduled by the OS, how long the command takes to execute, etc. It could work perfectly, or it could be run twice, or more times (if there are more than two targets). That's even worse. Really? why? I'm sure you are familiar with the notion of multi-tasking or multi-threading, and know that independent parallel execution threads are much less deterministic than sequential single-threaded ones. While that is a fact of life, this is the first time I hear someone say that it is such a bad thing. Anyway, you can always use -j1 if you don't want parallelism. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Inconsistent behaviour when building in parallel
On Sat, 2007-01-20 at 22:22 +0100, Dirk Heinrichs wrote: In the parallel case, make does exactly the same thing EXCEPT that instead of waiting for the first command to complete it immediately checks the second target and, since the first command is not done yet and hasn't updated it, make sees that it's still out of date so it runs that rule again to update it. So the left hand doesn't know what the right hand does? I consider this a bad thing. I do too, and so does make: that's why you shouldn't do it. After all, make cannot know what the commands you give it will REALLY do, it can only assume that they will do what you've told it they will do: namely, update the target file. However, in your makefile your commands don't just update the target file that you told make they would, they also update some OTHER target files that you didn't tell make about. So, make doesn't expect the commands to update those target files. In this case, the two hands you refer to are your commands that you wrote and added to your makefile, so you can hardly blame make for what they may be doing behind each others' backs. In the parallel case you can't tell exactly what will happen: it depends on your system; how busy it is, what order the various jobs are scheduled by the OS, how long the command takes to execute, etc. It could work perfectly, or it could be run twice, or more times (if there are more than two targets). That's even worse. Perhaps, but as Eli points out this is a feature of the operating system and is completely out of make's control. When you run multiple commands in parallel on a multitasking system, you cannot control the order in which they are executed. That's the entire point: if they always ran in a fixed order then they wouldn't be parallelized, would they? As Eli says, if you want serialized command execution it's trivial to get from make: just don't enable parallel builds. -- --- Paul D. Smith [EMAIL PROTECTED] Find some GNU make tips at: http://www.gnu.org http://make.paulandlesley.org Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Inconsistent behaviour when building in parallel
Hello, I'm using GNU make 3.81 (on Linux, if this is important) and see strange behaviour in parallel builds with the following Makefile: file1.txt file2.txt: file.in cat $ file1.txt|tee file2.txt test: file1.txt file2.txt cat $^ $@ In a normal build, make does the right thing: % make test cat file.in file1.txt|tee file2.txt cat file1.txt file2.txt test In case of a parallel build, the cat ... | tee ... command for file[12].txt is executed twice, which should not happen: % make -j 2 test cat file.in file1.txt|tee file2.txt cat file.in file1.txt|tee file2.txt cat file1.txt file2.txt test Bye... Dirk pgpkLFJzvU45z.pgp Description: PGP signature ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Inconsistent behaviour when building in parallel
On Fri, 2007-01-19 at 20:01 +0100, Dirk Heinrichs wrote: I'm using GNU make 3.81 (on Linux, if this is important) and see strange behaviour in parallel builds with the following Makefile: file1.txt file2.txt: file.in cat $ file1.txt|tee file2.txt test: file1.txt file2.txt cat $^ $@ In a normal build, make does the right thing: % make test cat file.in file1.txt|tee file2.txt cat file1.txt file2.txt test In case of a parallel build, the cat ... | tee ... command for file[12].txt is executed twice, which should not happen: % make -j 2 test cat file.in file1.txt|tee file2.txt cat file.in file1.txt|tee file2.txt cat file1.txt file2.txt test Make is behaving correctly here. Multiple targets in a single rule does not mean all these targets are built from one invocation of the rule. It means that each of these targets has the same prerequisites and rule, and will be built one at a time by running the rule. IOW, this: a b c : d e f ; command is identical to writing this: a : d e f ; command b : d e f ; command c : d e f ; command See the GNU make manual for details. You can use pattern rules: %1.txt %2.txt : %.in ... which do work as you expect. Or you have to use a sentinel file. -- --- Paul D. Smith [EMAIL PROTECTED] Find some GNU make tips at: http://www.gnu.org http://make.paulandlesley.org Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Inconsistent behaviour when building in parallel
Am Samstag, 20. Januar 2007 16:12 schrieb Paul Smith: On Fri, 2007-01-19 at 20:01 +0100, Dirk Heinrichs wrote: I'm using GNU make 3.81 (on Linux, if this is important) and see strange behaviour in parallel builds with the following Makefile: file1.txt file2.txt: file.in cat $ file1.txt|tee file2.txt test: file1.txt file2.txt cat $^ $@ In a normal build, make does the right thing: % make test cat file.in file1.txt|tee file2.txt cat file1.txt file2.txt test In case of a parallel build, the cat ... | tee ... command for file[12].txt is executed twice, which should not happen: % make -j 2 test cat file.in file1.txt|tee file2.txt cat file.in file1.txt|tee file2.txt cat file1.txt file2.txt test Make is behaving correctly here. Multiple targets in a single rule does not mean all these targets are built from one invocation of the rule. It means that each of these targets has the same prerequisites and rule, and will be built one at a time by running the rule. IOW, this: a b c : d e f ; command is identical to writing this: a : d e f ; command b : d e f ; command c : d e f ; command See the GNU make manual for details. So if make behaves correctly in the parallel case, does this mean that the non-parallel case is wrong? Bye... Dirk pgpUBh0pKtWPA.pgp Description: PGP signature ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Inconsistent behaviour when building in parallel
On Sat, 2007-01-20 at 17:35 +0100, Dirk Heinrichs wrote: So if make behaves correctly in the parallel case, does this mean that the non-parallel case is wrong? ?? No, both are correct. In the non-parallel case, make examines the first target and its prerequisites and sees that it needs to be rebuilt, so it runs the rule. The rule updates the first target and also updates the other targets. Next make looks at the second target and its prerequisites and sees that it does NOT need to be rebuilt (because it is newer than all the prerequisites, since it was just updated). So it's not rebuilt. Etc. In the parallel case, make does exactly the same thing EXCEPT that instead of waiting for the first command to complete it immediately checks the second target and, since the first command is not done yet and hasn't updated it, make sees that it's still out of date so it runs that rule again to update it. In the parallel case you can't tell exactly what will happen: it depends on your system; how busy it is, what order the various jobs are scheduled by the OS, how long the command takes to execute, etc. It could work perfectly, or it could be run twice, or more times (if there are more than two targets). -- --- Paul D. Smith [EMAIL PROTECTED] Find some GNU make tips at: http://www.gnu.org http://make.paulandlesley.org Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Inconsistent behaviour when building in parallel
Am Samstag, 20. Januar 2007 20:49 schrieb Paul Smith: No, both are correct. In the non-parallel case, make examines the first target and its prerequisites and sees that it needs to be rebuilt, so it runs the rule. The rule updates the first target and also updates the other targets. Next make looks at the second target and its prerequisites and sees that it does NOT need to be rebuilt (because it is newer than all the prerequisites, since it was just updated). So it's not rebuilt. Etc. In the parallel case, make does exactly the same thing EXCEPT that instead of waiting for the first command to complete it immediately checks the second target and, since the first command is not done yet and hasn't updated it, make sees that it's still out of date so it runs that rule again to update it. So the left hand doesn't know what the right hand does? I consider this a bad thing. In the parallel case you can't tell exactly what will happen: it depends on your system; how busy it is, what order the various jobs are scheduled by the OS, how long the command takes to execute, etc. It could work perfectly, or it could be run twice, or more times (if there are more than two targets). That's even worse. Thank you very much for this explanation. Bye... Dirk pgpb908DkGCP9.pgp Description: PGP signature ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
