Re: [HACKERS] build farm machine using make -j 8 mixed results
I wrote: And the answer is ... it's a gmake bug. Apparently introduced in 3.82. http://savannah.gnu.org/bugs/?30653 https://bugzilla.redhat.com/show_bug.cgi?id=835424 So I think .NOTPARALLEL is just masking the true problem, but nonetheless it's a problem. And given that the bug report on savannah has been ignored for two years, we should not hold our breath for a fix to appear upstream (much less propagate to everyone using 3.82). So no sooner do I complain about that, than the upstream maintainers wake up and commit it: http://lists.gnu.org/archive/html/bug-make/2012-09/msg00016.html No idea when a fixed release might appear, but at least somebody who knows the gmake code has signed off on the fix now. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/10/2012 02:44 PM, Tom Lane wrote: I wrote: And the answer is ... it's a gmake bug. Apparently introduced in 3.82. http://savannah.gnu.org/bugs/?30653 https://bugzilla.redhat.com/show_bug.cgi?id=835424 So I think .NOTPARALLEL is just masking the true problem, but nonetheless it's a problem. And given that the bug report on savannah has been ignored for two years, we should not hold our breath for a fix to appear upstream (much less propagate to everyone using 3.82). So no sooner do I complain about that, than the upstream maintainers wake up and commit it: http://lists.gnu.org/archive/html/bug-make/2012-09/msg00016.html No idea when a fixed release might appear, but at least somebody who knows the gmake code has signed off on the fix now. When it does appear in a release I guess we could make the .NOTPARALLEL conditional on make version. If not, we'll have to wait a long time before removing it. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Peter Eisentraut pete...@gmx.net writes: On Sat, 2012-09-08 at 19:54 -0400, Tom Lane wrote: Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, I'd hate that, because the ecpg build is one of the slowest parts of the build, so de-parallelizing it would slow down everything quite a bit. There's only one bit of it that's slow, which is the bison build + preproc.c compile, which is necessarily serial anyway. So I think trying to avoid .NOTPARALLEL there is a complete waste of effort. But if you wanna fix it some other way, step right up. (This all does remind me of the Recursive Make Considered Harmful thread from awhile back. Is anyone prepared to go down that path now? I'm not terribly excited by that, as yet --- yeah, it would be better, but the work involved seems out of proportion to the benefit.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/09/2012 03:29 AM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On Sat, 2012-09-08 at 19:54 -0400, Tom Lane wrote: Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, I'd hate that, because the ecpg build is one of the slowest parts of the build, so de-parallelizing it would slow down everything quite a bit. There's only one bit of it that's slow, which is the bison build + preproc.c compile, which is necessarily serial anyway. So I think trying to avoid .NOTPARALLEL there is a complete waste of effort. But if you wanna fix it some other way, step right up. Yeah. I am going to add a config parameter to the buildfarm to allow parallelism for the make and make contrib stages, but I'm not going to release it until this is fixed. (I suppose this is also a lesson to me that I should not ignore things like this that annoy me persistently as I did with these failures until Robert Creager started this discussion. It just didn't seem important enough.) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: On 09/09/2012 03:29 AM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On Sat, 2012-09-08 at 19:54 -0400, Tom Lane wrote: Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, I'd hate that, because the ecpg build is one of the slowest parts of the build, so de-parallelizing it would slow down everything quite a bit. There's only one bit of it that's slow, which is the bison build + preproc.c compile, which is necessarily serial anyway. So I think trying to avoid .NOTPARALLEL there is a complete waste of effort. But if you wanna fix it some other way, step right up. Yeah. I am going to add a config parameter to the buildfarm to allow parallelism for the make and make contrib stages, but I'm not going to release it until this is fixed. Well, why don't we stick .NOTPARALLEL in there for the moment, and then if Peter thinks of a better solution, he can revert that change in favor of something cleaner. I assume we need this for all active branches, if the buildfarm is going to be stressing it? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/09/2012 11:31 AM, Tom Lane wrote: Yeah. I am going to add a config parameter to the buildfarm to allow parallelism for the make and make contrib stages, but I'm not going to release it until this is fixed. Well, why don't we stick .NOTPARALLEL in there for the moment, and then if Peter thinks of a better solution, he can revert that change in favor of something cleaner. Works for me I assume we need this for all active branches, if the buildfarm is going to be stressing it? I can restrict it to only modern branches. Didn't we supposedly improve support for this during the 9.1 cycle? That seems like a sensible place to go back to. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: On 09/09/2012 11:31 AM, Tom Lane wrote: I assume we need this for all active branches, if the buildfarm is going to be stressing it? I can restrict it to only modern branches. Didn't we supposedly improve support for this during the 9.1 cycle? That seems like a sensible place to go back to. Mmm, yeah, the commit history shows several different fixes for parallel make back in the 9.1 time frame. I doubt we want to try back-porting all that work. Let's just say 9.1 and up, then. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
And the answer is ... it's a gmake bug. Apparently introduced in 3.82. http://savannah.gnu.org/bugs/?30653 https://bugzilla.redhat.com/show_bug.cgi?id=835424 So I think .NOTPARALLEL is just masking the true problem, but nonetheless it's a problem. And given that the bug report on savannah has been ignored for two years, we should not hold our breath for a fix to appear upstream (much less propagate to everyone using 3.82). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/09/2012 02:05 PM, Tom Lane wrote: And the answer is ... it's a gmake bug. Apparently introduced in 3.82. http://savannah.gnu.org/bugs/?30653 https://bugzilla.redhat.com/show_bug.cgi?id=835424 So I think .NOTPARALLEL is just masking the true problem, but nonetheless it's a problem. And given that the bug report on savannah has been ignored for two years, we should not hold our breath for a fix to appear upstream (much less propagate to everyone using 3.82). Thanks for pursuing this. Whether or not it masks the underlying problem, it's still something we should do, no? In fact, it seems to me like this makes it even less worth trying to do anything better. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: On 09/09/2012 02:05 PM, Tom Lane wrote: And the answer is ... it's a gmake bug. Thanks for pursuing this. Whether or not it masks the underlying problem, it's still something we should do, no? In fact, it seems to me like this makes it even less worth trying to do anything better. Yeah, exactly. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On Sun, 2012-09-09 at 14:05 -0400, Tom Lane wrote: And the answer is ... it's a gmake bug. Apparently introduced in 3.82. http://savannah.gnu.org/bugs/?30653 https://bugzilla.redhat.com/show_bug.cgi?id=835424 So I think .NOTPARALLEL is just masking the true problem, but nonetheless it's a problem. And given that the bug report on savannah has been ignored for two years, we should not hold our breath for a fix to appear upstream (much less propagate to everyone using 3.82). So does the issue go away when using GNU make 3.81? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On Sun, 2012-09-09 at 14:57 -0400, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: On 09/09/2012 02:05 PM, Tom Lane wrote: And the answer is ... it's a gmake bug. Thanks for pursuing this. Whether or not it masks the underlying problem, it's still something we should do, no? In fact, it seems to me like this makes it even less worth trying to do anything better. Yeah, exactly. But then the answer could be, if you want to use parallel make, use a version that's not broken. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Peter Eisentraut pete...@gmx.net writes: On Sun, 2012-09-09 at 14:05 -0400, Tom Lane wrote: So I think .NOTPARALLEL is just masking the true problem, but nonetheless it's a problem. And given that the bug report on savannah has been ignored for two years, we should not hold our breath for a fix to appear upstream (much less propagate to everyone using 3.82). So does the issue go away when using GNU make 3.81? I couldn't reproduce it in a few tries on my Mac laptop, which has 3.81. But that's not necessarily a comparable test, given the different hardware, OS, and compiler version. I'd put more faith in the multiple statements in the upstream bugs that people weren't seeing this with 3.81. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Peter Eisentraut pete...@gmx.net writes: But then the answer could be, if you want to use parallel make, use a version that's not broken. That's not a terribly practical answer for people who use the make supplied by their OS vendor, which is approximately 99.9% of people. It's even less practical for packagers, who don't have a choice about what tool set to use. Even if I wanted to use a locally-patched make, I'm not sure I'd trust a patch that doesn't seem to have been signed off on by any actual gmake developer or maintainer. That sort of cure is frequently worse than the disease. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/09/2012 05:00 PM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: But then the answer could be, if you want to use parallel make, use a version that's not broken. That's not a terribly practical answer for people who use the make supplied by their OS vendor, which is approximately 99.9% of people. It's even less practical for packagers, who don't have a choice about what tool set to use. Even if I wanted to use a locally-patched make, I'm not sure I'd trust a patch that doesn't seem to have been signed off on by any actual gmake developer or maintainer. That sort of cure is frequently worse than the disease. Right. Meanwhile, I have buildfarm animal crake building with the experimental buildfarm code, and so far the results look good. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: This seems totally stupid, but it happens when the path to the current directory includes a cross-device symlink. If I cd following the link, then this effect doesn't happen. Weird. Huh. So maybe a gmake bug, or maybe there's something wrong with our make rules for the case that `pwd` doesn't match what gmake thinks the current path is. Could you specify the setup more fully? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/08/2012 11:06 AM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: This seems totally stupid, but it happens when the path to the current directory includes a cross-device symlink. If I cd following the link, then this effect doesn't happen. Weird. Huh. So maybe a gmake bug, or maybe there's something wrong with our make rules for the case that `pwd` doesn't match what gmake thinks the current path is. Could you specify the setup more fully? Scratch that theory, that was just a transient. If anything it looks like it is related to system load. When almost nothing was running on the machine it worked fine. When I started up a Browser and an MUA the problem occurred. This VM has 4 CPUs and 4Gb of memory and a load average around 0.4 right now. I'm a bit perplexed. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: Scratch that theory, that was just a transient. If anything it looks like it is related to system load. When almost nothing was running on the machine it worked fine. When I started up a Browser and an MUA the problem occurred. This VM has 4 CPUs and 4Gb of memory and a load average around 0.4 right now. I'm a bit perplexed. Hm ... you weren't using the -l (--max-load) option were you? That would make system load affect gmake's scheduling. Although it's clear when I test it that it is waiting for the bison run to finish before launching the dependent builds, so it still seems like it must be a bug if your copy isn't waiting for that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: And it's the stock Fedora build of make. Which Fedora branch exactly? The package version I was trying to reproduce with here is make-3.82-8.fc16.x86_64. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/08/2012 04:54 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: And it's the stock Fedora build of make. Which Fedora branch exactly? The package version I was trying to reproduce with here is make-3.82-8.fc16.x86_64. Same: $ rpm -q make make-3.82-8.fc16.x86_64 cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/08/2012 04:46 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Scratch that theory, that was just a transient. If anything it looks like it is related to system load. When almost nothing was running on the machine it worked fine. When I started up a Browser and an MUA the problem occurred. This VM has 4 CPUs and 4Gb of memory and a load average around 0.4 right now. I'm a bit perplexed. Hm ... you weren't using the -l (--max-load) option were you? That would make system load affect gmake's scheduling. Although it's clear when I test it that it is waiting for the bison run to finish before launching the dependent builds, so it still seems like it must be a bug if your copy isn't waiting for that. No. just make -j 4 And it's the stock Fedora build of make. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/08/2012 04:52 PM, Andrew Dunstan wrote: On 09/08/2012 04:46 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Scratch that theory, that was just a transient. If anything it looks like it is related to system load. When almost nothing was running on the machine it worked fine. When I started up a Browser and an MUA the problem occurred. This VM has 4 CPUs and 4Gb of memory and a load average around 0.4 right now. I'm a bit perplexed. Hm ... you weren't using the -l (--max-load) option were you? That would make system load affect gmake's scheduling. Although it's clear when I test it that it is waiting for the bison run to finish before launching the dependent builds, so it still seems like it must be a bug if your copy isn't waiting for that. No. just make -j 4 And it's the stock Fedora build of make. I have just repeated this on an absolutely fresh up to date F17 machine, with no symlink stuff in play. Steps to recreate: CC=ccache gcc ../postgres/configure --enable-depend --enable-debug --enable-cassert --with-perl --with-python --with-tcl --with-libxml --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap make -j 4 cheers andrew q -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: I have just repeated this on an absolutely fresh up to date F17 machine, with no symlink stuff in play. Steps to recreate: CC=ccache gcc ../postgres/configure --enable-depend --enable-debug --enable-cassert --with-perl --with-python --with-tcl --with-libxml --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap make -j 4 Huh ... that recipe works (er, fails) for me too, at least some of the time. I wonder what exactly is the key difference between the working and failing cases? Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, because there are several reasons not to run its sub-makes in parallel: * preproc/Makefile casually does this: ../ecpglib/typename.o: ../ecpglib/typename.c $(MAKE) -C $(dir $@) $(notdir $@) which is very likely to screw up any make proceeding in parallel in ecpglib. * compatlib and ecpglib will equally casually invoke make all in other subdirectories; ditto. And that's not even counting the bison-output problem you were seeing. I'm not entirely sure what's causing that, but I'm suspicious that the ultimate cause is the extra rules for the all...recurse targets in ecpg/Makefile, which look like they could result in additional instances of multiple make processes running in the same subdirectory. After adding the .NOTPARALLEL marker, I don't see these failures anymore. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/08/2012 07:54 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: I have just repeated this on an absolutely fresh up to date F17 machine, with no symlink stuff in play. Steps to recreate: CC=ccache gcc ../postgres/configure --enable-depend --enable-debug --enable-cassert --with-perl --with-python --with-tcl --with-libxml --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap make -j 4 Huh ... that recipe works (er, fails) for me too, at least some of the time. I wonder what exactly is the key difference between the working and failing cases? Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, because there are several reasons not to run its sub-makes in parallel: * preproc/Makefile casually does this: ../ecpglib/typename.o: ../ecpglib/typename.c $(MAKE) -C $(dir $@) $(notdir $@) which is very likely to screw up any make proceeding in parallel in ecpglib. * compatlib and ecpglib will equally casually invoke make all in other subdirectories; ditto. And that's not even counting the bison-output problem you were seeing. I'm not entirely sure what's causing that, but I'm suspicious that the ultimate cause is the extra rules for the all...recurse targets in ecpg/Makefile, which look like they could result in additional instances of multiple make processes running in the same subdirectory. After adding the .NOTPARALLEL marker, I don't see these failures anymore. Well, I'm glad it's not just me. :-) This fix works for me too. I guess it should be applied back to 9.1, when it looks like we started using .NOTPARALLEL cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On Sat, 2012-09-08 at 19:54 -0400, Tom Lane wrote: Anyway, what I notice is that I get different types of failures, but they are all under ecpg/. What I think we need to do is insert .NOTPARALLEL in ecpg/Makefile, I'd hate that, because the ecpg build is one of the slowest parts of the build, so de-parallelizing it would slow down everything quite a bit. because there are several reasons not to run its sub-makes in parallel: * preproc/Makefile casually does this: ../ecpglib/typename.o: ../ecpglib/typename.c $(MAKE) -C $(dir $@) $(notdir $@) which is very likely to screw up any make proceeding in parallel in ecpglib. That should probably be fixed by symlinking the source file and building it in the preproc directory. And that's not even counting the bison-output problem you were seeing. I'm not entirely sure what's causing that, but I'm suspicious that the ultimate cause is the extra rules for the all...recurse targets in ecpg/Makefile, which look like they could result in additional instances of multiple make processes running in the same subdirectory. I think the point of these targets is exactly to prevent that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/04/2012 08:51 PM, Andrew Dunstan wrote: On 09/04/2012 08:37 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Frankly, I have had enough failures of parallel make that I think doing this would generate a significant number of non-repeatable failures (I had one just the other day that took three invocations of make to get right). So I'm not sure doing this would advance us much, although I'm open to persuasion. Really? I routinely use -j4 for building, and it's been a long time since I've seen failures. I can believe that for instance make check in contrib would have a problem running in parallel, but the build process per se seems reliable enough from here. Both cases were vpath builds, which is what I usually use, if that's a useful data point. Maybe I run on lower level hardware than you do. I saw this again this afternoon after I posted the above. In both cases this was the machine that runs the buildfarm's crake. I'll try to get a handle on it. Well, it looks like it's always failing on ecpg, with preproc.h not being made in the right order. Here is the last bit of a make log starting from when it starts on ecpg. This is pretty repeatable. cheers andrew - make -C ecpg all make[3]: Entering directory `/home/pgl/npgl/vpath.testpar/src/interfaces/ecpg' make -C include all make[4]: Entering directory `/home/pgl/npgl/vpath.testpar/src/interfaces/ecpg/include' make[4]: Nothing to be done for `all'. make[4]: Leaving directory `/home/pgl/npgl/vpath.testpar/src/interfaces/ecpg/include' make -C pgtypeslib all make -C preproc all make[4]: Entering directory `/home/pgl/npgl/vpath.testpar/src/interfaces/ecpg/pgtypeslib' ccache gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -pthread -D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS -fpic -I../include -I/home/andrew/pgl/pg_head/src/interfaces/ecpg/include -I/home/andrew/pgl/pg_head/src/include/utils -I/home/andrew/pgl/pg_head/src/interfaces/libpq -I../../../../src/include -I/home/andrew/pgl/pg_head/src/include -D_GNU_SOURCE -I/usr/include/libxml2 -DSO_MAJOR_VERSION=3 -c -o numeric.o /home/andrew/pgl/pg_head/src/interfaces/ecpg/pgtypeslib/numeric.c -MMD -MP -MF .deps/numeric.Po ccache gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -pthread -D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS -fpic -I../include -I/home/andrew/pgl/pg_head/src/interfaces/ecpg/include -I/home/andrew/pgl/pg_head/src/include/utils -I/home/andrew/pgl/pg_head/src/interfaces/libpq -I../../../../src/include -I/home/andrew/pgl/pg_head/src/include -D_GNU_SOURCE -I/usr/include/libxml2 -DSO_MAJOR_VERSION=3 -c -o common.o /home/andrew/pgl/pg_head/src/interfaces/ecpg/pgtypeslib/common.c -MMD -MP -MF .deps/common.Po ccache gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -pthread -D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS -fpic -I../include -I/home/andrew/pgl/pg_head/src/interfaces/ecpg/include -I/home/andrew/pgl/pg_head/src/include/utils -I/home/andrew/pgl/pg_head/src/interfaces/libpq -I../../../../src/include -I/home/andrew/pgl/pg_head/src/include -D_GNU_SOURCE -I/usr/include/libxml2 -DSO_MAJOR_VERSION=3 -c -o datetime.o /home/andrew/pgl/pg_head/src/interfaces/ecpg/pgtypeslib/datetime.c -MMD -MP -MF .deps/datetime.Po make[4]: Entering directory `/home/pgl/npgl/vpath.testpar/src/interfaces/ecpg/preproc' make -C ../../../../src/port all make[5]: Entering directory `/home/pgl/npgl/vpath.testpar/src/port' make -C ../backend submake-errcodes make[6]: Entering directory `/home/pgl/npgl/vpath.testpar/src/backend' make[6]: Nothing to be done for `submake-errcodes'. make[6]: Leaving directory `/home/pgl/npgl/vpath.testpar/src/backend' make[5]: Leaving directory `/home/pgl/npgl/vpath.testpar/src/port' '/usr/bin/perl' /home/andrew/pgl/pg_head/src/interfaces/ecpg/preproc/parse.pl /home/andrew/pgl/pg_head/src/interfaces/ecpg/preproc /home/andrew/pgl/pg_head/src/interfaces/ecpg/preproc/../../../backend/parser/gram.y preproc.y ccache gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -pthread -D_REENTRANT -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS -fpic -I../include -I/home/andrew/pgl/pg_head/src/interfaces/ecpg/include -I/home/andrew/pgl/pg_head/src/include/utils -I/home/andrew/pgl/pg_head/src/interfaces/libpq -I../../../../src/include -I/home/andrew/pgl/pg_head/src/include
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: Well, it looks like it's always failing on ecpg, with preproc.h not being made in the right order. Here is the last bit of a make log starting from when it starts on ecpg. This is pretty repeatable. Hmph. I can't reproduce it at all on my Fedora 16 box. What version of make are you using? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/07/2012 08:43 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Well, it looks like it's always failing on ecpg, with preproc.h not being made in the right order. Here is the last bit of a make log starting from when it starts on ecpg. This is pretty repeatable. Hmph. I can't reproduce it at all on my Fedora 16 box. What version of make are you using? $ make -v GNU Make 3.82 Built for x86_64-redhat-linux-gnu cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/07/2012 09:55 PM, Andrew Dunstan wrote: On 09/07/2012 08:43 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Well, it looks like it's always failing on ecpg, with preproc.h not being made in the right order. Here is the last bit of a make log starting from when it starts on ecpg. This is pretty repeatable. Hmph. I can't reproduce it at all on my Fedora 16 box. What version of make are you using? $ make -v GNU Make 3.82 Built for x86_64-redhat-linux-gnu OK, I just tried on a different F16 machine and it didn't happen. I wonder what's different. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/07/2012 10:46 PM, Andrew Dunstan wrote: On 09/07/2012 09:55 PM, Andrew Dunstan wrote: On 09/07/2012 08:43 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Well, it looks like it's always failing on ecpg, with preproc.h not being made in the right order. Here is the last bit of a make log starting from when it starts on ecpg. This is pretty repeatable. Hmph. I can't reproduce it at all on my Fedora 16 box. What version of make are you using? $ make -v GNU Make 3.82 Built for x86_64-redhat-linux-gnu OK, I just tried on a different F16 machine and it didn't happen. I wonder what's different. This seems totally stupid, but it happens when the path to the current directory includes a cross-device symlink. If I cd following the link, then this effect doesn't happen. Weird. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] build farm machine using make -j 8 mixed results
I change the build-farm.conf file to have the following make line: make = 'make -j 8', # or gmake if required. can include path if necessary. 2 pass, 4 fail. Is this a build configuration you want to pursue? I can either create a new machine, or change one of my existing machines. Makes no difference to me. roberts-imac:build-farm-4.7-j robert$ time ./run_branches.pl --run-all --test Sat Sep 1 09:26:49 2012: buildfarm run for CHANGEME:HEAD starting [09:26:49] checking out source ... [09:27:59] checking if build run needed ... [09:27:59] copying source to pgsql.54279 ... [09:28:03] running configure ... [09:28:25] running make ... [09:28:45] running make check ... Branch: HEAD Stage Check failed with status 2 Sat Sep 1 09:29:09 2012: buildfarm run for CHANGEME:REL9_2_STABLE starting [09:29:09] checking out source ... [09:29:57] checking if build run needed ... [09:29:57] copying source to pgsql.70926 ... [09:29:58] running configure ... [09:30:18] running make ... [09:30:45] running make check ... [09:31:09] running make contrib ... [09:31:12] running make install ... [09:31:15] running make contrib install ... [09:31:16] setting up db cluster (C)... [09:31:19] starting db (C)... [09:31:20] running make installcheck (C)... [09:31:39] restarting db (C)... [09:31:51] running make isolation check ... [09:32:00] restarting db (C)... [09:32:12] running make PL installcheck (C)... Branch: REL9_2_STABLE Stage PLCheck-C failed with status 2 Sat Sep 1 09:32:29 2012: buildfarm run for CHANGEME:REL9_1_STABLE starting [09:32:29] checking out source ... [09:33:08] checking if build run needed ... [09:33:08] copying source to pgsql.94883 ... [09:33:10] running configure ... [09:33:30] running make ... [09:33:54] running make check ... [09:34:18] running make contrib ... [09:34:21] running make install ... [09:34:23] running make contrib install ... [09:34:24] setting up db cluster (C)... [09:34:26] starting db (C)... [09:34:27] running make installcheck (C)... [09:34:47] restarting db (C)... [09:34:59] running make isolation check ... [09:35:16] restarting db (C)... [09:35:28] running make PL installcheck (C)... Branch: REL9_1_STABLE Stage PLCheck-C failed with status 2 Sat Sep 1 09:35:46 2012: buildfarm run for CHANGEME:REL9_0_STABLE starting [09:35:46] checking out source ... [09:36:08] checking if build run needed ... [09:36:08] copying source to pgsql.18851 ... [09:36:10] running configure ... [09:36:28] running make ... [09:37:00] running make check ... [09:37:23] running make contrib ... [09:37:27] running make install ... [09:37:30] running make contrib install ... [09:37:32] setting up db cluster (C)... [09:37:34] starting db (C)... [09:37:35] running make installcheck (C)... [09:37:52] restarting db (C)... [09:38:04] running make PL installcheck (C)... [09:38:06] restarting db (C)... [09:38:18] running make contrib installcheck (C)... [09:38:30] stopping db (C)... [09:38:31] running make ecpg check ... [09:38:43] OK Branch: REL9_0_STABLE All stages succeeded Sat Sep 1 09:38:43 2012: buildfarm run for CHANGEME:REL8_4_STABLE starting [09:38:43] checking out source ... [09:38:57] checking if build run needed ... [09:38:57] copying source to pgsql.45071 ... [09:38:59] running configure ... [09:39:19] running make ... [09:39:46] running make check ... [09:40:14] running make contrib ... [09:40:17] running make install ... [09:40:23] running make contrib install ... [09:40:25] setting up db cluster (C)... [09:40:30] starting db (C)... [09:40:31] running make installcheck (C)... [09:40:47] restarting db (C)... [09:40:59] running make PL installcheck (C)... [09:41:01] restarting db (C)... [09:41:13] running make contrib installcheck (C)... [09:41:25] stopping db (C)... [09:41:26] running make ecpg check ... [09:41:44] OK Branch: REL8_4_STABLE All stages succeeded Sat Sep 1 09:41:44 2012: buildfarm run for CHANGEME:REL8_3_STABLE starting [09:41:44] checking out source ... [09:42:39] checking if build run needed ... [09:42:39] copying source to pgsql.80222 ... [09:42:42] running configure ... [09:43:03] running make ... [09:43:38] running make check ... [09:44:04] running make contrib ... [09:44:09] running make install ... [09:44:14] running make contrib install ... [09:44:17] setting up db cluster (C)... [09:44:20] starting db (C)... [09:44:21] running make installcheck (C)... [09:44:39] restarting db (C)... [09:44:51] running make PL installcheck (C)... [09:44:52] restarting db (C)... [09:45:05] running make contrib installcheck (C)... Branch: REL8_3_STABLE Stage ContribCheck-C failed with status 2
Re: [HACKERS] build farm machine using make -j 8 mixed results
Excerpts from Robert Creager's message of sáb sep 01 12:12:51 -0400 2012: I change the build-farm.conf file to have the following make line: make = 'make -j 8', # or gmake if required. can include path if necessary. 2 pass, 4 fail. Is this a build configuration you want to pursue? Sure, why not? Parallel building is going to become more common, so it's a good idea to investigate the failures. I would have it build only HEAD though, because we're not likely to backpatch these fixes. I can either create a new machine, or change one of my existing machines. Makes no difference to me. If we want to have it run only HEAD I would say you should create a new animal. Perhaps you should wait until after 9.2 has been released, though, to avoid distracting people :-) -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 9/1/12 12:12 PM, Robert Creager wrote: I change the build-farm.conf file to have the following make line: make = 'make -j 8', # or gmake if required. can include path if necessary. 2 pass, 4 fail. Is this a build configuration you want to pursue? Sure that would be useful, but it's pretty clear that the check stages don't work in parallel. It think it's because the ports conflict, but there could be any number of other problems. That said, it would be useful, in my mind, to support parallel checks. But unless someone is going to put in the work first, you should restrict your parallel runs to the build and install phases. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Excerpts from Peter Eisentraut's message of mar sep 04 18:49:46 -0300 2012: On 9/1/12 12:12 PM, Robert Creager wrote: I change the build-farm.conf file to have the following make line: make = 'make -j 8', # or gmake if required. can include path if necessary. 2 pass, 4 fail. Is this a build configuration you want to pursue? Sure that would be useful, but it's pretty clear that the check stages don't work in parallel. It think it's because the ports conflict, but there could be any number of other problems. Is that really the problem? As far as I know, buildfarm doesn't use anything like installcheck-world or similar targets; each check target is run separately, serially, by the BF script. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/04/2012 05:49 PM, Peter Eisentraut wrote: On 9/1/12 12:12 PM, Robert Creager wrote: I change the build-farm.conf file to have the following make line: make = 'make -j 8', # or gmake if required. can include path if necessary. 2 pass, 4 fail. Is this a build configuration you want to pursue? Sure that would be useful, but it's pretty clear that the check stages don't work in parallel. It think it's because the ports conflict, but there could be any number of other problems. That said, it would be useful, in my mind, to support parallel checks. But unless someone is going to put in the work first, you should restrict your parallel runs to the build and install phases. The buildfarm code doesn't contain a facility to use a different make incantation for each step. It's pretty much an all or nothing deal. Of course, you can hack run_build.pl to make it do that, but I highly discourage that. For one thing, it makes upgrading that much more difficult. All the tweaking is supposed to be done vie the config file. I guess I could add a setting that allowed for per step make flags. Frankly, I have had enough failures of parallel make that I think doing this would generate a significant number of non-repeatable failures (I had one just the other day that took three invocations of make to get right). So I'm not sure doing this would advance us much, although I'm open to persuasion. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On Sep 4, 2012 6:06 PM, Andrew Dunstan and...@dunslane.net wrote: Frankly, I have had enough failures of parallel make that I think doing this would generate a significant number of non-repeatable failures (I had one just the other day that took three invocations of make to get right). So I'm not sure doing this would advance us much, although I'm open to persuasion. Seeing as most PostgreSQL bugs appear with concurrency, I think we should leave our default config with 1 for max connections. ;-) Parallel make failures are bugs in the dependencies as described in our make files. For the build phase, I don't recall parallel problems and as a habit I usually use parallel makes. I would like that to be supported and I think I've seen fixes applied when we had issues before. Cutting build times to 1/2 to 1/4 is good. Checks and tests are harder because often they can't run in parallel. But then we shouldn't have them listed as independent prerequisites for targets. Ideally. But fixing it might not be worth the cost since an acceptable work around (rely upon make to serialize the test sequences in the particular order) is pretty painless (so far). Of course, having the ability to run the tests 8 at a time (or more) and reduce the time by 80% would be nice .;-) On Sep 4, 2012 6:06 PM, Andrew Dunstan and...@dunslane.net wrote: On 09/04/2012 05:49 PM, Peter Eisentraut wrote: On 9/1/12 12:12 PM, Robert Creager wrote: I change the build-farm.conf file to have the following make line: make = 'make -j 8', # or gmake if required. can include path if necessary. 2 pass, 4 fail. Is this a build configuration you want to pursue? Sure that would be useful, but it's pretty clear that the check stages don't work in parallel. It think it's because the ports conflict, but there could be any number of other problems. That said, it would be useful, in my mind, to support parallel checks. But unless someone is going to put in the work first, you should restrict your parallel runs to the build and install phases. The buildfarm code doesn't contain a facility to use a different make incantation for each step. It's pretty much an all or nothing deal. Of course, you can hack run_build.pl to make it do that, but I highly discourage that. For one thing, it makes upgrading that much more difficult. All the tweaking is supposed to be done vie the config file. I guess I could add a setting that allowed for per step make flags. Frankly, I have had enough failures of parallel make that I think doing this would generate a significant number of non-repeatable failures (I had one just the other day that took three invocations of make to get right). So I'm not sure doing this would advance us much, although I'm open to persuasion. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/**mailpref/pgsql-hackershttp://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
Andrew Dunstan and...@dunslane.net writes: Frankly, I have had enough failures of parallel make that I think doing this would generate a significant number of non-repeatable failures (I had one just the other day that took three invocations of make to get right). So I'm not sure doing this would advance us much, although I'm open to persuasion. Really? I routinely use -j4 for building, and it's been a long time since I've seen failures. I can believe that for instance make check in contrib would have a problem running in parallel, but the build process per se seems reliable enough from here. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] build farm machine using make -j 8 mixed results
On 09/04/2012 08:37 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Frankly, I have had enough failures of parallel make that I think doing this would generate a significant number of non-repeatable failures (I had one just the other day that took three invocations of make to get right). So I'm not sure doing this would advance us much, although I'm open to persuasion. Really? I routinely use -j4 for building, and it's been a long time since I've seen failures. I can believe that for instance make check in contrib would have a problem running in parallel, but the build process per se seems reliable enough from here. Both cases were vpath builds, which is what I usually use, if that's a useful data point. Maybe I run on lower level hardware than you do. I saw this again this afternoon after I posted the above. In both cases this was the machine that runs the buildfarm's crake. I'll try to get a handle on it. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
