Re: [HACKERS] DTrace build dependency rules
On Tue, Oct 13, 2015 at 3:47 PM, Robert Haas wrote: > On Tue, Aug 18, 2015 at 12:08 PM, Alvaro Herrera > wrote: >> Robert Haas wrote: >>> On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston wrote: >> >>> > The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated >>> > object file, depends on the objfiles.txt for each of the backend >>> > subdirs. These files depend in turn on the object files themselves; if >>> > objfiles.txt is out of date with respect to one of its object files, the >>> > mtime of objfiles.txt is updated with "touch" (see backend/common.mk). >>> > The problem is that dtrace -G, which runs at the end of the build, >>> > modifies a number of object files (it overwrites their probe sites with >>> > NOPs), thus making their corresponding objfiles.txt out of date. Then, >>> > when "make install" traverses the backend subdirs, it updates >>> > objfiles.txt, which causes probes.o to be rebuilt, resulting in an error >>> > from dtrace(1). >>> >>> Gosh, that's pretty ugly. I would have thought it would be a real >>> no-no to update the .o file once it got generated. If nothing else, a >>> modification to the .c file concurrent with a make invocation might >>> lead to the .o not getting rebuilt the next time make is run. >> >> I had the same thought, and wondered for a bit whether we should instead >> have the compilation rules produce some intermediate file (prior to >> dtrace fumbling), then emit the .o from dtrace -G. OTOH this might be >> more trouble than is worth for a feature that doesn't see a lot of use. > > Given the lack of further interest from the PostgreSQL community, > that's my guess. I've pushed this patch to master; let's see if we > get any complaints. If it makes life better for FreeBSD without > making life worse for anyone else, I suppose we might as well do it > until something better comes along. Per http://www.postgresql.org/message-id/24541.1444928...@sss.pgh.pa.us I have had to revert this patch, because it's breaking parallel builds even for non-dtrace users. Sorry for the inconvenience. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DTrace build dependency rules
On Tue, Aug 18, 2015 at 12:08 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston wrote: > >> > The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated >> > object file, depends on the objfiles.txt for each of the backend >> > subdirs. These files depend in turn on the object files themselves; if >> > objfiles.txt is out of date with respect to one of its object files, the >> > mtime of objfiles.txt is updated with "touch" (see backend/common.mk). >> > The problem is that dtrace -G, which runs at the end of the build, >> > modifies a number of object files (it overwrites their probe sites with >> > NOPs), thus making their corresponding objfiles.txt out of date. Then, >> > when "make install" traverses the backend subdirs, it updates >> > objfiles.txt, which causes probes.o to be rebuilt, resulting in an error >> > from dtrace(1). >> >> Gosh, that's pretty ugly. I would have thought it would be a real >> no-no to update the .o file once it got generated. If nothing else, a >> modification to the .c file concurrent with a make invocation might >> lead to the .o not getting rebuilt the next time make is run. > > I had the same thought, and wondered for a bit whether we should instead > have the compilation rules produce some intermediate file (prior to > dtrace fumbling), then emit the .o from dtrace -G. OTOH this might be > more trouble than is worth for a feature that doesn't see a lot of use. Given the lack of further interest from the PostgreSQL community, that's my guess. I've pushed this patch to master; let's see if we get any complaints. If it makes life better for FreeBSD without making life worse for anyone else, I suppose we might as well do it until something better comes along. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DTrace build dependency rules
Robert Haas wrote: > On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston wrote: > > The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated > > object file, depends on the objfiles.txt for each of the backend > > subdirs. These files depend in turn on the object files themselves; if > > objfiles.txt is out of date with respect to one of its object files, the > > mtime of objfiles.txt is updated with "touch" (see backend/common.mk). > > The problem is that dtrace -G, which runs at the end of the build, > > modifies a number of object files (it overwrites their probe sites with > > NOPs), thus making their corresponding objfiles.txt out of date. Then, > > when "make install" traverses the backend subdirs, it updates > > objfiles.txt, which causes probes.o to be rebuilt, resulting in an error > > from dtrace(1). > > Gosh, that's pretty ugly. I would have thought it would be a real > no-no to update the .o file once it got generated. If nothing else, a > modification to the .c file concurrent with a make invocation might > lead to the .o not getting rebuilt the next time make is run. I had the same thought, and wondered for a bit whether we should instead have the compilation rules produce some intermediate file (prior to dtrace fumbling), then emit the .o from dtrace -G. OTOH this might be more trouble than is worth for a feature that doesn't see a lot of use. > > The attached patch fixes the problem by having probes.o depend on object > > files directly, rather than on objfiles.txt. I've tested it with > > PostgreSQL 9.0-9.4 on FreeBSD CURRENT. > > I don't see a particular reason not to make this change, though. I wanted to test it on Linux yesterday but didn't get any further than installing a couple of packages. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DTrace build dependency rules
On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston wrote: > There seems to be a bug in the make rules when DTrace is enabled. It > causes dtrace -G to be invoked twice when building PostgreSQL as a > FreeBSD port: once during the build itself, and once during > installation. For a long time this has been worked around on FreeBSD > with a change to libdtrace itself, but this workaround is proving > problematic and I'd like to fix the problem properly. I'm not sure > whether the problem has been observed on other operating systems that > support DTrace. > > The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated > object file, depends on the objfiles.txt for each of the backend > subdirs. These files depend in turn on the object files themselves; if > objfiles.txt is out of date with respect to one of its object files, the > mtime of objfiles.txt is updated with "touch" (see backend/common.mk). > The problem is that dtrace -G, which runs at the end of the build, > modifies a number of object files (it overwrites their probe sites with > NOPs), thus making their corresponding objfiles.txt out of date. Then, > when "make install" traverses the backend subdirs, it updates > objfiles.txt, which causes probes.o to be rebuilt, resulting in an error > from dtrace(1). Gosh, that's pretty ugly. I would have thought it would be a real no-no to update the .o file once it got generated. If nothing else, a modification to the .c file concurrent with a make invocation might lead to the .o not getting rebuilt the next time make is run. > The attached patch fixes the problem by having probes.o depend on object > files directly, rather than on objfiles.txt. I've tested it with > PostgreSQL 9.0-9.4 on FreeBSD CURRENT. I don't see a particular reason not to make this change, though. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] DTrace build dependency rules
Hi, There seems to be a bug in the make rules when DTrace is enabled. It causes dtrace -G to be invoked twice when building PostgreSQL as a FreeBSD port: once during the build itself, and once during installation. For a long time this has been worked around on FreeBSD with a change to libdtrace itself, but this workaround is proving problematic and I'd like to fix the problem properly. I'm not sure whether the problem has been observed on other operating systems that support DTrace. The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated object file, depends on the objfiles.txt for each of the backend subdirs. These files depend in turn on the object files themselves; if objfiles.txt is out of date with respect to one of its object files, the mtime of objfiles.txt is updated with "touch" (see backend/common.mk). The problem is that dtrace -G, which runs at the end of the build, modifies a number of object files (it overwrites their probe sites with NOPs), thus making their corresponding objfiles.txt out of date. Then, when "make install" traverses the backend subdirs, it updates objfiles.txt, which causes probes.o to be rebuilt, resulting in an error from dtrace(1). The attached patch fixes the problem by having probes.o depend on object files directly, rather than on objfiles.txt. I've tested it with PostgreSQL 9.0-9.4 on FreeBSD CURRENT. Thanks! -Mark diff --git a/src/backend/Makefile b/src/backend/Makefile index 98b978f..b1e3969 100644 --- a/src/backend/Makefile +++ b/src/backend/Makefile @@ -180,8 +180,8 @@ $(top_builddir)/src/include/utils/probes.h: utils/probes.h $(LN_S) "../../../$(subdir)/utils/probes.h" . -utils/probes.o: utils/probes.d $(SUBDIROBJS) - $(DTRACE) $(DTRACEFLAGS) -C -G -s $(call expand_subsys,$^) -o $@ +utils/probes.o: utils/probes.d $(call expand_subsys,$(SUBDIROBJS)) + $(DTRACE) $(DTRACEFLAGS) -C -G -s $< $(call expand_subsys,$(SUBDIROBJS)) -o $@ ## -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers