Re: [HACKERS] static or dynamic libpgport
On 12/12/2011 02:59 PM, Tom Lane wrote: Peter Eisentrautpete...@gmx.net writes: On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote: Right now, libpq laboriously rebuilds all the .o files it needs from src/port/ so as to get them with -fpic. It would be nice if we could clean that up while we're doing this. It might be all right to always build the client-side version of libpgport with -fpic, though I'd be sad if that leaked into the server-side build. So would we continue to build the client binaries (psql, pg_dump, etc.) against the static libpgport.a, thus keeping it invisible there, or would we dynamically link them, thus creating a new dependency. I think that if possible we should avoid creating a new dependency for either the client binaries or libpq.so itself; what I suggest above is only a simplification of the build process for libpq. If we create a new dependency we risk packagers breaking things by forgetting to include it. The Fedora/RHEL rule against static libraries is meant to prevent situations where changes in a library would require rebuilding other packages to get the fixes in place. If we had to make a quick security fix in libpq, for example, it would suck if dozens of other packages had to be rebuilt to propagate the change everywhere. However, I don't think that concern applies to programs that are in the same source package as the library --- they'd get rebuilt anyway. So I see nothing wrong with continuing to statically link these .o files into files belonging to the postgresql package. It's just that I can't export them in a .a file for *other* source packages to use. (Whether a security issue in libpgport is really likely to happen is not a question that this policy concerns itself with ...) OK, my possibly naïve approach is shown in the attached. Essentially it builds libpgport-shared.so and then installs it as libpgport.so. That ensures that the library is not used in building any postgres binaries or libraries. Places such as libpq that formerly symlinked and recompiled the sources in a way that is suitable for a shared library now just link in the already built object files. Is there a better way to do this? cheers andrew *** src/port/Makefile 2011-12-03 17:21:59.944509111 -0500 --- src/port/GNUmakefile 2011-12-12 22:32:50.875312294 -0500 *** *** 37,47 # foo_srv.o and foo.o are both built from foo.c, but only foo.o has -DFRONTEND OBJS_SRV = $(OBJS:%.o=%_srv.o) ! all: libpgport.a libpgport_srv.a ! # libpgport is needed by some contrib install: all installdirs ! $(INSTALL_STLIB) libpgport.a '$(DESTDIR)$(libdir)/libpgport.a' installdirs: $(MKDIR_P) '$(DESTDIR)$(libdir)' --- 37,55 # foo_srv.o and foo.o are both built from foo.c, but only foo.o has -DFRONTEND OBJS_SRV = $(OBJS:%.o=%_srv.o) ! NAME = pgport-shared ! SO_MAJOR_VERSION= 1 ! SO_MINOR_VERSION= 1 ! include $(top_srcdir)/src/Makefile.shlib ! ! all: all-lib libpgport_srv.a ! rm -f libpgport.a $(LN_S) libpgport-shared.a libpgport.a ! ! # libpgport is needed by any exe built with pgxs install: all installdirs ! rm -f libpgport.so* $(LN_S) libpgport-shared.so.$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION) libpgport.so.$(SO_MAJOR_VERSION).$(SO_MINOR_VERSION) ! $(MAKE) -f GNUmakefile.installshared install installdirs: $(MKDIR_P) '$(DESTDIR)$(libdir)' *** *** 49,57 uninstall: rm -f '$(DESTDIR)$(libdir)/libpgport.a' - libpgport.a: $(OBJS) - $(AR) $(AROPT) $@ $^ - # thread.o needs PTHREAD_CFLAGS (but thread_srv.o does not) thread.o: thread.c $(CC) $(CFLAGS) $(CPPFLAGS) $(PTHREAD_CFLAGS) -c $ --- 57,62 *** *** 64,70 $(AR) $(AROPT) $@ $^ %_srv.o: %.c ! $(CC) $(CFLAGS) $(subst -DFRONTEND,, $(CPPFLAGS)) -c $ -o $@ $(OBJS_SRV): | submake-errcodes --- 69,75 $(AR) $(AROPT) $@ $^ %_srv.o: %.c ! $(CC) $(subst $(CFLAGS_SL),,$(CFLAGS)) $(subst -DFRONTEND,, $(CPPFLAGS)) -c $ -o $@ $(OBJS_SRV): | submake-errcodes *** *** 97,100 echo #define MANDIR \$(mandir)\ $@ clean distclean maintainer-clean: ! rm -f libpgport.a libpgport_srv.a $(OBJS) $(OBJS_SRV) pg_config_paths.h --- 102,105 echo #define MANDIR \$(mandir)\ $@ clean distclean maintainer-clean: ! rm -f libpgport.so* libpgport.a libpgport_srv.a $(OBJS) $(OBJS_SRV) pg_config_paths.h *** /dev/null 2011-12-12 18:04:00.41099 -0500 --- src/port/GNUmakefile.installshared 2011-12-12 22:29:29.508512245 -0500 *** *** 0 --- 1,11 + subdir = src/port + top_builddir = ../.. + include $(top_builddir)/src/Makefile.global + + NAME = pgport + SO_MAJOR_VERSION= 1 + SO_MINOR_VERSION= 1 + + include $(top_srcdir)/src/Makefile.shlib + + install: install-lib *** src/interfaces/libpq/Makefile 2011-11-02 18:40:03.040342172 -0400 --- src/interfaces/libpq/GNUmakefile 2011-12-13 15:42:16.477592793 -0500 *** *** 88,94 # For some libpgport
Re: [HACKERS] static or dynamic libpgport
On 12/10/2011 08:26 PM, Tom Lane wrote: The other thing is we'd need to turn on flags that make the object suitable for a dynamic library (e.g. -fpic). Right now, libpq laboriously rebuilds all the .o files it needs from src/port/ so as to get them with -fpic. It would be nice if we could clean that up while we're doing this. It might be all right to always build the client-side version of libpgport with -fpic, though I'd be sad if that leaked into the server-side build. Here's a small diff that seems to build things the right way. No leakage of -fpic into the server side code. Still a deal of work to do, but it's a start. Would we want to link our own non-backend executables against the shared lib? That would almost certainly break the buildfarm for Windows builds, as it only currently copies the libpq DLL into the bin directory. That's no reason on its own not to do it, of course, and there are only a couple of owners other than me anyway, so it would be easy to fix. How do you want to proceed for libpq (and the ecpg library cases that do the same thing)? Just link in the object files directly? cheers andrew *** Makefile2011-12-03 17:21:59.944509111 -0500 --- GNUmakefile2011-12-12 12:39:43.176260505 -0500 *** *** 37,47 # foo_srv.o and foo.o are both built from foo.c, but only foo.o has -DFRONTEND OBJS_SRV = $(OBJS:%.o=%_srv.o) ! all: libpgport.a libpgport_srv.a ! # libpgport is needed by some contrib ! install: all installdirs ! $(INSTALL_STLIB) libpgport.a '$(DESTDIR)$(libdir)/libpgport.a' installdirs: $(MKDIR_P) '$(DESTDIR)$(libdir)' --- 37,52 # foo_srv.o and foo.o are both built from foo.c, but only foo.o has -DFRONTEND OBJS_SRV = $(OBJS:%.o=%_srv.o) ! NAME = pgport ! SO_MAJOR_VERSION= 1 ! SO_MINOR_VERSION= 1 ! include $(top_srcdir)/src/Makefile.shlib ! ! all: all-lib libpgport_srv.a ! ! # libpgport is needed by any exe built with pgxs ! install: all installdirs install-lib installdirs: $(MKDIR_P) '$(DESTDIR)$(libdir)' *** *** 49,57 uninstall: rm -f '$(DESTDIR)$(libdir)/libpgport.a' - libpgport.a: $(OBJS) - $(AR) $(AROPT) $@ $^ - # thread.o needs PTHREAD_CFLAGS (but thread_srv.o does not) thread.o: thread.c $(CC) $(CFLAGS) $(CPPFLAGS) $(PTHREAD_CFLAGS) -c $ --- 54,59 *** *** 64,70 $(AR) $(AROPT) $@ $^ %_srv.o: %.c ! $(CC) $(CFLAGS) $(subst -DFRONTEND,, $(CPPFLAGS)) -c $ -o $@ $(OBJS_SRV): | submake-errcodes --- 66,72 $(AR) $(AROPT) $@ $^ %_srv.o: %.c ! $(CC) $(subst $(CFLAGS_SL),,$(CFLAGS)) $(subst -DFRONTEND,, $(CPPFLAGS)) -c $ -o $@ $(OBJS_SRV): | submake-errcodes *** *** 97,100 echo #define MANDIR \$(mandir)\ $@ clean distclean maintainer-clean: ! rm -f libpgport.a libpgport_srv.a $(OBJS) $(OBJS_SRV) pg_config_paths.h --- 99,102 echo #define MANDIR \$(mandir)\ $@ clean distclean maintainer-clean: ! rm -f libpgport.so* libpgport.a libpgport_srv.a $(OBJS) $(OBJS_SRV) pg_config_paths.h -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote: The other thing is we'd need to turn on flags that make the object suitable for a dynamic library (e.g. -fpic). Right now, libpq laboriously rebuilds all the .o files it needs from src/port/ so as to get them with -fpic. It would be nice if we could clean that up while we're doing this. It might be all right to always build the client-side version of libpgport with -fpic, though I'd be sad if that leaked into the server-side build. So would we continue to build the client binaries (psql, pg_dump, etc.) against the static libpgport.a, thus keeping it invisible there, or would we dynamically link them, thus creating a new dependency. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
On fre, 2011-12-09 at 11:13 -0500, Andrew Dunstan wrote: Is there any good reason why we shouldn't build and install a dynamic libpgport.so? Just note, if you do this, you need to carefully manage API, ABI, soname, symbol list, and all that. Every time you tweak configure's decision about when to include a replacement function, you need to change the library version. Every time you remove a function, you need to change the soname. Every backpatched portability fix has the potential to escalate to a full shared library versioning dance. Downstream packagers will be delighted, especially if this requires changing the package name every three minor releases. To see what this can lead to in the extreme, check the dependencies that bind has on its internal libraries: bind9 depends: libbind9-60, libdns69, libisc62, libisccc60, libisccfg62, liblwres60 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
Peter Eisentraut pete...@gmx.net writes: On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote: Right now, libpq laboriously rebuilds all the .o files it needs from src/port/ so as to get them with -fpic. It would be nice if we could clean that up while we're doing this. It might be all right to always build the client-side version of libpgport with -fpic, though I'd be sad if that leaked into the server-side build. So would we continue to build the client binaries (psql, pg_dump, etc.) against the static libpgport.a, thus keeping it invisible there, or would we dynamically link them, thus creating a new dependency. I think that if possible we should avoid creating a new dependency for either the client binaries or libpq.so itself; what I suggest above is only a simplification of the build process for libpq. If we create a new dependency we risk packagers breaking things by forgetting to include it. The Fedora/RHEL rule against static libraries is meant to prevent situations where changes in a library would require rebuilding other packages to get the fixes in place. If we had to make a quick security fix in libpq, for example, it would suck if dozens of other packages had to be rebuilt to propagate the change everywhere. However, I don't think that concern applies to programs that are in the same source package as the library --- they'd get rebuilt anyway. So I see nothing wrong with continuing to statically link these .o files into files belonging to the postgresql package. It's just that I can't export them in a .a file for *other* source packages to use. (Whether a security issue in libpgport is really likely to happen is not a question that this policy concerns itself with ...) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
On 12/12/2011 02:59 PM, Tom Lane wrote: Peter Eisentrautpete...@gmx.net writes: On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote: Right now, libpq laboriously rebuilds all the .o files it needs from src/port/ so as to get them with -fpic. It would be nice if we could clean that up while we're doing this. It might be all right to always build the client-side version of libpgport with -fpic, though I'd be sad if that leaked into the server-side build. So would we continue to build the client binaries (psql, pg_dump, etc.) against the static libpgport.a, thus keeping it invisible there, or would we dynamically link them, thus creating a new dependency. I think that if possible we should avoid creating a new dependency for either the client binaries or libpq.so itself; what I suggest above is only a simplification of the build process for libpq. If we create a new dependency we risk packagers breaking things by forgetting to include it. OK, I'll work on this basis. The downside is that we'll be building it but not using it, but I can see the advantages. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
On 12/09/2011 06:27 PM, Bruce Momjian wrote:
In the Fedora world, a static lib would go in postgresql-devel, but a
dynamic lib would go in postgresql-libs, which is also where libpq is
shipped.
I am not against shipping a dynamic libpgport, but I will just point out
that this was never intended or anticipated. Are there any symbols in
there that might conflict with other software?
Possibly. Below is a list of symbols from a recent build. The other
thing is we'd need to turn on flags that make the object suitable for a
dynamic library (e.g. -fpic). I'm not sure if that has any significant
impact - probably not or else people would avoid using them far more.
I think we should deal with this. Just Peter's and Steve's responses
upthread indicate a demand, and I came into this because a customer
encountered enormous difficulty in doing an out of tree build of a well
known piece of support software, so this is clearly a pain point. If
we're not going to do it, we should probably think about adjusting
people's expectations.
[andrew@diego port]$ nm libpgport.a | grep ' T '
T strlcat
T strlcpy
T getpeereid
T pg_get_encoding_from_locale
T pgfnames
01a0 T pgfnames_cleanup
01e0 T rmtree
0240 T find_my_exec
05f0 T find_other_exec
0500 T pclose_check
0760 T set_pglocale_pgservice
T inet_net_ntop
0030 T pg_set_block
T pg_set_noblock
0280 T canonicalize_path
0110 T first_dir_separator
0140 T first_path_var_separator
0800 T get_doc_path
0720 T get_etc_path
0860 T get_home_path
0820 T get_html_path
0740 T get_include_path
0780 T get_includeserver_path
07a0 T get_lib_path
07e0 T get_locale_path
0840 T get_man_path
08e0 T get_parent_directory
0760 T get_pkginclude_path
07c0 T get_pkglib_path
0690 T get_progname
0700 T get_share_path
01b0 T join_path_components
0170 T last_dir_separator
01a0 T make_native_path
0560 T path_contains_parent_reference
0630 T path_is_prefix_of_path
0610 T path_is_relative_and_below_cwd
T pg_check_dir
T pg_mkdir_p
T pg_usleep
0330 T pg_ascii_tolower
0310 T pg_ascii_toupper
T pg_strcasecmp
0100 T pg_strncasecmp
0290 T pg_tolower
0210 T pg_toupper
00d0 T pg_qsort
00f0 T qsort_arg
T simple_prompt
0010 T pqGetpwuid
T pqStrerror
cheers
andrew
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
Andrew Dunstan and...@dunslane.net writes: On 12/09/2011 06:27 PM, Bruce Momjian wrote: I am not against shipping a dynamic libpgport, but I will just point out that this was never intended or anticipated. Are there any symbols in there that might conflict with other software? Possibly. Below is a list of symbols from a recent build. This doesn't seem like much of an issue to me, since anything wanting to link against libpgport would be designed to work with whatever it provides, no? The other thing is we'd need to turn on flags that make the object suitable for a dynamic library (e.g. -fpic). Right now, libpq laboriously rebuilds all the .o files it needs from src/port/ so as to get them with -fpic. It would be nice if we could clean that up while we're doing this. It might be all right to always build the client-side version of libpgport with -fpic, though I'd be sad if that leaked into the server-side build. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] static or dynamic libpgport
Recently I attempted to build an external package (pg_bulkload) against the latest Fedora packages. Unfortunately this fails, as pgxs adds -lpgport to any link line for an executable, and the corresponding libpgport.a isn't there. And in fact, pg_bulkload does use some of the functionality there (e.g. pg_strncasecmp), so just stripping -lpgport out doesn't work either. This happened because Fedora packaging guidelines http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries are strongly against shipping static libraries, and so all the PostgreSQL static libraries are excluded from the distribution (and I believe there are similar restrictions for RHEL). Of these libraries, I believe the only one that is *only* built as a static library is libpgport. Is there any good reason why we shouldn't build and install a dynamic libpgport.so? (Of course, you could say use the community RPMs, but that would be a bit of a cop out. Some organizations have a perfectly reasonable policy or requiring use of vendor packages wherever possible, since vendors are naturally only going to support packages they provide. So either we should be arguing to the Fedora/RedHat people that they should ship the static library, or we should be providing them with a dynamic one, ISTM.) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
On 9 December 2011 16:13, Andrew Dunstan and...@dunslane.net wrote: Is there any good reason why we shouldn't build and install a dynamic libpgport.so? +1 in favour of building and installing a dynamic libpgport.so. I generally agree with your analysis. I've seen this issue crop up a good few times now. I'm a Fedora user myself, but about 2 years ago I got into a he said she said situation with an OpenSUSE package maintainer over this, when I had to build Slony on that platform. I'm a bit hazy on the details now, but iirc he thought that it wasn't necessary to ship libpgport.a in particular (though I don't think that they have a beef with static libraries generally) - maybe they took a cue from Redhat there? -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
On 11-12-09 11:13 AM, Andrew Dunstan wrote: Recently I attempted to build an external package (pg_bulkload) against the latest Fedora packages. Unfortunately this fails, as pgxs adds -lpgport to any link line for an executable, and the corresponding libpgport.a isn't there. And in fact, pg_bulkload does use some of the functionality there (e.g. pg_strncasecmp), so just stripping -lpgport out doesn't work either. This happened because Fedora packaging guidelines http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries are strongly against shipping static libraries, and so all the PostgreSQL static libraries are excluded from the distribution (and I believe there are similar restrictions for RHEL). Of these libraries, I believe the only one that is *only* built as a static library is libpgport. Is there any good reason why we shouldn't build and install a dynamic libpgport.so? +1 We've struggled with slony and pgport because so many users have had problems with pgport not being included in some distributions. It has some useful functions, I think recent versions of slony use it on win32 but don't elsewhere. Wee have had at least one patch floating around that makes conditionally includes certain small behaviours in slony based on if pgport is available or not based on a configure check. What package would a shared static pgport be installed with? Slony requires a server + headers to build but slon and slonik only have a runtime dependency on libpq (I don't know if anyone installs slon/slonik on a machine without a postgresql server but you could) Steve cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
On 12/09/2011 01:01 PM, Steve Singer wrote: On 11-12-09 11:13 AM, Andrew Dunstan wrote: Recently I attempted to build an external package (pg_bulkload) against the latest Fedora packages. Unfortunately this fails, as pgxs adds -lpgport to any link line for an executable, and the corresponding libpgport.a isn't there. And in fact, pg_bulkload does use some of the functionality there (e.g. pg_strncasecmp), so just stripping -lpgport out doesn't work either. This happened because Fedora packaging guidelines http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries are strongly against shipping static libraries, and so all the PostgreSQL static libraries are excluded from the distribution (and I believe there are similar restrictions for RHEL). Of these libraries, I believe the only one that is *only* built as a static library is libpgport. Is there any good reason why we shouldn't build and install a dynamic libpgport.so? +1 We've struggled with slony and pgport because so many users have had problems with pgport not being included in some distributions. It has some useful functions, I think recent versions of slony use it on win32 but don't elsewhere. Wee have had at least one patch floating around that makes conditionally includes certain small behaviours in slony based on if pgport is available or not based on a configure check. What package would a shared static pgport be installed with? Slony requires a server + headers to build but slon and slonik only have a runtime dependency on libpq (I don't know if anyone installs slon/slonik on a machine without a postgresql server but you could) In the Fedora world, a static lib would go in postgresql-devel, but a dynamic lib would go in postgresql-libs, which is also where libpq is shipped. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] static or dynamic libpgport
Andrew Dunstan wrote: Is there any good reason why we shouldn't build and install a dynamic libpgport.so? +1 We've struggled with slony and pgport because so many users have had problems with pgport not being included in some distributions. It has some useful functions, I think recent versions of slony use it on win32 but don't elsewhere. Wee have had at least one patch floating around that makes conditionally includes certain small behaviours in slony based on if pgport is available or not based on a configure check. What package would a shared static pgport be installed with? Slony requires a server + headers to build but slon and slonik only have a runtime dependency on libpq (I don't know if anyone installs slon/slonik on a machine without a postgresql server but you could) In the Fedora world, a static lib would go in postgresql-devel, but a dynamic lib would go in postgresql-libs, which is also where libpq is shipped. I am not against shipping a dynamic libpgport, but I will just point out that this was never intended or anticipated. Are there any symbols in there that might conflict with other software? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
