RE: What hack in ld.so?
-Original Message- From: Alexandre Oliva [mailto:[EMAIL PROTECTED] Sent: Sunday, January 31, 1999 12:29 AM To: Jason Gunthorpe Cc: Debian Developers; [EMAIL PROTECTED] Subject: Re: What hack in ld.so? On Jan 30, 1999, Jason Gunthorpe [EMAIL PROTECTED] wrote: If you wish to make statement that binaries compiled with libtool work only on the host they were compiled for and even then only in specific conditions then fine - but that makes it totaly unsuitable for use by any of the binary distribution maker. That's not what I'd like libtool to do. I agree there is a problem to be fixed, I just think that libtool is not the only piece of software that may have to be changed to fix it, because it is not the only piece of software that uses -rpath. However, there is a risk that dropping -rpath will cause programs not to work in hosts other than the one in which they were installed. Assume libtool is modified so as to not hard-code directories listed in /etc/ld.so.conf. Then, the Devian (pun intended) distribution adds /dev/lib (*) to /etc/ld.so.conf, and configures some of their packages to install with --prefix=/dev (wouldn't that be beautiful? :-) (*) /dev is for Devian, not devices; I won't tell you that they have decided to move devices elsewhere too :-) When users of other distributions installed their pre-compiled .dev packages, programs that would run on Devian distributions would not run in any other, because /dev/lib is not listed in /etc/ld.so.conf, and libtool had failed to add /dev/lib to the rpath of Devian programs. Who's to blame for that? Of course, the solution is easy: adding the directory to ld.so.conf or to LD_LIBRARY_PATH. There's another solution: Do not use the Devian distribution, and stick to one of the (hopefully many) distributions of Linux, that happens to be quite standard :-) More seriously, what to describe is building the Devian distribution to provide something quite special (because if its placed in /dev I presume it's not available on other distributions), so the .dev packages, that use this Devian-specific stuff, will antway NOT work on any other distribution, till the other distribution makers decide to support this stuff... And I'm confident in the distribution makers intelligence to put the Devian-specific stuff in some Devian-compatible ;-) place (that is place it in /opt/devian because they do not like overloading /dev, and place /opt/devian in ld.so.conf). Once more this proves that -rpath is harmful: If Devian use -rpath thsi will *force* the other distribution makers to place Devian-specific stuff in /dev (and if Devian was crazy, placing standard stuff in /dev, their programs will only work on Devian distribution, which may be what they want but is definitely *NOT* what *I* would like). Regards, Bernard Bernard Dautrevaux Microprocess Ingéniérie 97 bis, rue de Colombes 92400 COURBEVOIE FRANCE Tel:+33 (0) 1 47 68 80 80 Fax:+33 (0) 1 47 88 97 85 e-mail: [EMAIL PROTECTED] [EMAIL PROTECTED]
RE: What hack in ld.so?
-Original Message- From: Marcus Brinkmann [mailto:[EMAIL PROTECTED] Sent: Sunday, January 31, 1999 12:53 AM To: Jason Gunthorpe; Alexandre Oliva Cc: Debian Developers; [EMAIL PROTECTED] Subject: Re: What hack in ld.so? Hi, On Sat, Jan 30, 1999 at 04:06:04PM -0700, Jason Gunthorpe wrote: On 30 Jan 1999, Alexandre Oliva wrote: snipped skipped Indeed libtool causes such a severe problem that if you take a libtool program, compile it on a libc5 Slackware and try to run it on -any- glibc system IT WILL NOT WORK. How could you expect it to work? I do *NOT* expect it to work: I everyday *USE* binaries that were *NOT* compiled by libtool but were indeed compiled on a libc5 system several years ago; they *STILL WORK* on my new libc6-based RedHat-5.2 system! I just want to be able to obtain the same functionality with libtool... No more, noless :-) If you wish to make statement that binaries compiled with libtool work only on the host they were compiled for and even then only in specific conditions then fine - but that makes it totaly unsuitable for use by any of the binary distribution maker. I think you got it wrong. You are right that Debian had hardly any alternative for the libc5-libc6 transition. But still, the problem is a missing feature: An implicit addition to the soname in dependence of the library dependencies. So you would have /usr/lib/libfoo.2 as rpath, but really would use /usr/lib/libfoo-libc6.2 or libfoo-libc5.2, you get the idea. However, the situation is not so easy, because you would need a multi dimensional table. As this is not possible (currently), the only thing you can do is to grant that the library is always compatible, as Alexandre correctly pointed out. We broke this, because it was more convenient for us not to change all Makefiles of our packages to use different sonames for libc6 libraries with the same version as their libc5 counterpart. However, the hack in the linker doesn't work for rpath binaries. libtool is not the cause for our problems, and rpath isn't, too. Our problems are the result of the following: * an incompatible upgrade path, together with an incomplete work-around in the linker. * the lack of a way to override rpath. The problem is neither -rpath or libc5/libc6; the fault is neither Alexandre nor Debian, RedHat or Devian :-). The problem is that I want to be able to obtain with libtool the same service I can obtain usually without it (although with a lot more difficulties): Build shared libraries and executables, that will work on various libc5 or libc6 based Linux distributions (as well as, as far as I'm concerned, on Solaris, hpUX, AIX and WindowsNT :-)) I do not ask Alexandre to devise *the perfect solution* that will allow me to build things without having to bother at all even in the very complicated cases where I link against totally non-standard stuff that I found in some exotic place on my system. I would like libtool to be able, without bothering me too much, to provide a correct answer on the maximum number of systems in the maximum number of cases. If I'm in a situation where -rpath is *needed*, OK, let *me* decide that (when I say me, let the package builder decide of this). If I'me pretty sure everything will work by default, without using -rpath, let me do it this way. The package builder is the one that knows, not Alexandre; and the package installer is the one that has to be provided with a mean to turn the problem if there is one. In the current situation Alexandre (that is libtool) is deciding in place of the package builder, and with a scheme where the package installer is stuck if things do not work right out-of-the-box. Regards, Bernard PS. Anyway Alexandre you've made a great job with libtool; why I'm a bit upset here is that I have the feeling that I will *not* be able to use this marvelous tool that is libtool... Bernard Dautrevaux Microprocess Ingéniérie 97 bis, rue de Colombes 92400 COURBEVOIE FRANCE Tel:+33 (0) 1 47 68 80 80 Fax:+33 (0) 1 47 88 97 85 e-mail: [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: What hack in ld.so?
On Feb 1, 1999, Bernard Dautrevaux [EMAIL PROTECTED] wrote: Once more this proves that -rpath is harmful: If Devian use -rpath thsi will *force* the other distribution makers to place Devian-specific stuff in /dev (and if Devian was crazy, placing standard stuff in /dev, their programs will only work on Devian distribution, which may be what they want but is definitely *NOT* what *I* would like). It appears that you haven't fully understood the meaning of -rpath. It adds a directory to the search path of a program, but it doesn't force the program to find libraries it needs in that directory. If /dev/lib does not exist, or the required libraries are not found in /dev/lib, they will be searched in directories specified in LD_LIBRARY_PATH and in the default search path. The only problem for Devian packages in non-Devian distributions, if libtool failed to -rpath /dev/lib, would be that, even if libraries that are part of Devian packages are properly installed in /dev/lib, they will not be found by Devian programs, because /dev/lib will not be searched. -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
Re: What hack in ld.so?
On Feb 1, 1999, Bernard Dautrevaux [EMAIL PROTECTED] wrote: The problem is that I want to be able to obtain with libtool the same service I can obtain usually without it (although with a lot more difficulties): Build shared libraries and executables, that will work on various libc5 or libc6 based Linux distributions (as well as, as far as I'm concerned, on Solaris, hpUX, AIX and WindowsNT :-)) You already can. However, because of the unfortunate choice of the Debian maintainers of replacing libraries with incompatible versions and failing to complete the ld.so hack that would have worked around this problem, you have to avoid hard-coding /lib, /usr/lib and /usr/X11R6/lib in your programs, and the only way you can currently do this is by: 1) using the script I have posted that modifies libtool so as not to hard-code any rpath at all; or 2) use another directory as your library directory, such as /usr/microprocess/lib If I'me pretty sure everything will work by default, without using -rpath, let me do it this way. The package builder is the one that knows, not Alexandre Agreed. I'm waiting for a patch that implements hard-coding exclusion of directories that are searched by default. PS. Anyway Alexandre you've made a great job with libtool Thank Gordon, not me. I fell like if all I was able to do was to start a war against one of our greatest users :-( ; why I'm a bit upset here is that I have the feeling that I will *not* be able to use this marvelous tool that is libtool... I'm pretty sure we'll be able to work it out :-) -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
RE: What hack in ld.so?
-Original Message- From: Alexandre Oliva [mailto:[EMAIL PROTECTED] Sent: Saturday, January 30, 1999 11:39 PM To: Buddha Buck Cc: Jason Gunthorpe; Debian Developers; [EMAIL PROTECTED] Subject: Re: What hack in ld.so? On Jan 29, 1999, Buddha Buck [EMAIL PROTECTED] wrote: [exact description of what I had assumed about the behavior of ld.so, based on previous postings to the libtool mailing list, snipped] This is not how I understand how the ld.so linker works on Linux systems. My understanding is that it caches the locations of all known versions of the libraries, and makes an intelligent decision as to which version to load. [description of what I now understand to b e the current behavior of ld.so snipped] This breaks in the presense of -rpath, because with rpath, bar is not dependent on libfoo, but on /usr/lib/libfoo. It shouldn't break, because -rpath is not associated with particular libraries, it is just a search path, just like the default search path. In order for ld.so to do an intelligent decision about whihc version to load, it should probably take into account the hard-coded rpath in addition to the default search path, preferring hard-coded rpaths as long as they do not introduce conflicts. Obviously, this would have never been needed if old libraries had not been replaced with (in)compatible versions, but the maintainers of Debian have already taken this step, despite the fact that this would break any previously-compiled programs that happened to hard-code /usr/lib or /usr/X11R6/lib. Therefore, new code must be written to support this step. I agree with you up to now, but Since modifying the next release of libtool won't contribute at all to fix the problem in already compiled programs, which are the only programs affected by this problem, I don't see much point in introducing a quick hack in libtool to prevent hard-coding of paths in libtool at this point. If/when someone contributes a portable and user/developer-configurable mechanism to do that, we may adopt it. THIS part of your statement is wrong, at least in my case :-( I usually still build production code with a libc5 setup, and will continue to do so for a while: I cannot force all my users to switch to libc6 (even if it's free, it cost quite a lot to migrate a production system with several people hooked on). I *have* to build programs that work on various, libc5 or libc6 based, linux boxes, so I build *new* programs with a libc5 setup, and I woudl like to be able to use libtool instead of having to had-craft shared library setup (remember I also have to work on Suns, HPs, etc...) Regards, Bernard Bernard Dautrevaux Microprocess Ingéniérie 97 bis, rue de Colombes 92400 COURBEVOIE FRANCE Tel:+33 (0) 1 47 68 80 80 Fax:+33 (0) 1 47 88 97 85 e-mail: [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: What hack in ld.so?
On Feb 1, 1999, Bernard Dautrevaux [EMAIL PROTECTED] wrote: Since modifying the next release of libtool won't contribute at all to fix the problem in already compiled programs, which are the only programs affected by this problem I usually still build production code with a libc5 setup, and will continue to do so for a while I understand, your point is a very good one. So, in addition to the two options I have listed in one of my previous messages, you have a third one to satisfy this need: start creating two distributions, one for backward-compatibility with libc5, another for users that have already upgraded to libc6. You'll have to do it at some time anyway, when standard distributors stop including libc5 in their default installations, so why not start doing it now? -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
Re: What hack in ld.so?
Ian Lance Taylor [EMAIL PROTECTED] writes: Date: Sat, 30 Jan 1999 16:52:48 -0700 (MST) From: Jason Gunthorpe [EMAIL PROTECTED] That's not what I'd like libtool to do. I agree there is a problem to be fixed, I just think that libtool is not the only piece of software that may have to be changed to fix it, because it is not the only piece of software that uses -rpath. libtool is however the only piece of software that we cannot easially change. I don't understand the reasoning here. The libtool files that a package will use are distributed with that package. If you are willing to modify a package's Makefile to remove -rpath, then it ought to be just as easy or difficult to modify the package's ltconfig file. I think that's the point - we don't want to have to modify each and every package that uses libtool. That's what we are doing now. It seems, somehow, wrong. There are valid reasons for Debian not to want -rpath. But we have to use a somewhat 'hacky' method to disable it currently. I think all we want is a somewhat cleaner method of achieving the same result. Cheers, - Jim
Re: What hack in ld.so?
On Jan 30, 1999, Jason Gunthorpe [EMAIL PROTECTED] wrote: On 30 Jan 1999, Alexandre Oliva wrote: I agree there is a problem to be fixed, I just think that libtool is not the only piece of software that may have to be changed to fix it, because it is not the only piece of software that uses -rpath. libtool is however the only piece of software that we cannot easially change. Huh? You (or someone else, I've (not) been introduced to too many Debian Developers lately to be able to remember who said what :-( mentioned editing Makefiles to adapt packages to Debian's policies, then I said one could just as easily edit the libtool script, and I have even posted a script that would do that for you. If you're talking about introducing a quick hack in libtool to never use rpath, forget about it, it won't work in general. If you're willing to contribute a patch to libtool that selectively drops rpath flags for directories listed in /etc/ld.so.conf, as I have suggested in another message I posted today, that's great. It doesn't really matter who is to blame Wouldn't the Devian developers complain that libtool isn't not properly arranging that their programs run even in non-Devian distributions? If they were to be as inflexible as you, we'd probably have to support switches like -devian-hardcoding-policies and -debian-hardcoding-policies, and I'd certainly have dropped the maintainership of libtool just after that (or just before) Thanks God I've got only one group of developers almost forcing me to change something that is correct, and whose change wouldn't even help them work around a problem they're facing. because it -can- be fixed and fixed fairly easially by the installer, the -rpath situation -cannot- be fixed at all by the installer, I think that is the key difference to realize. Ulrich Drepper has already mentioned that it's quite easy to modify the hard-coded rpath of a program without recompiling it, and that there's even a tool that will do that. joke sarcasm=off skip=(eval $mood==bad) It has just occurred to me that you should hack `ld' so that, whenever it finds a -rpath switch, it prints a warning message like: *** WARNING: YOU HAVE USED THE NEFAST --rpath SWITCH. THIS WILL CAUSE *** YOUR PROGRAM NOT TO WORK ON FUTURE Debian RELEASES BECAUSE WE *** DON'T CARE ABOUT (L)USERS WHO USE --rpath. YOU WERE ADVISED! /joke :-D -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
Re: What hack in ld.so?
On Jan 30, 1999, Marcus Brinkmann [EMAIL PROTECTED] wrote: So you would have /usr/lib/libfoo.2 as rpath, but really would use /usr/lib/libfoo-libc6.2 or libfoo-libc5.2, you get the idea. Just a minor nit: rpaths and sonames are independent of one-another. Although it is possible to arrange that sonames contain full pathnames, that's not what happens if you link a program with --rpath /usr/lib -lfoo. The --rpath switch will just add /usr/lib to a hard-coded search path, and -lfoo will just add the soname of libfoo to the list of run-time dependencies of a program. It is possible, though, to create a shared library with `-soname /usr/lib/libfoo.so.2', but then it becomes absolutely impossible to move the library around. There are times when this behavior is desirable, but libtool never does this, and there is no way to tell libtool to do it. -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
Re: What hack in ld.so?
On 30 Jan 1999, Alexandre Oliva wrote: libtool is however the only piece of software that we cannot easially change. then I said one could just as easily edit the libtool script, and I have even posted a script that would do that for you. Ah I missed that script, maybe we should just use that for the time being? (Sure we would be saying screw every other linux system, and binaries compiled with libtool on an old libc5 system will still not work, but hey) Thanks God I've got only one group of developers almost forcing me to change something that is correct, and whose change wouldn't even help them work around a problem they're facing. shrug I'm sure you have lots of screwball stuff to support defective and broken systems in libtool, I don't see how doing something 'wrong' to support another broken system like Linux is so bad. You cannot deny that it is necessary, and that it effects more that just Debian and our users but everyone using a libc5/libc6 linux system. It is not a problem we are 'facing' it is one we already faced and arguably made the wrong choice - So now all linux systems have this horrifying defect. Oh Well. That's Life - we can't fix it. Ulrich Drepper has already mentioned that it's quite easy to modify the hard-coded rpath of a program without recompiling it, and that there's even a tool that will do that. I actually think we have a program that does it as well, but somehow adding something to a file in /etc vs 'stripping' every binary you install on your system does not seem comparable. joke sarcasm=off skip=(eval $mood==bad) It has just occurred to me that you should hack `ld' so that, whenever it finds a -rpath switch, it prints a warning message like: At the very least the linker man page should have a warning to this effect, and as I said before, this is not a debian specific problem, your program could very will not work on any other linux system, including future releases of ours. Arguably that is what rpath is for, as strange as it seems : Say, has anyone asked the glibc people if they will change ld.so to have a different rpath behavior? Then we would be different than other systems, but it might be worth it. Jason
Re: What hack in ld.so?
Date: Sat, 30 Jan 1999 16:06:04 -0700 (MST)
From: Jason Gunthorpe [EMAIL PROTECTED]
On 30 Jan 1999, Alexandre Oliva wrote:
Obviously, this would have never been needed if old libraries had not
been replaced with (in)compatible versions, but the maintainers of
Debian have already taken this step, despite the fact that this would
Look, it was not us that decided to do this. It was the upstream
maintainers, other dists and a huge combination of factors. It is not in
our power to choose a different direction to solve these problems, we must
have libc6 xlib called libX11.so.6 and we must have libc5 called
libX11.so.6 - that is what all the other dists did, that is the default
and expected compilation behavoir of xlib and it is what all the new glibc
binary-only programs are using (ie netscape)
If you want to say that is a dumb way then fine, but you have not proposed
an alternative to solving the versioning problem and you have not proposed
an alternative way to handle the requirement of identical sonames and
libtool continues to perpetuate this 'bad' behavoir and makes it worse by
providing no way to get around it with the standard linux ld.so
What you are saying, I think, is that you have two programs with
A) DT_NEEDED libc.so.5, DT_NEEDED libX11.so.6
B) DT_NEEDED libc.so.6, DT_NEEDED libX11.so.6
Neither has DT_RPATH. The system has four libraries:
libc.so.5
libc.so.6
libX11.so.6 with DT_NEEDED libc.so.5
libX11.so.6 with DT_NEEDED libc.so.6
You want programs A and B to both work, without modification.
This was done by modifying the search algorithm used by the dynamic
linker so that it chooses the version of libX11.so.6 which matches the
version of libc.so.N used by the program. This was done by recording
in /etc/ld.so.cache the version of libc which the shared library uses.
The dynamic linker was modified to look in /etc/ld.so.cache for
libraries which were not found in DT_RPATH, but to only select
libraries listed in /etc/ld.so.cache which matched the version of the
dynamic linker being used. Programs which are linked against
different versions of libc must then use different dynamic linkers.
There are in fact three different dynamic linkers which understand
this ld.so.cache algorithm: the old a.out dynamic linker, the libc5
dynamic linker, and the glibc dynamic linker.
I just spent some time looking at the ld.so sources. Interestingly,
it seems to me that everything will work correctly in the sources I
was looking at. That is because the libc5 dynamic linker on my system
(RedHat 5.2) was modified to search the library cache before using the
application's DT_RPATH.
I think that is a hack that Debian is missing: it is the final hack to
the libc5 dynamic linker to change the search path to account for the
moved shared libraries even when rpath is used. I have appended the
RedHat patch below. This is to ld.so-1.9.5. Of course, this will
technically break the handling of DT_RPATH. However, we've already
determined that DT_RPATH binaries will not work correctly anyhow,
because the shared libraries were moved. So using this patch should
not make us any worse off.
Indeed libtool causes such a severe problem that if you take a libtool
program, compile it on a libc5 Slackware and try to run it on -any- glibc
system IT WILL NOT WORK.
This is not quite right. As I described above, glibc and libc5
binaries use a different dynamic linker. So the behaviour of your
libc5 binary depends upon the behaviour of your libc5 dynamic linker.
That linker does not come from glibc. Although I can not test this, I
now believe that if you take a libtool program, compile it on a libc5
Slackware and try to run it on a RedHat 5.2 system, it will work.
Ian
--- ld.so-1.9.5/d-link/readelflib1.c.ewtMon Nov 17 10:04:15 1997
+++ ld.so-1.9.5/d-link/readelflib1.cMon Nov 17 10:23:15 1997
@@ -179,38 +179,10 @@
goto goof;
}
- /*
- * The ABI specifies that RPATH is searched before LD_*_PATH or
- * the default path of /usr/lib.
- * Check in rpath directories
- */
- for (tpnt = _dl_loaded_modules; tpnt; tpnt = tpnt-next) {
-if (tpnt-libtype == elf_executable) {
- pnt1 = (char *)tpnt-dynamic_info[DT_RPATH];
- if(pnt1) {
- pnt1 += (unsigned int) tpnt-loadaddr + tpnt-dynamic_info[DT_STRTAB];
- while(*pnt1){
- pnt2 = mylibname;
- while(*pnt1 *pnt1 != ':') {
- if (pnt2 - mylibname 1024)
- *pnt2++ = *pnt1++;
- else
- pnt1++;
- }
- if (pnt2 - mylibname = 1024)
- break;
- if(pnt2[-1] != '/') *pnt2++ = '/';
- pnt = libname;
- while(*pnt) *pnt2++ = *pnt++;
- *pnt2++ = 0;
- tpnt1 = _dl_load_elf_shared_library(mylibname, 0);
- if(tpnt1) return tpnt1;
- if(*pnt1 == ':') pnt1++;
- }
- }
-}
- }
-
+ /* EWT - change things around a
Re: What hack in ld.so?
Date: Sat, 30 Jan 1999 16:52:48 -0700 (MST) From: Jason Gunthorpe [EMAIL PROTECTED] That's not what I'd like libtool to do. I agree there is a problem to be fixed, I just think that libtool is not the only piece of software that may have to be changed to fix it, because it is not the only piece of software that uses -rpath. libtool is however the only piece of software that we cannot easially change. I don't understand the reasoning here. The libtool files that a package will use are distributed with that package. If you are willing to modify a package's Makefile to remove -rpath, then it ought to be just as easy or difficult to modify the package's ltconfig file. Ian
Re: What hack in ld.so?
On Jan 30, 1999, Jason Gunthorpe [EMAIL PROTECTED] wrote: On 30 Jan 1999, Alexandre Oliva wrote: Thanks God I've got only one group of developers almost forcing me to change something that is correct, and whose change wouldn't even help them work around a problem they're facing. shrug I'm sure you have lots of screwball stuff to support defective and broken systems in libtool, I don't see how doing something 'wrong' to support another broken system like Linux is so bad. The point is that you've just been asking for libtool not to use -rpath at all, but this would only work for people who create .deb or .rpm binary packages, and not for any user that would be interested in building a package from the sources and installing it in non-standard directories. I have been somewhat hesitant about selectively dropping directories that are searched by default from the hardcoded rpath, but I think we can do it without much damage. There will be some, but whenever such a problem occurs, it is only because there was potential for its occurrence before, so we can live with it. It's just waiting for someone to implement it. You cannot deny that it is necessary, and that it effects more that just Debian and our users but everyone using a libc5/libc6 linux system. I can, that that's what I've been doing since the `Debian x Libtool War' started, a few days ago. I've insisted that the problem is not in the fact that libtool uses rpath, it is that libraries have been replaced with incompatible ones, and the code in ld.so that tries to accomodate for this replacement does not work when a program has a hard-coded library path. It is not a problem we are 'facing' it is one we already faced and arguably made the wrong choice - So now all linux systems have this horrifying defect. Oh Well. That's Life - we can't fix it. You can, but the problem cannot be fixed within libtool (which doesn't mean libtool can't help); it can only be fixed in ld.so Ulrich Drepper has already mentioned that it's quite easy to modify the hard-coded rpath of a program without recompiling it, and that there's even a tool that will do that. I actually think we have a program that does it as well, but somehow adding something to a file in /etc vs 'stripping' every binary you install on your system does not seem comparable. You don't have to strip every binary you install, only binaries that stop working after an upgrade. The upgrade program itself might search for programs linked with the old version of libc and with a (now) wrong hard-coded library path and fix them. -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
Re: What hack in ld.so?
On 30 Jan 1999, Alexandre Oliva wrote: You cannot deny that it is necessary, and that it effects more that just Debian and our users but everyone using a libc5/libc6 linux system. I can, that that's what I've been doing since the `Debian x Libtool War' started, a few days ago. I've insisted that the problem is not in the fact that libtool uses rpath, it is that libraries have been replaced with incompatible ones, and the code in ld.so that tries to accomodate for this replacement does not work when a program has a hard-coded library path. So you think we should be harrassing the ld.so people to change the meaning of rpath instead of you for using it? Jason
Re: What hack in ld.so?
Alexandre Oliva wrote: The point is that you've just been asking for libtool not to use -rpath at all, Yes, I think this is the correct solution. but this would only work for people who create .deb or .rpm binary packages, You fill this house with lies. It works for anyone putting libraries in standard places, standard being defined by the admin, who has write access to /etc/ld.so.cache. Users can work around this by, at worst, creating a wrapper script to set LD_LIBRARY_PATH. For example: frantica:~/apt/build/bin$ ./apt-get ./apt-get: error in loading shared libraries libapt-pkg.so.2.0: cannot open shared object file: No such file or directory frantica:~/apt/build/bin$ cat apt-get-wrapper #!/bin/sh LD_LIBRARY_PATH=/home/rcw/apt/build/bin exec /home/rcw/apt/build/bin/apt-get $@ frantica:~/apt/build/bin$ ls -l apt-get-wrapper -rwxr-xr-x 1 rcw rcw87 Jan 31 02:11 apt-get-wrapper* frantica:~/apt/build/bin$ ./apt-get-wrapper apt 0.3.0 for i386 [...] This is what our ld.so has done for years, this is what our users expect, and this is what has determined Linux's library placement for every other transition it has had to make. It's safe to say that will not change. Hardcoded pathnames to libraries are evil, you can't blame people for trying to get rid of them, especially when they start breaking left and right the minute things move around. -- Robert Woodcock - [EMAIL PROTECTED] It's like a love-hate relationship, but without the love. -- jwz, on linux
Re: What hack in ld.so?
Buddha Buck [EMAIL PROTECTED] writes: does it know that libc5 and libc6 are incompatable versions of the same library (different sonames), or does it feel that loading two libraries (libfoo, libc6) is better than loading three (libfoo, libc5, libc6). It recognizes libc, libm, and libdl in special code. That is why every shared library must be linked against at least one of those. Guy
Re: What hack in ld.so?
On Jan 29, 1999, Buddha Buck [EMAIL PROTECTED] wrote: [exact description of what I had assumed about the behavior of ld.so, based on previous postings to the libtool mailing list, snipped] This is not how I understand how the ld.so linker works on Linux systems. My understanding is that it caches the locations of all known versions of the libraries, and makes an intelligent decision as to which version to load. [description of what I now understand to b e the current behavior of ld.so snipped] This breaks in the presense of -rpath, because with rpath, bar is not dependent on libfoo, but on /usr/lib/libfoo. It shouldn't break, because -rpath is not associated with particular libraries, it is just a search path, just like the default search path. In order for ld.so to do an intelligent decision about whihc version to load, it should probably take into account the hard-coded rpath in addition to the default search path, preferring hard-coded rpaths as long as they do not introduce conflicts. Obviously, this would have never been needed if old libraries had not been replaced with (in)compatible versions, but the maintainers of Debian have already taken this step, despite the fact that this would break any previously-compiled programs that happened to hard-code /usr/lib or /usr/X11R6/lib. Therefore, new code must be written to support this step. Since modifying the next release of libtool won't contribute at all to fix the problem in already compiled programs, which are the only programs affected by this problem, I don't see much point in introducing a quick hack in libtool to prevent hard-coding of paths in libtool at this point. If/when someone contributes a portable and user/developer-configurable mechanism to do that, we may adopt it. -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
Re: What hack in ld.so?
On 30 Jan 1999, Alexandre Oliva wrote: Obviously, this would have never been needed if old libraries had not been replaced with (in)compatible versions, but the maintainers of Debian have already taken this step, despite the fact that this would Look, it was not us that decided to do this. It was the upstream maintainers, other dists and a huge combination of factors. It is not in our power to choose a different direction to solve these problems, we must have libc6 xlib called libX11.so.6 and we must have libc5 called libX11.so.6 - that is what all the other dists did, that is the default and expected compilation behavoir of xlib and it is what all the new glibc binary-only programs are using (ie netscape) If you want to say that is a dumb way then fine, but you have not proposed an alternative to solving the versioning problem and you have not proposed an alternative way to handle the requirement of identical sonames and libtool continues to perpetuate this 'bad' behavoir and makes it worse by providing no way to get around it with the standard linux ld.so Indeed libtool causes such a severe problem that if you take a libtool program, compile it on a libc5 Slackware and try to run it on -any- glibc system IT WILL NOT WORK. If you wish to make statement that binaries compiled with libtool work only on the host they were compiled for and even then only in specific conditions then fine - but that makes it totaly unsuitable for use by any of the binary distribution maker. Jason
Re: What hack in ld.so?
On Jan 30, 1999, Jason Gunthorpe [EMAIL PROTECTED] wrote: If you wish to make statement that binaries compiled with libtool work only on the host they were compiled for and even then only in specific conditions then fine - but that makes it totaly unsuitable for use by any of the binary distribution maker. That's not what I'd like libtool to do. I agree there is a problem to be fixed, I just think that libtool is not the only piece of software that may have to be changed to fix it, because it is not the only piece of software that uses -rpath. However, there is a risk that dropping -rpath will cause programs not to work in hosts other than the one in which they were installed. Assume libtool is modified so as to not hard-code directories listed in /etc/ld.so.conf. Then, the Devian (pun intended) distribution adds /dev/lib (*) to /etc/ld.so.conf, and configures some of their packages to install with --prefix=/dev (wouldn't that be beautiful? :-) (*) /dev is for Devian, not devices; I won't tell you that they have decided to move devices elsewhere too :-) When users of other distributions installed their pre-compiled .dev packages, programs that would run on Devian distributions would not run in any other, because /dev/lib is not listed in /etc/ld.so.conf, and libtool had failed to add /dev/lib to the rpath of Devian programs. Who's to blame for that? Of course, the solution is easy: adding the directory to ld.so.conf or to LD_LIBRARY_PATH. -- Alexandre Oliva http://www.dcc.unicamp.br/~oliva [EMAIL PROTECTED] [EMAIL PROTECTED],gnu.org,egcs.cygnus.com,samba.org} Universidade Estadual de Campinas, SP, Brasil
Re: What hack in ld.so?
On 30 Jan 1999, Alexandre Oliva wrote: On Jan 30, 1999, Jason Gunthorpe [EMAIL PROTECTED] wrote: If you wish to make statement that binaries compiled with libtool work only on the host they were compiled for and even then only in specific conditions then fine - but that makes it totaly unsuitable for use by any of the binary distribution maker. That's not what I'd like libtool to do. I agree there is a problem to be fixed, I just think that libtool is not the only piece of software that may have to be changed to fix it, because it is not the only piece of software that uses -rpath. libtool is however the only piece of software that we cannot easially change. When users of other distributions installed their pre-compiled .dev packages, programs that would run on Devian distributions would not run in any other, because /dev/lib is not listed in /etc/ld.so.conf, and libtool had failed to add /dev/lib to the rpath of Devian programs. Who's to blame for that? It doesn't really matter who is to blame because it -can- be fixed and fixed fairly easially by the installer, the -rpath situation -cannot- be fixed at all by the installer, I think that is the key difference to realize. Jason
Re: What hack in ld.so?
Hi, On Sat, Jan 30, 1999 at 04:06:04PM -0700, Jason Gunthorpe wrote: On 30 Jan 1999, Alexandre Oliva wrote: Obviously, this would have never been needed if old libraries had not been replaced with (in)compatible versions, but the maintainers of Debian have already taken this step, despite the fact that this would Look, it was not us that decided to do this. Debian was the first distribution to make a decision about the libc6 transition. Certainly we have decided to do it the way we did. It was the upstream maintainers, other dists and a huge combination of factors. It is true that these determined the options and Debians final decision, but still we decided to follow this. It is not in our power to choose a different direction to solve these problems, we must have libc6 xlib called libX11.so.6 and we must have libc5 called libX11.so.6 - that is what all the other dists did, that is the default and expected compilation behavoir of xlib and it is what all the new glibc binary-only programs are using (ie netscape) And Alexandre is right that this is - in general - the cause of our problems with rpath, and it is indeed _wrong_ (in general). That it works in Linux is only due to a smart hack in the linker, and the hack does obviously not take rpath into account. So, ask yourself the question: Why do you expect it to work? Why do you expect it to be fixed in libtool, when libtool has nothing to do with it? If you want to say that is a dumb way then fine, but you have not proposed an alternative to solving the versioning problem and you have not proposed an alternative way to handle the requirement of identical sonames and libtool continues to perpetuate this 'bad' behavoir and makes it worse by providing no way to get around it with the standard linux ld.so There is no right way to get identical sonames but different libraries. The bad behaviour has to be searched for (and can be found) on the Linux side. The only solution to this problem would be to allow multiple libraries with the same soname but different dependencies staying in the same place. There is currently no way doing so. The way the Linux linker does circumvent the missing feature does work surprisingly well, but it is still not the correct thing. Indeed libtool causes such a severe problem that if you take a libtool program, compile it on a libc5 Slackware and try to run it on -any- glibc system IT WILL NOT WORK. How could you expect it to work? If you wish to make statement that binaries compiled with libtool work only on the host they were compiled for and even then only in specific conditions then fine - but that makes it totaly unsuitable for use by any of the binary distribution maker. I think you got it wrong. You are right that Debian had hardly any alternative for the libc5-libc6 transition. But still, the problem is a missing feature: An implicit addition to the soname in dependence of the library dependencies. So you would have /usr/lib/libfoo.2 as rpath, but really would use /usr/lib/libfoo-libc6.2 or libfoo-libc5.2, you get the idea. However, the situation is not so easy, because you would need a multi dimensional table. As this is not possible (currently), the only thing you can do is to grant that the library is always compatible, as Alexandre correctly pointed out. We broke this, because it was more convenient for us not to change all Makefiles of our packages to use different sonames for libc6 libraries with the same version as their libc5 counterpart. However, the hack in the linker doesn't work for rpath binaries. libtool is not the cause for our problems, and rpath isn't, too. Our problems are the result of the following: * an incompatible upgrade path, together with an incomplete work-around in the linker. * the lack of a way to override rpath. Thanks, Marcus -- Rhubarb is no Egyptian god.Debian GNU/Linuxfinger brinkmd@ Marcus Brinkmann http://www.debian.orgmaster.debian.org [EMAIL PROTECTED]for public PGP Key http://homepage.ruhr-uni-bochum.de/Marcus.Brinkmann/ PGP Key ID 36E7CD09
