Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
But creating correct symlinks IS one of the tasks of a system administrator... His up to know the distro he is using and the software he wants to install. I bet a in the field sysadmin will be much more flexible than a hardcoded implementation that would need to take zillions of obscure distros in consideration from the limited perspective of compile time... 2012/8/15 Graeme Geldenhuys graemeg.li...@gmail.com: Hi, This issue has come up before in a fcl-db discussion regarding the Firebird Database, where the libfbclient.so was missing from many Linux distros (eg: Ubuntu 10.04), but a libfbclient.so.2.0 was available instead. FCL-DB only checks for the unversioned shared libraries, thus your application will probably not run out of the box. I've stumbled across this problem again today with OpenSUSE 12.1 and Synapse where I send emails from inside my applications via a secure SMTP connection. The Synapse library is looking for libssl.so, but by default OpenSUSE only has libssl.so.1.0.0. In both these cases, I manually created unversioned symlinks to those libraries, and that got my applications working again. This is not ideal, but I don't know how else to handle this. Does any body know what is the most correct way of handling this? Am I supposed to modify my copies of fcl-db and synapse to look for specific versions of these shared libraries, or should I somehow add a function in my application installation that checks in unversioned shared libraries exist, and if not, try and create those (which would require root access - and might cause problems an client installs). Anybody know of any Linux documentation URL that explain when unversioned shared libraries are used and when not, and how applications are supposed to handle this? -- Regards, - Graeme - ___ fpGUI - a cross-platform Free Pascal GUI toolkit http://fpgui.sourceforge.net ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
On 15 Aug 2012, at 12:43, Graeme Geldenhuys wrote: This issue has come up before in a fcl-db discussion regarding the Firebird Database, where the libfbclient.so was missing from many Linux distros (eg: Ubuntu 10.04), but a libfbclient.so.2.0 was available instead. FCL-DB only checks for the unversioned shared libraries, thus your application will probably not run out of the box. In both these cases, I manually created unversioned symlinks to those libraries, and that got my applications working again. This is not ideal, but I don't know how else to handle this. The official way to get the unversioned symbolic links is to install the -dev or -devel package for that library. Of course, you're not supposed to require end-users to do that. Does any body know what is the most correct way of handling this? Determine which major version of the library you require (e.g. 1.x, or 2.x, or even something like 5.1.x if the library in question only supports API compatibility across minor releases), and explicitly load that one. That is also how it works when linking against a library at compile time: the unversioned name is simply a symbolic link to a versioned file name, and it's that versioned file name that the linker will hardcode in your application. The reasoning is that your application was written and tested against that version of the library, so letting it use arbitrary other versions is not a good idea. If you can work with multiple major versions, you can try to dynamically load them all until you have found one that exists. At least most (if not all) FCL units that dynamically load libraries have a function that you can call to explicitly specify the name of the dynamic library to use. Since the units themselves have presumably been written to work with certain versions of those libraries, it would however probably be best if that code were directly added to those units (and to remove any *default* dynamic loading of unversioned file names, since such libraries usually only exist on a developer's machine -- except maybe on non-Unix platforms). Jonas___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
Graeme Geldenhuys wrote: Hi, This issue has come up before in a fcl-db discussion regarding the Firebird Database, where the libfbclient.so was missing from many Linux distros (eg: Ubuntu 10.04), but a libfbclient.so.2.0 was available instead. FCL-DB only checks for the unversioned shared libraries, thus your application will probably not run out of the box. I've stumbled across this problem again today with OpenSUSE 12.1 and Synapse where I send emails from inside my applications via a secure SMTP connection. The Synapse library is looking for libssl.so, but by default OpenSUSE only has libssl.so.1.0.0. In both these cases, I manually created unversioned symlinks to those libraries, and that got my applications working again. This is not ideal, but I don't know how else to handle this. Quite a long way from ideal, since it implies that an administrator has to be involved even if the program is only sitting in an unprivileged user's home directory (or is a symlink in ~ good enough?). I'm tempted to say that this is a distro issue, and that if an upstream project (FireBird, OpenSSL, PostgreSQL) normally publishes an unversioned library that a distro is at fault if it decorates it with an appended version number without providing a symlink chain. I'd suggest that the best solution would be for FPC policy to mandate that all shared library (.so etc.) usage in the RTL/FCL/LCL should be capable of taking a library name parameter, and for application code to allow this name to be specified in a .ini file. I think this would be safer than having the app try to deduce the location of the appropriate file, since sooner or later there will be a case where it sees multiple versions and selects the wrong one, or assumes that e.g. /usr/local/opt is permanently available when in fact it's a short-term NFS mount. And doing it that way means that the user would be aware of the problem, and if enough users are aware of the problem it might trickle through to the awareness of the distreaux maintainers. -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
Hi, On 15 August 2012 12:10, Jonas Maebe jonas.ma...@elis.ugent.be wrote: The official way to get the unversioned symbolic links is to install the -dev or -devel package for that library. Of course, you're not supposed to require end-users to do that. Yes, I know that bit, but even as a developer, if I don't do actual Firebird or OpenSSL development (I don't work on those project, I simply use there libraries), I don't need to install those libraries. That is why my Ubuntu and OpenSUSE system didn't have the -devel packages for them installed. FCL-db and Synapse dynamically load those libraries on my development machines, so no -devel package requirement. ... that the linker will hardcode in your application. The reasoning is that your application was written and tested against that version of the library, so letting it use arbitrary other versions is not a good idea. In a way I understand the reasoning for the unversioned symlink, I just don't know how best to handle it in my apps. I also don't know the rules as to which version the unversioned symlib will normally point to.I know my apps work with Firebird 2.1 and Firebird 2.5, so should I then change fcl-db to look for libfbclient.so.2 or libfbclient.so.2.0 - instead of the unversioned default. Because the unversioned default could very easily point to a v1.x library of Firebird, and that would be untested with my app (it will probably not work). If you can work with multiple major versions, you can try to dynamically load them all until you have found one that exists. OK, I'll take a look at this idea. That seems better, because then I know I am trying to load library versions my app is known to work with. I'm not so much concerned about my development system (I can fix that) - I am thinking more in line of when I deploy my apps. I want them to obviously work as easily as possible. -- Regards, - Graeme - ___ fpGUI - a cross-platform Free Pascal GUI toolkit http://fpgui.sourceforge.net ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
Jonas Maebe wrote: In both these cases, I manually created unversioned symlinks to those libraries, and that got my applications working again. This is not ideal, but I don't know how else to handle this. The official way to get the unversioned symbolic links is to install the -dev or -devel package for that library. Of course, you're not supposed to require end-users to do that. Does not always work, e.g. in the case of Debian. -dev will install the .h and .a files, but won't necessarily set up unversioned symlinks. Anybody: what's the position with Ubuntu, as the leading Debian derivative? -- Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues] ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
Hi, On 15 August 2012 12:27, Mark Morgan Lloyd markmll.fpc-pas...@telemetry.co.uk wrote: Quite a long way from ideal, since it implies that an administrator has to be involved even if the program is only sitting in an unprivileged user's home directory (or is a symlink in ~ good enough?). Correct, definitely not ideal. I was just thinking about the $HOME directory. I'll try and play with that now, and see what happens to my apps. Maybe my apps should all have startup scripts that define local LD_LIBRARY_PATH options? I'm tempted to say that this is a distro issue, and that if an upstream project (FireBird, OpenSSL, PostgreSQL) normally publishes an unversioned library that a distro is at fault if it decorates it with an appended version number without providing a symlink chain. They explicitly mention that general runtime libraries should not include the unversioned symlink. Only the -devel packages should include those. That makes me think, why do we then need the unversioned symlink in the first place! Fedora: http://fedoraproject.org/wiki/Packaging:Guidelines#Devel_Packages OpenSUSE: http://en.opensuse.org/openSUSE:Shared_library_packaging_policy#Unversioned_packages Ubuntu/Debian: ?? I can't remember the URL now. -- Regards, - Graeme - ___ fpGUI - a cross-platform Free Pascal GUI toolkit http://fpgui.sourceforge.net ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
On 15/08/12 12:37, Mark Morgan Lloyd wrote: Jonas Maebe wrote: In both these cases, I manually created unversioned symlinks to those libraries, and that got my applications working again. This is not ideal, but I don't know how else to handle this. The official way to get the unversioned symbolic links is to install the -dev or -devel package for that library. Of course, you're not supposed to require end-users to do that. Does not always work, e.g. in the case of Debian. -dev will install the .h and .a files, but won't necessarily set up unversioned symlinks. I've never found a -dev package that doesn't set up symlinks in many years of using Debian for development. (Except if only static versions of a specific library exists, obviously). Could you give an example? Henry ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
On 15 Aug 2012, at 13:32, Graeme Geldenhuys wrote: On 15 August 2012 12:10, Jonas Maebe jonas.ma...@elis.ugent.be wrote: The official way to get the unversioned symbolic links is to install the -dev or -devel package for that library. Of course, you're not supposed to require end-users to do that. Yes, I know that bit, but even as a developer, if I don't do actual Firebird or OpenSSL development (I don't work on those project, I simply use there libraries), I don't need to install those libraries. The dev/devel packages are not for people working on those libraries, they are for building programs that use those libraries. It's developers that do work on those packages themselves that don't necessarily need those packages, since they can just use their own source code. That is why my Ubuntu and OpenSUSE system didn't have the -devel packages for them installed. FCL-db and Synapse dynamically load those libraries on my development machines, so no -devel package requirement. You said that you manually created the symbolic link. I simply explained that you should never do that, and instead install the development packages because they will do that (correctly) for you. Whether or not it is desirable for the FCL/Synapse units to require an unversioned symlink to be present is a separate issue. ... that the linker will hardcode in your application. The reasoning is that your application was written and tested against that version of the library, so letting it use arbitrary other versions is not a good idea. In a way I understand the reasoning for the unversioned symlink, I just don't know how best to handle it in my apps. I also don't know the rules as to which version the unversioned symlib will normally point to. It will point to the library of the version corresponding to the dev/devel package that has been installed. I know my apps work with Firebird 2.1 and Firebird 2.5, so should I then change fcl-db to look for libfbclient.so.2 or libfbclient.so.2.0 - instead of the unversioned default. libfbclient.so.2.0 means Firebird 2.0.x. libfbclient.so.2 means Firebird 2.x. If you only support 2.1 and 2.5, you should link against libfbclient.so.2 and then call a function to get the actual version. If the library does not contain such a function, then there is simply no way to dynamically load that library and use it in a safe way in your scenario. Jonas___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
Hi, On 15 August 2012 12:49, Jonas Maebe jonas.ma...@elis.ugent.be wrote: You said that you manually created the symbolic link. I simply explained that you should never do that, and instead install the development packages because they will do that (correctly) for you. So my applications that I deploy to clients will now require -devel packages? This just doesn't sound sane, sorry. Maybe there is just a fundamental flaw in all distros and how they package software. Initially a great idea (in theory), but in reality it is seriously flawed. Whether or not it is desirable for the FCL/Synapse units to require an unversioned symlink to be present is a separate issue. FCL-DB uses dynamic linking by default, and looks for the unversioned shared library. So what specific Firebird version is the FCL-DB coded too? Because remember, libfbclient.so can point to a v1.x or a v2.x - depending on what -devel package was installed. This uncertain (version mismatch) is what I would like to avoid, plus the fact that my clients now need to install -devel packages too. Maybe I should simply install my apps into /home/user/appname. My installer would then ship the required/expected *.so files, and setup the unversioned symlink. My app would then be launched via a script which sets the LD_LIBRARY_PATH to the application directory. My app would then be totally self-contained. This seems like the only way to get around these flawed package management rules and dependencies. It will point to the library of the version corresponding to the dev/devel package that has been installed. So one minute it could point to a v1.x library, and the next it could point to a v2.x library. This will surely break many apps that only look at the unversioned symlink. libfbclient.so.2.0 means Firebird 2.0.x. libfbclient.so.2 means Firebird 2.x. OK, thanks for that. My apps are tiOPF based, and tiOPF uses the bare minimum functionality from each backend database server. Basically just the CRUD (create, read, update and delete) functions. So even though I say our apps use Firebird DB v2.x, it will most probably run on a v1.x system too, but like I said, that would be untested. We only test with v2.x Firebird servers. -- Regards, - Graeme - ___ fpGUI - a cross-platform Free Pascal GUI toolkit http://fpgui.sourceforge.net ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
On 15/08/12 13:43, Mark Morgan Lloyd wrote:
I do agree with Graeme's position though: a -dev is described as
containing files for developers, and it should not be necessary for a
non-developer user to start encumbering his system with .h files etc.
What's more, part of the reason that I've had to create symlinks
manually in the past is because things like (if I recall correctly)
OpenOffice's interface to PostgreSQL require libpq.so without declaring
-dev as a prerequisite... which rather reinforces my position that this
is a distro issue even if they prefer to disclaim it.
I'd say that it's the OpenOffice interface that's broken. They
shouldn't link to a library that doesn't specify a version. They need
to link to a specific version by using the naming conventions as Jonas
explained. This is to protect yourself from api/feature breakage.
Linking directly to, e.g. libpq.so, is only o.k. if you can check the
version by means of some api, which the library needs to supply in all
the versions that exist - past, present and future. Which is probably
what they're doing. This isn't very good practice, in my opinion, but
can work.
If they really support multiple versions of libpq, then they will have
to try to dynamically link against all the versions they support. This
won't be much more effort than checking the version in the first place.
Even on Windows some libraries are using names that reflect the version,
e.g. I link to some opencv libs, which are named
libopencv_{module}231.dll. I suspect this is much less of a problem in
the Windows world, since library reuse isn't that common.
Henry
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
On 15 Aug 2012, at 15:14, Graeme Geldenhuys wrote: On 15 August 2012 12:49, Jonas Maebe jonas.ma...@elis.ugent.be wrote: You said that you manually created the symbolic link. I simply explained that you should never do that, and instead install the development packages because they will do that (correctly) for you. So my applications that I deploy to clients will now require -devel packages? They always have required them until now (as have any other applications that use fcl-db). This just doesn't sound sane, sorry. Maybe there is just a fundamental flaw in all distros and how they package software. Initially a great idea (in theory), but in reality it is seriously flawed. It's the fcl-db/Synapse units that are flawed. As explained in my first reply, the unversioned symlink is only for use by the linker (ld), and they work perfectly well for that purpose. Whether or not it is desirable for the FCL/Synapse units to require an unversioned symlink to be present is a separate issue. FCL-DB uses dynamic linking by default, and looks for the unversioned shared library. So what specific Firebird version is the FCL-DB coded too? I have no idea, which is why I also said in my first reply: *** At least most (if not all) fcl units that dynamically load libraries have a function that you can call to explicitly specify the name of the dynamic library to use. Since the units themselves have presumably been written to work with certain versions of those libraries, it would however probably be best if that code were directly added to those units (and to remove any *default* dynamic loading of unversioned file names, since such libraries usually only exist on a developer's machine -- except maybe on non-Unix platforms). *** Maybe I should simply install my apps into /home/user/appname. My installer would then ship the required/expected *.so files, and setup the unversioned symlink. My app would then be launched via a script which sets the LD_LIBRARY_PATH to the application directory. My app would then be totally self-contained. This seems like the only way to get around these flawed package management rules and dependencies. While shipping all libraries yourself is an option, the package management rules and dependencies are not flawed (or at least the scenario you are describing currently does not point to a flaw in them). If you start shipping libraries yourself, you'd better make sure that you also ship all libraries they in turn depend on as well, because otherwise you're more likely to make the problem worse than solve it. It will point to the library of the version corresponding to the dev/devel package that has been installed. So one minute it could point to a v1.x library, and the next it could point to a v2.x library. This will surely break many apps that only look at the unversioned symlink. As mentioned in my previous replies, applications should never look at the unversioned link in the first place when dynamically loading a library. Jonas___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
In our previous episode, Graeme Geldenhuys said:
In both these cases, I manually created unversioned symlinks to those
libraries, and that got my applications working again. This is not
ideal, but I don't know how else to handle this.
Does any body know what is the most correct way of handling this?
(1) avoid _dyn versions.
(2) override the built-in naming by calling
ibase60dyn.initializeibase60('whatever.so');
before connection or other db parts initializes.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
In our previous episode, Graeme Geldenhuys said: You said that you manually created the symbolic link. I simply explained that you should never do that, and instead install the development packages because they will do that (correctly) for you. So my applications that I deploy to clients will now require -devel packages? This just doesn't sound sane, sorry. Maybe there is just a fundamental flaw in all distros and how they package software. No. It all works fine WITHIN a distribution (in a particular version). And basically that is the only supported way on Linux. Anything crossdistribution and cross-distroversion is assumed to be on source level, not binary level (IOW rebuild your app). According to common linux philosophy distributing crossdistro binaries is asking for pain. Basically if you link to a .so the normal way, to simplify build systems, a symlink from a generic name to a versioned name is used. But the generic binary contains a reference to the versioned name, and thus will work without symlink, which is why end-users systems don't have the link. But this way of linking is more dependent on the exact version of the library, so in the past some people changed everything to dynamically loaded via dlload/dlsym. This alleviates the versioning problem somewhat, but discards the only distribution provided way to fix this. I still think that changing everything to _dyn created more problems than it solved. (except for special disaster case mysql maybe) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] linux: should we hard-code versioned or unversioned shared libraries in our apps?
Hi, On 15 August 2012 14:39, Jonas Maebe jonas.ma...@elis.ugent.be wrote: FCL-DB uses dynamic linking by default, and looks for the unversioned shared library. So what specific Firebird version is the FCL-DB coded too? I have no idea, which is why I also said in my first reply: Umm, so ideally the ibconnection.pp unit should really be split into various units with version numbers in their names. That way we will know to which Firebird version they are referring too. eg: ibconnection.pp // old interbase only (eg: Delphi 7) firebird-1.x.pp firebird-2.1.pp firebird-2.5.pp Include files could probably be used to reduce some duplication of code. As mentioned in my previous replies, applications should never look at the unversioned link in the first place when dynamically loading a library. OK, for now I'll modify my fcl-db and synapse code to look for the versioned shared libraries that I know I have tested with. eg: libfbclient.so.2 Thanks for all the help guys. -- Regards, - Graeme - ___ fpGUI - a cross-platform Free Pascal GUI toolkit http://fpgui.sourceforge.net ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
