Re: [CMake] Does find_library check that a found library does in fact link?

[Michael Hertling]

On 09/29/2011 09:22 AM, Hendrik Sattler wrote:
> Zitat von Michael Wild :
>> Just a few of my thoughts on this:
> 
> Same for me.
> 
>> - There are several ways to handle dead symlinks:
>>   1. Don't check, let the linker complain (status quo)
>>   2. Check whether the found library is a symlink, and if not valid,
>>  remove it silently from the list of candidates. Can be very
>>  surprising and will likely result in quite a few bug reports
>>  about CMake not finding a certain library that is "clearly" there.
>>   3. Like 2, but warn about the issue. Possibly very annoying.
>> I don't know which option to prefer (perhaps somebody finds another one
>> that is better?). IMHO bogus symlinks to shared libraries constitute a
>> real problem of the library installation, so CMake shouldn't just paper
>> over it by silently skipping the dead link, proceeding to the next
>> candidate which rules out option 2 for me. Also, option 3 can be
>> troublesome in automated setups where no user actually reads the CMake
>> output.
> 
> People tend to misunderstand the purpose of build systems like CMake  
> and autotools. They try to make the detection bullet-proof and most  
> likely fail. You can see that this is wrong by looking at those auto*  
> setups that take ages to detect all possible header files and use an  
> ifdef hell to handle that in the code, run lots of link and  
> symbol-finding tests, and still fail on conditions that the author  
> simple didn't consider or know.

A concrete example: Several years ago, I worked with an X11 system
which - besides TCP/IP - was aware of DECnet, and pretty nobody has
taken into account that there might be a dependency of libX11.so on
libdnet.so. Consequently, I had countless packages - also from highly
respected projects - that needed to be configured with LDFLAGS=-ldnet.
Usually, that's all there was to it, and I was perfectly okay with it,
i.e. I wouldn't expect from a project's configuration to take DECnet
into consideration just because the project is configured for X11.

> I prefer option 1. It's not CMake's duty to detect or paper over  
> messed-up systems. That's the administrator's job!
> BTW: running "ldconfig -v" on Linux likely tells you about the  
> dangling symlink.

Absolutely. Moreover, there are trivial day-to-day situations where
even the most sophisticated configuration can't succeed, e.g. if the
project needs a package that's simply not installed. In such a case,
administrative action must be taken anyway, either one installs the
package by oneself or asks the admin to do so. Therefore, it is my
personal opinion that one should expend a reasonable effort to fit
a project for the common configurational pitfalls, but trying to do
as much as possible in this regard will become counterproductive at
a certain point, in particular if one could easily counteract with
administrative means.

   There is no such thing as portable software,
only software that has been ported.
 -- Unknown

[Annotation to chapter 1 of the imake book]

> What's next? Asking for a syntax check of found header files so that  
> the compiler doesn't complain on broken header files?

Equipping FIND_PATH() with a check if the file for which the path is
to be returned is a valid header, and equipping FIND_PROGRAM() with
a check if the program to be returned is a valid executable - with
the same justification as for library checks in FIND_LIBRARY().

Regards,

Michael
--
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Michael Hertling]

On 09/29/2011 07:15 AM, Clifford Yapp wrote:
> On Wed, Sep 28, 2011 at 8:52 PM, Michael Hertling wrote:
> 
> What do you do on systems which have no idea of symbolic links, e.g.
>> previous Windows versions? Adding more platform-specific code to the
>> sources of the FIND_LIBRARY() function?
> 
> 
> If the problem isn't solved (or readily solvable) on some platforms, we can
> just fall back to doing what we do now in those cases... we don't have to
> solve the problem on all possible platforms to improve the results on
> platforms that can be supported.  The point wouldn't be to *guarantee* the
> result of find_library is a valid working library (it doesn't do that now),
> just improve the results by doing what can be done to avoid returning
> results that can be tagged as non-working (since a build trying to use such
> results is guaranteed to fail anyway.)  Does cmake know (or can it tell)
> when it is cross compiling?  If so, it could decide how much to test and not
> test in those cases.

There's the CMAKE_CROSSCOMPILING variable which indicates if CMake is
configuring a project for cross-compilation, but that's not my actual
point here. Even for a non-cross-compiling project, FIND_LIBRARY()'s
candidate might be a valid library in one of several native binary or
even textual formats, or it can be in a foreign binary format and is
to be used in a combined native- and cross-compiling project. What's
about the opposite case, i.e. FIND_LIBRARY() returns a working native
library, but you intended to find a library in a foreign format? How
should FIND_LIBRARY() ever check if a library is valid, and what does
"valid" mean in this context? Nobody says that a library must be used
with the compilers specified in the CMAKE__COMPILER variables;
it might also be processed in a custom command/target by a different
tool. How would you implement the checks you envision if you can not
rely on CMAKE__COMPILER as linker front-end? How do you know
when you must check a library with the C++ compiler, and not the C
compiler? In short: For which criteria and by use of which means do
you decide whether a library is valid? IMO, FIND_LIBRARY() would
hardly be able to check for any criterion in a reliable manner,
so it's best to not have such checks in this function at all.

>> Furthermore, the kernels of
>> *nix systems hardly distinguish between binary files and text files;
>> usually, they know just "files" with the limited exception of being
>> able to recognize the native executable formats and the #! shebang.
>> The detection of the diverse file types is typically implemented
>> in utility programs, notably the file(1) utility.
>>
> 
> A linker test when possible (non-cross-compilation scenarios) with a
> fallback testing option using file introspection would handle a lot of
> situations, if I'm understanding correctly how this works.  To the best of
> my knowledge and that of one of our other devs (he knows a lot more than me)
> there aren't any platforms where actual library files (as opposed to linker
> scripts) consist solely of character values <128.  Checking a found file for
> at least one character over 128 would avoid at least a few failure cases -
> empty files, plain text files, and dead links.  Such a check would be
> reliable, consist, and would be useful in eliminating files we might find
> that are not actually library files.

"Why doesn't my linker script pass this test?"
"Because it contains comments in German."
"Häää?" ("48c3a4c3a4c3a43f" in UTF-8)

> Such a test  of course wouldn't pass linker scripts like ubuntu's libc.so,
> but such a script *would* pass the actual linker test and would never get to
> file introspection. Actually if the script failed the linker test, then
> there is a real problem that should be a failure case.  This would also be
> helpful in spotting early on the hypothetical case of a linker being tested
> that doesn't understand the .so linker script.

OTOH, an arbitrary binary file would not pass the linker test, but
the fallback introspection test, wouldn't it? A valid library as well
as an invalid one might consist of a binary as well as a textual file,
so this test cannot even rudimentarily be considered as reliable, IMO.

 But how do you test whether a library is "linkable"?
>>
> 
> If not cross-compiling, wouldn't it be possible to have find_library try the
> link as part of its test?  Even in the case where linking isn't viable,
> falling back on the check on file contents would still be helpful.

Again, trying to link with *which* linker?

>> What do you do if the library found by FIND_LIBRARY() has a non-native
>> binary format, e.g. for cross-compiling purposes? How do you select the
>> right set of tools to check the library in question without the user's
>> ado? IMO, FIND_LIBRARY()'s job is to find library files, and a general
>> check whether these files are valid/usable/linkable and not dangling
>> symlinks or whatever is beyond the scope of this function; ensurin

Re: [CMake] Does find_library check that a found library does in fact link?

[Clifford Yapp]

On Thu, Sep 29, 2011 at 3:22 AM, Hendrik Sattler wrote:


> People tend to misunderstand the purpose of build systems like CMake and
> autotools. They try to make the detection bullet-proof and most likely fail.
> You can see that this is wrong by looking at those auto* setups that take
> ages to detect all possible header files and use an ifdef hell to handle
> that in the code, run lots of link and symbol-finding tests, and still fail
> on conditions that the author simple didn't consider or know.
>

The problem is in real world situations the software developer often has
virtually zero control over system administration issues on systems their
users are working with, and when your trying to be very portable you have to
spot and handle as many as possible of the wide variety of situations that
exist in the wild.  The decision on how much effort to put into working
around the existing issues (or at the very least detecting them before
putting all the time and resources into a full compilation) is determined by
the goals of the project.  Knowing that *all* situations can't be handled is
not a reason (at least for some projects) to avoid handling as much as they
can - they keep improving their handling when they run into new issues.


> I prefer option 1. It's not CMake's duty to detect or paper over messed-up
> systems. That's the administrator's job!
>

Except there are cases where a software developer has to do their best to
make their program work with the system as it is.  Spotting results that
aren't valid libraries in configuration is a time saver under those
conditions.

Couldn't CMake provide the means to let developers turn on testing
mechanisms (just disable them by default)?  A project could explicitly
enable CMAKE_CHECK_SHARED_LIBRARY_LINKING if they wanted to run the tests,
and projects who don't want it could ignore the issue and never know the
difference.  As long as CMake provided the core mechanism and hooks needed
to perform the test and feed in options, each project could decide what to
do from there.

Cheers,
CY
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Clifford Yapp]

On Thu, Sep 29, 2011 at 2:59 AM, Michael Wild  wrote:

>
> Just a few of my thoughts on this:
>
> - There are several ways to handle dead symlinks:
>  1. Don't check, let the linker complain (status quo)
>  2. Check whether the found library is a symlink, and if not valid,
> remove it silently from the list of candidates. Can be very
> surprising and will likely result in quite a few bug reports
> about CMake not finding a certain library that is "clearly" there.
>  3. Like 2, but warn about the issue. Possibly very annoying.
> I don't know which option to prefer (perhaps somebody finds another one
> that is better?). IMHO bogus symlinks to shared libraries constitute a
> real problem of the library installation, so CMake shouldn't just paper
> over it by silently skipping the dead link, proceeding to the next
> candidate which rules out option 2 for me. Also, option 3 can be
> troublesome in automated setups where no user actually reads the CMake
> output.
>

What about warning by default, but accepting a flag to be quiet about it if
the user specifically requests those warnings not be issued?

The issue may "surprise" people, but if they don't understand what a bad
symlink implies they'll be just as surprised (and confused) when their build
fails but the file is "clearly there."  Better to know up front before
putting the extra time into running the compile, IMHO...


>
> - Concerning link-tests:
>  1. Only possible with shared libraries
>

That's OK - we can still do it for those libraries.


>  2. Some shared libraries require special linker flags which CMake
> would require the user to provide. Not so much of an issue, since
> the user needs to provide them anyways.
>

Right.  In fact, a linker failure because of flags not supplied is
information the dev needs in any case, so I don't really see that as a
problem.


>  3. Some shared libraries on *NIX systems are under-linked, i.e. they
> require additional libraries to be specified on the link line.
> Again, CMake can't know about those (as with static libraries).
>

It might be worth looking at AC_CHECK_LIB and how it handles such cases.
One approach might be to support a flag to find_library to specifically skip
link checking for that particular test, if the dev knows in advance it
doesn't link in isolation.


>  4. Last, but not least, linking can be a very expensive operation.
>

That's the tradeoff - better results or more time.  Perhaps a global
CMAKE_CHECK_SHARED_LIBRARY_LINKING variable could turn testing on or off
globally?


> All of these factors make it very hard to come up with a set of tests
> that produces no false negatives (i.e. *never* discards a library that
> is actually valid). I don't care about false positives so much, because
> the linker will complain loudly about those, but if CMake "refuses" to
> find a valid library, that is a *real* problem.
>

What about having the shared library checking variable off by default, but
allowing a project to enable it if they want to deal with the issues of
making sure the tests are valid?  Once the fundamental mechanisms to enable
such tests are in place, the burden then shifts to the project to make sure
they're feeding in the right linker flags and such to correctly test a valid
library.  Some of them will want to pay both the configure time costs and
the developer time costs to have those checks in place.

Cheers,
CY
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Micha Renner]

Am Donnerstag, den 29.09.2011, 10:32 +0200 schrieb Hendrik Sattler:
> Zitat von Micha Renner :
> 
> > Am Donnerstag, den 29.09.2011, 09:22 +0200 schrieb Hendrik Sattler:
> >> Zitat von Michael Wild :
> >> > Just a few of my thoughts on this:
> >>
> >> Same for me.
> >>
> >> > - There are several ways to handle dead symlinks:
> >> >   1. Don't check, let the linker complain (status quo)
> >> >   2. Check whether the found library is a symlink, and if not valid,
> >> >  remove it silently from the list of candidates. Can be very
> >> >  surprising and will likely result in quite a few bug reports
> >> >  about CMake not finding a certain library that is "clearly" there.
> >> >   3. Like 2, but warn about the issue. Possibly very annoying.
> >> > I don't know which option to prefer (perhaps somebody finds another one
> >> > that is better?). IMHO bogus symlinks to shared libraries constitute a
> >> > real problem of the library installation, so CMake shouldn't just paper
> >> > over it by silently skipping the dead link, proceeding to the next
> >> > candidate which rules out option 2 for me. Also, option 3 can be
> >> > troublesome in automated setups where no user actually reads the CMake
> >> > output.
> >>
> >> People tend to misunderstand the purpose of build systems like CMake
> >> and autotools. They try to make the detection bullet-proof and most
> >> likely fail. You can see that this is wrong by looking at those auto*
> >> setups that take ages to detect all possible header files and use an
> >> ifdef hell to handle that in the code, run lots of link and
> >> symbol-finding tests, and still fail on conditions that the author
> >> simple didn't consider or know.
> >>
> >> I prefer option 1. It's not CMake's duty to detect or paper over
> >> messed-up systems. That's the administrator's job!
> >> BTW: running "ldconfig -v" on Linux likely tells you about the
> >> dangling symlink.
> >>
> >> What's next? Asking for a syntax check of found header files so that
> >> the compiler doesn't complain on broken header files?
> >
> > This is exactly what CHECK_INCLUDE_FILES does (via TRY_COMPLIE).
> > It works good as long as the header file has no predecessor. In that
> > case CHECK_INCLUDE_FILES reports FILE-NOT-FOUND. It would save a lot of
> > time, if there would be notice that CHECK_INCLUDE_FILES tests the header
> > file.
> > "Check if the files can be included" is not enough.
> 
> And what does it gain you to do this check except wasting time during  
> build configuration? Do you really check that any add_definitions()  
> does not alter the result? After all, you only get the real result  
> during project compilation.
That is not my point. 
I only say that CHECK_INCLUDE_FILES already tests header files and
CHECK_INCLUDE_FILES reports FILE-NOT_FOUND if the tested header file
needs a predecessor header file.
I bet, that in this case many users rack their brains why
CHECK_INCLUDE_FILES does not find the header file, despite they can see
it in the directory and this should be documented.

The rest is a decision of the maintainers, which I do not comment.

Greetings
Micha

> 
> HS
> 
> 
> --
> 
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
> 
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake


--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Hendrik Sattler]

Zitat von Micha Renner :


Am Donnerstag, den 29.09.2011, 09:22 +0200 schrieb Hendrik Sattler:

Zitat von Michael Wild :
> Just a few of my thoughts on this:

Same for me.

> - There are several ways to handle dead symlinks:
>   1. Don't check, let the linker complain (status quo)
>   2. Check whether the found library is a symlink, and if not valid,
>  remove it silently from the list of candidates. Can be very
>  surprising and will likely result in quite a few bug reports
>  about CMake not finding a certain library that is "clearly" there.
>   3. Like 2, but warn about the issue. Possibly very annoying.
> I don't know which option to prefer (perhaps somebody finds another one
> that is better?). IMHO bogus symlinks to shared libraries constitute a
> real problem of the library installation, so CMake shouldn't just paper
> over it by silently skipping the dead link, proceeding to the next
> candidate which rules out option 2 for me. Also, option 3 can be
> troublesome in automated setups where no user actually reads the CMake
> output.

People tend to misunderstand the purpose of build systems like CMake
and autotools. They try to make the detection bullet-proof and most
likely fail. You can see that this is wrong by looking at those auto*
setups that take ages to detect all possible header files and use an
ifdef hell to handle that in the code, run lots of link and
symbol-finding tests, and still fail on conditions that the author
simple didn't consider or know.

I prefer option 1. It's not CMake's duty to detect or paper over
messed-up systems. That's the administrator's job!
BTW: running "ldconfig -v" on Linux likely tells you about the
dangling symlink.

What's next? Asking for a syntax check of found header files so that
the compiler doesn't complain on broken header files?


This is exactly what CHECK_INCLUDE_FILES does (via TRY_COMPLIE).
It works good as long as the header file has no predecessor. In that
case CHECK_INCLUDE_FILES reports FILE-NOT-FOUND. It would save a lot of
time, if there would be notice that CHECK_INCLUDE_FILES tests the header
file.
"Check if the files can be included" is not enough.


And what does it gain you to do this check except wasting time during  
build configuration? Do you really check that any add_definitions()  
does not alter the result? After all, you only get the real result  
during project compilation.


HS


--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Micha Renner]

Am Donnerstag, den 29.09.2011, 09:22 +0200 schrieb Hendrik Sattler:
> Zitat von Michael Wild :
> > Just a few of my thoughts on this:
> 
> Same for me.
> 
> > - There are several ways to handle dead symlinks:
> >   1. Don't check, let the linker complain (status quo)
> >   2. Check whether the found library is a symlink, and if not valid,
> >  remove it silently from the list of candidates. Can be very
> >  surprising and will likely result in quite a few bug reports
> >  about CMake not finding a certain library that is "clearly" there.
> >   3. Like 2, but warn about the issue. Possibly very annoying.
> > I don't know which option to prefer (perhaps somebody finds another one
> > that is better?). IMHO bogus symlinks to shared libraries constitute a
> > real problem of the library installation, so CMake shouldn't just paper
> > over it by silently skipping the dead link, proceeding to the next
> > candidate which rules out option 2 for me. Also, option 3 can be
> > troublesome in automated setups where no user actually reads the CMake
> > output.
> 
> People tend to misunderstand the purpose of build systems like CMake  
> and autotools. They try to make the detection bullet-proof and most  
> likely fail. You can see that this is wrong by looking at those auto*  
> setups that take ages to detect all possible header files and use an  
> ifdef hell to handle that in the code, run lots of link and  
> symbol-finding tests, and still fail on conditions that the author  
> simple didn't consider or know.
> 
> I prefer option 1. It's not CMake's duty to detect or paper over  
> messed-up systems. That's the administrator's job!
> BTW: running "ldconfig -v" on Linux likely tells you about the  
> dangling symlink.
> 
> What's next? Asking for a syntax check of found header files so that  
> the compiler doesn't complain on broken header files?

This is exactly what CHECK_INCLUDE_FILES does (via TRY_COMPLIE).
It works good as long as the header file has no predecessor. In that
case CHECK_INCLUDE_FILES reports FILE-NOT-FOUND. It would save a lot of
time, if there would be notice that CHECK_INCLUDE_FILES tests the header
file.
"Check if the files can be included" is not enough.

Greetings

Micha



> 
> HS
> 
> 
> --
> 
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
> 
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake


--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Hendrik Sattler]

Zitat von Michael Wild :

Just a few of my thoughts on this:


Same for me.


- There are several ways to handle dead symlinks:
  1. Don't check, let the linker complain (status quo)
  2. Check whether the found library is a symlink, and if not valid,
 remove it silently from the list of candidates. Can be very
 surprising and will likely result in quite a few bug reports
 about CMake not finding a certain library that is "clearly" there.
  3. Like 2, but warn about the issue. Possibly very annoying.
I don't know which option to prefer (perhaps somebody finds another one
that is better?). IMHO bogus symlinks to shared libraries constitute a
real problem of the library installation, so CMake shouldn't just paper
over it by silently skipping the dead link, proceeding to the next
candidate which rules out option 2 for me. Also, option 3 can be
troublesome in automated setups where no user actually reads the CMake
output.


People tend to misunderstand the purpose of build systems like CMake  
and autotools. They try to make the detection bullet-proof and most  
likely fail. You can see that this is wrong by looking at those auto*  
setups that take ages to detect all possible header files and use an  
ifdef hell to handle that in the code, run lots of link and  
symbol-finding tests, and still fail on conditions that the author  
simple didn't consider or know.


I prefer option 1. It's not CMake's duty to detect or paper over  
messed-up systems. That's the administrator's job!
BTW: running "ldconfig -v" on Linux likely tells you about the  
dangling symlink.


What's next? Asking for a syntax check of found header files so that  
the compiler doesn't complain on broken header files?


HS


--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Rolf Eike Beer]

> On 09/29/2011 07:15 AM, Clifford Yapp wrote:
>> On Wed, Sep 28, 2011 at 8:52 PM, Michael Hertling
>> mailto:mhertl...@online.de>> wrote:

>> I guess the question revolves around the expectation of find_library
>> being different from find_file - as a user, my expectation would be
>> that find_library is doing something to distinguish libraries from
>> files (when that's technically workable, of course - clearly solving
>>  that problem in general is hard.)  If something tricky like
>> cross-compiling is going on then the simpler testing behavior is in
>> order, but couldn't CMake scrub the results looking for library
>> validity as much as possible?
>>
>> Cheers, CY
>
> Just a few of my thoughts on this:
>
> - There are several ways to handle dead symlinks:
>   1. Don't check, let the linker complain (status quo)
>   2. Check whether the found library is a symlink, and if not valid,
>  remove it silently from the list of candidates. Can be very
>  surprising and will likely result in quite a few bug reports
>  about CMake not finding a certain library that is "clearly" there.
>   3. Like 2, but warn about the issue. Possibly very annoying.

3a) Warn about skipped links if no library is found at the end (e.g.
"foolib not found, dead symlinks ignored during search") or when called
with some debug flag.

Eike
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Michael Wild]

On 09/29/2011 07:15 AM, Clifford Yapp wrote:
> On Wed, Sep 28, 2011 at 8:52 PM, Michael Hertling 
> mailto:mhertl...@online.de>> wrote:
> 
> What do you do on systems which have no idea of symbolic links, e.g. 
> previous Windows versions? Adding more platform-specific code to the 
> sources of the FIND_LIBRARY() function?
> 
> 
> If the problem isn't solved (or readily solvable) on some platforms,
>  we can just fall back to doing what we do now in those cases... we 
> don't have to solve the problem on all possible platforms to improve
>  the results on platforms that can be supported.  The point wouldn't
>  be to *guarantee* the result of find_library is a valid working 
> library (it doesn't do that now), just improve the results by doing 
> what can be done to avoid returning results that can be tagged as 
> non-working (since a build trying to use such results is guaranteed 
> to fail anyway.)  Does cmake know (or can it tell) when it is cross 
> compiling?  If so, it could decide how much to test and not test in 
> those cases.
> 
> 
> Furthermore, the kernels of *nix systems hardly distinguish between 
> binary files and text files; usually, they know just "files" with the
> limited exception of being able to recognize the native executable
> formats and the #! shebang. The detection of the diverse file types
> is typically implemented in utility programs, notably the file(1)
> utility.
> 
> 
> A linker test when possible (non-cross-compilation scenarios) with a 
> fallback testing option using file introspection would handle a lot 
> of situations, if I'm understanding correctly how this works.  To the
> best of my knowledge and that of one of our other devs (he knows a
> lot more than me) there aren't any platforms where actual library 
> files (as opposed to linker scripts) consist solely of character 
> values <128. Checking a found file for at least one character over 
> 128 would avoid at least a few failure cases - empty files, plain 
> text files, and dead links.  Such a check would be reliable, consist,
> and would be useful in eliminating files we might find that are not
> actually library files.
> 
> Such a test  of course wouldn't pass linker scripts like ubuntu's 
> libc.so, but such a script *would* pass the actual linker test and 
> would never get to file introspection. Actually if the script failed
>  the linker test, then there is a real problem that should be a 
> failure case.  This would also be helpful in spotting early on the 
> hypothetical case of a linker being tested that doesn't understand 
> the .so linker script.
> 
> 
>>> But how do you test whether a library is "linkable"?
> 
> 
> If not cross-compiling, wouldn't it be possible to have find_library
>  try the link as part of its test?  Even in the case where linking 
> isn't viable, falling back on the check on file contents would still
>  be helpful.
> 
> 
> What do you do if the library found by FIND_LIBRARY() has a 
> non-native binary format, e.g. for cross-compiling purposes? How do 
> you select the right set of tools to check the library in question 
> without the user's ado? IMO, FIND_LIBRARY()'s job is to find library
>  files, and a general check whether these files are 
> valid/usable/linkable and not dangling symlinks or whatever is beyond
> the scope of this function; ensuring that is rather the realm of
> system administration.
> 
> 
> I guess the question revolves around the expectation of find_library 
> being different from find_file - as a user, my expectation would be 
> that find_library is doing something to distinguish libraries from 
> files (when that's technically workable, of course - clearly solving
>  that problem in general is hard.)  If something tricky like 
> cross-compiling is going on then the simpler testing behavior is in 
> order, but couldn't CMake scrub the results looking for library 
> validity as much as possible?
> 
> Cheers, CY

Just a few of my thoughts on this:

- There are several ways to handle dead symlinks:
  1. Don't check, let the linker complain (status quo)
  2. Check whether the found library is a symlink, and if not valid,
 remove it silently from the list of candidates. Can be very
 surprising and will likely result in quite a few bug reports
 about CMake not finding a certain library that is "clearly" there.
  3. Like 2, but warn about the issue. Possibly very annoying.
I don't know which option to prefer (perhaps somebody finds another one
that is better?). IMHO bogus symlinks to shared libraries constitute a
real problem of the library installation, so CMake shouldn't just paper
over it by silently skipping the dead link, proceeding to the next
candidate which rules out option 2 for me. Also, option 3 can be
troublesome in automated setups where no user actually reads the CMake
output.

- Concerning link-tests:
  1. Only possible with shared libraries
  2. Some shared libraries require special linker flags which CMake
 would require the user 

Re: [CMake] Does find_library check that a found library does in fact link?

[Clifford Yapp]

On Wed, Sep 28, 2011 at 8:52 PM, Michael Hertling wrote:

What do you do on systems which have no idea of symbolic links, e.g.
> previous Windows versions? Adding more platform-specific code to the
> sources of the FIND_LIBRARY() function?


If the problem isn't solved (or readily solvable) on some platforms, we can
just fall back to doing what we do now in those cases... we don't have to
solve the problem on all possible platforms to improve the results on
platforms that can be supported.  The point wouldn't be to *guarantee* the
result of find_library is a valid working library (it doesn't do that now),
just improve the results by doing what can be done to avoid returning
results that can be tagged as non-working (since a build trying to use such
results is guaranteed to fail anyway.)  Does cmake know (or can it tell)
when it is cross compiling?  If so, it could decide how much to test and not
test in those cases.


> Furthermore, the kernels of
> *nix systems hardly distinguish between binary files and text files;
> usually, they know just "files" with the limited exception of being
> able to recognize the native executable formats and the #! shebang.
> The detection of the diverse file types is typically implemented
> in utility programs, notably the file(1) utility.
>

A linker test when possible (non-cross-compilation scenarios) with a
fallback testing option using file introspection would handle a lot of
situations, if I'm understanding correctly how this works.  To the best of
my knowledge and that of one of our other devs (he knows a lot more than me)
there aren't any platforms where actual library files (as opposed to linker
scripts) consist solely of character values <128.  Checking a found file for
at least one character over 128 would avoid at least a few failure cases -
empty files, plain text files, and dead links.  Such a check would be
reliable, consist, and would be useful in eliminating files we might find
that are not actually library files.

Such a test  of course wouldn't pass linker scripts like ubuntu's libc.so,
but such a script *would* pass the actual linker test and would never get to
file introspection. Actually if the script failed the linker test, then
there is a real problem that should be a failure case.  This would also be
helpful in spotting early on the hypothetical case of a linker being tested
that doesn't understand the .so linker script.


> >> But how do you test whether a library is "linkable"?
>

If not cross-compiling, wouldn't it be possible to have find_library try the
link as part of its test?  Even in the case where linking isn't viable,
falling back on the check on file contents would still be helpful.


> What do you do if the library found by FIND_LIBRARY() has a non-native
> binary format, e.g. for cross-compiling purposes? How do you select the
> right set of tools to check the library in question without the user's
> ado? IMO, FIND_LIBRARY()'s job is to find library files, and a general
> check whether these files are valid/usable/linkable and not dangling
> symlinks or whatever is beyond the scope of this function; ensuring
> that is rather the realm of system administration.
>

I guess the question revolves around the expectation of find_library being
different from find_file - as a user, my expectation would be that
find_library is doing something to distinguish libraries from files (when
that's technically workable, of course - clearly solving that problem in
general is hard.)  If something tricky like cross-compiling is going on then
the simpler testing behavior is in order, but couldn't CMake scrub the
results looking for library validity as much as possible?

Cheers,
CY
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Michael Wild]

On 09/29/2011 02:52 AM, Michael Hertling wrote:
> On 09/29/2011 01:30 AM, Clifford Yapp wrote:
>> On Wed, Sep 28, 2011 at 2:47 AM, Michael Wild  wrote:
>>
>>>
>>> Only if your installation is broken ;-) If the symlink is broken, I
>>> consider this to be a user-error. Period. OTOH, CMake /could/ check
>>> whether the library is a symlink, and if it is, check that it is valid.
>>>
>>
>> Oh, no question the installation is broken.  I'd just expect find_library to
>> do whatever minimal validation it can easily do and not return invalid cases
>> it can spot - checking for symlink and whether it's valid would catch one
>> general class of error, and perhaps a quick check to see if the file is a
>> binary or a text file would be another.   Not perfect, but such tests should
>> be relatively simple and would improve the utility of find_library.
> 
> What do you do on systems which have no idea of symbolic links, e.g.
> previous Windows versions? Adding more platform-specific code to the
> sources of the FIND_LIBRARY() function? Furthermore, the kernels of
> *nix systems hardly distinguish between binary files and text files;
> usually, they know just "files" with the limited exception of being
> able to recognize the native executable formats and the #! shebang.
> The detection of the diverse file types is typically implemented
> in utility programs, notably the file(1) utility.

And to add to the confusion, GNU ld also accepts linker scripts which
are text files, but are named like libraries. E.g. on my Ubuntu 11.04
box /usr/lib/x86_64-linux-gnu/libc.so is actually a linker script.

file(1) uses "magic" numbers and heuristics to detect the file type, and
it certainly isn't bulletproof...

Michael

--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Michael Hertling]

On 09/29/2011 01:30 AM, Clifford Yapp wrote:
> On Wed, Sep 28, 2011 at 2:47 AM, Michael Wild  wrote:
> 
>>
>> Only if your installation is broken ;-) If the symlink is broken, I
>> consider this to be a user-error. Period. OTOH, CMake /could/ check
>> whether the library is a symlink, and if it is, check that it is valid.
>>
> 
> Oh, no question the installation is broken.  I'd just expect find_library to
> do whatever minimal validation it can easily do and not return invalid cases
> it can spot - checking for symlink and whether it's valid would catch one
> general class of error, and perhaps a quick check to see if the file is a
> binary or a text file would be another.   Not perfect, but such tests should
> be relatively simple and would improve the utility of find_library.

What do you do on systems which have no idea of symbolic links, e.g.
previous Windows versions? Adding more platform-specific code to the
sources of the FIND_LIBRARY() function? Furthermore, the kernels of
*nix systems hardly distinguish between binary files and text files;
usually, they know just "files" with the limited exception of being
able to recognize the native executable formats and the #! shebang.
The detection of the diverse file types is typically implemented
in utility programs, notably the file(1) utility.

>> But how do you test whether a library is "linkable"?
>>
> 
> Not sure - autoconf has some sort of test that works in at least some cases
> in their AC_CHECK_LIB macro, but I'm not really clear on what it does.  Even
> if such a test wouldn't catch all cases, mightn't it be useful to fail when
> available tests do detect a problem?

What do you do if the library found by FIND_LIBRARY() has a non-native
binary format, e.g. for cross-compiling purposes? How do you select the
right set of tools to check the library in question without the user's
ado? IMO, FIND_LIBRARY()'s job is to find library files, and a general
check whether these files are valid/usable/linkable and not dangling
symlinks or whatever is beyond the scope of this function; ensuring
that is rather the realm of system administration.

> Cheers,
> CY

Regards,

Michael
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Clifford Yapp]

On Wed, Sep 28, 2011 at 2:47 AM, Michael Wild  wrote:

>
> Only if your installation is broken ;-) If the symlink is broken, I
> consider this to be a user-error. Period. OTOH, CMake /could/ check
> whether the library is a symlink, and if it is, check that it is valid.
>

Oh, no question the installation is broken.  I'd just expect find_library to
do whatever minimal validation it can easily do and not return invalid cases
it can spot - checking for symlink and whether it's valid would catch one
general class of error, and perhaps a quick check to see if the file is a
binary or a text file would be another.   Not perfect, but such tests should
be relatively simple and would improve the utility of find_library.


> But how do you test whether a library is "linkable"?
>

Not sure - autoconf has some sort of test that works in at least some cases
in their AC_CHECK_LIB macro, but I'm not really clear on what it does.  Even
if such a test wouldn't catch all cases, mightn't it be useful to fail when
available tests do detect a problem?

Cheers,
CY
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Eric Noulard]

2011/9/28 Michael Wild :
> On 09/28/2011 07:47 AM, Clifford Yapp wrote:
>>
>>
>> On Wed, Sep 28, 2011 at 12:13 AM, Michael Wild > > wrote:
>>
>>     On 09/28/2011 02:44 AM, Clifford Yapp wrote:
>>     > I've run into a situation where find_library is returning a symlink:
>>     >
>>     > /usr/lib/libblah.so -> libblah.so.1
>>     >
>>     >  but libblah.so.1 does not actually exist (e.g. the symlink is bad).
>>     >
>>     > Is there an option I can set to have find_library ensure that a found
>>     > library file is valid and links?
>>     >
>>     > Cheers,
>>     > CY
>>     >
>>
>>     Yes, but you need to do it yourself, e.g. use the CheckFunctionExists
>>     module.
>>
>>     Michael
>>
>>
>> Urm.  That's surprising, and not so hot in that it leads to "false
>> positives" in find_library reports.  Has anyone wrapped find_library
>> with a generic linking test successfully?
>>
>> CY
>
> Only if your installation is broken ;-) If the symlink is broken, I
> consider this to be a user-error. Period. OTOH, CMake /could/ check
> whether the library is a symlink, and if it is, check that it is valid.

I agree that the breakage is on user side and not CMake.
However CMake may just warn that the found symlink is broken,
and nothing more.
At least the user won't be puzzled by a non-working build.


> But how do you test whether a library is "linkable"? Specifically, I'm
> thinking about static libraries. First, only referenced symbols are
> linked, so that's kind of difficult to handle as CMake would need to
> know at least one symbol name to force the library to be linked at all,
> secondly static libraries may have transitive dependencies which CMake
> doesn't know about, so it's not possible to link an executable. On Unix
> you could create a shared library for the link test, but that depends on
> the linker flags being used, and on Windows you can't, since DLL's may
> not contain unresolved symbols.
>
> I don't think that there is any good way to handle this in find_library().

Agreed as well but this is another story.


-- 
Erk
Membre de l'April - « promouvoir et défendre le logiciel libre » -
http://www.april.org
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Michael Wild]

On 09/28/2011 07:47 AM, Clifford Yapp wrote:
> 
> 
> On Wed, Sep 28, 2011 at 12:13 AM, Michael Wild  > wrote:
> 
> On 09/28/2011 02:44 AM, Clifford Yapp wrote:
> > I've run into a situation where find_library is returning a symlink:
> >
> > /usr/lib/libblah.so -> libblah.so.1
> >
> >  but libblah.so.1 does not actually exist (e.g. the symlink is bad).
> >
> > Is there an option I can set to have find_library ensure that a found
> > library file is valid and links?
> >
> > Cheers,
> > CY
> >
> 
> Yes, but you need to do it yourself, e.g. use the CheckFunctionExists
> module.
> 
> Michael
> 
> 
> Urm.  That's surprising, and not so hot in that it leads to "false
> positives" in find_library reports.  Has anyone wrapped find_library
> with a generic linking test successfully?
> 
> CY

Only if your installation is broken ;-) If the symlink is broken, I
consider this to be a user-error. Period. OTOH, CMake /could/ check
whether the library is a symlink, and if it is, check that it is valid.

But how do you test whether a library is "linkable"? Specifically, I'm
thinking about static libraries. First, only referenced symbols are
linked, so that's kind of difficult to handle as CMake would need to
know at least one symbol name to force the library to be linked at all,
secondly static libraries may have transitive dependencies which CMake
doesn't know about, so it's not possible to link an executable. On Unix
you could create a shared library for the link test, but that depends on
the linker flags being used, and on Windows you can't, since DLL's may
not contain unresolved symbols.

I don't think that there is any good way to handle this in find_library().

Michael
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Rolf Eike Beer]

Michael Wild wrote:
> On 09/28/2011 02:44 AM, Clifford Yapp wrote:
> > I've run into a situation where find_library is returning a symlink:
> > 
> > /usr/lib/libblah.so -> libblah.so.1
> > 
> >  but libblah.so.1 does not actually exist (e.g. the symlink is bad).
> > 
> > Is there an option I can set to have find_library ensure that a found
> > library file is valid and links?

> Yes, but you need to do it yourself, e.g. use the CheckFunctionExists
> module.

Which is unreliable on it's own as we found out during the FindThreads pthread 
stuff. I think this should be considered as a bug.

Eike

signature.asc
Description: This is a digitally signed message part.
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Clifford Yapp]

On Wed, Sep 28, 2011 at 12:13 AM, Michael Wild  wrote:

> On 09/28/2011 02:44 AM, Clifford Yapp wrote:
> > I've run into a situation where find_library is returning a symlink:
> >
> > /usr/lib/libblah.so -> libblah.so.1
> >
> >  but libblah.so.1 does not actually exist (e.g. the symlink is bad).
> >
> > Is there an option I can set to have find_library ensure that a found
> > library file is valid and links?
> >
> > Cheers,
> > CY
> >
>
> Yes, but you need to do it yourself, e.g. use the CheckFunctionExists
> module.
>
> Michael
>

Urm.  That's surprising, and not so hot in that it leads to "false
positives" in find_library reports.  Has anyone wrapped find_library with a
generic linking test successfully?

CY
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

Re: [CMake] Does find_library check that a found library does in fact link?

[Michael Wild]

On 09/28/2011 02:44 AM, Clifford Yapp wrote:
> I've run into a situation where find_library is returning a symlink:
> 
> /usr/lib/libblah.so -> libblah.so.1
> 
>  but libblah.so.1 does not actually exist (e.g. the symlink is bad).
> 
> Is there an option I can set to have find_library ensure that a found
> library file is valid and links?
> 
> Cheers,
> CY
> 

Yes, but you need to do it yourself, e.g. use the CheckFunctionExists
module.

Michael
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake

[CMake] Does find_library check that a found library does in fact link?

[Clifford Yapp]

I've run into a situation where find_library is returning a symlink:

/usr/lib/libblah.so -> libblah.so.1

 but libblah.so.1 does not actually exist (e.g. the symlink is bad).

Is there an option I can set to have find_library ensure that a found
library file is valid and links?

Cheers,
CY
--

Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake