Re: dlopen()ing shared libraries considered harmful (was Re: Depends/Recommends from libraries)

[Jens Reyer]

On 03/26/2017 09:37 AM, Florian Weimer wrote:
> * Guillem Jover:
> 
>>> dlopen()ing dependencies in the way that is most commonly implemented,
>>> with dlopen("libimobiledevice.so.6") and dlsym(handle, "idevice_new")
>>> or similar, has some practical problems for Debian:
>>>
>>> * The libraries used aren't visible to dpkg-shlibdeps. The maintainer has
>>>   to know what dlopen() calls the code will make, and either hard-code an
>>>   appropriate Depends (or in this case Recommends), or link a dummy
>>>   executable against the same things that are dlopen()ed (as is done
>>>   in packages like wine and openal-soft) and use that for dpkg-shlibdeps.
>>>   Either way, they have to remember to update it for every new upstream
>>>   release. This is the sort of tedious-but-subtle work that we automate
>>>   because otherwise it will inevitably be incorrect after a few releases.
>>>
>>> * The exact symbols used aren't visible to dpkg-shlibdeps. The maintainer
>>>   has to hard-code a version number that has all the required symbols.
>>>   Again, this is the sort of thing that we automate because it will
>>>   inevitably go wrong if done manually.
>>
>> And:
>>
>>  * Because the shared library SONAME is hard-coded in both code and
>>packaging metadata, whenever there's a transition with a SOVERSION
>>bump that could have been handled with a simple mass binNMU, instead
>>it requires manual intervention to patch those, while making sure it
>>still works due to the points above.

At least for Wine the SONAME of dlopen'ed libraries is not hardcoded,
but computed during configure.  We then grep these SONAMEs and build a
helper elf binary to compute the needed dependencies with dpkg-shlibdeps.

So this solution is fully automatic both upstream and in the packaging,
therefore it allows for binNMUs.

Building the helper binary could even be omitted if someone implements
this in dpkg-shlibdeps (see #596715 "dpkg-shlibdeps: Please allow to
manually add library dependencies via shlibdeps").

Greets
jre


> And:
> 
> * For regular dynamic linking, the static linker (ld) puts the default
>   symbol version seen in the linked DSO into the object which refers
>   to that DSO, so that each symbol refers to a fixed symbol version.
>   This symbol version matches the header file contents of the library
>   at the time, and is consistent with the link.  With dlopen/dlsym,
>   this step does not happen, and dlsym will pick the newest (*) symbol
>   version.  This version might not match what was in the header files
>   at compilation time.  (Theoretically, it would be possible to
>   determine the applicable symbol version at compile time and use
>   dlvsym, but no one does that.)
> 
> (*) Not quite because we have a bug:
> 
> 
>>> The pedantically correct way to have weak library dependencies is this
>>> multi-level chain of linking, analogous to what is documented for APIs
>>> that wrap dlopen() such as libltdl and GLib's GModule:
>>>
>>> - eventual API user, e.g. /usr/bin/gnome-software
>>>   dlopen()s:
>>> - a plugin intended to be dlopen()ed, e.g. 
>>> libgs_plugin_flatpak-user.so
>>>   has DT_NEEDED on:
>>> - libflatpak0
>>
>> I don't think this is pedantical at all, it's IMO the only sane and
>> correct way to handle this.
> 
> Right, this way also allows the static linker to do its job and fix
> the symbol version.

Re: dlopen()ing shared libraries considered harmful (was Re: Depends/Recommends from libraries)

[Wouter Verhelst]

On Sun, Mar 26, 2017 at 05:44:20AM +0200, Guillem Jover wrote:
> As I've also noted in the past [B], I'd go even further and say that
> we need at least to very strongly discourage, but ideally outright ban
> the dlopen()ing of shared libraries that are not part of the same
> source package or at least under the control of the same upstream
> project, or are shared libraries that define themselves to the ABI
> and SONAME level (but those are very few and rare).

This wording as written bans third-party plugins. I'm sure you didn't mean
that, and meant instead something like wanting to have .so files be used
*either* for dlopen() *or* for compile-time dynamic linking.

Unfortunately, that too would forbid some legitimate uses. For example,
the PKCS#11 standard defines an API which can be used to access
cryptographic routines and is popular for accessing hardware security
modules (HSMs) or smart cards. It's supported by OpenSSL (using an
engine), OpenSSH (for converting RSA keys on smartcards to SSH keys),
Firefox, Libreoffice, and Thunderbird, as well as various other bits of
non-free software. OpenSC is one example of a package in Debian which
ships a PKCS#11 module. An application would typically use the PKCS#11
module by dlopen()ing it, based on configuration related to the specific
piece of hardware you're trying to use.

At the same time, the API is flexible enough that it can be used by a
specific implementation to export extra data beyond the cryptographic
material. An application using the API in that way would probably be
written for one specific device, rather than any random device, and
hence it might be easier for the programmer to link to the shared object
at compile time.

I realize this is a bit extreme, but it's not a hypothetical scenario;
the support software for the Belgian electronic ID card that I work on
at $DAYJOB works in the above-described fashion. Having said that, I
would agree with the general feeling that in most cases dlopen()ing
random shared libraries is a bad idea; but at best, this can be
specified as a SHOULD (in the RFC sense, not the Policy sense).

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: Depends/Recommends from libraries

[Guillem Jover]

Hi!

On Thu, 2017-03-23 at 17:34:58 +, Ian Jackson wrote:
> Russ Allbery writes ("Re: Depends/Recommends from libraries"):
> >   It still feels like needless complexity to me,
> 
> Here is an example I just found.

It does not turn out to be a very good example, I'm afraid. :)

> Try, in a fresh stretch chroot
>   apt-get --no-install-recommends install libgtkspell-dev
> Which you might reasonably do because you were building something
> which linked against libgtk-spell.  Or because you installed some
> portmanteau package like ocaml-libs.
> 
> -dev libraries are typically installed because you want to build
> something, not necessarily because you wanted to _use_ the features of
> the runtime library.
> 
> The list that comes out is humungous and contains many things that you
> probably didn't want.

But after checking that list, most of it comes from other -dev
packages and their shared library counterparts. The packages that you
list below are relatively tiny in comparison! Around ~15-20 MiB vs the
total ~280 MiB installed size.

> aspell ?  Well, at least the connection is clear, although it's
> clearly wrong to get it.

Yes, the problem is that dictionaries such as aspell-en, depend on
aspell, when they should just Recommend/Suggest it, primarily because
this adds an unnecessary dependency cycle, and because the data does
not require the program. Needs a bug filed.

> fontconfig ?  Probably pulled in pointlessly by one of the other
> librararies.

Pulled by libpango-1.0-0. I've checked and it seems currently
unnecessary, nothing in the source calls any «fc-*» command. It seems
to originate from the very beginning of the pango1.0 packaging source
history in svn, so probably just uncollected cruft. Needs a bug filed.

> dictionaries-common ?  That's a waste of a download.

Pulled by aspell-en → pulled by libenchant. AFAIK used to register
the dictionaries, which seems legitimate to me. (750 KiB)

> emacsen-common ?  Wow.

Pulled by dictionaries-common. Yeah I've always wondered about this
one, perhaps the packaging could be reworked to avoid it, but the
complexity implied might be too great for the few KiB that this
dependency introduces. (135 KiB)

> libgtk2.0-dev, which lots of things obviously build-depend on, pulls
> in fontconfig and gnome-icon-theme (a 10Mby download).  I can't
> believe you need gnome-icon-theme installed to compile gtk2 programs.

I could probably imagine a tool built during the build making use of it,
or while running test suites or similar. And while this is the biggest
of the non lib/-dev packages, trying to remove this dependeny would seem
wrong to me, as this is an implementation detail.

Perhaps one solution here could be to find a smaller compatible or mock
icon-theme, and depend on it as first option from the gtk -dev package.
But I'm not sure how that would play with apt when having to choose
between the first options in the libgtk-dev and libgtk. :)

In any case this is still inert data, and not huge in comparison to
the total size to be installed.

Thanks,
Guillem

Re: dlopen()ing shared libraries considered harmful (was Re: Depends/Recommends from libraries)

[Florian Weimer]

* Guillem Jover:

>> dlopen()ing dependencies in the way that is most commonly implemented,
>> with dlopen("libimobiledevice.so.6") and dlsym(handle, "idevice_new")
>> or similar, has some practical problems for Debian:
>> 
>> * The libraries used aren't visible to dpkg-shlibdeps. The maintainer has
>>   to know what dlopen() calls the code will make, and either hard-code an
>>   appropriate Depends (or in this case Recommends), or link a dummy
>>   executable against the same things that are dlopen()ed (as is done
>>   in packages like wine and openal-soft) and use that for dpkg-shlibdeps.
>>   Either way, they have to remember to update it for every new upstream
>>   release. This is the sort of tedious-but-subtle work that we automate
>>   because otherwise it will inevitably be incorrect after a few releases.
>> 
>> * The exact symbols used aren't visible to dpkg-shlibdeps. The maintainer
>>   has to hard-code a version number that has all the required symbols.
>>   Again, this is the sort of thing that we automate because it will
>>   inevitably go wrong if done manually.
>
> And:
>
>  * Because the shared library SONAME is hard-coded in both code and
>packaging metadata, whenever there's a transition with a SOVERSION
>bump that could have been handled with a simple mass binNMU, instead
>it requires manual intervention to patch those, while making sure it
>still works due to the points above.

And:

* For regular dynamic linking, the static linker (ld) puts the default
  symbol version seen in the linked DSO into the object which refers
  to that DSO, so that each symbol refers to a fixed symbol version.
  This symbol version matches the header file contents of the library
  at the time, and is consistent with the link.  With dlopen/dlsym,
  this step does not happen, and dlsym will pick the newest (*) symbol
  version.  This version might not match what was in the header files
  at compilation time.  (Theoretically, it would be possible to
  determine the applicable symbol version at compile time and use
  dlvsym, but no one does that.)

(*) Not quite because we have a bug:


>> The pedantically correct way to have weak library dependencies is this
>> multi-level chain of linking, analogous to what is documented for APIs
>> that wrap dlopen() such as libltdl and GLib's GModule:
>> 
>> - eventual API user, e.g. /usr/bin/gnome-software
>>   dlopen()s:
>> - a plugin intended to be dlopen()ed, e.g. 
>> libgs_plugin_flatpak-user.so
>>   has DT_NEEDED on:
>> - libflatpak0
>
> I don't think this is pedantical at all, it's IMO the only sane and
> correct way to handle this.

Right, this way also allows the static linker to do its job and fix
the symbol version.

Re: Depends/Recommends from libraries

[Guillem Jover]

On Tue, 2017-03-21 at 12:36:16 +, Ian Jackson wrote:
> Russ Allbery writes ("Re: Depends/Recommends from libraries"):
> > I think this would be a great way of introducing spurious bugs in our
> > distribution from [developers] who don't happen to read the README
> > file and miss dependencies they actually need because they're used
> > to Debian properly picking up shared library dependencies and to the
> > dependencies of any given package being fully self-contained.  Both
> > of which, I should add, are major *features* of our distribution
> > that many of us have worked very hard to achieve.  I'm opposed.
> > 
> > Now, if this were taken a further step so that dpkg-shlibdeps would
> > provide some mechanism to *automatically* add those downstream
> > dependencies to packages that depend on the library unless the
> > dependencies were explicitly suppressed, I wouldn't be as strongly
> > opposed.
> 
> This seems doable to me.

I don't think it's a matter of being doable or not, it certainly is.
I think it's a matter of whether this is correct or not. I think it
might be in extremely few cases, but in most it would be outright
wrong.

What this will end up doing is pushing dependencies for implementation
details towards far away transitive dependencies.

Whenever those implementation details change all those other packages
will need to be rebuilt, although this would still does not solve
artial upgrades.

> >  It still feels like needless complexity to me,
> 
> Others have explained why this is a real problem.

I think in many cases those problems can be solved in other ways, of
course that might imply work, but they are still solvable in better
ways.

Thanks,
Guillem

dlopen()ing shared libraries considered harmful (was Re: Depends/Recommends from libraries)

[Guillem Jover]

Hi!

On Fri, 2017-03-10 at 10:16:58 +, Simon McVittie wrote:
> I do not agree that dlopen()ing dependencies (what you have called "dynamic
> loading") is something we should encourage over normal linking with -lfoo
> (resulting in a DT_NEEDED entry, what you have called "static loading").

Thanks for this write-up! I've had pending in my queue something like
this for a long time, but never found the time. It's still one of my
long-term personal crusades [C]. :)

As I've also noted in the past [B], I'd go even further and say that
we need at least to very strongly discourage, but ideally outright ban
the dlopen()ing of shared libraries that are not part of the same
source package or at least under the control of the same upstream
project, or are shared libraries that define themselves to the ABI
and SONAME level (but those are very few and rare).

 [C] 
 [B] 
 

> dlopen()ing dependencies in the way that is most commonly implemented,
> with dlopen("libimobiledevice.so.6") and dlsym(handle, "idevice_new")
> or similar, has some practical problems for Debian:
> 
> * The libraries used aren't visible to dpkg-shlibdeps. The maintainer has
>   to know what dlopen() calls the code will make, and either hard-code an
>   appropriate Depends (or in this case Recommends), or link a dummy
>   executable against the same things that are dlopen()ed (as is done
>   in packages like wine and openal-soft) and use that for dpkg-shlibdeps.
>   Either way, they have to remember to update it for every new upstream
>   release. This is the sort of tedious-but-subtle work that we automate
>   because otherwise it will inevitably be incorrect after a few releases.
> 
> * The exact symbols used aren't visible to dpkg-shlibdeps. The maintainer
>   has to hard-code a version number that has all the required symbols.
>   Again, this is the sort of thing that we automate because it will
>   inevitably go wrong if done manually.

And:

 * Because the shared library SONAME is hard-coded in both code and
   packaging metadata, whenever there's a transition with a SOVERSION
   bump that could have been handled with a simple mass binNMU, instead
   it requires manual intervention to patch those, while making sure it
   still works due to the points above.

> The pedantically correct way to have weak library dependencies is this
> multi-level chain of linking, analogous to what is documented for APIs
> that wrap dlopen() such as libltdl and GLib's GModule:
> 
> - eventual API user, e.g. /usr/bin/gnome-software
>   dlopen()s:
> - a plugin intended to be dlopen()ed, e.g. 
> libgs_plugin_flatpak-user.so
>   has DT_NEEDED on:
> - libflatpak0

I don't think this is pedantical at all, it's IMO the only sane and
correct way to handle this.

Thanks,
Guillem

Re: Depends/Recommends from libraries

[Ian Jackson]

Russ Allbery writes ("Re: Depends/Recommends from libraries"):
>   It still feels like needless complexity to me,

Here is an example I just found.

Try, in a fresh stretch chroot
  apt-get --no-install-recommends install libgtkspell-dev
Which you might reasonably do because you were building something
which linked against libgtk-spell.  Or because you installed some
portmanteau package like ocaml-libs.

-dev libraries are typically installed because you want to build
something, not necessarily because you wanted to _use_ the features of
the runtime library.

The list that comes out is humungous and contains many things that you
probably didn't want.

aspell ?  Well, at least the connection is clear, although it's
clearly wrong to get it.

fontconfig ?  Probably pulled in pointlessly by one of the other
librararies.

dictionaries-common ?  That's a waste of a download.

emacsen-common ?  Wow.

libgtk2.0-dev, which lots of things obviously build-depend on, pulls
in fontconfig and gnome-icon-theme (a 10Mby download).  I can't
believe you need gnome-icon-theme installed to compile gtk2 programs.

Ian.

Re: Rethinking dynamic linking a bit (was: Re: Depends/Recommends from libraries)

[Paul Wise]

On Wed, Mar 22, 2017 at 6:17 PM, Christian Seiler wrote:

> For $DAYJOB I had to work on Mac OS X a bit, and they have an interesting
> feature there: weakly binding to a shared library.

Apparently Solaris has support for optional shared libraries:

https://lists.debian.org/debian-devel/2015/02/msg00261.html
https://sourceware.org/ml/libc-help/2013-02/msg00017.html
http://comments.gmane.org/gmane.comp.handhelds.tizen.devel/4892

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Rethinking dynamic linking a bit (was: Re: Depends/Recommends from libraries)

[Christian Seiler]

On 03/08/2017 11:33 PM, Adam Borowski wrote:
> I'd like to discuss (and then propose to -policy) the following rule:
> 
> # Libraries which don't provide a convenient means of conditionally loading
> # at runtime (this includes most libraries for languages such as C), SHOULD
> # NOT declare a "Depends:" or "Recommends:" relationship, directly or
> # indirectly, on packages containing anything more than dormant files. 
> # Those include, among others, daemons, executables in $PATH, etc.  Any such
> # relationship should be instead declared by programs that use the library
> # in question -- it is up to them to decide how important the relationship
> # is.

For $DAYJOB I had to work on Mac OS X a bit, and they have an interesting
feature there: weakly binding to a shared library. The main reason this
exists is the following: suppose you want to create a binary that will
work on earlier versions of the operating system, but can also make use of
newer features if they're available on the target system. There are two
cases how new features can be added:

 - new symbols added to libraries
 - new libraries added

To handle the first case, the compiler defines a minimum version that the
program wants to support, and the standard include headers contain some
preprocessor magic that will mark all symbols as weak symbols if they're
from a newer operating system version.

To handle the second case there's a linker flag that allows you to weakly
bind to an entire library. The dynamic linker will not error out if the
library doesn't exist.

To make that work in the program, the code needs to check for symbol
availability, e.g.:

if (some_fancy_function)
some_fancy_function();

The first case is already supported with the current linker infrastructure
on Debian: you can compile a library with a symbol (without the weak
marker), and then link a program where you import that symbol, but the
header (when included in the program) marks the symbol as weak, so the
absence of that symbol is not an error for the dynamic linker. See the
postscript of this email for an example.

However, the second case might be even more interesting here: if the
library is imported "weakly", and it doesn't exist when running the
program, the program will start regardless, and all the symbols from the
library will be NULL pointers for the underlying code, and that condition
can be detected.



Now on Mac OS X this is used to support targeting older version of the
operating system for proprietary software. And the first part about
using weak symbols (but linking fully against the library) is not very
interesting to Debian (even though that's the part that already works),
since we don't care that binaries support older versions of the same
library. (In contrast to people writing proprietary software.)



But for Debian we could use the second idea for optional runtime library
dependencies. For example, a lot of software in the core system
currently links against libselinux.so - while most people don't use
that at all. On the other hand, there have been several cases in the
past where maintainers of core software declined to build against another
shared library to not increase the size of the most minimal installation
any further. With weakly linking against optional shared libraries, we
can decrease the size of the base system further without sacrificing
functionality.

The plan related to infrastructure would actually be quite reasonable:

 - Add a new ELF dynamic header tag that has the same semantics that
   DT_NEEDED has, except for the fact that if the library is not found,
   it is not considered an error.

 - Add a new linker flag to weakly link against a library and generate
   this tag instead of DT_NEEDED. All symbols imported from that
   library will automatically be considered weak imports in the final
   executable.

 - Update the dynamic linker in glibc to support this.

 - Make sure gcc doesn't optimize away if (symbol) checks. I believe it
   already doesn't do that for symbols explicitly marked as weak, but it
   may still do that for symbols that are not marked as weak during
   compilation. (One would have to check.) And when linking an entire
   library weakly, one doesn't know at compile time that all of these
   symbols will be weak imports.

 - Update shlibdeps to produce Recommends: instead of Depends: for
   libraries that are linked weakly, with the option for the package
   Maintainer to change that to Suggests.

 - Update shlibdeps to Pre-Depend: on glibc with the dynamic linker
   that supports this if the new header type is found.

With that in place, one could then start to modify software to make
use of this feature where this is applicable.

Benefits (in the long term):

 - Smaller base system
 - Easier to add new features to core components without stepping on
   people's toes. (And since we now have build profiles, bootstrapping
   shouldn't be a problem.)
 - Executables that need to be copied into the

Re: Depends/Recommends from libraries

[Ian Jackson]

Russ Allbery writes ("Re: Depends/Recommends from libraries"):
> I think this would be a great way of introducing spurious bugs in our
> distribution from [developers] who don't happen to read the README
> file and miss dependencies they actually need because they're used
> to Debian properly picking up shared library dependencies and to the
> dependencies of any given package being fully self-contained.  Both
> of which, I should add, are major *features* of our distribution
> that many of us have worked very hard to achieve.  I'm opposed.
> 
> Now, if this were taken a further step so that dpkg-shlibdeps would
> provide some mechanism to *automatically* add those downstream
> dependencies to packages that depend on the library unless the
> dependencies were explicitly suppressed, I wouldn't be as strongly
> opposed.

This seems doable to me.

>  It still feels like needless complexity to me,

Others have explained why this is a real problem.

> relabeling all of
> those libraries to have a new dependency field

The onward-dependency information for use by dpkg-shlibdeps wouldn't
be a control file field, I think.  It would be elsewhere in bar's
source package, and end up as a dropping in libbar-dev.deb.

Ian.

Re: AppStream Re: Depends/Recommends from libraries

[Jeremy Bicha]

On Fri, Mar 10, 2017 at 3:05 AM, Rebecca N. Palmer
 wrote:
> Upstream have since granted permission for the metadata to be MIT, but too
> late for the freeze.

I'm not on the Release Team, but I think a one-line change to fix the
appstream metadata should be easy to unblock for stretch at this
point.

Jeremy

Re: Depends/Recommends from libraries

[Russ Allbery]

Wouter Verhelst  writes:
> On Thu, Mar 09, 2017 at 10:19:10AM -0800, Russ Allbery wrote:

>> Now, if this were taken a further step so that dpkg-shlibdeps would
>> provide some mechanism to *automatically* add those downstream
>> dependencies to packages that depend on the library unless the
>> dependencies were explicitly suppressed, I wouldn't be as strongly
>> opposed.

> I think this is probably the best way forward, and it doesn't even need
> to be too complicated:

> - Modify dpkg-shlibdeps so it supports the type "opt" in the shlibs file
>   (see "man deb-shlibs" for details);
> - Make dpkg-shlibdeps emit the conjunction of the regular shared library
>   dependencies (i.e., typeless dependencies) and the "opt" type ones
>   (the daemons etc that the library would need if used) in the
>   ${shlibs:Depends} substvar, if no special command line parameter was
>   passed to it;
> - If a special command line was passed to it of the form
>   "--suggests=libfoo0,libbar0" or "--recommends=libfoo0,libbar0", then
>   pass the optional dependencies for the given packages in new
>   ${shlibs:Suggests} or ${shlibs:Recommends} substvars instead (and pass
>   any optional dependencies for packages not so given still in the
>   ${shlibs:Depends} substvars).

> With that, maintainers who consume libraries but don't do anything
> special will continue to produce working packages; and maintainers who
> consume libraries and care can use ${shlibs:Suggests} and
> ${shlibs:Recommends} substvars to decide where the optional dependencies
> must go without breaking anything for users.

Yeah, I'm kind of coming around to something like this, although I suspect
you'd need more granularity than just one pair of variables, at least
eventually.  But that's an implementation detail.

I think it's worth remembering, though, that this is unlikely to change
anything about the particular use case that triggered this whole thread.
I don't want us to lose track of the fact that the canonical example
(upower -> usbmuxd via a library for Apple support) isn't a mistake or
accident; it's a tradeoff disagreement between people with different
preferences about what the ideal state should look like.

If we put in place this mechanism or something like it, I fully expect
that the upower maintainer would think "hm, actually, I want my package to
work on Apple hardware out of the box without requiring anyone do
something special, particularly since otherwise I get bug reports about it
not working on that hardware, so adding the Recommends that brings in
usbmuxd looks correct -- if people don't like it, they can always remove
it or turn off Recommends."  And then we would be back to exactly where we
are right now.

We should make sure that we're actually solving a real problem here before
doing work, which includes being sure that a solution, if implemented,
would actually change things.  I continue to think that the disagreement
isn't a theoretical one over how to handle library to binary dependencies,
but an actual technical disagreement over the importance of out-of-the-box
hardware support versus system bloat when Recommends is enabled.  We're
not going to resolve that disagreement by writing more Policy or more code
in dpkg-shlibdeps.

(That said, the above system seems rather neat to me.)

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Marvin Renich]

* Simon McVittie  [170310 05:17]:
> On Thu, 09 Mar 2017 at 17:52:05 -0500, Marvin Renich wrote:
> > If more upstreams were careful to use dynamic loading in these
> > situations, it would be less of a problem.  In a perfect world, the
> > solution would be for foo's maintainer to convince upstream to switch to
> > dynamic loading.
> 
> (For context, I maintain several game engines that default to dlopen()ing
> their dependencies, some of which I have patched to stop doing that.)
> 
> I do not agree that dlopen()ing dependencies (what you have called "dynamic
> loading") is something we should encourage over normal linking with -lfoo
> (resulting in a DT_NEEDED entry, what you have called "static loading").

I'm sorry if I wasn't clear.  By "in these situations" I meant when the
library is only being used for a feature that is not likely to be used
by most users of the package, and only when the library has additional
dependencies that the user may want to avoid if he does not want the
feature provided by the library.

> Having done that, either the plugin can either be split out into its own
> package that is recommended or suggested by the main package (as was done
> for gnome-software support for Flatpak), or the plugin's dependencies can
> be downgraded to Recommends or Suggests with something like
> "dh_shlibdeps -- -e/usr/lib/myplugins/myplugin.so -dRecommends" and it
> will fail to dlopen if they are missing (as is done in at least iproute2 and
> thunar, see https://codesearch.debian.net/search?q=-dRecommends).

I believe I was suggesting something along those lines...

> If a library dependency is sufficiently "heavy" that it needs to become
> optional, please consider that approach.

...and only in this specific case.

...Marvin

Re: Depends/Recommends from libraries

[Wouter Verhelst]

On Thu, Mar 09, 2017 at 10:19:10AM -0800, Russ Allbery wrote:
> Now, if this were taken a further step so that dpkg-shlibdeps would
> provide some mechanism to *automatically* add those downstream
> dependencies to packages that depend on the library unless the
> dependencies were explicitly suppressed, I wouldn't be as strongly
> opposed.

I think this is probably the best way forward, and it doesn't even need
to be too complicated:

- Modify dpkg-shlibdeps so it supports the type "opt" in the shlibs file
  (see "man deb-shlibs" for details);
- Make dpkg-shlibdeps emit the conjunction of the regular shared library
  dependencies (i.e., typeless dependencies) and the "opt" type ones
  (the daemons etc that the library would need if used) in the
  ${shlibs:Depends} substvar, if no special command line parameter was
  passed to it;
- If a special command line was passed to it of the form
  "--suggests=libfoo0,libbar0" or "--recommends=libfoo0,libbar0", then
  pass the optional dependencies for the given packages in new
  ${shlibs:Suggests} or ${shlibs:Recommends} substvars instead (and pass
  any optional dependencies for packages not so given still in the
  ${shlibs:Depends} substvars).

With that, maintainers who consume libraries but don't do anything
special will continue to produce working packages; and maintainers who
consume libraries and care can use ${shlibs:Suggests} and
${shlibs:Recommends} substvars to decide where the optional dependencies
must go without breaking anything for users.

The only problem with that scheme is that there will need to be a
transition for libraries where all their users will need to be rebuilt
before they themselves can drop the new optional dependencies, otherwise
new packages will fail to pick up the dependencies that become their
responsibility. This is something that can be done by triggering binNMUs
though, so shouldn't be a showstopper.

This doesn't cover symbols files (because there's currently no way to
specify a "type" in a symbols file), but I don't think that's impossible
to fix.

(you'll also need to modify helpers such as CDBS and debhelper so that
the given parameters can be passed; but none of that needs to be a
showstopper...)

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: AppStream Re: Depends/Recommends from libraries

[Matthias Klumpp]

2017-03-10 9:05 GMT+01:00 Rebecca N. Palmer :
> On 10/03/17 00:10, Jeremy Bicha wrote:
>>
>> I think a lot of those appstream installs are from KDE and GNOME which
>> install plasma-discover and gnome-software by default.
>
> Do those things display AppStream "packages related to this hardware" by
> default?

I think GNOME Software doesn't, and Plasma Discover only displays them
as regular install-items. A feature like that would be really nice to
have though, it's maybe worth filing a feature request.

>> But beignet-opencli-icd's appstream metadata is invalid [copyleft not
>> allowed].
>
> This was done because parts of the metadata are automatically extracted from
> the (LGPL) code; it's only a warning in 'appstreamcli validate', and the
> people I discussed it with at the time thought it would be OK.

It's a warning because the data isn't completely useless (in case you
don't want to aggregate it, for example), but in general a warning
means a failed validation - to be safe, the data needs to be warning
and error free (exit-code of appstreamcli validate needs to be zero).

Debian's appstream-generator didn't reject metadata with
non-permissive licenses until August 2016 though.

> Upstream have since granted permission for the metadata to be MIT, but too
> late for the freeze.

Nice :-) The license restraints are annoying, but it might keep us
from trouble when mixing data with multiple different licenses in one
file.

>> Also, do you have a link for more information about what
>> beignet-opencl-icd does?
>
> The homepage field of the metadata is set, to
> https://www.freedesktop.org/wiki/Software/Beignet/
>
>

-- 
Debian Developer | Freedesktop-Developer
I welcome VSRE emails. See http://vsre.info/

Re: Depends/Recommends from libraries

[Simon McVittie]

On Thu, 09 Mar 2017 at 17:52:05 -0500, Marvin Renich wrote:
> If more upstreams were careful to use dynamic loading in these
> situations, it would be less of a problem.  In a perfect world, the
> solution would be for foo's maintainer to convince upstream to switch to
> dynamic loading.

(For context, I maintain several game engines that default to dlopen()ing
their dependencies, some of which I have patched to stop doing that.)

I do not agree that dlopen()ing dependencies (what you have called "dynamic
loading") is something we should encourage over normal linking with -lfoo
(resulting in a DT_NEEDED entry, what you have called "static loading").

dlopen()ing dependencies in the way that is most commonly implemented,
with dlopen("libimobiledevice.so.6") and dlsym(handle, "idevice_new")
or similar, has some practical problems for Debian:

* The libraries used aren't visible to dpkg-shlibdeps. The maintainer has
  to know what dlopen() calls the code will make, and either hard-code an
  appropriate Depends (or in this case Recommends), or link a dummy
  executable against the same things that are dlopen()ed (as is done
  in packages like wine and openal-soft) and use that for dpkg-shlibdeps.
  Either way, they have to remember to update it for every new upstream
  release. This is the sort of tedious-but-subtle work that we automate
  because otherwise it will inevitably be incorrect after a few releases.

* The exact symbols used aren't visible to dpkg-shlibdeps. The maintainer
  has to hard-code a version number that has all the required symbols.
  Again, this is the sort of thing that we automate because it will
  inevitably go wrong if done manually.

It also has some practical problems for upstreams:

* It adds a pile of code complexity. Maintainer attention is a scarce
  resource :-)

* It moves any compatibility issues from compile-time (where only the
  developer and packagers will experience them) to runtime (where end users
  will experience them).

* If the library is present at runtime, but with an older version than at
  compile-time, the code needs to be robust against that. This is the sort
  of thing that a typical upstream will never test.

* It isn't entirely portable. On common OSs like GNU/* and Windows
  (and I think macOS, but I'm less sure about that one), it happens to work,
  but in some OSs there is a distinction between loadable modules and
  shared libraries, and some upstreams aim to be portable to those OSs.
  (See the libtool documentation for more than you could possibly want
  to know about this topic.)

The pedantically correct way to have weak library dependencies is this
multi-level chain of linking, analogous to what is documented for APIs
that wrap dlopen() such as libltdl and GLib's GModule:

- eventual API user, e.g. /usr/bin/gnome-software
  dlopen()s:
- a plugin intended to be dlopen()ed, e.g. libgs_plugin_flatpak-user.so
  has DT_NEEDED on:
- libflatpak0

(For best portability, you'd use libltdl or GModule or similar instead
of direct dlopen().)

Having done that, either the plugin can either be split out into its own
package that is recommended or suggested by the main package (as was done
for gnome-software support for Flatpak), or the plugin's dependencies can
be downgraded to Recommends or Suggests with something like
"dh_shlibdeps -- -e/usr/lib/myplugins/myplugin.so -dRecommends" and it
will fail to dlopen if they are missing (as is done in at least iproute2 and
thunar, see https://codesearch.debian.net/search?q=-dRecommends).

If a library dependency is sufficiently "heavy" that it needs to become
optional, please consider that approach.

S

Re: Depends/Recommends from libraries

[Thibaut Paumard]

Dear Ian,

Le 09/03/2017 à 18:39, Ian Jackson a écrit :
> Thibaut Paumard writes ("Re: Depends/Recommends from libraries"):
>> There are quite legitimate uses for dependencies or recommendations in
>> libraries. For instance, tne library that I maintain (libgyoto) has the
>> option to provide MPI paralellisation. This requires an external
>> executable, which is provided in a separate package. The external tool
>> is in a separate package because it can exist only for one architecture
>> at a time on the system (it lives in /usr/bin), while the library lives
>> in a multi-arch directory.
> 
> This is perhaps a nice example.  Mind if I ask some questions ?
> 
> You say "an option to provide MPI paralellisation".  Is this option
> enabled by default ?  Does it occur with all useful functions in the
> library ?  Do library callers control over whether it occurs ?

The option can be turned off completely at build time, but the default
build-time behaviour is to enable it. The default run-time behaviour is
to not use MPI parallelisation, but the user can switch it on. It is not
used by all the functions. Actually there is only one high-level
function that makes use of this possibility (the function that executes
the ray-tracing for all pixels of an image, see below).

> Are there any programs linked against libgyoto which actually call
> into libgyoto only some of the time (perhaps, rarely) ?  (I did a
> quick search and it seems that libgyoto's rdependency stack is quite
> short.  There's some python modules and something to do with yorick,
> but they are all explicitly gyoto-related; and there are some blend
> metapackages, which are fine.  So maybe the answer to this is no, but
> maybe it will become yes in the future.)
> 

Indeed, Gyoto is somewhat of an academic example, because it is
currently only used by binary packages built from the same source. All
of the programs currently linked with Gyoto are low or high level
interfaces to the library, they do nothing else than calling the library.

However it is meant to be usable by third party applications. libgyoto
is a library to compute the trajectory of bodies in general relativity,
including photons to perform ray-traycing in curved spacetimes. Its
primary (and currently unique) audience is researchers. However I would
find it quite cool if games, for instance, would start using it to
produce realistic images of black-holes (which could benefit from MPI
parallelisation) or to implement realistic physics for space-crafts
(which could not). One could also easily use libgyoto or its Python
bindings to build applications for educational or artistic purposes.

In any case, I think the mechanics at work in this package is a good
example of why a library may need to call external programs:

The MPI parallelisation comes almost for free for any program that uses
libgyoto. The main program needs to initialise the MPI environment and
let libgyoto know to use it. The user typically needs to call the
application through 'mpirun'. That's all. The main program does not need
to have any parallel computing semantics in it. The cost is that the
additional processes needed to work in parallel are spawned by libgyoto
itself, so the corresponding executable needs to be somewhere where
libgyoto can find it reliably, even in an heterogeneous environment.

In conclusion: the generic part of my example is that a library may
implement a multiprocess algorithm using the singleton paradigm (i.e. a
single process is started, this process spawns workers). Since clusters
can be heterogeneous, it is not necessarily desirable to hard-code the
location of the worker process, so it must be in the PATH. We don't have
any multi-arch location for executables, that is in the PATH by default.
So the executable cannot be in a multi-arch package. If you want your
library to be multi-arch, the executable that is called from the library
cannot be in the same package. The lib package need to recommend the
'worker' executable package. Their are a few niche cases that I have not
worked out, specifically I am not sure whether on a single machine, it
is possible for an i386 process to spawn an amd64 process using
MPI_Comm_Spawn. I think not but I may be wrong. I do believe it is
possible over the network, though.

Yes, I think my use-case boils down to that. There are certainly other
legitimate uses of library-calls-executable, but that's mine.


Regards, Thibaut.

Re: AppStream Re: Depends/Recommends from libraries

[Rebecca N. Palmer]

On 10/03/17 00:10, Jeremy Bicha wrote:

I think a lot of those appstream installs are from KDE and GNOME which
install plasma-discover and gnome-software by default.
Do those things display AppStream "packages related to this hardware" by 
default?


(beignet-opencl-icd isn't a valid test because of...)


But beignet-opencli-icd's appstream metadata is invalid [copyleft not allowed].
This was done because parts of the metadata are automatically extracted 
from the (LGPL) code; it's only a warning in 'appstreamcli validate', 
and the people I discussed it with at the time thought it would be OK.


Upstream have since granted permission for the metadata to be MIT, but 
too late for the freeze.



Also, do you have a link for more information about what
beignet-opencl-icd does?
The homepage field of the metadata is set, to 
https://www.freedesktop.org/wiki/Software/Beignet/

Re: Depends/Recommends from libraries

[Russ Allbery]

Marvin Renich  writes:

> If libbar-dev documents that it requires bar-daemon (and under what
> circumstances, if appropriate), but libbar does not declare the Depends,
> then it becomes the Debian maintainer of foo who decides to add an
> appropriate Depends, Recommends, or Suggests for bar-daemon, in addition
> to the Depends (that should be, but can't be, a Recommends or Suggests)
> on libbar.

And if you think the maintainer of foo is going to remember to do that,
well damn, why do we even bother with dpkg-shlibdeps?  Our maintainers can
just hand-craft this stuff -- it's so easy!

Humans don't work this way.

> Every Debian maintainer whose package links libbar would then be
> required to read the documentation of libbar-dev, and act on that to add
> a dependency that libbar would have used.  I would certainly expect a
> Debian maintainer to read said documentation (irregardless of Ian's
> proposal).

I would absolutely not expect that.

Let's please not create more situations where packaging requires careful
memory, multiple checklists, and human judgement.  It is absolutely not
okay to require maintainers read all of the documentation of all dev
packages they depend on in order to hand-craft dependencies.  This is not
how we make a better distribution or make Debian packaging easier or more
likely to be correct; this is how we end up with a bunch of frustrating
bugs.

-- 
Russ Allbery (r...@debian.org)   

Symbol specific dependencies (was Re: Depends/Recommends from libraries)

[Guillem Jover]

Hi!

On Thu, 2017-03-09 at 17:29:09 +, Ian Jackson wrote:
> I think the right way to solve this problem is to declare that:
[…]
>  * If a library needs or wants additional software installed,
>if and when functions in that library are called, this
>should be documented in the /usr/share/doc/BAR/README.Debian.gz for
>the corresponding -dev library package.  (If churn is
>likely, a library-specific virtual package name may need
>to be documented, and provided as appropriate.)

>  * Programs which call functions in libraries (directly or indirectly)
>should arrange to Depend on, Recommend, or Suggest, the appropriate
>infrastructure, as documented by the -dev package(s).

We can already represent symbol specific dependencies, so we can have
a library that ships this following made-up symbols file:

,---
libservice.so.0 libservice0 #MINVER#
| serviced
 start_service_daemon 0 1
 generic_serviceless_action 0
`---

Of course one problem is that libraries might not be delimited in this
way, and they might just start such services when merely initializing
the library, when calling any of its functions, or at any point inside
some state machine or whatever. Also a program might call such function
but that might be conditional on a configuration parameter or something
else, so the fact that a program uses that symbol does not mean it might
even call it.

Thanks,
Guillem

Re: AppStream Re: Depends/Recommends from libraries

[Jeremy Bicha]

On Thu, Mar 9, 2017 at 6:23 PM, Rebecca N. Palmer
 wrote:
> When beignet-opencl-icd added AppStream metadata (black line in [1]), there
> was no noticeable increase in its installs.  As it's for popular hardware

I think a lot of those appstream installs are from KDE and GNOME which
install plasma-discover and gnome-software by default.

But beignet-opencli-icd's appstream metadata is invalid.

https://appstream.debian.org/sid/main/issues/beignet-opencl-icd.html

https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html#tag-metadata_license

When you fix this issue, it may take a day or so for Debian's
appstream generator to pick up the change and publish it. If there are
no issues, the issues page won't be live but
https://appstream.debian.org/sid/main/metainfo/beignet-opencl-icd.html will be.

Also, do you have a link for more information about what
beignet-opencl-icd does?

Thanks,
Jeremy

AppStream Re: Depends/Recommends from libraries

[Rebecca N. Palmer]

appstream itself is installed on ~60% of sid/stretch desktops [0], but 
isenkram on only ~5% (and most of those are the -cli version).


When beignet-opencl-icd added AppStream metadata (black line in [1]), 
there was no noticeable increase in its installs.  As it's for popular 
hardware (~33% of systems [2]) but a sufficiently behind-the-scenes 
feature that few users manually install it, this is a fairly sensitive 
test: <10% of new (including newly upgraded) sid/stretch desktop 
installations with the hardware installed it.


Hence, either most users don't see AppStream suggestions, or most users 
don't want GPU compute when offered the option.  (beignet-opencl-icd's 
AppStream summary/description are "OpenCL (GPU compute) driver for Intel 
GPUs"/"This allows using Intel integrated GPUs for general computation, 
speeding up some applications."  At least libreoffice-calc and probably 
ffmpeg can take advantage of it, but it's very possible that typical use 
of these is either on too little data to notice the difference, or has a 
better option (libva/libvdpau).)


[0] 
https://qa.debian.org/popcon-graph.php?packages=libvpx4+libllvm3.9+appstream+libappstream4+beignet-opencl-icd&show_installed=on&want_legend=on&want_ticks=on&from_date=2016-01-01&to_date=&hlght_date=2017-01-18&date_fmt=%25Y-%25m&beenhere=1
libvpx4 (video decoding, dependency of firefox and chromium) and 
libllvm3.9 (compiler library, dependency of libgl1-mesa-dri) were chosen 
as approximations of "all stretch/sid desktops", as they have changed 
soname since jessie.


[1] 
https://qa.debian.org/popcon-graph.php?packages=beignet-opencl-icd+mesa-opencl-icd+pocl-opencl-icd&show_installed=on&want_legend=on&want_ticks=on&from_date=2016-01-01&to_date=&hlght_date=2017-01-18&date_fmt=%25Y-%25m&beenhere=1


[2] from recent bug reports:
http://www.mail-archive.com/search?a=1&l=debian-bugs-dist%40lists.debian.org&haswords=VGA+compatible+controller+[0300]&x=16&y=26&from=&subject=&datewithin=1y&date=2017-01-01¬words=&o=newest

Re: Depends/Recommends from libraries

[Nick Phillips]

On Thu, 2017-03-09 at 10:19 -0800, Russ Allbery wrote:
> 
> I think this would be a great way of introducing spurious bugs in our
> distribution from people who don't happen to read the README file and
> miss
> dependencies they actually need because they're used to Debian
> properly
> picking up shared library dependencies and to the dependencies of any
> given package being fully self-contained.  Both of which, I should
> add,
> are major *features* of our distribution that many of us have worked
> very
> hard to achieve.  I'm opposed.
> 

Can we just clarify - in the setup that Ian proposed, a "normal" user
would have experience no different to now (except for less bloat);
package maintainers and those using -dev libs are the ones who would
need to read those docs. Package maintainers in order to ensure they
set the correct deps on their packages, and -dev package users to
ensure they are aware of which features of a library need extra
packages installed in order to function.


Cheers,


Nick
-- 
Nick Phillips / nick.phill...@otago.ac.nz / 03 479 4195
# These statements are mine, not those of the University of Otago

Re: Depends/Recommends from libraries

[Marvin Renich]

* Russ Allbery  [170309 13:19]:
> I think this would be a great way of introducing spurious bugs in our
> distribution from people who don't happen to read the README file and miss
> dependencies they actually need...

I think you are missing Ian's meaning.  Currently foo Depends libbar,
and libbar Depends bar-daemon.  If libbar were dynamically loaded, the
maintainer of foo would have used either a Recommends or Suggests for
libbar, but must instead use Depends because it is statically loaded.
If libbar-dev documents that it requires bar-daemon (and under what
circumstances, if appropriate), but libbar does not declare the Depends,
then it becomes the Debian maintainer of foo who decides to add an
appropriate Depends, Recommends, or Suggests for bar-daemon, in addition
to the Depends (that should be, but can't be, a Recommends or Suggests)
on libbar.  This is not pushed to the user at all, except in the normal
way that a user currently chooses to install Recommends or Suggests in
other circumstances.

Every Debian maintainer whose package links libbar would then be
required to read the documentation of libbar-dev, and act on that to add
a dependency that libbar would have used.  I would certainly expect a
Debian maintainer to read said documentation (irregardless of Ian's
proposal).  This has nothing to do with an end user reading a README
file to get the dependencies right (at least not any differently than
the current situation for other non-lib Recommends or Suggests).

I have not decided which side of the fence I am on, but I am certainly
empathetic to Ian's, Adam's, and others' desire to improve the
situation, as I have been bitten by this myself on more than one
occasion.  This is a situation where the maintainer of foo has no choice
but to use Depends, even if the library "can perhaps enhance [foo's]
usefulness, but installing [foo] without [libbar] is perfectly
reasonable."  You end up with a daemon that you don't want because
libbar Depends bar-daemon is appropriate, even if you, the end user, are
never going to use foo in a way that uses the libbar functionality.

If more upstreams were careful to use dynamic loading in these
situations, it would be less of a problem.  In a perfect world, the
solution would be for foo's maintainer to convince upstream to switch to
dynamic loading.  I'm not convinced that Ian's proposal is the right
approach, but I definitely agree that it is an attempt to solve a real
problem, and I believe it has more merit than you give it.

...Marvin

Re: Depends/Recommends from libraries

[Russ Allbery]

Adam Borowski  writes:

> What's wrong in the current state is that it looks only from the point
> of view of the library: libwrap1 is useless without tcpd, thus it's
> natural for it to have an elevanted severity.  But that dependency is
> wrong from a more global point of view.  That's why I'm proposing making
> the decision a responsibility of programs which link to the library.

I don't think these issues are as hard to reason about one by one as
you're saying.  This is a good example: libwrap0 is *not* useless without
tcpd.  All that tcpd is providing it are the default /etc/hosts.allow and
/etc/hosts.deny files (and maybe some binaries that are used in unusual --
these days -- situations like safe_finger).  You can use libwrap0 easily
without having tcpd installed if you write those files yourself.

Therefore, I think it would be reasonable to downgrade this Recommends to
Suggests.  (However, I seem to recall that we talked about this before and
there was some specific problem Marco was worried about that we haven't
captured in this discussion.)

That said, while I get that every little bit counts, installing tcpd on a
system has remarkably minimal impact.  It's just a few binaries in a tiny
package that doesn't start any processes and doesn't do anything.  It
takes all of roughly 92KB on disk.

The difficulty with the usbmuxd issue is precisely that it's just on the
borderline.  It only supports one specific (and not particularly common
for Debian, if common out in the world) type of hardware, but it's kind of
important on that piece of hardware.  There isn't a communication problem
here, there isn't a Policy problem here, there's a *hard tradeoff* problem
here, and talking about this as if it were an issue with missing Policy
loses the fact that this is a thoughtful choice the maintainers have made
with an understanding of all of the consequences.

I get that you disagree with their decision, but their decision isn't
obviously wrong.  We don't have a better way of supporting unusual
hardware than this.  It's worth noting that some steps have been taken to
minimize the impact of the additional dependency (it's small, it's only
started on demand, etc.), which I at least find entirely reasonable and
therefore feel fairly comfortable with still having it in the Recommends
set because otherwise people with Apple hardware have to do fairly obscure
things to get their hardware working.

I think everyone involved would be happy to switch to a better solution if
we had one (some package that detects packages needed for specific
hardware profiles and installs them, for instance).

It feels like you're unhappy with a decision that was made thoughtfully,
not accidentally, and want to use Policy as a way to override that
decision, which makes me quite uncomfortable.

And, with my Policy Editor hat on, I will say that this is not the purpose
of Policy; Policy is to document consensus about how we build the
distribution.  If you have a technical disagreement with maintainers about
their package metadata that involves thoughtful and principled
disagreement between two valid approaches with different tradeoffs (which
appears to be the case here), then for better or worse that's what the
Technical Committee is for.

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Russ Allbery]

Andrey Rahmatullin  writes:
> On Thu, Mar 09, 2017 at 12:22:17PM -0800, Russ Allbery wrote:

>> Sure, but hopefully we find and report those as bugs.  I personally run
>> without recommends on Debian unstable on several different types of
>> systems and report these problems whenever I run into them.

> But how do you decide if a specific problem caused by Recommends not
> installed is serious enough to be worth a bug report? "all but unusual
> installations" after all.

Mostly I use my judgement.

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Adam Borowski]

On Thu, Mar 09, 2017 at 10:14:13AM -0800, Russ Allbery wrote:
> Jonas Smedegaard  writes:
> > Quoting Russ Allbery (2017-03-09 04:24:09)
> 
> >> In general, I don't want to see us place too many restrictions on 
> >> Recommends.  If you don't want additional helpful programs, disable 
> >> installing Recommends by default.  I think it's very odd to worry 
> >> about bloat while simultaneously installing Recommends by default; 
> >> those aren't really consistent things to do.
> 
> > I strongly disagree: We _can_ improve on the default behaviour of our 
> > package handling - we need not give up and leave it to power users able 
> > to handle arguably broken systems.
> 
> Well, I strongly disagree with you.  I think this would take things in the
> wrong direction; I like that software is fully useful when Recommends are
> enabled at the cost of some bloat.

Yeah, but how the _library_ can make a meaningful decision about how
important the dependency is?  Only the program that links with the library
knows whether it's a fringe option or something important.

> If you don't want possibly unused software installed, we have a supported
> mechanism for that: disable automatic installation of Recommends.

The policy says:
#  `Recommends'
#  This declares a strong, but not absolute, dependency.
#
#  The `Recommends' field should list packages that would be found
#  together with this one in all but unusual installations.

Ie, if a substantial part of users don't want that package installed, then,
by the Policy's wording, that dependency is wrong.


What's wrong in the current state is that it looks only from the point of
view of the library: libwrap1 is useless without tcpd, thus it's natural for
it to have an elevanted severity.  But that dependency is wrong from a more
global point of view.  That's why I'm proposing making the decision a
responsibility of programs which link to the library.

-- 
⢀⣴⠾⠻⢶⣦⠀ Meow!
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Collisions shmolisions, let's see them find a collision or second
⠈⠳⣄ preimage for double rot13!

Re: Depends/Recommends from libraries

[Michael Lustfield]

On Mar 9, 2017 12:22 PM, "Russ Allbery"  wrote:


Sure, but hopefully we find and report those as bugs.  I personally run
without recommends on Debian unstable on several different types of
systems and report these problems whenever I run into them.


I'm in this same boat. I disable installing recommends on all but one of my
systems, including my laptop's and other devices. If (when) I have issues
or check out new software, I take a look at those lists to see what I might
want.


Heh.. I remember using gnome, kde, xfce, etc. and they all seemed to have
this problem in one way or another. After enough battling, I eventually
arrived at the conclusion that these desktop environments were written to
be super complete, not super lean. They were for everybody, not for me.

I ended up switching to openbox and a whole different selection of
"desktop" utilities and even had to write some myself. Issues like this
have completely evaporated for me.

Re: Depends/Recommends from libraries

[Andrey Rahmatullin]

On Thu, Mar 09, 2017 at 12:22:17PM -0800, Russ Allbery wrote:
> >> If you don't want possibly unused software installed, we have a
> >> supported mechanism for that: disable automatic installation of
> >> Recommends.
> 
> > Which explodes from time to time, like when ntpdate and ntpd only
> > recommended lockfile-progs (#731976).
> 
> Sure, but hopefully we find and report those as bugs.  I personally run
> without recommends on Debian unstable on several different types of
> systems and report these problems whenever I run into them.
But how do you decide if a specific problem caused by Recommends not
installed is serious enough to be worth a bug report? "all but unusual
installations" after all.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Depends/Recommends from libraries

[Russ Allbery]

Andrey Rahmatullin  writes:
> On Thu, Mar 09, 2017 at 10:14:13AM -0800, Russ Allbery wrote:

>> If you don't want possibly unused software installed, we have a
>> supported mechanism for that: disable automatic installation of
>> Recommends.

> Which explodes from time to time, like when ntpdate and ntpd only
> recommended lockfile-progs (#731976).

Sure, but hopefully we find and report those as bugs.  I personally run
without recommends on Debian unstable on several different types of
systems and report these problems whenever I run into them.

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Andrey Rahmatullin]

On Thu, Mar 09, 2017 at 10:14:13AM -0800, Russ Allbery wrote:
> If you don't want possibly unused software installed, we have a supported
> mechanism for that: disable automatic installation of Recommends.
Which explodes from time to time, like when ntpdate and ntpd only
recommended lockfile-progs (#731976).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Depends/Recommends from libraries

[Holger Levsen]

On Thu, Mar 09, 2017 at 10:14:13AM -0800, Russ Allbery wrote:
> Well, I strongly disagree with you.  I think this would take things in the
> wrong direction; I like that software is fully useful when Recommends are
> enabled at the cost of some bloat.
> 
> If you don't want possibly unused software installed, we have a supported
> mechanism for that: disable automatic installation of Recommends.
> 
> That said, it's certainly still fine to question individual choices, and
> to request a downgrade to Suggests if the provided benefit is marginal.
 
FWIW, +1 on everything here.


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Depends/Recommends from libraries

[Russ Allbery]

Ian Jackson  writes:

> I think the right way to solve this problem is to declare that:

>  * When a library package is installed, the Depends and Recommends
>of the library should be appropriate on the assumption that:

>  - the library package is only installed because it is the dormant
>runtime-link-dependency of an executable or other library;

>  - none of the functions in the library are going to be
>called.

>Normally this will mean that the library will reference only other
>library packages, on which it has runtime-link dependencies.

>  * If a library needs or wants additional software installed,
>if and when functions in that library are called, this
>should be documented in the /usr/share/doc/BAR/README.Debian.gz for
>the corresponding -dev library package.  (If churn is
>likely, a library-specific virtual package name may need
>to be documented, and provided as appropriate.)

>  * Programs which call functions in libraries (directly or indirectly)
>should arrange to Depend on, Recommend, or Suggest, the appropriate
>infrastructure, as documented by the -dev package(s).

I think this would be a great way of introducing spurious bugs in our
distribution from people who don't happen to read the README file and miss
dependencies they actually need because they're used to Debian properly
picking up shared library dependencies and to the dependencies of any
given package being fully self-contained.  Both of which, I should add,
are major *features* of our distribution that many of us have worked very
hard to achieve.  I'm opposed.

Now, if this were taken a further step so that dpkg-shlibdeps would
provide some mechanism to *automatically* add those downstream
dependencies to packages that depend on the library unless the
dependencies were explicitly suppressed, I wouldn't be as strongly
opposed.  It still feels like needless complexity to me, but at least we
would default to the known-working dependencies but provide an easy way
for a library consumer to relax those dependencies as needed.  But that
would require writing some additional infrastructure and relabeling all of
those libraries to have a new dependency field, and I strongly suspect
it's more effort than it's worth.

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Russ Allbery]

Jonas Smedegaard  writes:
> Quoting Russ Allbery (2017-03-09 04:24:09)

>> In general, I don't want to see us place too many restrictions on 
>> Recommends.  If you don't want additional helpful programs, disable 
>> installing Recommends by default.  I think it's very odd to worry 
>> about bloat while simultaneously installing Recommends by default; 
>> those aren't really consistent things to do.

> I strongly disagree: We _can_ improve on the default behaviour of our 
> package handling - we need not give up and leave it to power users able 
> to handle arguably broken systems.

Well, I strongly disagree with you.  I think this would take things in the
wrong direction; I like that software is fully useful when Recommends are
enabled at the cost of some bloat.

If you don't want possibly unused software installed, we have a supported
mechanism for that: disable automatic installation of Recommends.

That said, it's certainly still fine to question individual choices, and
to request a downgrade to Suggests if the provided benefit is marginal.

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Ian Jackson]

Russ Allbery writes ("Re: Depends/Recommends from libraries"):
> I feel like the problem here is that people are failing to fix bugs in
> their packages (unnecessary dependencies on libraries that have heavy
> dependencies),

No.  The problem is that for an ordinary library, if package foo ever
wants to call functions in library bar, /usr/bin/foo ends up linked
against libbar.so.  If the libbar runtime is not installed, foo fails
to start at all.

So foo ends up with a Depends on libbar - even if the bar
functionality in foo is an extreme niche feature, or even is only
relevant for some other package wombat (depending on foo).

If libbar has a Depends or Recommends on bar-daemon then default
installations of foo all have bar-daemon installed.

This is a problem because:

 * bar-daemon may have undesirable security properties

 * bar-daemon may need or want configuration, resulting in unnecessary
   config management (including potential exposure of foo's users to
   defects or lacunae in bar's config manageent, migrations, etc.)

 * bar-daemon may be large (disk space, backup, and bandwidth costs,
   especially if bar gets many updates)

 * bar-daemon's dependency stack may itself be large, causing large
   quantities of further unnecessary software to be installed,
   increasing the risks of problems I've just discussed

If foo executed /usr/bin/bar directly, or connected to
/run/bar/socket, or something, then the maintainers of foo (and maybe
wombat) could control the strength of the dependencies themselves,
based on their knowledge of the likely need of foo's (or wombat's)
users for bar, and the error behaviour if the bar-related
functionality is requested without the right pieces installed.

But because foo must Depend on libbar, with our current arrangements
the strength of the dependency is determined by the dependencies of
libbar.

But the bar authors:

 * do not know how or why libbar is called in any particular
circumstances

 * have a natural tendency to assume that their package is more
important than perhaps it is in the global scheme of things

 * are in any case unable to set the dependencies differently
for different callers


I think the right way to solve this problem is to declare that:

 * When a library package is installed, the Depends and Recommends
   of the library should be appropriate on the assumption that:

 - the library package is only installed because it is the dormant
   runtime-link-dependency of an executable or other library;

 - none of the functions in the library are going to be
   called.

   Normally this will mean that the library will reference only other
   library packages, on which it has runtime-link dependencies.

 * If a library needs or wants additional software installed,
   if and when functions in that library are called, this
   should be documented in the /usr/share/doc/BAR/README.Debian.gz for
   the corresponding -dev library package.  (If churn is
   likely, a library-specific virtual package name may need
   to be documented, and provided as appropriate.)

 * Programs which call functions in libraries (directly or indirectly)
   should arrange to Depend on, Recommend, or Suggest, the appropriate
   infrastructure, as documented by the -dev package(s).


This applies to libraries in C and C++.  For libraries in other
languages, it depends on whether the conventional calling pattern for
the library, and the language, is to load it unconditionally (so that
anyone who might call it, no matter how rarely, must require it to be
installed), or to load it on demand.


Ian.

Re: Depends/Recommends from libraries

[Ian Jackson]

Thibaut Paumard writes ("Re: Depends/Recommends from libraries"):
> There are quite legitimate uses for dependencies or recommendations in
> libraries. For instance, tne library that I maintain (libgyoto) has the
> option to provide MPI paralellisation. This requires an external
> executable, which is provided in a separate package. The external tool
> is in a separate package because it can exist only for one architecture
> at a time on the system (it lives in /usr/bin), while the library lives
> in a multi-arch directory.

This is perhaps a nice example.  Mind if I ask some questions ?

You say "an option to provide MPI paralellisation".  Is this option
enabled by default ?  Does it occur with all useful functions in the
library ?  Do library callers control over whether it occurs ?

Are there any programs linked against libgyoto which actually call
into libgyoto only some of the time (perhaps, rarely) ?  (I did a
quick search and it seems that libgyoto's rdependency stack is quite
short.  There's some python modules and something to do with yorick,
but they are all explicitly gyoto-related; and there are some blend
metapackages, which are fine.  So maybe the answer to this is no, but
maybe it will become yes in the future.)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Depends/Recommends from libraries

[Thibaut Paumard]

Le 08/03/2017 à 23:33, Adam Borowski a écrit :
> Hi, mortals and paultag!
> 
> I'd like to discuss (and then propose to -policy) the following rule:
> 
> # Libraries which don't provide a convenient means of conditionally loading
> # at runtime (this includes most libraries for languages such as C), SHOULD
> # NOT declare a "Depends:" or "Recommends:" relationship, directly or
> # indirectly, on packages containing anything more than dormant files. 
> # Those include, among others, daemons, executables in $PATH, etc.  Any such
> # relationship should be instead declared by programs that use the library
> # in question -- it is up to them to decide how important the relationship
> # is.

Hi,

This doesn't seem to make sense to me. You list a number of examples
where the outcome of a dependency or recommendation chain is not
desirable. That means that those packages may be improved by breaking
this chain, which doesn't need to go through a policy amendment.

There are quite legitimate uses for dependencies or recommendations in
libraries. For instance, tne library that I maintain (libgyoto) has the
option to provide MPI paralellisation. This requires an external
executable, which is provided in a separate package. The external tool
is in a separate package because it can exist only for one architecture
at a time on the system (it lives in /usr/bin), while the library lives
in a multi-arch directory.

Kind regards, Thibaut.

Re: Depends/Recommends from libraries

[Adrian Bunk]

On Wed, Mar 08, 2017 at 11:33:21PM +, Ian Jackson wrote:
> Adam Borowski writes ("Depends/Recommends from libraries"):
> > I'd like to discuss (and then propose to -policy) the following rule:
> > 
> > # Libraries which don't provide a convenient means of conditionally loading
> > # at runtime (this includes most libraries for languages such as C), SHOULD
> > # NOT declare a "Depends:" or "Recommends:" relationship, directly or
> > # indirectly, on packages containing anything more than dormant files. 
> > # Those include, among others, daemons, executables in $PATH, etc.  Any such
> > # relationship should be instead declared by programs that use the library
> > # in question -- it is up to them to decide how important the relationship
> > # is.
> 
> This seems like a non-brainer to me.  Can anyone come up with a reason
> why this would be wrong in general ?
>...

I can even come up with two:


First, it sounds like a layering violation, forcing programs to know 
about implementation details several layers of dependencies away.

As an example, look at the "libdbus-1-3 -> dbus" Recommends.

When libdbus-1-3 itself is used by a library, I don't see any reasonable
way to push the implementation detail that dbus is used all the way up
to all applications.


Second, what should this achieve?

The mentioned usbmuxd is 0.1 MB.

libglib2.0-0 recommends libglib2.0-data.
The sole contents of libglib2.0-data are 9.5 MB translations.
This recommends would still be permitted, and by default these
translations should be installed.

In the common case disk space is plenty and not a problem.
Except for some cases of daemons, installing too much is
therefore usually not a problem.[1]

Recommends are basically dependencies you are allowed to break when
you know what you are doing, and that's the level of expertise someone 
is expected to have when optimizing disk space usage.


> Ian.

cu
Adrian

[1] "libfoo pulls in 100 MB of data" cases are rare

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Depends/Recommends from libraries

[Jonas Smedegaard]

Quoting Russ Allbery (2017-03-09 04:24:09)
> Adam Borowski  writes:
> 
> > I'd like to discuss (and then propose to -policy) the following 
> > rule:
> 
> > # Libraries which don't provide a convenient means of conditionally 
> > # loading at runtime (this includes most libraries for languages 
> > # such as C), SHOULD NOT declare a "Depends:" or "Recommends:" 
> > # relationship, directly or indirectly, on packages containing 
> > # anything more than dormant files. Those include, among others, 
> > # daemons, executables in $PATH, etc.  Any such relationship should 
> > # be instead declared by programs that use the library in question 
> > # -- it is up to them to decide how important the relationship is.
> 
> This feels to me like the wrong tool.  It's entirely plausible for 
> there to be a library whose entire purpose is to execute some external 
> helper program with elevated privileges, which obviously should depend 
> on the package that provides that program.  If we had such a 
> requirement in Policy, we would end up with libraries that don't 
> Depend on their actual dependencies and then callers have to add the 
> dependency and then it's all just a mess.
> 
> I feel like the problem here is that people are failing to fix bugs in 
> their packages (unnecessary dependencies on libraries that have heavy 
> dependencies), and you're trying to use Policy as a stick to beat them 
> with to get them to fix their bugs.  I don't think this is a good 
> idea, and I don't want us to end up with weird and incorrect 
> dependencies for libraries that really do require external helper 
> programs (which is not particularly rare).

What I believe could be improved in policy is to clarify/emphasize that 
package relations are _directional_.

I have reported several bugs related to too strong relationships, where 
the main arugment of mine was that the relationship more accurately was 
the reverse of what was declared.

That said, I agree it not not sensible to (yet) impose a strict rule, 
only a clarification to existing rules on declaring relationships.


> Particularly since I don't think this requirement is actually targeted at
> the dependency that's bothering you.  In your example:
> 
> > It'd help disarm dependency chains such as:
> > xfce4-power-manager -> upower -> libimobiledevice4 -> usbmuxd
> > ie, wanting XFCE currently pulls in a daemon that's of no use to anyone not
> > using a piece of iJunk (and what it even has to do with power management?).
> 
> this would outlaw the Recommends from libimobiledevice4 to usbmuxd, but my
> understanding is that dependency is *correct*, and the actual extraneous
> dependency that's upsetting you is the one from upower to
> libimobiledevice4, which this Policy change would not affect at all.  If I
> were fixing this bug, that's the change I would make: have upower
> dynamically load libimobiledevice4 iff it exists, since that's fairly
> niche functionality.  (Or, really, given that it's a Recommends, just not
> worry about it.)

What you describe above is how a _different_ software design could 
result in simpler package dependencies - for _existing_ software I find 
it correct to identify the libimobiledevice4 → usbmuxd relation as the 
problematic one.


> In general, I don't want to see us place too many restrictions on 
> Recommends.  If you don't want additional helpful programs, disable 
> installing Recommends by default.  I think it's very odd to worry 
> about bloat while simultaneously installing Recommends by default; 
> those aren't really consistent things to do.

I strongly disagree: We _can_ improve on the default behaviour of our 
package handling - we need not give up and leave it to power users able 
to handle arguably broken systems.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: Depends/Recommends from libraries

[Paul Wise]

On Thu, Mar 9, 2017 at 11:48 AM, Michael Biebl wrote:

> out of the box. Unfortunately we don't have a well supported mechanism
> which would install such hardware-enablement packages when the hardware
> is plugged in.

Is the AppStream hardware support stuff not well supported in the desktops?

https://wiki.debian.org/AppStream/Guidelines#Announcing_supported_hardware

We do have at least one tool in Debian supporting it:

http://people.skolelinux.org/pere/blog/Using_appstream_with_isenkram_to_install_hardware_related_packages_in_Debian.html

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Depends/Recommends from libraries

[Michael Biebl]

Am 09.03.2017 um 04:24 schrieb Russ Allbery:
> Adam Borowski  writes:
> 
>> I'd like to discuss (and then propose to -policy) the following rule:
> 
>> # Libraries which don't provide a convenient means of conditionally loading
>> # at runtime (this includes most libraries for languages such as C), SHOULD
>> # NOT declare a "Depends:" or "Recommends:" relationship, directly or
>> # indirectly, on packages containing anything more than dormant files. 
>> # Those include, among others, daemons, executables in $PATH, etc.  Any such
>> # relationship should be instead declared by programs that use the library
>> # in question -- it is up to them to decide how important the relationship
>> # is.
> 
> This feels to me like the wrong tool.  It's entirely plausible for there
> to be a library whose entire purpose is to execute some external helper
> program with elevated privileges, which obviously should depend on the
> package that provides that program.  If we had such a requirement in
> Policy, we would end up with libraries that don't Depend on their actual
> dependencies and then callers have to add the dependency and then it's all
> just a mess.
> 
> I feel like the problem here is that people are failing to fix bugs in
> their packages (unnecessary dependencies on libraries that have heavy
> dependencies), and you're trying to use Policy as a stick to beat them
> with to get them to fix their bugs.  I don't think this is a good idea,
> and I don't want us to end up with weird and incorrect dependencies for
> libraries that really do require external helper programs (which is not
> particularly rare).
> 
> Particularly since I don't think this requirement is actually targeted at
> the dependency that's bothering you.  In your example:
> 
>> It'd help disarm dependency chains such as:
>> xfce4-power-manager -> upower -> libimobiledevice4 -> usbmuxd
>> ie, wanting XFCE currently pulls in a daemon that's of no use to anyone not
>> using a piece of iJunk (and what it even has to do with power management?).
> 
> this would outlaw the Recommends from libimobiledevice4 to usbmuxd, but my
> understanding is that dependency is *correct*, and the actual extraneous
> dependency that's upsetting you is the one from upower to
> libimobiledevice4, which this Policy change would not affect at all.  If I
> were fixing this bug, that's the change I would make: have upower
> dynamically load libimobiledevice4 iff it exists, since that's fairly
> niche functionality.  (Or, really, given that it's a Recommends, just not
> worry about it.)

As the question was asked why upower links against libimobiledevice4: it
is to support Apple hardware and get the battery state from those devices.

usbmuxd is hardware activated via a udev rule, so there is no extraneous
running daemon unless the hardware is plugged in.
Disk space is rather cheap and I prefer if hardware is just supported
out of the box. Unfortunately we don't have a well supported mechanism
which would install such hardware-enablement packages when the hardware
is plugged in.


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Re: Depends/Recommends from libraries

[Russ Allbery]

Ian Jackson  writes:
> Adam Borowski writes ("Depends/Recommends from libraries"):

>> I'd like to discuss (and then propose to -policy) the following rule:
>> 
>> # Libraries which don't provide a convenient means of conditionally loading
>> # at runtime (this includes most libraries for languages such as C), SHOULD
>> # NOT declare a "Depends:" or "Recommends:" relationship, directly or
>> # indirectly, on packages containing anything more than dormant files. 
>> # Those include, among others, daemons, executables in $PATH, etc.  Any such
>> # relationship should be instead declared by programs that use the library
>> # in question -- it is up to them to decide how important the relationship
>> # is.

> This seems like a non-brainer to me.  Can anyone come up with a reason
> why this would be wrong in general ?

> Of course there may be exceptions, but you're proposing a SHOULD.

Policy doesn't have an RFC 2119 SHOULD.  A Policy "should" means that
something is generally a bug, just not an RC one.  It would be nice to
have the equivalent of SHOULD ("this is normally not a good idea but may
be the correct thing to do in specific situations"), but we don't
currently have that.

This would definitely declare lots of existing packages buggy, which is
something we normally try not to do because usually packages are doing
this for some good reason (and I think that's obviously the case here).

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Russ Allbery]

Adam Borowski  writes:

> I'd like to discuss (and then propose to -policy) the following rule:

> # Libraries which don't provide a convenient means of conditionally loading
> # at runtime (this includes most libraries for languages such as C), SHOULD
> # NOT declare a "Depends:" or "Recommends:" relationship, directly or
> # indirectly, on packages containing anything more than dormant files. 
> # Those include, among others, daemons, executables in $PATH, etc.  Any such
> # relationship should be instead declared by programs that use the library
> # in question -- it is up to them to decide how important the relationship
> # is.

This feels to me like the wrong tool.  It's entirely plausible for there
to be a library whose entire purpose is to execute some external helper
program with elevated privileges, which obviously should depend on the
package that provides that program.  If we had such a requirement in
Policy, we would end up with libraries that don't Depend on their actual
dependencies and then callers have to add the dependency and then it's all
just a mess.

I feel like the problem here is that people are failing to fix bugs in
their packages (unnecessary dependencies on libraries that have heavy
dependencies), and you're trying to use Policy as a stick to beat them
with to get them to fix their bugs.  I don't think this is a good idea,
and I don't want us to end up with weird and incorrect dependencies for
libraries that really do require external helper programs (which is not
particularly rare).

Particularly since I don't think this requirement is actually targeted at
the dependency that's bothering you.  In your example:

> It'd help disarm dependency chains such as:
> xfce4-power-manager -> upower -> libimobiledevice4 -> usbmuxd
> ie, wanting XFCE currently pulls in a daemon that's of no use to anyone not
> using a piece of iJunk (and what it even has to do with power management?).

this would outlaw the Recommends from libimobiledevice4 to usbmuxd, but my
understanding is that dependency is *correct*, and the actual extraneous
dependency that's upsetting you is the one from upower to
libimobiledevice4, which this Policy change would not affect at all.  If I
were fixing this bug, that's the change I would make: have upower
dynamically load libimobiledevice4 iff it exists, since that's fairly
niche functionality.  (Or, really, given that it's a Recommends, just not
worry about it.)

In general, I don't want to see us place too many restrictions on
Recommends.  If you don't want additional helpful programs, disable
installing Recommends by default.  I think it's very odd to worry about
bloat while simultaneously installing Recommends by default; those aren't
really consistent things to do.

> Other prominent examples include:
> openssh-server -> libwrap1 -> tcpd
> using tcpd for ssh is a very niche use.

I assume this is for /etc/hosts.allow and /etc/hosts.deny.  I think that
could be lowered to a Suggests without losing much assuming the library
behaves correctly when those files are missing; have you already reported
that as a bug?

> e2fsprogs -> libuuid1 -> uuid-runtime
> that daemon is useful only if you need to rapidly generate MANY uuids.  Not
> just a single uuid per filesystem.  If your package actually needs that, it
> can declare the dependency itself, like ceph-base does.

Here too, Recommends -> Suggests seems to make sense to me; is that a
conversation anyone has already had with the maintainer?

-- 
Russ Allbery (r...@debian.org)   

Re: Depends/Recommends from libraries

[Josh Triplett]

Adam Borowski wrote:
> I'd like to discuss (and then propose to -policy) the following rule:
>
> # Libraries which don't provide a convenient means of conditionally loading
> # at runtime (this includes most libraries for languages such as C), SHOULD
> # NOT declare a "Depends:" or "Recommends:" relationship, directly or
> # indirectly, on packages containing anything more than dormant files.
> # Those include, among others, daemons, executables in $PATH, etc.  Any such
> # relationship should be instead declared by programs that use the library
> # in question -- it is up to them to decide how important the relationship
> # is.

In general, I'd like to see something like this; I'd love to see the
default install reduced, and Recommends made less painful to leave
enabled.  I think it'll need some more precise definition, though, on a
few points:

- Regarding packages that install running daemons, I agree with this
  entirely.  I wonder if we could establish some kind of substvar
  mechanism similar to shlibdeps to make it easy for an application
  package to say "I want a Depends/Recommends/Suggests on whatever this
  -dev package's corresponding library needs to run".

- A Depends with multiple alternatives, where the primary alternative
  meets this requirement, seems fine overall even if some of the
  alternatives would not meet this requirement.

- I don't actually see any issue with a library having a Depends or
  Recommends on a package that provides a binary executable, such as a
  helper program.  Most of the time such an executable would live in a
  library-specific directory under /usr/lib/, but if the program also
  has value for the end user to run directly, having it in $PATH seems
  fine.  (In which case it *has* to go in a separate package, to avoid
  conflicts that would make the library not work with multiarch.)
  Installing a library and ending up with an additional executable seems
  far more reasonable than installing a library and ending up with a
  running daemon.

  As an example, consider something like libguestfs.  Or consider any
  library that exists to wrap some command-line tool and give it an API.
  (Yes, the right answer is "refactor or rewrite the tool to put its
  implementation in a library", but that isn't always doable overnight.)

- Configuration files seem fine, such as default configuration for
  programs using the library, even though in some cases I wouldn't
  describe such configuration as "dormant".

- Plugins for other applications might consist of "libraries", but in
  those packages should not fall under this restriction.  Likewise for
  PAM and NSS modules and similar.

- While in general I would prefer not to see library packages asking
  debconf questions, I do see various libraries with dependencies on
  debconf.  This would seem to prohibit that.


And, of course, the usual issue of "policy shouldn't change to make
large numbers of packages insta-buggy" applies.  It's relatively easy to
construct a package limit (hit 'l') to browse such packages in aptitude;
just use this expression:

?section(libs) ?depends(?not(?or(?section(libs), ?name(^lib

(You may want to refine it further from there.)

Re: Depends/Recommends from libraries

[Ian Jackson]

Adam Borowski writes ("Depends/Recommends from libraries"):
> I'd like to discuss (and then propose to -policy) the following rule:
> 
> # Libraries which don't provide a convenient means of conditionally loading
> # at runtime (this includes most libraries for languages such as C), SHOULD
> # NOT declare a "Depends:" or "Recommends:" relationship, directly or
> # indirectly, on packages containing anything more than dormant files. 
> # Those include, among others, daemons, executables in $PATH, etc.  Any such
> # relationship should be instead declared by programs that use the library
> # in question -- it is up to them to decide how important the relationship
> # is.

This seems like a non-brainer to me.  Can anyone come up with a reason
why this would be wrong in general ?

Of course there may be exceptions, but you're proposing a SHOULD.

Ian.