Re: document symbols

[Jonathan Nieder]

Russ Allbery wrote:

> Let's go with 1:1.2.3.3.dfsg in the example to show the common case
> instead of the unusual case.  I've applied this:

Thanks.  Looks good.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120812233845.GA4447@mannheim-rule.local

Re: document symbols

[Russ Allbery]

Jonathan Nieder  writes:

> Hoorah! :)  I don't see any problems in the normative content, so I'd
> second this if I could.  Cosmetic nits (patch below):

Thanks, applied.

>> +  In our example, if the last change to the zlib1g
>> +  package that could change behavior for a client of that
>> +  library was in version 1:1.2.3.3.dfsg-1, then
>> +  the shlibs entry for this library could say:
>> +  
>> +libz 1 zlib1g (>= 1:1.2.3.3.dfsg-1)
>> +  

> Should this say (>= 1:1.2.3.3.dfsg-1~) or (>= 1:1.2.3.3.dfsg) to be
> kind to backporters?  Before the patch, the example said ">= 1:1.1.3".

Let's go with 1:1.2.3.3.dfsg in the example to show the common case
instead of the unusual case.  I've applied this:

commit 29e3fc2e05b59a7e13913a263a1e22d40cbc9918
Author: Russ Allbery 
Date:   Sun Aug 12 16:32:35 2012 -0700

Reflect the common case in the shlibs example

diff --git a/policy.sgml b/policy.sgml
index 050c688..3c863dc 100644
--- a/policy.sgml
+++ b/policy.sgml
@@ -6796,10 +6796,10 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)

  In our example, if the last change to the zlib1g
  package that could change behavior for a client of that
- library was in version 1:1.2.3.3.dfsg-2, then
+ library was in version 1:1.2.3.3.dfsg-1, then
  the shlibs entry for this library could say:
  
-   libz 1 zlib1g (>= 1:1.2.3.3.dfsg-2~)
+   libz 1 zlib1g (>= 1:1.2.3.3.dfsg)
  
  This version restriction must be new enough that any binary
  built against the current version of the library will work
@@ -6811,7 +6811,7 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
  As zlib1g also provides a udeb containing the shared
  library, there would also be a second line:
  
-   udeb: libz 1 zlib1g-udeb (>= 1:1.2.3.3.dfsg-2~)
+   udeb: libz 1 zlib1g-udeb (>= 1:1.2.3.3.dfsg)
  

  

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sjbrsn2a@windlord.stanford.edu

Re: document symbols

[Jonathan Nieder]

Russ Allbery wrote:

> Okay, once more for the win.

Hoorah! :)  I don't see any problems in the normative content, so I'd
second this if I could.  Cosmetic nits (patch below):

[...]
> +++ b/policy.sgml
[...]
> @@ -5633,17 +5634,29 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
[...]
>   
> +   To determine the soversion, look at
> +   the SONAME of the library, stored in the
> +   ELF SONAME attribute.  it is usually of the
  ^^

Capitalization: s/it/It/

[...]
> + A common example of when a change to the is required is a
  ^

Missing word: s/to the/to the dependency version/.  (In a previous
version of the patch, this passage was discussing symbols files only
and said minimal-version.)

[...]
> +   no symbol version.  minimal-version is the most
> +   recent version of the shared library that changed the
> +   behavior of that symbol, whether by adding it, changing its
  ~~ 
> +   function signature (the parameters, their types, or the
> +   return type), or its behavior in a way that is visible to a
~~~
> +   caller.  id-of-dependency-template is an optional

Maintaining parallel construction: s/its behavior/changing its
behavior/.

[...]
> +   In our example, if the last change to the zlib1g
> +   package that could change behavior for a client of that
> +   library was in version 1:1.2.3.3.dfsg-1, then
> +   the shlibs entry for this library could say:
> +   
> + libz 1 zlib1g (>= 1:1.2.3.3.dfsg-1)
> +   

Should this say (>= 1:1.2.3.3.dfsg-1~) or (>= 1:1.2.3.3.dfsg) to be
kind to backporters?  Before the patch, the example said ">= 1:1.1.3".

-- >8 --
Subject: symbols/shlibs policy: cosmetic fixes

Use "zlib1g (>= 1:1.2.3.3.dfsg-2~)" in the sample shlibs dependency
field to emphasize the backport-friendly convention described in the
sharedlibs-updates section.

Also correct two small typos --- one sentence is uncapitalized and
another missing a noun --- and rephrase a sentence that describes when
to bump the dependency-version to make it easier to read.
---
diff --git a/policy.sgml b/policy.sgml
index fa1c39a..050c688 100644
--- a/policy.sgml
+++ b/policy.sgml
@@ -5646,7 +5646,7 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)

  To determine the soversion, look at
  the SONAME of the library, stored in the
- ELF SONAME attribute.  it is usually of the
+ ELF SONAME attribute.  It is usually of the
  form name.so.major-version (for
  example, libz.so.1).  The version part is the part
  which comes after .so., so in that example it
@@ -6238,9 +6238,9 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
  
 
  
-   A common example of when a change to the is required is a
-   function that takes an enum or struct argument that controls
-   what the function does.  For example:
+   A common example of when a change to the dependency version
+   is required is a function that takes an enum or struct
+   argument that controls what the function does.  For example:

  enum library_op { OP_FOO, OP_BAR };
  int library_do_operation(enum library_op);
@@ -6489,8 +6489,9 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
  recent version of the shared library that changed the
  behavior of that symbol, whether by adding it, changing its
  function signature (the parameters, their types, or the
- return type), or its behavior in a way that is visible to a
- caller.  id-of-dependency-template is an optional
+ return type), or changing its behavior in a way that is
+ visible to a caller.
+ id-of-dependency-template is an optional
  field that references
  an alternative-dependency-template; see below for
  a full description.
@@ -6795,10 +6796,10 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)

  In our example, if the last change to the zlib1g
  package that could change behavior for a client of that
- library was in version 1:1.2.3.3.dfsg-1, then
+ library was in version 1:1.2.3.3.dfsg-2, then
  the shlibs entry for this library could say:
  
-   libz 1 zlib1g (>= 1:1.2.3.3.dfsg-1)
+   libz 1 zlib1g (>= 1:1.2.3.3.dfsg-2~)
  
  This version restriction must be new enough that any binary
  built against the current version of the library will work
@@ -6810,7 +6811,7 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
  As zlib1g al

Re: document symbols

[Russ Allbery]

Jonathan Nieder  writes:
> Russ Allbery wrote:

>> I'm therefore including here the complete SGML source of that section
>> not in diff format, followed by the diff of everything *outside* of
>> that section.  I think this will be easier to review.

> Thanks!  I would have preferred a diff since it shows the text that is
> being replaced, too, but let's go with this for a first pass.

Yeah, it's frustrating to review something this large, and none of the
normal tools do a particularly good job at it.  A side-by-side contextual
diff tool is probably best.

Anyway, thank you for the detailed review, and apologies for taking so
long to get back to this.  The amount of work required is intimidating,
and I kept putting it off.

For the most part, I adopted your changes; assume that if I don't comment
here specifically, I've incorporated that change.  (I started by applying
your interdiff and then only changing the bits that I thought I could
further clarify.)

> [...]
>>  
>>If a package contains a binary or library which links to a
>>shared library, we must ensure that, when the package is
>>installed on the system, all of the libraries needed are also
>>installed.

> This text is carried over from before and contains a requirement I never
> noticed before.  Suppose my package contains two binaries:  maintool and
> side-tool.  The latter is not very important and links to libbiglibrary.
> I might be tempted to make the dependency by my package on libbiglibrary
> a Recommends instead of a Depends.  The above says I must not.

> Intentional?  It seems like good policy, anyway.

Could you open a separate bug about this?  I think we should allow
Recommends, but as you say it's already in the current wording and this
change is already too complicated.  We should discuss it separately.

> This means packages must not hard-code library dependencies.  It also
> seems like good policy, but I suspect it would render packages such as
> chromium that use dlopen() and hard-code the corresponding library name
> in dependencies RC-buggy.

Your fix (making dependencies for dlopen a should instead of a must)
looked like a good way of fixing this problem to me.  Thanks!

>>To allow these dependencies to be constructed, shared libraries
>>must provide either a symbols file or
>>a shlibs file, which provide information on the
>>package dependencies required to ensure the presence of this
>>library.

> Subject/verb agreement: s/provide/provides/

While that's technically correct, it looks completely wrong to me.  I
reworded to make this two sentences instead, so that it's both formally
correct and "feels" right.

> If I remove a symbol that was documented to be private or change the
> behavior of a function when given invalid arguments, is that a
> backward-compatible change?

> If I add change the implementation in such a way that the library
> becomes so large that some large programs cannot use it any more, is
> that a backward-incompatible change?

You addressed this by introducing the concept of a "reasonable" program
but not defining it.  That sounded like the right approach to me, but I
felt the need to say more, so I added a footnote explaining the intent:

There are two types of ABI changes: ones that are
backward-compatible and ones that are not.  An ABI change is
backward-compatible if any reasonable program or library that
was linked with the previous version of the shared library
will still work correctly with the new version of the shared
library.
  An example of an "unreasonable" program is one that uses library
  interfaces that are documented as internal and unsupported.  If the
  only programs or libraries affected by a change are "unreasonable"
  ones, other techniques, such as declaring Breaks
  relationships with affected packages or treating their usage of the
  library as bugs in those packages, may be appropriate instead of
  changing the SONAME.  However, the default approach is to change the
  SONAME for any change to the ABI that could break a program.


> Unrelated change.  The patch would have been easier to review if this
> were a separate commit, which could have gone straight to master since
> it doesn't change the output.

Yes, sorry.  I really hate the whole diff system for making changes to
text documents, since I always reformat text documents as I work on them.
I'll try to avoid this when people find it confusing, but as long as I'm
writing the patches, you may have to just live with some of this, since
putting more barriers in the way of writing text for Policy will mean that
I'll do even less work than I do now.  :/  That said, I agree that it's
kind of annoying for review, and I'll try to get better about not doing
it.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.deb

Re: document symbols

[Jonathan Nieder]

Jonathan Nieder wrote:

> I'll reply with an interdiff relative to the last version of the
> patch.

Here it is.
Subject: Clarifications to symbols and shlibs policy

subject/verb agreement: s/provide/provides/

Packages with libraries or binaries linking to a shared library must
use symbols or shlibs files to compute their dependencies.  Packages
that dlopen() a shared library should do so as well, but since that
is not typical practice and the tools to do that don't exist, it is
not made a policy "must" yet.

The minimal version for a symbol can be bumped after the version of
the package in which the symbol was introduced.

Add a footnote explaining why shlibs files cannot be used for
libraries with unusual sonames.

The shlibs file for a library udeb goes in the corresponding deb.

The library deb corresponding to a udeb is supposed to provide a
shlibs file, rather than consuming (using) one.

Add "for example" when talking about dpkg-shlibdeps -T.  This is
just an illustration and not meant to be normative.

If a library is used both directly and indirectly, the direct
dependency still needs to be declared.

Backward-compatibility is defined in terms of what reasonable
programs and libraries need.

In the normal case, symbols files go in dpkg's admindir as package
control files.

wording fix: "dependency on" avoids some of the ambiguity in
"dependency of".
---
 policy.sgml |   86 +--
 1 file changed, 48 insertions(+), 38 deletions(-)

diff --git a/policy.sgml b/policy.sgml
index 1eb039f8..50ae756c 100644
--- a/policy.sgml
+++ b/policy.sgml
@@ -5838,25 +5838,30 @@ Replaces: mail-transport-agent
  whether new library interfaces are available and can be called).
  To allow these dependencies to be constructed, shared libraries
  must provide either a symbols file or
- a shlibs file, which provide information on the
- package dependencies required to ensure the presence of this
- library.  Any package which uses a shared library must use these
- files to determine the required dependencies when it is built.
+ a shlibs file, which provides information on the
+ package dependencies required to ensure the presence of
+ interfaces provided by this library.  Any package with binaries
+ or libraries linking to a shared library must use these files
+ to determine the required dependencies when it is built.  Other
+ packages which use a shared library (for example using
+ dlopen()) should compute appropriate dependencies
+ using these files at build time as well.

 

- These two mechanisms differ in the degree of detail that they
- provide.  A symbols file documents every symbol
- that is part of the library ABI and, for each, the version of
- the package in which it was introduced.  This permits detailed
- analysis of the symbols used by a particular package and
- construction of an accurate dependency, but it requires the
- package maintainer to track more information about the shared
- library.  A shlibs file, in contrast, only
- documents the last time the library ABI changed in any way.  It
- only provides information about the library as a whole, not
- individual symbols.  When a package is built using a shared
- library with only a shlibs file, the generated
+ The two mechanisms differ in the degree of detail that they
+ provide.  A symbols file documents for each symbol
+ exported by a library the minimal version of the package any
+ binary using this symbol will need, which is typically the
+ version of the package in which the symbol was introduced.
+ This permits detailed analysis of the symbols used by a
+ particular package and construction of an accurate dependency,
+ but it requires the package maintainer to track more information
+ about the shared library.  A shlibs file, in
+ contrast, only documents the last time the library ABI changed
+ in any way.  It only provides information about the library as a
+ whole, not individual symbols.  When a package is built using a
+ shared library with only a shlibs file, the generated
  dependency will require a version of the shared library equal to
  or newer than the version of the last ABI change.  This
  generates unnecessarily restrictive dependencies compared
@@ -5869,7 +5874,11 @@ Replaces: mail-transport-agent

  shlibs files also have a flawed representation of
  library SONAMEs, making it difficult to use shlibs
- files in some unusual corner cases.
+ files in some unusual corner cases.
+   libfooN.shlibs says 'libfoo N' instead of the actual SONAME,
+   so if the SONAME doesn't match

Re: document symbols

[Jonathan Nieder]

Julien Cristau wrote:
> On Mon, Mar 19, 2012 at 17:26:04 -0500, Jonathan Nieder wrote:

>> What about libraries like glib (assuming one only uses old symbols)
>> that are never supposed to change soname?
>
> What about them?

I wanted to make sure that forbidding hard-coded dependencies on them
is intentional.  It seems like a good choice to me, but it should be a
deliberate choice (and it does not seem obvious to me that a patch
documenting symbols would automatically do that).

>> [...]
>>>   shlibs files also have a flawed representation of
>>>   library SONAMEs, making it difficult to use shlibs
>>>   files in some unusual corner cases.
>>
>> I'm not sure what this passage is referring to.  Can you say more?
>> (Maybe in a footnote.)
>
> libfooN.shlibs says 'libfoo N' not the actual SONAME, so if the SONAME
> doesn't match one of the two expected formats (libfoo-N.so or
> libfoo.so.N) it can't be represented.

Thanks.  Sounds like good text for a footnote.

[...]
>> To avoid confusion it might be worth forbidding symbols files in
>> udebs, or at least symbols files without a corresponding shlibs file
>> accompanying them.
>
> That makes no sense.  udebs don't have those files, when building an
> udeb the dependency information is read from the shlibs files of the
> debs corresponding to the libraries you depend on.

Oh, good catch.  Russ's text said:

  symbols files are therefore recommended for most
  shared library packages since they provide more accurate
  dependencies.  For most C libraries, the additional detail
  required by symbols files is not too difficult to
  maintain.  However, maintaining exhaustive symbols information
  for a C++ library can be quite onerous, so shlibs
  files may be more appropriate for most C++ libraries.  udebs
  must also use shlibs, since the udeb infrastructure
  does not use symbols.

which sounded like it was saying that most shared libraries should
provide symbols files but udebs should not since the infrastructure
does not support it.

If I understand you correctly, the actual rule would be:

 - symbols files are always recommended

 - the deb corresponding to a shared library udeb must provide a
   shlibs file to support udeb infrastructure

 - udebs provide neither shlibs nor symbols files

[...]
>>>If you have
>>> multiple binary packages, you will need to
>>> call dpkg-shlibdeps on each one which contains
>>> compiled libraries or binaries, using the -T option
>>> to the dpkg utilities to specify a
>>> different substvars file for each binary
>>> package.
>>
>> An alternative is to clear substvars between builds of different
>> binary packages.
>
> Who does that?

I did before I saw this patch, in a package not yet proposed for
upload to Debian.  Should I be ashamed?

>>> There are two types of ABI changes: ones that are
>>> backward-compatible and ones that are not.  An ABI change is
>>> backward-compatible if any binary was linked with the previous
>>> version of the shared library will still work correctly with
>>> the new version of the shared library.  Adding new symbols to
>>> the shared library is a backward-compatible change.  Removing
>>> symbols from the shared library is not.
>>
>> If I remove a symbol that was documented to be private or change
>> the behavior of a function when given invalid arguments, is that a
>> backward-compatible change?
>>
>> If I add change the implementation in such a way that the library
>> becomes so large that some large programs cannot use it any more, is
>> that a backward-incompatible change?
>
> I'm not sure policy should go into such details.

Sorry for the lack of clarity.  I never meant to suggest that policy
should speak to these cases directly.  That would be insane, and among
other consequences it would result in a very long policy manual.

What I was trying to hint at is that the above definition gives the
wrong answer to both questions.

> And anyway, that's
> answered by the previous sentence (an incompatible change is one that
> breaks reverse deps).  The last two are simple examples.

The definition says a change is backward-compatible when "any binary
[that] was linked with the previous version of the shared library will
still work correctly with the new version of the shared library".  If
I understand it correctly, that means that the answer to the first
question is "no" (a binary using private symbols is still a binary)
and the answer to the second question is "yes" (a binary whose process
image barely fits in address space is still a binary).

I believe the definition would need a word like "reasonable" before
"binary" to be accurate.

Thanks for your help,
Jonathan


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscrib

Re: document symbols

[Jonathan Nieder]

Russ Allbery wrote:

>   I'm therefore including here the complete
> SGML source of that section not in diff format, followed by the diff of
> everything *outside* of that section.  I think this will be easier to
> review.

Thanks!  I would have preferred a diff since it shows the text that is
being replaced, too, but let's go with this for a first pass.

[...]
>   
> If a package contains a binary or library which links to a
> shared library, we must ensure that, when the package is
> installed on the system, all of the libraries needed are also
> installed.

This text is carried over from before and contains a requirement I
never noticed before.  Suppose my package contains two binaries:
maintool and side-tool.  The latter is not very important and links
to libbiglibrary.  I might be tempted to make the dependency by my
package on libbiglibrary a Recommends instead of a Depends.  The above
says I must not.

Intentional?  It seems like good policy, anyway.

>These dependencies must be added to the binary
> package when it is built, since they may change

This means packages must not hard-code library dependencies.  It
also seems like good policy, but I suspect it would render packages
such as chromium that use dlopen() and hard-code the corresponding
library name in dependencies RC-buggy.

What about libraries like glib (assuming one only uses old symbols)
that are never supposed to change soname?

[...]
> To allow these dependencies to be constructed, shared libraries
> must provide either a symbols file or
> a shlibs file, which provide information on the
> package dependencies required to ensure the presence of this
> library.

Subject/verb agreement: s/provide/provides/

Clarity: s/this library/interfaces provided by this library/

>   
> These two mechanisms differ in the degree of detail that they
> provide.  A symbols file documents every symbol
> that is part of the library ABI and, for each, the version of
> the package in which it was introduced.

Maybe, since minimal-version is not always the version in which the
symbol was introduced:

and, for each, a minimal version of the library needed to use
that symbol, which is typically the version of the package in
which it was introduced.

[...]
> shlibs files also have a flawed representation of
> library SONAMEs, making it difficult to use shlibs
> files in some unusual corner cases.

I'm not sure what this passage is referring to.  Can you say more?
(Maybe in a footnote.)

[...]
>udebs
> must also use shlibs, since the udeb infrastructure
> does not use symbols.

To avoid confusion it might be worth forbidding symbols files in
udebs, or at least symbols files without a corresponding shlibs file
accompanying them.

[...]
>  If you have
>   multiple binary packages, you will need to
>   call dpkg-shlibdeps on each one which contains
>   compiled libraries or binaries, using the -T option
>   to the dpkg utilities to specify a
>   different substvars file for each binary
>   package.

An alternative is to clear substvars between builds of different
binary packages.

[...]
>   loads libbar.  A package should depend on the
>   libraries it directly uses, but not the libraries it
>   indirectly uses.

Pedantry: what if my package uses the same library both directly and
indirectly?  "but not the libraries it only uses indirectly" would
avoid that question.

>   There are two types of ABI changes: ones that are
>   backward-compatible and ones that are not.  An ABI change is
>   backward-compatible if any binary was linked with the previous
>   version of the shared library will still work correctly with
>   the new version of the shared library.  Adding new symbols to
>   the shared library is a backward-compatible change.  Removing
>   symbols from the shared library is not.

If I remove a symbol that was documented to be private or change
the behavior of a function when given invalid arguments, is that a
backward-compatible change?

If I add change the implementation in such a way that the library
becomes so large that some large programs cannot use it any more, is
that a backward-incompatible change?

[...]
> symbols files for a shared library are normally
> provided by the shared library package, but there are
> several override paths that are checked first in case that
> information is wrong or missing.

It's not obvious at first how the two clauses of this sentence relate
to each other.  (Why would which package provides the file and