Re: document symbols
Jonathan Nieder jrnie...@gmail.com writes: Russ Allbery wrote: I'm therefore including here the complete SGML source of that section not in diff format, followed by the diff of everything *outside* of that section. I think this will be easier to review. Thanks! I would have preferred a diff since it shows the text that is being replaced, too, but let's go with this for a first pass. Yeah, it's frustrating to review something this large, and none of the normal tools do a particularly good job at it. A side-by-side contextual diff tool is probably best. Anyway, thank you for the detailed review, and apologies for taking so long to get back to this. The amount of work required is intimidating, and I kept putting it off. For the most part, I adopted your changes; assume that if I don't comment here specifically, I've incorporated that change. (I started by applying your interdiff and then only changing the bits that I thought I could further clarify.) [...] p If a package contains a binary or library which links to a shared library, we must ensure that, when the package is installed on the system, all of the libraries needed are also installed. This text is carried over from before and contains a requirement I never noticed before. Suppose my package contains two binaries: maintool and side-tool. The latter is not very important and links to libbiglibrary. I might be tempted to make the dependency by my package on libbiglibrary a Recommends instead of a Depends. The above says I must not. Intentional? It seems like good policy, anyway. Could you open a separate bug about this? I think we should allow Recommends, but as you say it's already in the current wording and this change is already too complicated. We should discuss it separately. This means packages must not hard-code library dependencies. It also seems like good policy, but I suspect it would render packages such as chromium that use dlopen() and hard-code the corresponding library name in dependencies RC-buggy. Your fix (making dependencies for dlopen a should instead of a must) looked like a good way of fixing this problem to me. Thanks! To allow these dependencies to be constructed, shared libraries must provide either a filesymbols/file file or a fileshlibs/file file, which provide information on the package dependencies required to ensure the presence of this library. Subject/verb agreement: s/provide/provides/ While that's technically correct, it looks completely wrong to me. I reworded to make this two sentences instead, so that it's both formally correct and feels right. If I remove a symbol that was documented to be private or change the behavior of a function when given invalid arguments, is that a backward-compatible change? If I add change the implementation in such a way that the library becomes so large that some large programs cannot use it any more, is that a backward-incompatible change? You addressed this by introducing the concept of a reasonable program but not defining it. That sounded like the right approach to me, but I felt the need to say more, so I added a footnote explaining the intent: There are two types of ABI changes: ones that are backward-compatible and ones that are not. An ABI change is backward-compatible if any reasonable program or library that was linked with the previous version of the shared library will still work correctly with the new version of the shared library.footnote An example of an unreasonable program is one that uses library interfaces that are documented as internal and unsupported. If the only programs or libraries affected by a change are unreasonable ones, other techniques, such as declaring ttBreaks/tt relationships with affected packages or treating their usage of the library as bugs in those packages, may be appropriate instead of changing the SONAME. However, the default approach is to change the SONAME for any change to the ABI that could break a program. /footnote Unrelated change. The patch would have been easier to review if this were a separate commit, which could have gone straight to master since it doesn't change the output. Yes, sorry. I really hate the whole diff system for making changes to text documents, since I always reformat text documents as I work on them. I'll try to avoid this when people find it confusing, but as long as I'm writing the patches, you may have to just live with some of this, since putting more barriers in the way of writing text for Policy will mean that I'll do even less work than I do now. :/ That said, I agree that it's kind of annoying for review, and I'll try to get better about not doing it. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to
Re: document symbols
Russ Allbery wrote:
Okay, once more for the win.
Hoorah! :) I don't see any problems in the normative content, so I'd
second this if I could. Cosmetic nits (patch below):
[...]
+++ b/policy.sgml
[...]
@@ -5633,17 +5634,29 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
[...]
p
+ To determine the varsoversion/var, look at
+ the ttSONAME/tt of the library, stored in the
+ ELF ttSONAME/tt attribute. it is usually of the
^^
Capitalization: s/it/It/
[...]
+ A common example of when a change to the is required is a
^
Missing word: s/to the/to the dependency version/. (In a previous
version of the patch, this passage was discussing symbols files only
and said varminimal-version/var.)
[...]
+ no symbol version. varminimal-version/var is the most
+ recent version of the shared library that changed the
+ behavior of that symbol, whether by adding it, changing its
~~
+ function signature (the parameters, their types, or the
+ return type), or its behavior in a way that is visible to a
~~~
+ caller. varid-of-dependency-template/var is an optional
Maintaining parallel construction: s/its behavior/changing its
behavior/.
[...]
+ In our example, if the last change to the ttzlib1g/tt
+ package that could change behavior for a client of that
+ library was in version tt1:1.2.3.3.dfsg-1/tt, then
+ the ttshlibs/tt entry for this library could say:
+ example compact=compact
+ libz 1 zlib1g (= 1:1.2.3.3.dfsg-1)
+ /example
Should this say (= 1:1.2.3.3.dfsg-1~) or (= 1:1.2.3.3.dfsg) to be
kind to backporters? Before the patch, the example said = 1:1.1.3.
-- 8 --
Subject: symbols/shlibs policy: cosmetic fixes
Use zlib1g (= 1:1.2.3.3.dfsg-2~) in the sample shlibs dependency
field to emphasize the backport-friendly convention described in the
sharedlibs-updates section.
Also correct two small typos --- one sentence is uncapitalized and
another missing a noun --- and rephrase a sentence that describes when
to bump the dependency-version to make it easier to read.
---
diff --git a/policy.sgml b/policy.sgml
index fa1c39a..050c688 100644
--- a/policy.sgml
+++ b/policy.sgml
@@ -5646,7 +5646,7 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
p
To determine the varsoversion/var, look at
the ttSONAME/tt of the library, stored in the
- ELF ttSONAME/tt attribute. it is usually of the
+ ELF ttSONAME/tt attribute. It is usually of the
form ttvarname/var.so.varmajor-version/var/tt (for
example, ttlibz.so.1/tt). The version part is the part
which comes after tt.so./tt, so in that example it
@@ -6238,9 +6238,9 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
/p
p
- A common example of when a change to the is required is a
- function that takes an enum or struct argument that controls
- what the function does. For example:
+ A common example of when a change to the dependency version
+ is required is a function that takes an enum or struct
+ argument that controls what the function does. For example:
example
enum library_op { OP_FOO, OP_BAR };
int library_do_operation(enum library_op);
@@ -6489,8 +6489,9 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
recent version of the shared library that changed the
behavior of that symbol, whether by adding it, changing its
function signature (the parameters, their types, or the
- return type), or its behavior in a way that is visible to a
- caller. varid-of-dependency-template/var is an optional
+ return type), or changing its behavior in a way that is
+ visible to a caller.
+ varid-of-dependency-template/var is an optional
field that references
an varalternative-dependency-template/var; see below for
a full description.
@@ -6795,10 +6796,10 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1)
p
In our example, if the last change to the ttzlib1g/tt
package that could change behavior for a client of that
- library was in version tt1:1.2.3.3.dfsg-1/tt, then
+ library was in version tt1:1.2.3.3.dfsg-2/tt, then
the ttshlibs/tt entry for this library could say:
example compact=compact
- libz 1 zlib1g (= 1:1.2.3.3.dfsg-1)
+ libz 1 zlib1g (= 1:1.2.3.3.dfsg-2~)
/example
This version restriction must be new enough
Re: document symbols
Jonathan Nieder jrnie...@gmail.com writes: Hoorah! :) I don't see any problems in the normative content, so I'd second this if I could. Cosmetic nits (patch below): Thanks, applied. + In our example, if the last change to the ttzlib1g/tt + package that could change behavior for a client of that + library was in version tt1:1.2.3.3.dfsg-1/tt, then + the ttshlibs/tt entry for this library could say: + example compact=compact +libz 1 zlib1g (= 1:1.2.3.3.dfsg-1) + /example Should this say (= 1:1.2.3.3.dfsg-1~) or (= 1:1.2.3.3.dfsg) to be kind to backporters? Before the patch, the example said = 1:1.1.3. Let's go with 1:1.2.3.3.dfsg in the example to show the common case instead of the unusual case. I've applied this: commit 29e3fc2e05b59a7e13913a263a1e22d40cbc9918 Author: Russ Allbery r...@debian.org Date: Sun Aug 12 16:32:35 2012 -0700 Reflect the common case in the shlibs example diff --git a/policy.sgml b/policy.sgml index 050c688..3c863dc 100644 --- a/policy.sgml +++ b/policy.sgml @@ -6796,10 +6796,10 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1) p In our example, if the last change to the ttzlib1g/tt package that could change behavior for a client of that - library was in version tt1:1.2.3.3.dfsg-2/tt, then + library was in version tt1:1.2.3.3.dfsg-1/tt, then the ttshlibs/tt entry for this library could say: example compact=compact - libz 1 zlib1g (= 1:1.2.3.3.dfsg-2~) + libz 1 zlib1g (= 1:1.2.3.3.dfsg) /example This version restriction must be new enough that any binary built against the current version of the library will work @@ -6811,7 +6811,7 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1) As zlib1g also provides a udeb containing the shared library, there would also be a second line: example compact=compact - udeb: libz 1 zlib1g-udeb (= 1:1.2.3.3.dfsg-2~) + udeb: libz 1 zlib1g-udeb (= 1:1.2.3.3.dfsg) /example /p /sect2 -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87sjbrsn2a@windlord.stanford.edu
Re: document symbols
Russ Allbery wrote: Let's go with 1:1.2.3.3.dfsg in the example to show the common case instead of the unusual case. I've applied this: Thanks. Looks good. -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120812233845.GA4447@mannheim-rule.local
Re: document symbols
Jonathan Nieder wrote: I'll reply with an interdiff relative to the last version of the patch. Here it is. Subject: Clarifications to symbols and shlibs policy subject/verb agreement: s/provide/provides/ Packages with libraries or binaries linking to a shared library must use symbols or shlibs files to compute their dependencies. Packages that dlopen() a shared library should do so as well, but since that is not typical practice and the tools to do that don't exist, it is not made a policy must yet. The minimal version for a symbol can be bumped after the version of the package in which the symbol was introduced. Add a footnote explaining why shlibs files cannot be used for libraries with unusual sonames. The shlibs file for a library udeb goes in the corresponding deb. The library deb corresponding to a udeb is supposed to provide a shlibs file, rather than consuming (using) one. Add for example when talking about dpkg-shlibdeps -T. This is just an illustration and not meant to be normative. If a library is used both directly and indirectly, the direct dependency still needs to be declared. Backward-compatibility is defined in terms of what reasonable programs and libraries need. In the normal case, symbols files go in dpkg's admindir as package control files. wording fix: dependency on avoids some of the ambiguity in dependency of. --- policy.sgml | 86 +-- 1 file changed, 48 insertions(+), 38 deletions(-) diff --git a/policy.sgml b/policy.sgml index 1eb039f8..50ae756c 100644 --- a/policy.sgml +++ b/policy.sgml @@ -5838,25 +5838,30 @@ Replaces: mail-transport-agent whether new library interfaces are available and can be called). To allow these dependencies to be constructed, shared libraries must provide either a filesymbols/file file or - a fileshlibs/file file, which provide information on the - package dependencies required to ensure the presence of this - library. Any package which uses a shared library must use these - files to determine the required dependencies when it is built. + a fileshlibs/file file, which provides information on the + package dependencies required to ensure the presence of + interfaces provided by this library. Any package with binaries + or libraries linking to a shared library must use these files + to determine the required dependencies when it is built. Other + packages which use a shared library (for example using + ttdlopen()/tt) should compute appropriate dependencies + using these files at build time as well. /p p - These two mechanisms differ in the degree of detail that they - provide. A filesymbols/file file documents every symbol - that is part of the library ABI and, for each, the version of - the package in which it was introduced. This permits detailed - analysis of the symbols used by a particular package and - construction of an accurate dependency, but it requires the - package maintainer to track more information about the shared - library. A fileshlibs/file file, in contrast, only - documents the last time the library ABI changed in any way. It - only provides information about the library as a whole, not - individual symbols. When a package is built using a shared - library with only a fileshlibs/file file, the generated + The two mechanisms differ in the degree of detail that they + provide. A filesymbols/file file documents for each symbol + exported by a library the minimal version of the package any + binary using this symbol will need, which is typically the + version of the package in which the symbol was introduced. + This permits detailed analysis of the symbols used by a + particular package and construction of an accurate dependency, + but it requires the package maintainer to track more information + about the shared library. A fileshlibs/file file, in + contrast, only documents the last time the library ABI changed + in any way. It only provides information about the library as a + whole, not individual symbols. When a package is built using a + shared library with only a fileshlibs/file file, the generated dependency will require a version of the shared library equal to or newer than the version of the last ABI change. This generates unnecessarily restrictive dependencies compared @@ -5869,7 +5874,11 @@ Replaces: mail-transport-agent p fileshlibsfile files also have a flawed representation of library SONAMEs, making it difficult to use fileshlibs/file - files in some unusual corner cases. + files in some unusual corner cases.footnote +
Re: document symbols
Julien Cristau wrote: On Mon, Mar 19, 2012 at 17:26:04 -0500, Jonathan Nieder wrote: What about libraries like glib (assuming one only uses old symbols) that are never supposed to change soname? What about them? I wanted to make sure that forbidding hard-coded dependencies on them is intentional. It seems like a good choice to me, but it should be a deliberate choice (and it does not seem obvious to me that a patch documenting symbols would automatically do that). [...] fileshlibsfile files also have a flawed representation of library SONAMEs, making it difficult to use fileshlibs/file files in some unusual corner cases. I'm not sure what this passage is referring to. Can you say more? (Maybe in a footnote.) libfooN.shlibs says 'libfoo N' not the actual SONAME, so if the SONAME doesn't match one of the two expected formats (libfoo-N.so or libfoo.so.N) it can't be represented. Thanks. Sounds like good text for a footnote. [...] To avoid confusion it might be worth forbidding symbols files in udebs, or at least symbols files without a corresponding shlibs file accompanying them. That makes no sense. udebs don't have those files, when building an udeb the dependency information is read from the shlibs files of the debs corresponding to the libraries you depend on. Oh, good catch. Russ's text said: filesymbols/file files are therefore recommended for most shared library packages since they provide more accurate dependencies. For most C libraries, the additional detail required by filesymbols/file files is not too difficult to maintain. However, maintaining exhaustive symbols information for a C++ library can be quite onerous, so fileshlibs/file files may be more appropriate for most C++ libraries. udebs must also use fileshlibs/file, since the udeb infrastructure does not use filesymbols/file. which sounded like it was saying that most shared libraries should provide symbols files but udebs should not since the infrastructure does not support it. If I understand you correctly, the actual rule would be: - symbols files are always recommended - the deb corresponding to a shared library udeb must provide a shlibs file to support udeb infrastructure - udebs provide neither shlibs nor symbols files [...] If you have multiple binary packages, you will need to call prgndpkg-shlibdeps/prgn on each one which contains compiled libraries or binaries, using the tt-T/tt option to the ttdpkg/tt utilities to specify a different filesubstvars/file file for each binary package.footnote An alternative is to clear substvars between builds of different binary packages. Who does that? I did before I saw this patch, in a package not yet proposed for upload to Debian. Should I be ashamed? There are two types of ABI changes: ones that are backward-compatible and ones that are not. An ABI change is backward-compatible if any binary was linked with the previous version of the shared library will still work correctly with the new version of the shared library. Adding new symbols to the shared library is a backward-compatible change. Removing symbols from the shared library is not. If I remove a symbol that was documented to be private or change the behavior of a function when given invalid arguments, is that a backward-compatible change? If I add change the implementation in such a way that the library becomes so large that some large programs cannot use it any more, is that a backward-incompatible change? I'm not sure policy should go into such details. Sorry for the lack of clarity. I never meant to suggest that policy should speak to these cases directly. That would be insane, and among other consequences it would result in a very long policy manual. What I was trying to hint at is that the above definition gives the wrong answer to both questions. And anyway, that's answered by the previous sentence (an incompatible change is one that breaks reverse deps). The last two are simple examples. The definition says a change is backward-compatible when any binary [that] was linked with the previous version of the shared library will still work correctly with the new version of the shared library. If I understand it correctly, that means that the answer to the first question is no (a binary using private symbols is still a binary) and the answer to the second question is yes (a binary whose process image barely fits in address space is still a binary). I believe the definition would need a word like reasonable before binary to be accurate. Thanks for your help, Jonathan -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact
Re: document symbols
Russ Allbery wrote: I'm therefore including here the complete SGML source of that section not in diff format, followed by the diff of everything *outside* of that section. I think this will be easier to review. Thanks! I would have preferred a diff since it shows the text that is being replaced, too, but let's go with this for a first pass. [...] p If a package contains a binary or library which links to a shared library, we must ensure that, when the package is installed on the system, all of the libraries needed are also installed. This text is carried over from before and contains a requirement I never noticed before. Suppose my package contains two binaries: maintool and side-tool. The latter is not very important and links to libbiglibrary. I might be tempted to make the dependency by my package on libbiglibrary a Recommends instead of a Depends. The above says I must not. Intentional? It seems like good policy, anyway. These dependencies must be added to the binary package when it is built, since they may change This means packages must not hard-code library dependencies. It also seems like good policy, but I suspect it would render packages such as chromium that use dlopen() and hard-code the corresponding library name in dependencies RC-buggy. What about libraries like glib (assuming one only uses old symbols) that are never supposed to change soname? [...] To allow these dependencies to be constructed, shared libraries must provide either a filesymbols/file file or a fileshlibs/file file, which provide information on the package dependencies required to ensure the presence of this library. Subject/verb agreement: s/provide/provides/ Clarity: s/this library/interfaces provided by this library/ p These two mechanisms differ in the degree of detail that they provide. A filesymbols/file file documents every symbol that is part of the library ABI and, for each, the version of the package in which it was introduced. Maybe, since minimal-version is not always the version in which the symbol was introduced: and, for each, a minimal version of the library needed to use that symbol, which is typically the version of the package in which it was introduced. [...] fileshlibsfile files also have a flawed representation of library SONAMEs, making it difficult to use fileshlibs/file files in some unusual corner cases. I'm not sure what this passage is referring to. Can you say more? (Maybe in a footnote.) [...] udebs must also use fileshlibs/file, since the udeb infrastructure does not use filesymbols/file. To avoid confusion it might be worth forbidding symbols files in udebs, or at least symbols files without a corresponding shlibs file accompanying them. [...] If you have multiple binary packages, you will need to call prgndpkg-shlibdeps/prgn on each one which contains compiled libraries or binaries, using the tt-T/tt option to the ttdpkg/tt utilities to specify a different filesubstvars/file file for each binary package.footnote An alternative is to clear substvars between builds of different binary packages. [...] loads ttlibbar/tt. A package should depend on the libraries it directly uses, but not the libraries it indirectly uses. Pedantry: what if my package uses the same library both directly and indirectly? but not the libraries it only uses indirectly would avoid that question. There are two types of ABI changes: ones that are backward-compatible and ones that are not. An ABI change is backward-compatible if any binary was linked with the previous version of the shared library will still work correctly with the new version of the shared library. Adding new symbols to the shared library is a backward-compatible change. Removing symbols from the shared library is not. If I remove a symbol that was documented to be private or change the behavior of a function when given invalid arguments, is that a backward-compatible change? If I add change the implementation in such a way that the library becomes so large that some large programs cannot use it any more, is that a backward-incompatible change? [...] filesymbols/file files for a shared library are normally provided by the shared library package, but there are several override paths that are checked first in case that information is wrong or missing. It's not obvious at first how the two clauses of this sentence
