Re: Bug#571776: document symbols
Julien Cristau jcris...@debian.org writes: On Sun, Aug 12, 2012 at 14:25:12 -0700, Russ Allbery wrote: Okay, once more for the win. Here is the current version of the patch, incorporating substantial improvements from Jonathan Nieder and hopefully incorporating all the feedback in subsequent discussion. I'm looking for seconds so that we can finally merge this monster. Presented as a diff since that was the request last time, but the branch has also been pushed to the Policy Git repository, so if you want to review it various other ways, you can start at: http://anonscm.debian.org/gitweb/?p=dbnpolicy/policy.git;a=shortlog;h=refs/heads/bug571776-rra or with a Policy Git checkout and look at the bug571776-rra branch. Seconded. Thank you to everyone for the reviews! I've now merged this patch for the next release. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87r4qzkd0v@windlord.stanford.edu
Re: Bug#571776: document symbols
On Sun, Aug 12, 2012 at 14:25:12 -0700, Russ Allbery wrote: Okay, once more for the win. Here is the current version of the patch, incorporating substantial improvements from Jonathan Nieder and hopefully incorporating all the feedback in subsequent discussion. I'm looking for seconds so that we can finally merge this monster. Presented as a diff since that was the request last time, but the branch has also been pushed to the Policy Git repository, so if you want to review it various other ways, you can start at: http://anonscm.debian.org/gitweb/?p=dbnpolicy/policy.git;a=shortlog;h=refs/heads/bug571776-rra or with a Policy Git checkout and look at the bug571776-rra branch. Seconded. Cheers, Julien signature.asc Description: Digital signature
Re: Bug#571776: document symbols
Hi, On Sun, 12 Aug 2012, Russ Allbery wrote: I'm looking for seconds so that we can finally merge this monster. Presented as a diff since that was the request last time, but the branch has also been pushed to the Policy Git repository, so if you want to review it various other ways, you can start at: Seconded. Thank you very much for this work! Cheers, -- Raphaël Hertzog ◈ Debian Developer Get the Debian Administrator's Handbook: → http://debian-handbook.info/get/ signature.asc Description: Digital signature
Re: Bug#571776: document symbols
Jonathan Nieder jrnie...@gmail.com writes: Jonathan Nieder wrote: I'll reply with an interdiff relative to the last version of the patch. Here it is. And here is the interdiff between your patch and what I currently have, to make it easier for you and anyone who was familiar with your version of the patch to review what I further changed. I'm going to reply to this thread in a moment with the whole current patch, for hopefully the last time, and then we can try to get seconds and (at least) merge this monster. commit 97cb027db4afab774ea4f4ff9e7bef7a6dcbbda0 Author: Russ Allbery r...@debian.org Date: Sun Aug 12 14:14:23 2012 -0700 Further wording changes on top of Jonathan Neider's work Add a footnote explaining what a reasonable program is. Clarify the shlibs versioning text. Other minor textual changes for clarity. diff --git a/policy.sgml b/policy.sgml index 0965b76..fa1c39a 100644 --- a/policy.sgml +++ b/policy.sgml @@ -5978,11 +5978,11 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1) whether new library interfaces are available and can be called). To allow these dependencies to be constructed, shared libraries must provide either a filesymbols/file file or - a fileshlibs/file file, which provides information on the + a fileshlibs/file file. These provide information on the package dependencies required to ensure the presence of interfaces provided by this library. Any package with binaries or libraries linking to a shared library must use these files to determine the required dependencies when it is built. Other packages which use a shared library (for example using ttdlopen()/tt) should compute appropriate dependencies using these files at build time as well. @@ -5990,21 +5990,25 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1) p The two mechanisms differ in the degree of detail that they - provide. A filesymbols/file file documents for each symbol - exported by a library the minimal version of the package any - binary using this symbol will need, which is typically the - version of the package in which the symbol was introduced. - This permits detailed analysis of the symbols used by a + provide. A filesymbols/file file documents, for each symbol + exported by a library, the minimal version of the package any + binary using this symbol will need. This is typically the + version of the package in which the symbol was introduced. This + information permits detailed analysis of the symbols used by a particular package and construction of an accurate dependency, but it requires the package maintainer to track more information - about the shared library. A fileshlibs/file file, in - contrast, only documents the last time the library ABI changed - in any way. It only provides information about the library as a - whole, not individual symbols. When a package is built using a - shared library with only a fileshlibs/file file, the generated - dependency will require a version of the shared library equal to - or newer than the version of the last ABI change. This - generates unnecessarily restrictive dependencies compared + about the shared library. + /p + + p + A fileshlibs/file file, in contrast, only documents the last + time the library ABI changed in any way. It only provides + information about the library as a whole, not individual + symbols. When a package is built using a shared library with + only a fileshlibs/file file, the generated dependency will + require a version of the shared library equal to or newer than + the version of the last ABI change. This generates + unnecessarily restrictive dependencies compared to filesymbols/file files if none of the symbols used by the package have changed. This, in turn, may make upgrades needlessly complex and unnecessarily restrict use of the package @@ -6012,12 +6016,15 @@ Built-Using: grub2 (= 1.99-9), loadlin (= 1.6e-1) /p p - fileshlibsfile files also have a flawed representation of + fileshlibsfile files also only support a limited range of library SONAMEs, making it difficult to use fileshlibs/file files in some unusual corner cases.footnote - libfooN.shlibs says 'libfoo N' instead of the actual SONAME, - so if the SONAME doesn't match one of the two expected - formats (libfoo-N.so or libfoo.so.N) it can't be represented. + A fileshlibs/file file represents an SONAME as a library + name and version number, such as ttlibfoo VERSION/tt, + instead of recording the actual SONAME.
Re: Bug#571776: document symbols
Russ Allbery r...@debian.org writes: commit 97cb027db4afab774ea4f4ff9e7bef7a6dcbbda0 Author: Russ Allbery r...@debian.org Date: Sun Aug 12 14:14:23 2012 -0700 Further wording changes on top of Jonathan Neider's work I fixed the spelling of your name in Git before I pushed. Sorry about that. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fw7ru7j7@windlord.stanford.edu
Re: Bug#571776: document symbols
* Russ Allbery r...@debian.org [120317 19:17]: These two mechanisms differ in the degree of detail that they provide. A filesymbols/file file documents every symbol that is part of the library ABI and, for each, the version of the package in which it was introduced. [...] This is misleading. It's about when the symbol with its current meaning was introduced but could be easily misunderstood to mean the first introduction (and some people might not read the later explanations). How about something like | [...] A filesymbols/file file documents for each symbol | exported by a library the minimal version of the package any | binary using this symbol will need. [...] fileshlibsfile files also have a flawed representation of library SONAMEs, making it difficult to use fileshlibs/file files in some unusual corner cases. Only stylistics: How about not using flawed? Something like Also the way library SONAMEs are represented in fileshlibsfile files makes it difficult to use them in some unusual corner cases.? Bernhard R. Link -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120324152733.ga20...@client.brlink.eu
Re: Bug#571776: document symbols
On Mon, Mar 19, 2012 at 17:26:04 -0500, Jonathan Nieder wrote: These dependencies must be added to the binary package when it is built, since they may change This means packages must not hard-code library dependencies. It also seems like good policy, but I suspect it would render packages such as chromium that use dlopen() and hard-code the corresponding library name in dependencies RC-buggy. They're already broken. What about libraries like glib (assuming one only uses old symbols) that are never supposed to change soname? What about them? [...] To allow these dependencies to be constructed, shared libraries must provide either a filesymbols/file file or a fileshlibs/file file, which provide information on the package dependencies required to ensure the presence of this library. Subject/verb agreement: s/provide/provides/ Clarity: s/this library/interfaces provided by this library/ p These two mechanisms differ in the degree of detail that they provide. A filesymbols/file file documents every symbol that is part of the library ABI and, for each, the version of the package in which it was introduced. Maybe, since minimal-version is not always the version in which the symbol was introduced: and, for each, a minimal version of the library needed to use that symbol, which is typically the version of the package in which it was introduced. [...] fileshlibsfile files also have a flawed representation of library SONAMEs, making it difficult to use fileshlibs/file files in some unusual corner cases. I'm not sure what this passage is referring to. Can you say more? (Maybe in a footnote.) libfooN.shlibs says 'libfoo N' not the actual SONAME, so if the SONAME doesn't match one of the two expected formats (libfoo-N.so or libfoo.so.N) it can't be represented. [...] udebs must also use fileshlibs/file, since the udeb infrastructure does not use filesymbols/file. To avoid confusion it might be worth forbidding symbols files in udebs, or at least symbols files without a corresponding shlibs file accompanying them. That makes no sense. udebs don't have those files, when building an udeb the dependency information is read from the shlibs files of the debs corresponding to the libraries you depend on. [...] If you have multiple binary packages, you will need to call prgndpkg-shlibdeps/prgn on each one which contains compiled libraries or binaries, using the tt-T/tt option to the ttdpkg/tt utilities to specify a different filesubstvars/file file for each binary package.footnote An alternative is to clear substvars between builds of different binary packages. Who does that? I don't think it's necessary to document all the twisted ways to make things not break. [...] loads ttlibbar/tt. A package should depend on the libraries it directly uses, but not the libraries it indirectly uses. Pedantry: what if my package uses the same library both directly and indirectly? but not the libraries it only uses indirectly would avoid that question. There are two types of ABI changes: ones that are backward-compatible and ones that are not. An ABI change is backward-compatible if any binary was linked with the previous version of the shared library will still work correctly with the new version of the shared library. Adding new symbols to the shared library is a backward-compatible change. Removing symbols from the shared library is not. If I remove a symbol that was documented to be private or change the behavior of a function when given invalid arguments, is that a backward-compatible change? If I add change the implementation in such a way that the library becomes so large that some large programs cannot use it any more, is that a backward-incompatible change? I'm not sure policy should go into such details. And anyway, that's answered by the previous sentence (an incompatible change is one that breaks reverse deps). The last two are simple examples. Cheers, Julien signature.asc Description: Digital signature
Re: Bug#571776: document symbols
Le Sat, Mar 17, 2012 at 11:17:29AM -0700, Russ Allbery a écrit : Here is a new proposed patch that incorporates the feedback to date with some other, substantial changes. Due to the reformatting, the diff is even longer and is now really just the complete removal of the current shlibs section followed by the addition of the new section. I'm therefore including here the complete SGML source of that section not in diff format, followed by the diff of everything *outside* of that section. I think this will be easier to review. Hi Russ, after reading your chages, I have not found potential problems or errors. Have a nice day, -- Charles -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120320022804.gc10...@falafel.plessy.net
Re: Bug#571776: document symbols
Here is a new proposed patch that incorporates the feedback to date with some other, substantial changes. It's apparent to me from hands-on experimentation with C++ libraries that, at least at the moment, shlibs is likely to have an ongoing existence in the archive. Accordingly, some of the layout decisions I made to keep the shlibs section separate in a way that would let it potentially be dropped later weren't a good idea. This version therefore substantially reorganizes and somewhat expands the documentation to: * Describe the difference between symbols and shlibs in more detail and provide advice on how to choose between them. * Document calling dpkg-shlibdeps in a separate section independent of either symbols or shlibs, since the details of how to use it doesn't vary based on the dependency system in play. * Provide separate documentation for how to handle ABI changes and determine dependency versions independent of either system, since the same process should be used either way. The only difference between the systems is whether the version information is per-symbol or per-library. * Move the symbols and shlibs documentation to subsections of the general section on shared library dependency management. * Move the information on how to determine the soversion out of the shlibs section and to the runtime shared library section so that it is next to the discussion of shared library package names. The shlibs documentation now just references it. Due to the reformatting, the diff is even longer and is now really just the complete removal of the current shlibs section followed by the addition of the new section. I'm therefore including here the complete SGML source of that section not in diff format, followed by the diff of everything *outside* of that section. I think this will be easier to review. sect id=sharedlibs-depends headingDependencies between the library and other packages/heading p If a package contains a binary or library which links to a shared library, we must ensure that, when the package is installed on the system, all of the libraries needed are also installed. These dependencies must be added to the binary package when it is built, since they may change based on which version of a shared library the binary or library was linked with even if there are no changes to the source of the binary (for example, symbol versions change, macros become functions or vice versa, or the binary package may determine at compile-time whether new library interfaces are available and can be called). To allow these dependencies to be constructed, shared libraries must provide either a filesymbols/file file or a fileshlibs/file file, which provide information on the package dependencies required to ensure the presence of this library. Any package which uses a shared library must use these files to determine the required dependencies when it is built. /p p These two mechanisms differ in the degree of detail that they provide. A filesymbols/file file documents every symbol that is part of the library ABI and, for each, the version of the package in which it was introduced. This permits detailed analysis of the symbols used by a particular package and construction of an accurate dependency, but it requires the package maintainer to track more information about the shared library. A fileshlibs/file file, in contrast, only documents the last time the library ABI changed in any way. It only provides information about the library as a whole, not individual symbols. When a package is built using a shared library with only a fileshlibs/file file, the generated dependency will require a version of the shared library equal to or newer than the version of the last ABI change. This generates unnecessarily restrictive dependencies compared to filesymbols/file files if none of the symbols used by the package have changed. This, in turn, may make upgrades needlessly complex and unnecessarily restrict use of the package on systems with older versions of the shared libraries. /p p fileshlibsfile files also have a flawed representation of library SONAMEs, making it difficult to use fileshlibs/file files in some unusual corner cases. /p p filesymbols/file files are therefore recommended for most shared library packages since they provide more accurate dependencies. For most C libraries, the additional detail required by filesymbols/file files is not too difficult to maintain.
Re: Bug#571776: document symbols
Raphael Hertzog hert...@debian.org writes: On Mon, 02 Jan 2012, Russ Allbery wrote: [...] p fileshlibs/file files were the original mechanism for handling library dependencies. They are documented in ref id=sharedlibs-shlibdeps. filesymbols/file files, documented in this section, are recommended for most packages, since they provide dependency information for each exported symbol and therefore generate more accurate dependencies for binaries that do not use symbols from newer versions of the shared library. However, fileshlibs/file files must be used for udebs. Packages which provide a filesymbols/file file are not required to provide a fileshlibs/file file. /p In practice I don't think that we have any package providing a symbols files without a shlibs file. Yes, but there was some discussion in the Policy bug asking why shlibs files were required when they're not used if a symbols file is present, and while I originally argued that keeping them both made sense, I came around to that position after reviewing the bug discussion. It doesn't hurt anything, but there doesn't really seem to be any point in providing shlibs if symbols is present. If your source package builds only a single binary package that contains only compiled binaries and libraries (but no scripts) and is not multiarch, you can use a command such as: example compact=compact dpkg-shlibdeps debian/tmp/usr/bin/* debian/tmp/usr/sbin/* \ debian/tmp/usr/lib/* /example but normally finding all of the binaries is more complex. I would drop the part above (up to the 4 dashes that I added). This example will only mislead people IMO because debian/tmp/ is far from being the norm. debian/pkg is much more common for single binary packages. And as you said, it doesn't cover all cases. Yeah, good point. This was retained from the old shlibs documentation, but at this point I don't think anyone really looks here to see how to do this. Dropped. binary package.footnote Again, prgndh_shlibdeps/prgn and prgndh_gencontrol/prgn will handle all of this for you if you're using ttdebhelper/tt. /footnote I would indicate in the footnote that dh_shlibdeps/dh_gencontrol use an alternate substvars file by default (debian/pkg.substvars). I now have: binary package.footnote Again, prgndh_shlibdeps/prgn and prgndh_gencontrol/prgn will handle all of this for you if you're using ttdebhelper/tt, including generating separate filesubstvars/file files for each binary package and calling prgndpkg-gencontrol/prgn with the appropriate flags. /footnote sect1 id=symbols headingThe filesymbols/file File Format/heading [...] For our example, the ttzlib1g/tt filesymbols/file file would contain: example compact=compact * Build-Depends-Package: zlib1g-dev /example (Don't forget the leading space.) What leading space are you referring to ? I now have: (Don't forget the space before the tt*/tt so that it will be parsed as part of the entry for that library.) Due to the way that the formatting of Policy works, it's very hard to tell that there's a space there, and unlike with symbols where the indentation is fairly obvious, it's not completely obvious that it's required. [...] sect1 id=shlibs-paths headingThe fileshlibs/file files present on the system/heading [...] item pfileDEBIAN/shlibs/file files in the build directory/p p When packages are being built, any filedebian/shlibs/file files are copied into the control information file area of the temporary build directory and given the name fileshlibs/file. These files give details of any shared libraries included in the same package. /p /item debian/shlibs is (no longer) a standard file... debhelper doesn't install such files, it generates shlibs files on the fly and I don't believe that any modern package still has debian/shlibs. So I would rewrite this paragraph to just say that DEBIAN/shlibs is the shlibs file produced by the current package build. I now have: p These files are generated as part of the package build process and staged for inclusion as control files in the binary packages being built. They provide details of any shared libraries included in the same package. /p I would shorten the explanation here and drop the example of how to install a
Re: Bug#571776: document symbols
Russ Allbery r...@debian.org writes: I tried sending a unified diff, but the new sections are largely unreadable since they're intermixed with the old sections being removed. Hence, for review purposes, here are the symbols and shlibs sections in their entirety, followed by a diff for the changes elsewhere in Policy. You can also refer to branch bug571776-rra in the Policy repository. Here is an updated diff with the changes from Raphael's review. git diff --patience actually generates a decent diff (I learn something new every day!), so I didn't have to separate out the new sections. diff --git a/policy.sgml b/policy.sgml index 23c1913..b57dd76 100644 --- a/policy.sgml +++ b/policy.sgml @@ -840,10 +840,11 @@ Among those files are the package maintainer scripts and filecontrol/file, the qref id=binarycontrolfilesbinary package control file/qref that contains the control fields for - the package. Other control information files - include qref id=sharedlibs-shlibdepsthe fileshlibs/file - file/qref used to store shared library dependency information - and the fileconffiles/file file that lists the package's + the package. Other control information files include + the qref id=sharedlibs-symbolsfilesymbols/file file/qref + or qref id=sharedlibs-shlibdepsfileshlibs/file file/qref + used to store shared library dependency information and + the fileconffiles/file file that lists the package's configuration files (described in ref id=config-files). /p @@ -5521,9 +5522,9 @@ Replaces: mail-transport-agent linked against the old shared library. Correct versioning of dependencies on the newer shared library by binaries that use the new interfaces is handled via - the qref id=sharedlibs-shlibdepsttshlibs/tt - system/qref or via symbols files (see - manref name=deb-symbols section=5). + the qref id=sharedlibs-symbolsttsymbols/tt system/qref + or the qref id=sharedlibs-shlibdepsttshlibs/tt + system/qref. /p p @@ -5792,361 +5793,789 @@ Replaces: mail-transport-agent /p /sect + sect id=sharedlibs-symbols + headingDependencies between the library and other packages - + the ttsymbols/tt system/heading + + p + If a package contains a binary or library which links to a + shared library, we must ensure that, when the package is + installed on the system, all of the libraries needed are also + installed. These dependencies must be added to the binary + package when it is built, since they may change based on which + version of a shared library the binary or library was linked + with. To allow these dependencies to be constructed, shared + libraries must provide either a filesymbols/file file or + a fileshlibs/file file, which provide information on the + package dependencies required to ensure the presence of this + library. Any package which uses a shared library must use these + files to determine the required dependencies when it is built. + /p + + p + fileshlibs/file files were the original mechanism for + handling library dependencies. They are documented + in ref id=sharedlibs-shlibdeps. filesymbols/file files, + documented in this section, are recommended for most packages, + since they provide dependency information for each exported + symbol and therefore generate more accurate dependencies for + binaries that do not use symbols from newer versions of the + shared library. However, fileshlibs/file files must be used + for udebs. Packages which provide a filesymbols/file file + are not required to provide a fileshlibs/file file. + /p + + p + When a package that contains any shared libraries or compiled + binaries is built, it must run prgndpkg-shlibdeps/prgn on + each shared library and compiled binary to determine the + libraries used and hence the dependencies needed by the + package.footnote + prgndpkg-shlibdeps/prgn will use a program + like prgnobjdump/prgn or prgnreadelf/prgn to find the + libraries and the symbols in those libraries directly needed + by the binaries or shared libraries in the package. + /footnote + /p + + p + We say that a binary ttfoo/tt emdirectly/em uses a + library ttlibbar/tt if it is explicitly linked with that + library (that is, the library is listed in the + ELF ttNEEDED/tt attribute, caused by adding tt-lbar/tt + to the link line when the binary is created). Other libraries + that are needed by ttlibbar/tt are + linked emindirectly/em to ttfoo/tt, and the dynamic + linker will load
Re: Bug#571776: document symbols
On Fri, 13 Jan 2012, Russ Allbery wrote: For our example, the ttzlib1g/tt filesymbols/file file would contain: example compact=compact * Build-Depends-Package: zlib1g-dev /example (Don't forget the leading space.) What leading space are you referring to ? I now have: (Don't forget the space before the tt*/tt so that it will be parsed as part of the entry for that library.) Due to the way that the formatting of Policy works, it's very hard to tell that there's a space there, and unlike with symbols where the indentation is fairly obvious, it's not completely obvious that it's required. There is no leading space before the *. Just like | it must be on the first column to differentiate with symbol definitions which do have a leading space on their lines. Thanks for the review! All the other corrections you made are fine. Thanks! Cheers, -- Raphaël Hertzog ◈ Debian Developer Pre-order a copy of the Debian Administrator's Handbook and help liberate it: http://debian-handbook.info/liberation/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120113193823.gd14...@rivendell.home.ouaza.com
Re: Bug#571776: document symbols
On Fri, 13 Jan 2012, Russ Allbery wrote: + p + example +varlibrary-soname/var varmain-dependency-template/var +[ | varalternative-dependency-template/var ] +[ ... ] +[ * varfield-name/var: varfield-value/var ] +[ ... ] + varsymbol/var varminimal-version/var[ varid-of-dependency-template/var ] + /example I think this description adapted from the deb-symbols(5) manual page mislead you into thinking that there were leading spaces before | or * when in fact there are none. I have updated the manual page to make it look like this now: library-soname main-dependency-template [| alternative-dependency-template] [...] [* field-name: field-value] [...] symbol minimal-version [id-of-dependency-template] + example +libGL.so.1 libgl1 + | libgl1-mesa-glx #MINVER# Drop the leading space on that last line. + publicGlSymbol@Base 6.3-1 + [...] + implementationSpecificSymbol@Base 6.5.2-7 1 + [...] + For our example, the ttzlib1g/tt filesymbols/file file + would contain: + example compact=compact + * Build-Depends-Package: zlib1g-dev And here too. + /example + (Don't forget the space before the tt*/tt so that it will + be parsed as part of the entry for that library.) And that sentence is then useless (or needs to be reworded). Cheers, -- Raphaël Hertzog ◈ Debian Developer Pre-order a copy of the Debian Administrator's Handbook and help liberate it: http://debian-handbook.info/liberation/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120113195534.ge14...@rivendell.home.ouaza.com
Re: Bug#571776: document symbols
Raphael Hertzog hert...@debian.org writes: There is no leading space before the *. Just like | it must be on the first column to differentiate with symbol definitions which do have a leading space on their lines. Oh, then deb-symbols(5) is wrong for both * and |... oh, I see, I was misreading how the syntax definition worked and the spaces around [] weren't literal. Aie. I'll fix. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87vcof8h7x@windlord.stanford.edu
Re: Bug#571776: document symbols
Raphael Hertzog hert...@debian.org writes: I think this description adapted from the deb-symbols(5) manual page mislead you into thinking that there were leading spaces before | or * when in fact there are none. I have updated the manual page to make it look like this now: library-soname main-dependency-template [| alternative-dependency-template] [...] [* field-name: field-value] [...] symbol minimal-version [id-of-dependency-template] Thanks, I've now fixed my draft text as well. + example +libGL.so.1 libgl1 + | libgl1-mesa-glx #MINVER# Drop the leading space on that last line. Done. +For our example, the ttzlib1g/tt filesymbols/file file +would contain: +example compact=compact + * Build-Depends-Package: zlib1g-dev And here too. Done. +/example +(Don't forget the space before the tt*/tt so that it will +be parsed as part of the entry for that library.) And that sentence is then useless (or needs to be reworded). Dropped. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87ehv35h9k@windlord.stanford.edu
Re: Bug#571776: document symbols
Dear Russ and Raphaël,
here are some comments about the current patch. I agree with the other changes
made subsequently in that thread.
+ If a package contains a binary or library which links to a
+ shared library, we must ensure that, when the package is
+ installed on the system, all of the libraries needed are also
+ installed. These dependencies must be added to the binary
+ package when it is built, since they may change based on which
+ version of a shared library the binary or library was linked
+ with.
I have a difficulty with that sentence : isn't the goal of the symbols system
to actually avoid that a dependancy changes when a package is built on a newer
version of a library ? What rather changes is that the dependancy is modified
when a new version of the package makes new calls to the library, which may
have been implemented only in newer versions of the library.
If the misunderstanding is on my side, please consider it a hint that the
explanations can be expanded :) [PS: after reading the whole patch, maybe
this has something to do with minimal-version ?]
+ p
+ When a package that contains any shared libraries or compiled
+ binaries is built, it must run prgndpkg-shlibdeps/prgn on
+ each shared library and compiled binary to determine the
+ libraries used and hence the dependencies needed by the
+ package.footnote
+ prgndpkg-shlibdeps/prgn will use a program
+ like prgnobjdump/prgn or prgnreadelf/prgn to find the
+ libraries and the symbols in those libraries directly needed
+ by the binaries or shared libraries in the package.
+ /footnote
+ /p
(I understand that this is carried over from 3.9.2, but…) Just running
dpkg-shlibdeps is not enough, as this program only populates debian/substvars.
This is actually documented later in this chapter, which makes this paragraph
redundant. If its purpose is to lock the use of dpkg-shlibdeps and prevent
parallel implementations, it could be slimmed as follows.
p
Packages that contain any shared libraries or compiled binaries must
determine their dependencies by using the prgndpkg-shlibdeps/prgn
program and the qref id=substvarstt${shlibs:Depends}/tt/qref
substvar. See ref id=dpkg-shlibdeps for detailed instructions.
/p
+ p
+ During the package build, if the package itself contains
+ shared libraries with filesymbols/file files, they
+ will be generated in these staging directories
+ by prgndpkg-gensymbols/prgn. filesymbols/file
With a linear reading, it is the first time dpkg-gensymbols appears in the
Policy. How about linking to the symbols section, that gives a few more
words and refers to the manual page ?
+ p
+ If your package contains any compiled binaries or shared
+ libraries, put a call to prgndpkg-shlibdeps/prgn into
+ your filedebian/rules/file file in the source package.
The footnote removed above could be added here:
footnote
prgndpkg-shlibdeps/prgn will use a program
like prgnobjdump/prgn or prgnreadelf/prgn to find the
libraries and the symbols in those libraries directly needed
by the binaries or shared libraries in the package.
/footnote
+ p
+ This command puts the dependency information into
+ the filedebian/substvars/file file, which is then used
+ by prgndpkg-gencontrol/prgn.
I propose to mitigate this by adding “by default”, because when using
debhelper, which is very frequent, debian/substvars is actually not used. (It
took me hours to understand why I could not populate new substvars in
debian/control by simply adding their values in debian/substvars in my
debhelper-managed packages).
- sect1 id=pkg-dpkg-shlibdeps
- heading
- prgndpkg-shlibdeps/prgn - calculates shared library
- dependencies
- /heading
I propose to keep a placeholder so that the next subsections are not
renumbered. Once all these sections are trimmed, we could consider removing
the appendix altogether.
Have a nice day,
--
Charles Plessy
Tsurumi, Kanagawa, Japan
--
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120114044940.gd9...@merveille.plessy.net
Re: Bug#571776: document symbols
Charles Plessy ple...@debian.org writes: here are some comments about the current patch. I agree with the other changes made subsequently in that thread. + If a package contains a binary or library which links to a + shared library, we must ensure that, when the package is + installed on the system, all of the libraries needed are also + installed. These dependencies must be added to the binary + package when it is built, since they may change based on which + version of a shared library the binary or library was linked + with. I have a difficulty with that sentence : isn't the goal of the symbols system to actually avoid that a dependancy changes when a package is built on a newer version of a library ? What rather changes is that the dependancy is modified when a new version of the package makes new calls to the library, which may have been implemented only in newer versions of the library. That's true, but somewhat deceptive because this can happen with no changes to the client code. In some cases, changes only to the library with no changes to the client code can require tighter dependencies when the client is built against the newer library. The obvious example is replacing a macro in a header with an actual function entry point. In this case, the client code will build and work fine with either the older or the newer version of the library, but when built against the newer library, it won't work with the older library. That's what this is trying to capture here. If the exact same code is rebuilt against a newer version of a library, it may have to depend on that newer version of the library. If the misunderstanding is on my side, please consider it a hint that the explanations can be expanded :) [PS: after reading the whole patch, maybe this has something to do with minimal-version ?] I'm not really sure what else to say, but I'm happy to add more if someone can take a shot at letting me know what's needed But I was intentionally trying to keep this vague and not get into enumerating the reasons why dependencies may need to be tightened. That's really a topic for a whole separate guide on how shared libraries should be packaged. The rest of your changes look good, although I haven't had a chance to incorporate them yet. I'll take a look, hopefully soon, although I may not have time this weekend (which is a long holiday weekend in the US). -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87boq6512a@windlord.stanford.edu
Re: Bug#571776: document symbols
Russ Allbery r...@debian.org (13/01/2012):
Yes, but there was some discussion in the Policy bug asking why shlibs
files were required when they're not used if a symbols file is present,
and while I originally argued that keeping them both made sense, I came
around to that position after reviewing the bug discussion. It doesn't
hurt anything, but there doesn't really seem to be any point in providing
shlibs if symbols is present.
Last I checked, one needs an shlibs file to get proper dependencies into
udebs, which explains why libx* packages still use some -V flag in their
dh_makeshlibs call.
Demo: xorg-server/experimental build-depends on libpciaccess (= version-
in-unstable), and the symbols added in this version are really needed (see
below).
Rebuilding libpciaccess to drop the -V flag, and rebuilding xorg-server
against it generates this change in xserver-xorg-core-udeb's dependencies:
| Depends: […], [-libpciaccess0-udeb (= 0.12.902),-] {+libpciaccess0-udeb,+}
[…]
→ Fail.
“Those new symbols really are needed”:
| $ grep 902 ../lib/libpciaccess.git/debian/*symbols
| pci_device_map_legacy@Base 0.12.902
| pci_device_unmap_legacy@Base 0.12.902
| $ for i in $(find debian/xserver-xorg-core-udeb -type f); do if nm -D $i
21|grep -qs pci_device_'.*'map_legacy; then echo $i; fi; done
| debian/xserver-xorg-core-udeb/usr/lib/xorg/modules/libvgahw.so
| debian/xserver-xorg-core-udeb/usr/lib/xorg/modules/libint10.so
Diff in the libpciaccess package to exhibit that:
| - dh_makeshlibs -V'libpciaccess0 (= 0.12.902)'
--add-udeb=$(PACKAGE)-udeb -- -c4
| + dh_makeshlibs --add-udeb=$(PACKAGE)-udeb -- -c4
Mraw,
KiBi.
signature.asc
Description: Digital signature
Re: Bug#571776: document symbols
Cyril Brulebois k...@debian.org writes: Russ Allbery r...@debian.org (13/01/2012): Yes, but there was some discussion in the Policy bug asking why shlibs files were required when they're not used if a symbols file is present, and while I originally argued that keeping them both made sense, I came around to that position after reviewing the bug discussion. It doesn't hurt anything, but there doesn't really seem to be any point in providing shlibs if symbols is present. Last I checked, one needs an shlibs file to get proper dependencies into udebs, which explains why libx* packages still use some -V flag in their dh_makeshlibs call. Yes, udebs are called out as an exception and it's still stated that shlibs must be used for udebs. They still are consistent with the property above, though, since symbols aren't supported for udebs, or at least that's what I understood. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87obu63ey2@windlord.stanford.edu
Re: Bug#571776: document symbols
Hi,
On Mon, 02 Jan 2012, Russ Allbery wrote:
p
If a package contains a binary or library which links to a
shared library, we must ensure that, when the package is
installed on the system, all of the libraries needed are also
installed. These dependencies must be added to the binary
package when it is built, since they may change based on which
version of a shared library the binary or library was linnked
s/linnked/linked/
[...]
p
fileshlibs/file files were the original mechanism for
handling library dependencies. They are documented
in ref id=sharedlibs-shlibdeps. filesymbols/file files,
documented in this section, are recommended for most packages,
since they provide dependency information for each exported
symbol and therefore generate more accurate dependencies for
binaries that do not use symbols from newer versions of the
shared library. However, fileshlibs/file files must be used
for udebs. Packages which provide a filesymbols/file file
are not required to provide a fileshlibs/file file.
/p
In practice I don't think that we have any package providing a symbols
files without a shlibs file.
[...]
sect1 id=dpkg-shlibdeps
headingHow to use prgndpkg-shlibdeps/prgn and the
ttsymbols/tt files/heading
p
If your package contains any compiled binaries or shared
libraries, put a call to prgndpkg-shlibdeps/prgn into
your filedebian/rules/file file in the source package.
List all of the compiled binaries, libraries, or loadable
modules in your package.
If your source package builds only a
single binary package that contains only compiled binaries and
libraries (but no scripts) and is not multiarch, you can use a
command such as:
example compact=compact
dpkg-shlibdeps debian/tmp/usr/bin/* debian/tmp/usr/sbin/* \
debian/tmp/usr/lib/*
/example
but normally finding all of the binaries is more
complex.
I would drop the part above (up to the 4 dashes that I added). This
example will only mislead people IMO because debian/tmp/ is far from being
the norm. debian/pkg is much more common for single binary packages. And
as you said, it doesn't cover all cases.
footnote
The easiest way to do this is to use a package helper
framework such as ttdebhelper/tt. If you are
using ttdebhelper/tt, the prgndh_shlibdeps/prgn
program will do this work for you. It will also correctly
handle multi-binary packages.
/footnote
/p
p
This command puts the dependency information into
the filedebian/substvars/file file, which is then used
by prgndpkg-gencontrol/prgn. You will need to place
a tt${shlibs:Depends}/tt variable in the ttDepends/tt
field in the control file of every binary package built by
this source package that contains compiled binaries,
libraries, or loadable modules. If you have multiple binary
packages, you will need to call prgndpkg-shlibdeps/prgn on
each one which contains compiled libraries or binaries, using
the tt-T/tt option to the ttdpkg/tt utilities to
specify a different filesubstvars/file file for each
binary package.footnote
Again, prgndh_shlibdeps/prgn
and prgndh_gencontrol/prgn will handle all of this for
you if you're using ttdebhelper/tt.
/footnote
I would indicate in the footnote that dh_shlibdeps/dh_gencontrol use an
alternate substvars file by default (debian/pkg.substvars).
sect1 id=symbols
headingThe filesymbols/file File Format/heading
[...]
For our example, the ttzlib1g/tt filesymbols/file file
would contain:
example compact=compact
* Build-Depends-Package: zlib1g-dev
/example
(Don't forget the leading space.)
What leading space are you referring to ?
[...]
sect1 id=shlibs-paths
headingThe fileshlibs/file files present on the
system/heading
[...]
item
pfileDEBIAN/shlibs/file files in the build
directory/p
p
When packages are being built,
any filedebian/shlibs/file files are copied into the
control information file area of the temporary build
directory and given the name fileshlibs/file. These
files give details of any shared libraries included in
the same package.
/p
/item
debian/shlibs is (no longer) a standard file... debhelper doesn't install
such files, it generates shlibs
Re: Bug#571776: document symbols
* Russ Allbery r...@debian.org, 2012-01-02, 13:51:
p
A common example of when a change
to varminimal-version/var is required is a function that
takes an enum or struct argument that controls what the
function does. For example:
example
enum library_op { OP_FOO, OP_BAR };
int library_do_operation(enum library_op);
/example
If a new operation, ttOP_BAZ/tt, is added,
the varminimal-version/var
of ttlibrary_do_operation/tt must be increased to the
version at which ttOP_BAZ/tt was introduced. Otherwise, a
binary built against the new version of the library (having
detected at compile-time that the library
supports ttOP_BAZ/tt) may be installed with a shared
library that doesn't support ttOP_BAZ/tt and will fail at
runtime when it tries to pass ttOP_BAZ/tt into this
function.
/p
Hmm. It was my understanding that one of the purposes of
Build-Depends-Package is to avoid bumping symbol versions in situations
like this.
--
Jakub Wilk
--
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120103232901.ga9...@jwilk.net
Re: Bug#571776: document symbols
Hello folks, I took some time today and wrote a first draft of a new section of Policy documenting symbols files, and the revisions to shlibs for their interaction. Please review. There's quite a lot of material here, including details from dpkg-shlibdeps, dpkg-gensymbols, and deb-symbols documentation as well as additional requirements and recommendations around how to maintain the minimal-version field. I tried sending a unified diff, but the new sections are largely unreadable since they're intermixed with the old sections being removed. Hence, for review purposes, here are the symbols and shlibs sections in their entirety, followed by a diff for the changes elsewhere in Policy. You can also refer to branch bug571776-rra in the Policy repository. sect id=sharedlibs-symbols headingDependencies between the library and other packages - the ttsymbols/tt system/heading p If a package contains a binary or library which links to a shared library, we must ensure that, when the package is installed on the system, all of the libraries needed are also installed. These dependencies must be added to the binary package when it is built, since they may change based on which version of a shared library the binary or library was linnked with. To allow these dependencies to be constructed, shared libraries must provide either a filesymbols/file file or a fileshlibs/file file, which provide information on the package dependencies required to ensure the presence of this library. Any package which uses a shared library must use these files to determine the required dependencies when it is built. /p p fileshlibs/file files were the original mechanism for handling library dependencies. They are documented in ref id=sharedlibs-shlibdeps. filesymbols/file files, documented in this section, are recommended for most packages, since they provide dependency information for each exported symbol and therefore generate more accurate dependencies for binaries that do not use symbols from newer versions of the shared library. However, fileshlibs/file files must be used for udebs. Packages which provide a filesymbols/file file are not required to provide a fileshlibs/file file. /p p When a package that contains any shared libraries or compiled binaries is built, it must run prgndpkg-shlibdeps/prgn on each shared library and compiled binary to determine the libraries used and hence the dependencies needed by the package.footnote prgndpkg-shlibdeps/prgn will use a program like prgnobjdump/prgn or prgnreadelf/prgn to find the libraries and the symbols in those libraries directly needed by the binaries or shared libraries in the package. /footnote /p p We say that a binary ttfoo/tt emdirectly/em uses a library ttlibbar/tt if it is explicitly linked with that library (that is, the library is listed in the ELF ttNEEDED/tt attribute, caused by adding tt-lbar/tt to the link line when the binary is created). Other libraries that are needed by ttlibbar/tt are linked emindirectly/em to ttfoo/tt, and the dynamic linker will load them automatically when it loads ttlibbar/tt. A package should depend on the libraries it directly uses, but not the libraries it indirectly uses. The dependencies for those libraries will automatically pull in the other libraries. prgndpkg-shlibdeps/prgn will handle this logic automatically, but package maintainers need to be aware of this distinction between directly and indirectly using a library if they have to override its results for some reason. footnote A good example of where this helps is the following. We could update ttlibimlib/tt with a new version that supports a new graphics format called dgf (but retaining the same major version number) and depends on ttlibdgf/tt. If we used prgnldd/prgn to add dependencies for every library directly or indirectly linked with a binary, every package that uses ttlibimlib/tt would need to be recompiled so it would also depend on ttlibdgf/tt or it wouldn't run due to missing symbols. Since dependencies are only added based on ELF ttNEEDED/tt attribute, packages using ttlibimlib/tt can rely on ttlibimlib/tt itself having the dependency on ttlibdgf/tt and so they would not need rebuilding. /footnote /p p In the
