Re: Storing package metadata in ELF objects
On Wed, 2021-05-19 at 16:58 +0200, Mark Wielaard wrote: > Hi Guillem, > > On Wed, 2021-05-19 at 02:19 +0200, Guillem Jover wrote: > > So this is where I guess I'm missing something. To be able to make > > sense of the coredumps there are two things that might end up being > > relevant, backtraces and source code. systemd-coredump might already > > emit a backtrace, and depending on the information provided it might > > be more or less useful. If one needs the actual debug symbols there's > > already some external querying/fetching required, and if distribution > > specific source code is required because many distributions patch > > upstream source, then even more querying/fetching will be required. > > > > Which is why I'm not seeing why this standalone and isolated metadata > > would be of much help by itself. As in, the way I see it, either the > > information from systemd (w/o the extra metadata) is sufficient to > > track down bugs, or that querying/fetching would be needed anyway, at > > which point the metadata can be inferred too then? > > Because without that metadata you cannot easily figure out where/how to > get the files needed to properly track down the bugs. But using that > metadata you can figure out where the debuginfod server is that can > provide all that information for the binaries/core files (or a distro > specific method if the distributor doesn't have a debuginfod server > yet). Yes, that is a good use case. Another use case is that the person (or bot) who first looks at the journal with the crash notification and the metadata might not necessarily be the one that does the full in-depth debugging. First level support will look at things and go "oh this is from package X at version Y, therefore team Z needs to chime in". Or it can see that it's version A.B.C, which is listed as known buggy, and ignore the issue. Or a myriad of other combinations. Then there are automated systems, parsing logs and creating tickets. Having the structured metadata available in the journal in the crash message means much more complex and useful querying and ticket creation systems can be set up. You _cannot_ just go and query random remote internet servers from these systems, they need to be network-isolated, as logs can have confidential data from customers. The main point is, there are many many things that happen between a crash and the team owning the component looking up debugging symbols and loading core files. This feature is very very useful for a whole lot of those things. > > Oh, I was thinking about those mixed environments, full chroots or > > stripped down containers, from different vendors, but affecting Debian > > installations. What I'm also probably missing here is how does the > > metadata help for a third-party that is not expected to track/keep/upload > > debug symbols nor source packages into some repository, because otherwise > > I'm not seeing how they'd make use of the cores (if they are insufficient > > by themselves) to debug issues? I guess my question back would be what > > would they do with the metadata if they do not have the debug symbols > > nor the sources readily available? Also assuming of course they exercise > > good practices such as never reusing the same package-version-arch tuple > > for different builds, and similar. :) > > Different builds will have different build-ids, so they can be kept > apart. But even if without the distributor having added debuginfod > meta-data and providing a server to fetch the extra symbols, debuginfo, > sources, the extra meta-data is useful to administrators. Just seeing > that crashes are associated with specific distro/version/packages helps > narrow down instabilities. Precisely - again this is a very much real use case. In my case, the engineers doing support and looking at the logs (and only at the logs - no access to the running systems is permitted) _want_ this, because it helps them. Being able to know at a glance to the journal exactly what is borken, with version info, is extremely valuable to them. Yes the version info might not be precise for a minority of use cases that override the binary version with something different than the source version, but that's fine as it's far and few, mostly affects metapackages, and even then it can be documented clearly that the reference is to the source version, not the binary one. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Re: Storing package metadata in ELF objects
Hi Guillem, On Wed, 2021-05-19 at 02:19 +0200, Guillem Jover wrote: > So this is where I guess I'm missing something. To be able to make > sense of the coredumps there are two things that might end up being > relevant, backtraces and source code. systemd-coredump might already > emit a backtrace, and depending on the information provided it might > be more or less useful. If one needs the actual debug symbols there's > already some external querying/fetching required, and if distribution > specific source code is required because many distributions patch > upstream source, then even more querying/fetching will be required. > > Which is why I'm not seeing why this standalone and isolated metadata > would be of much help by itself. As in, the way I see it, either the > information from systemd (w/o the extra metadata) is sufficient to > track down bugs, or that querying/fetching would be needed anyway, at > which point the metadata can be inferred too then? Because without that metadata you cannot easily figure out where/how to get the files needed to properly track down the bugs. But using that metadata you can figure out where the debuginfod server is that can provide all that information for the binaries/core files (or a distro specific method if the distributor doesn't have a debuginfod server yet). > Oh, I was thinking about those mixed environments, full chroots or > stripped down containers, from different vendors, but affecting Debian > installations. What I'm also probably missing here is how does the > metadata help for a third-party that is not expected to track/keep/upload > debug symbols nor source packages into some repository, because otherwise > I'm not seeing how they'd make use of the cores (if they are insufficient > by themselves) to debug issues? I guess my question back would be what > would they do with the metadata if they do not have the debug symbols > nor the sources readily available? Also assuming of course they exercise > good practices such as never reusing the same package-version-arch tuple > for different builds, and similar. :) Different builds will have different build-ids, so they can be kept apart. But even if without the distributor having added debuginfod meta-data and providing a server to fetch the extra symbols, debuginfo, sources, the extra meta-data is useful to administrators. Just seeing that crashes are associated with specific distro/version/packages helps narrow down instabilities. Cheers, Mark
Re: Storing package metadata in ELF objects
On Fri, 2021-05-14 at 14:41:32 +0100, Luca Boccassi wrote: > On Fri, 2021-05-14 at 12:41 +0200, Guillem Jover wrote: > > On Sat, 2021-04-10 at 13:38:31 +0100, Luca Boccassi wrote: > > > On Sat, 2021-04-10 at 13:29 +0100, Luca Boccassi wrote: > > > > After an initial discussion [0], recently we have been working on a new > > > > specification [0] to encode rich package-level metadata inside ELF > > > > objects, so that it can be included automatically in generated coredump > > > > files. The prototype to parse this in systemd-coredump and store the > > > > information in systemd-journal is ready for testing and merged > > > > upstream. We are now seeking further comments/opinions/suggestions, as > > > > we have a few months before the next release and thus there's plenty of > > > > time to make incompatible changes to the format and implementation, if > > > > required. > > > > I've skimmed over the discussion at [0], and while having this data > > seems like it might be "nice", I've to agree with the comments there > > voicing that there does not really seem to be an actual need and the > > overhead and additional work do not seem worth it, TBH, at least > > in the Debian context. > Hi Guillem, thanks for having a look, much appreciated! > > Just to clarify, the need is there - this is not an experimental > exercise, but it is borne out of an actual need&requirement, and it is > undergoing testing right now before deployment in a large scale > production infrastructure. > Not _everybody_ will need it, and not everywhere - that's absolutely > fair, and discussions on whether the ovearhead is worth it for > something that is not universally needed, but only in certain use > cases, is perfectly reasonable and welcome. I know Zbigniew is going to > try and get some raw numbers on the kind of overhead we are talking > about, that will hopefully help frame the discussion with more > precision. Sorry, I think I expressed myself sloppily. My impression is not that there's no need behind this, but that there's no need for this specific implementation at least in the Debian context, as I think the information should be already available by other means. Also when I mentioned overhead here, it was more about packaging and integration work than say actual size increase, even though that might have an adverse effect on .udeb's for example, and perhaps a non-insignificant one on .deb's depending on the amount of objects included. > > > > The Fedora Wiki and the systemd.io document have more details, but to > > > > make a long story short, a new .notes.package section with a JSON > > > > payload will be included in ELF objects, encoding various package- > > > > build-time information like distro name&version, package name&version, > > > > etc. > > > > > > > > To summarize from the discussion, the main reasons why we believe this > > > > is useful are as following: > > > > > > > > 1) minimal containers: the rpm database is not installed in the > > > > containers. The information about build-ids needs to be stored > > > > externally, so package name information is not available immediately, > > > > but only after offline processing. The new note doesn't depend on the > > > > rpm db in any way. > > > > In the Debian context, the build-ids data is going to be available > > in the affected executables, and in debug symbols packages and the > > Packages metaindices listing them, so there's no need for access to > > any local dpkg database. Given that someone needing to install debug > > packages will need access to those indices (either with outgoing network > > access or with a repository mirror), these can be queried at that time. > > Not to mention that AFAIR the Debian debug symbol servers make it > > possible to query for specific build-ids. > > This is not strictly related to debug packages, though? That was actually the impression I was getting from reading the refs though. :) But I'll expand below. > In fact, on > systems where this could be of most use you explicitly do _not_ install > debug packages (or anything at all). Or even if you wanted to, you > could not - corefiles are not handled inside the container, but > outside. Even if you wanted to and were allowed to (which for many > environments it's not the case), you can't install a Debian debug > package on a CoreOS host or Mariner host or a Flatcar host. Sure, but precisely dbgsym .deb's are the things that are trivial to unpack with dpkg-deb (or ar+tar) w/o needing to actually install them. > > > > 2) handling of a core from a container, where the container and host > > > > have different distros > > > > How each distribution handles debug packages and debug symbols is > > going to be different, so it seems there will be a need for explicit > > handling of these, at which point the above mentioned querying can be > > implemented as well, w/o the need to encode the packaging data inside > > the executable. > > Again, matching to debug symbols is n
Re: Storing package metadata in ELF objects
On Fri, 2021-05-14 at 12:41 +0200, Guillem Jover wrote: > On Sat, 2021-04-10 at 13:38:31 +0100, Luca Boccassi wrote: > > On Sat, 2021-04-10 at 13:29 +0100, Luca Boccassi wrote: > > > After an initial discussion [0], recently we have been working on a new > > > specification [0] to encode rich package-level metadata inside ELF > > > objects, so that it can be included automatically in generated coredump > > > files. The prototype to parse this in systemd-coredump and store the > > > information in systemd-journal is ready for testing and merged > > > upstream. We are now seeking further comments/opinions/suggestions, as > > > we have a few months before the next release and thus there's plenty of > > > time to make incompatible changes to the format and implementation, if > > > required. > > I've skimmed over the discussion at [0], and while having this data > seems like it might be "nice", I've to agree with the comments there > voicing that there does not really seem to be an actual need and the > overhead and additional work do not seem worth it, TBH, at least > in the Debian context. Hi Guillem, thanks for having a look, much appreciated! Just to clarify, the need is there - this is not an experimental exercise, but it is borne out of an actual need&requirement, and it is undergoing testing right now before deployment in a large scale production infrastructure. Not _everybody_ will need it, and not everywhere - that's absolutely fair, and discussions on whether the ovearhead is worth it for something that is not universally needed, but only in certain use cases, is perfectly reasonable and welcome. I know Zbigniew is going to try and get some raw numbers on the kind of overhead we are talking about, that will hopefully help frame the discussion with more precision. > > > The Fedora Wiki and the systemd.io document have more details, but to > > > make a long story short, a new .notes.package section with a JSON > > > payload will be included in ELF objects, encoding various package- > > > build-time information like distro name&version, package name&version, > > > etc. > > > > > > To summarize from the discussion, the main reasons why we believe this > > > is useful are as following: > > > > > > 1) minimal containers: the rpm database is not installed in the > > > containers. The information about build-ids needs to be stored > > > externally, so package name information is not available immediately, > > > but only after offline processing. The new note doesn't depend on the > > > rpm db in any way. > > In the Debian context, the build-ids data is going to be available > in the affected executables, and in debug symbols packages and the > Packages metaindices listing them, so there's no need for access to > any local dpkg database. Given that someone needing to install debug > packages will need access to those indices (either with outgoing network > access or with a repository mirror), these can be queried at that time. > Not to mention that AFAIR the Debian debug symbol servers make it > possible to query for specific build-ids. This is not strictly related to debug packages, though? In fact, on systems where this could be of most use you explicitly do _not_ install debug packages (or anything at all). Or even if you wanted to, you could not - corefiles are not handled inside the container, but outside. Even if you wanted to and were allowed to (which for many environments it's not the case), you can't install a Debian debug package on a CoreOS host or Mariner host or a Flatcar host. > > > 2) handling of a core from a container, where the container and host > > > have different distros > > How each distribution handles debug packages and debug symbols is > going to be different, so it seems there will be a need for explicit > handling of these, at which point the above mentioned querying can be > implemented as well, w/o the need to encode the packaging data inside > the executable. Again, matching to debug symbols is not the main goal here, build-id works for that. The main goal is to have useful metadata immediately available in all occasions, regardless of where the core was generated on the host, without reaching out to external services, so that it is directly included and collated in the system journal when the core file is handled. With a common metadata definition, there's no need to query or explicitly handle anything - this already works if you use systemd- coredump built from the main branch, and handle a core file from different containers running different distros with binaries having this metadata in the ELF file, and it just works. This is tested, not theoretical. > > > 3) self-built and external packages: unless a lot of care is taken to > > > keep access to the debuginfo packages, this information may be lost. > > > The new note is available even if the repository metadata gets lost. > > > Users can easily provide equivalent information in a format that makes > > > sense i
Re: Storing package metadata in ELF objects
On Sat, 2021-04-10 at 13:38:31 +0100, Luca Boccassi wrote: > On Sat, 2021-04-10 at 13:29 +0100, Luca Boccassi wrote: > > After an initial discussion [0], recently we have been working on a new > > specification [0] to encode rich package-level metadata inside ELF > > objects, so that it can be included automatically in generated coredump > > files. The prototype to parse this in systemd-coredump and store the > > information in systemd-journal is ready for testing and merged > > upstream. We are now seeking further comments/opinions/suggestions, as > > we have a few months before the next release and thus there's plenty of > > time to make incompatible changes to the format and implementation, if > > required. I've skimmed over the discussion at [0], and while having this data seems like it might be "nice", I've to agree with the comments there voicing that there does not really seem to be an actual need and the overhead and additional work do not seem worth it, TBH, at least in the Debian context. > > The Fedora Wiki and the systemd.io document have more details, but to > > make a long story short, a new .notes.package section with a JSON > > payload will be included in ELF objects, encoding various package- > > build-time information like distro name&version, package name&version, > > etc. > > > > To summarize from the discussion, the main reasons why we believe this > > is useful are as following: > > > > 1) minimal containers: the rpm database is not installed in the > > containers. The information about build-ids needs to be stored > > externally, so package name information is not available immediately, > > but only after offline processing. The new note doesn't depend on the > > rpm db in any way. In the Debian context, the build-ids data is going to be available in the affected executables, and in debug symbols packages and the Packages metaindices listing them, so there's no need for access to any local dpkg database. Given that someone needing to install debug packages will need access to those indices (either with outgoing network access or with a repository mirror), these can be queried at that time. Not to mention that AFAIR the Debian debug symbol servers make it possible to query for specific build-ids. > > 2) handling of a core from a container, where the container and host > > have different distros How each distribution handles debug packages and debug symbols is going to be different, so it seems there will be a need for explicit handling of these, at which point the above mentioned querying can be implemented as well, w/o the need to encode the packaging data inside the executable. > > 3) self-built and external packages: unless a lot of care is taken to > > keep access to the debuginfo packages, this information may be lost. > > The new note is available even if the repository metadata gets lost. > > Users can easily provide equivalent information in a format that makes > > sense in their own environment. It should work even when rpms and debs > > and other formats are mixed, e.g. during container image creation. I'm not sure I see the problem here. Either these self-built 3rd-party packages are kept in repos that also provide the debug symbols somewhere for all historically released versions or these will not be accessible anyway. If they are, they can as well be located as per above from the Packages metaindices, and even if the repository metadata gets lost, as long as the debug symbol packages are present (if they are not what's the point anyway) the build-ids can always be re-scanned from them as they are part of the Build-Ids field in the .deb control file. > > Other than in Fedora, we are already making the required code changes > > at Microsoft to use the same format&specification for internally-built > > binaries, and for tools that parse core files and logs. > > > > Tools for RPM and DEB (debhelper) integration are also available [3]. So, to conclude, I don't really see the point of this in the Debian context. (Not to mention the problems with encoding binary versions that might be wrong, and the busy work involved.) > > [0] https://github.com/systemd/systemd/issues/18433 > > [1] https://systemd.io/COREDUMP_PACKAGE_METADATA/ > > [2] > > https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects > > [3] https://github.com/systemd/package-notes Thanks, Guillem
Re: Storing package metadata in ELF objects
On Thu, 2021-05-06 at 03:17 +0200, Mark Wielaard wrote: > Hi Luca, > > On Tue, 2021-05-04 at 14:43 +0100, Luca Boccassi wrote: > > On Fri, 2021-04-30 at 19:57 +0200, Mark Wielaard wrote: > > > Is there a list of default keys (and their canonical spelling, upper- > > > lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key > > > with as value an URL pointing to the debuginfod server URL where the > > > embedded build-id executable, debuginfo and sources can be found? > > > https://sourceware.org/elfutils/Debuginfod.html > > > > The "Implementation" section of the spec lists the "main" fields: > > > > https://systemd.io/COREDUMP_PACKAGE_METADATA/ > > > > (source for that is > > https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md > > ) > > > > Would you like to send a PR to update it and add that field? > > Sorry, I don't have a github account. But attached is a patch for to > document it and one for the package-notes generator to add an -- > debuginfod argument (maybe the distro should set a default value for > that?) Hopefully those patches could be applied somehow. Hi, Thanks, opened PRs with your commits: https://github.com/systemd/systemd/pull/19523 https://github.com/systemd/package-notes/pull/8 Yes, if the distro has a debuginfod server, it should definitely be included. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Re: Storing package metadata in ELF objects
Hi Luca,
On Tue, 2021-05-04 at 14:43 +0100, Luca Boccassi wrote:
> On Fri, 2021-04-30 at 19:57 +0200, Mark Wielaard wrote:
> > Is there a list of default keys (and their canonical spelling, upper-
> > lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key
> > with as value an URL pointing to the debuginfod server URL where the
> > embedded build-id executable, debuginfo and sources can be found?
> > https://sourceware.org/elfutils/Debuginfod.html
>
> The "Implementation" section of the spec lists the "main" fields:
>
> https://systemd.io/COREDUMP_PACKAGE_METADATA/
>
> (source for that is
> https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md
> )
>
> Would you like to send a PR to update it and add that field?
Sorry, I don't have a github account. But attached is a patch for to
document it and one for the package-notes generator to add an --
debuginfod argument (maybe the distro should set a default value for
that?) Hopefully those patches could be applied somehow.
Thanks,
Mark
From 176dde09fcfeff9bff97fbf65d47a641dbd2fa3a Mon Sep 17 00:00:00 2001
From: Mark Wielaard
Date: Thu, 6 May 2021 03:05:02 +0200
Subject: [PATCH] docs/COREDUMP_PACKAGE_METADATA.md: Add debuginfod key
Signed-off-by: Mark Wielaard
---
docs/COREDUMP_PACKAGE_METADATA.md | 5 +
1 file changed, 5 insertions(+)
diff --git a/docs/COREDUMP_PACKAGE_METADATA.md b/docs/COREDUMP_PACKAGE_METADATA.md
index ab7c4596de..b7be9ca797 100644
--- a/docs/COREDUMP_PACKAGE_METADATA.md
+++ b/docs/COREDUMP_PACKAGE_METADATA.md
@@ -26,6 +26,10 @@ Thus it becomes desirable to add additional metadata to a binary at build time,
`systemd-coredump` and other services analyzing core files are able to extract said
metadata simply from the core file itself, without external dependencies.
+And in case external dependencies and network are available the core file
+should also include the debuginfod server URL that can provide the original
+executable, debuginfo and sources to facilitate debugging.
+
## Implementation
This document will attempt to define a common metadata format specification, so that
@@ -58,6 +62,7 @@ Value: a JSON string with the structure described below
"name":"coreutils",
"version": "4711.0815.fc13.arm32",
"osCpe": # A CPE name for the operating system, `CPE_NAME` from os-release is a good default
+ "debuginfod": "https://debuginfod.fedoraproject.org";
}
```
--
2.18.4
From 6264605699cbac02ed3060cc1ff6497b32e90382 Mon Sep 17 00:00:00 2001
From: Mark Wielaard
Date: Thu, 6 May 2021 03:13:57 +0200
Subject: [PATCH] generate-package-notes.py: Add --debuginfod argument
Signed-off-by: Mark Wielaard
---
generate-package-notes.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/generate-package-notes.py b/generate-package-notes.py
index c0b5d98..92126f7 100755
--- a/generate-package-notes.py
+++ b/generate-package-notes.py
@@ -71,6 +71,7 @@ def parse_args():
p.add_argument('--package-version')
p.add_argument('--cpe')
p.add_argument('--rpm', metavar='NEVRA')
+p.add_argument('--debuginfod')
opts = p.parse_args()
@@ -143,6 +144,8 @@ def generate_section(opts):
else:
data['os'] = read_os_release('ID')
data['osVersion'] = read_os_release('VERSION_ID')
+if opts.debuginfod:
+data['debuginfod'] = opts.debuginfod
json = json_serialize(data)
--
2.18.4
Re: Storing package metadata in ELF objects
On Fri, 2021-04-30 at 19:57 +0200, Mark Wielaard wrote: > Hi, > > On Sat, 2021-04-10 at 18:44 +, Zbigniew Jędrzejewski-Szmek wrote: > > [I'm forwarding the mail from Luca who is not subscribed to fedora- > > devel] > > On Sat, Apr 10, 2021 at 01:38:31PM +0100, Luca Boccassi wrote: > > Cross-posting to the mailing lists of a few relevant projects. > > Note that in this version of the email the [N] references in your email > don't seem to point anywhere. I found an older variant of the same > email which contained: > > [0] https://github.com/systemd/systemd/issues/18433 > [1] https://systemd.io/COREDUMP_PACKAGE_METADATA/ > [2] https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects > [3] https://github.com/systemd/package-notes Sorry about that! Must have messed up the copy&paste. > > After an initial discussion [0], recently we have been working on a new > > specification [0] to encode rich package-level metadata inside ELF > > objects, so that it can be included automatically in generated coredump > > files. The prototype to parse this in systemd-coredump and store the > > information in systemd-journal is ready for testing and merged > > upstream. We are now seeking further comments/opinions/suggestions, as > > we have a few months before the next release and thus there's plenty of > > time to make incompatible changes to the format and implementation, if > > required. > > > > A proposal to use this by default for all packages built in Fedora 35 > > has been submitted [1]. > > > > The Fedora Wiki and the systemd.io document have more details, but to > > make a long story short, a new .notes.package section with a JSON > > payload will be included in ELF objects, encoding various package- > > build-time information like distro name&version, package > > name&version, > > etc. > > Is there a list of default keys (and their canonical spelling, upper- > lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key > with as value an URL pointing to the debuginfod server URL where the > embedded build-id executable, debuginfo and sources can be found? > https://sourceware.org/elfutils/Debuginfod.html The "Implementation" section of the spec lists the "main" fields: https://systemd.io/COREDUMP_PACKAGE_METADATA/ (source for that is https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md ) Would you like to send a PR to update it and add that field? > > To summarize from the discussion, the main reasons why we believe this > > is useful are as following: > > > > 1) minimal containers: the rpm database is not installed in the > > containers. The information about build-ids needs to be stored > > externally, so package name information is not available immediately, > > but only after offline processing. The new note doesn't depend on the > > rpm db in any way. > > > > 2) handling of a core from a container, where the container and host > > have different distros > > > > 3) self-built and external packages: unless a lot of care is taken to > > keep access to the debuginfo packages, this information may be lost. > > The new note is available even if the repository metadata gets lost. > > Users can easily provide equivalent information in a format that makes > > sense in their own environment. It should work even when rpms and debs > > and other formats are mixed, e.g. during container image creation. > > > > Other than in Fedora, we are already making the required code changes > > at Microsoft to use the same format&specification for internally-built > > binaries, and for tools that parse core files and logs. > > > > Tools for RPM and DEB (debhelper) integration are also available [3]. > > > > > -- > > > Kind regards, > > > Luca Boccassi -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Re: Storing package metadata in ELF objects
Hi, On Sat, 2021-04-10 at 18:44 +, Zbigniew Jędrzejewski-Szmek wrote: > [I'm forwarding the mail from Luca who is not subscribed to fedora- > devel] > On Sat, Apr 10, 2021 at 01:38:31PM +0100, Luca Boccassi wrote: > Cross-posting to the mailing lists of a few relevant projects. Note that in this version of the email the [N] references in your email don't seem to point anywhere. I found an older variant of the same email which contained: [0] https://github.com/systemd/systemd/issues/18433 [1] https://systemd.io/COREDUMP_PACKAGE_METADATA/ [2] https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects [3] https://github.com/systemd/package-notes > After an initial discussion [0], recently we have been working on a new > specification [0] to encode rich package-level metadata inside ELF > objects, so that it can be included automatically in generated coredump > files. The prototype to parse this in systemd-coredump and store the > information in systemd-journal is ready for testing and merged > upstream. We are now seeking further comments/opinions/suggestions, as > we have a few months before the next release and thus there's plenty of > time to make incompatible changes to the format and implementation, if > required. > > A proposal to use this by default for all packages built in Fedora 35 > has been submitted [1]. > > The Fedora Wiki and the systemd.io document have more details, but to > make a long story short, a new .notes.package section with a JSON > payload will be included in ELF objects, encoding various package- > build-time information like distro name&version, package > name&version, > etc. Is there a list of default keys (and their canonical spelling, upper- lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key with as value an URL pointing to the debuginfod server URL where the embedded build-id executable, debuginfo and sources can be found? https://sourceware.org/elfutils/Debuginfod.html > To summarize from the discussion, the main reasons why we believe this > is useful are as following: > > 1) minimal containers: the rpm database is not installed in the > containers. The information about build-ids needs to be stored > externally, so package name information is not available immediately, > but only after offline processing. The new note doesn't depend on the > rpm db in any way. > > 2) handling of a core from a container, where the container and host > have different distros > > 3) self-built and external packages: unless a lot of care is taken to > keep access to the debuginfo packages, this information may be lost. > The new note is available even if the repository metadata gets lost. > Users can easily provide equivalent information in a format that makes > sense in their own environment. It should work even when rpms and debs > and other formats are mixed, e.g. during container image creation. > > Other than in Fedora, we are already making the required code changes > at Microsoft to use the same format&specification for internally-built > binaries, and for tools that parse core files and logs. > > Tools for RPM and DEB (debhelper) integration are also available [3]. > > > -- > > Kind regards, > > Luca Boccassi > >
Re: Storing package metadata in ELF objects
[I'm forwarding the mail from Luca who is not subscribed to fedora-devel] On Sat, Apr 10, 2021 at 01:38:31PM +0100, Luca Boccassi wrote: Hello, Cross-posting to the mailing lists of a few relevant projects. After an initial discussion [0], recently we have been working on a new specification [0] to encode rich package-level metadata inside ELF objects, so that it can be included automatically in generated coredump files. The prototype to parse this in systemd-coredump and store the information in systemd-journal is ready for testing and merged upstream. We are now seeking further comments/opinions/suggestions, as we have a few months before the next release and thus there's plenty of time to make incompatible changes to the format and implementation, if required. A proposal to use this by default for all packages built in Fedora 35 has been submitted [1]. The Fedora Wiki and the systemd.io document have more details, but to make a long story short, a new .notes.package section with a JSON payload will be included in ELF objects, encoding various package- build-time information like distro name&version, package name&version, etc. To summarize from the discussion, the main reasons why we believe this is useful are as following: 1) minimal containers: the rpm database is not installed in the containers. The information about build-ids needs to be stored externally, so package name information is not available immediately, but only after offline processing. The new note doesn't depend on the rpm db in any way. 2) handling of a core from a container, where the container and host have different distros 3) self-built and external packages: unless a lot of care is taken to keep access to the debuginfo packages, this information may be lost. The new note is available even if the repository metadata gets lost. Users can easily provide equivalent information in a format that makes sense in their own environment. It should work even when rpms and debs and other formats are mixed, e.g. during container image creation. Other than in Fedora, we are already making the required code changes at Microsoft to use the same format&specification for internally-built binaries, and for tools that parse core files and logs. Tools for RPM and DEB (debhelper) integration are also available [3]. > -- > Kind regards, > Luca Boccassi
Re: Storing package metadata in ELF objects
On Sat, 2021-04-10 at 13:29 +0100, Luca Boccassi wrote: > Hello, > > Cross-posting to the mailing lists of a few relevant projects. > > After an initial discussion [0], recently we have been working on a new > specification [0] to encode rich package-level metadata inside ELF > objects, so that it can be included automatically in generated coredump > files. The prototype to parse this in systemd-coredump and store the > information in systemd-journal is ready for testing and merged > upstream. We are now seeking further comments/opinions/suggestions, as > we have a few months before the next release and thus there's plenty of > time to make incompatible changes to the format and implementation, if > required. > > A proposal to use this by default for all packages built in Fedora 35 > has been submitted [1]. > > The Fedora Wiki and the systemd.io document have more details, but to > make a long story short, a new .notes.package section with a JSON > payload will be included in ELF objects, encoding various package- > build-time information like distro name&version, package name&version, > etc. > > To summarize from the discussion, the main reasons why we believe this > is useful are as following: > > 1) minimal containers: the rpm database is not installed in the > containers. The information about build-ids needs to be stored > externally, so package name information is not available immediately, > but only after offline processing. The new note doesn't depend on the > rpm db in any way. > > 2) handling of a core from a container, where the container and host > have different distros > > 3) self-built and external packages: unless a lot of care is taken to > keep access to the debuginfo packages, this information may be lost. > The new note is available even if the repository metadata gets lost. > Users can easily provide equivalent information in a format that makes > sense in their own environment. It should work even when rpms and debs > and other formats are mixed, e.g. during container image creation. > > Other than in Fedora, we are already making the required code changes > at Microsoft to use the same format&specification for internally-built > binaries, and for tools that parse core files and logs. > > Tools for RPM and DEB (debhelper) integration are also available [3]. Wrong Fedora list address - off to a great start already :-) (fixed now) -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Storing package metadata in ELF objects
Hello, Cross-posting to the mailing lists of a few relevant projects. After an initial discussion [0], recently we have been working on a new specification [0] to encode rich package-level metadata inside ELF objects, so that it can be included automatically in generated coredump files. The prototype to parse this in systemd-coredump and store the information in systemd-journal is ready for testing and merged upstream. We are now seeking further comments/opinions/suggestions, as we have a few months before the next release and thus there's plenty of time to make incompatible changes to the format and implementation, if required. A proposal to use this by default for all packages built in Fedora 35 has been submitted [1]. The Fedora Wiki and the systemd.io document have more details, but to make a long story short, a new .notes.package section with a JSON payload will be included in ELF objects, encoding various package- build-time information like distro name&version, package name&version, etc. To summarize from the discussion, the main reasons why we believe this is useful are as following: 1) minimal containers: the rpm database is not installed in the containers. The information about build-ids needs to be stored externally, so package name information is not available immediately, but only after offline processing. The new note doesn't depend on the rpm db in any way. 2) handling of a core from a container, where the container and host have different distros 3) self-built and external packages: unless a lot of care is taken to keep access to the debuginfo packages, this information may be lost. The new note is available even if the repository metadata gets lost. Users can easily provide equivalent information in a format that makes sense in their own environment. It should work even when rpms and debs and other formats are mixed, e.g. during container image creation. Other than in Fedora, we are already making the required code changes at Microsoft to use the same format&specification for internally-built binaries, and for tools that parse core files and logs. Tools for RPM and DEB (debhelper) integration are also available [3]. -- Kind regards, Luca Boccassi [0] https://github.com/systemd/systemd/issues/18433 [1] https://systemd.io/COREDUMP_PACKAGE_METADATA/ [2] https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects [3] https://github.com/systemd/package-notes signature.asc Description: This is a digitally signed message part
