[RFCv2 PATCH 0/3] Salted build ids via linker sections
Hi, This is v2 of my proposal to allow unique build-ids in the kernel. from last time: "" In Fedora, the debug information is packaged separately (foo-debuginfo) and can be installed separately. There's been a long standing issue where only one version of a debuginfo info package can be installed at a time. Mark Wielaard made an effort for Fedora 27 to allow parallel installation of debuginfo (see https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo for more details) Part of the requirement to allow this to work is that build ids are unique between builds. The existing upstream rpm implementation ensures this by re-calculating the build-id using the version and release as a seed. This doesn't work 100% for the kernel because of the vDSO which is its own binary and doesn't get updated. After poking holes in a few of my ideas, there was a discussion with some people from the binutils team about adding --build-id-salt to let ld do the calculation debugedit is doing. There was a counter proposal made about adding some extra information via a .comment which will affect the build id calculation but just get stripped out. "" This v2 cleans up the naming to be consistent and also switches to a config option vs. an environment variable. I've seen some sporadic failures about missing the generated header so I think I'm still missing a dependency somewhere. I'm still mostly looking for feedback whether this would be acceptable for merging or if we should just persue a --build-id-salt in binutils. Thanks, Laura Laura Abbott (3): kbuild: Introduce build-salt generated header kbuild: Link with generated build-salt header x86/vdso: Add build salt to the vDSO Makefile | 13 +++-- arch/x86/entry/vdso/vdso-layout.lds.S | 3 +++ init/Kconfig | 8 scripts/.gitignore| 1 + scripts/Makefile | 2 +- scripts/build-salt.lds.S | 5 + scripts/gensalt | 21 + scripts/link-vmlinux.sh | 3 ++- 8 files changed, 52 insertions(+), 4 deletions(-) create mode 100644 scripts/build-salt.lds.S create mode 100755 scripts/gensalt -- 2.16.2
Re: [RFCv2 PATCH 0/3] Salted build ids via linker sections
On 05/07/2018 07:49 PM, Andy Lutomirski wrote: On Sun, May 6, 2018 at 11:59 PM Masahiro Yamada < yamada.masah...@socionext.com> wrote: 2018-03-30 21:40 GMT+09:00 Mark Wielaard : On Thu, 2018-03-29 at 11:01 -0700, Laura Abbott wrote: I'm still mostly looking for feedback whether this would be acceptable for merging or if we should just persue a --build-id-salt in binutils. Personally I would go with this approach. It seems simple and it might take years before a new linker option is available everywhere. Indeed. This series is easier than --build-id-salt. If you do not see any better solution, I can accept this. BTW, when I read https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo I thought "we could reverse the symlink direction from debug file to build-id file)" sensible (but I understand it is not easy to change this way). If two packages share an identical image, one package can borrow the image from the other, then the storage space will be saved. So, having identical ID should be advantage, but we actually see only disadvantage... To simplify things I think you could just always add the extra vdso .comment initialized to something like KERNELRELEASE. Which distros seem to update anyway to include their build number, so they wouldn't need to do anything special to "update the build salt". That's what I was thinking, too. Would that solve Fedora's problem? Yes, that seems reasonable.
Re: [RFCv2 PATCH 0/3] Salted build ids via linker sections
On 05/06/2018 11:28 PM, Masahiro Yamada wrote: 2018-03-30 3:01 GMT+09:00 Laura Abbott : Hi, This is v2 of my proposal to allow unique build-ids in the kernel. from last time: "" In Fedora, the debug information is packaged separately (foo-debuginfo) and can be installed separately. There's been a long standing issue where only one version of a debuginfo info package can be installed at a time. Mark Wielaard made an effort for Fedora 27 to allow parallel installation of debuginfo (see https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo for more details) Part of the requirement to allow this to work is that build ids are unique between builds. The existing upstream rpm implementation ensures this by re-calculating the build-id using the version and release as a seed. This doesn't work 100% for the kernel because of the vDSO which is its own binary and doesn't get updated. After poking holes in a few of my ideas, there was a discussion with some people from the binutils team about adding --build-id-salt to let ld do the calculation debugedit is doing. There was a counter proposal made about adding some extra information via a .comment which will affect the build id calculation but just get stripped out. "" I think you already know '--build-id=uuid' linker option. Doesn't this solve your problem? The disadvantage of this option is, we will lose reproducible building because --build-id=uuid adds every time random salt. The advantage is, the implementation is even simpler, and easier to migrate to --build-id-salt once it is supported in the future. It could, theoretically. The reproducibility is nice though and I'd like to keep the kernel close to matching what other packages do though. Thanks, Laura This v2 cleans up the naming to be consistent and also switches to a config option vs. an environment variable. I've seen some sporadic failures about missing the generated header so I think I'm still missing a dependency somewhere. Right. There is no dependency between 'prepare' and 'scripts' in the top Makefile. Therefore, Kbuild can run them simultaneously, which would cause a race in parallel building. I'm still mostly looking for feedback whether this would be acceptable for merging or if we should just persue a --build-id-salt in binutils. Thanks, Laura Laura Abbott (3): kbuild: Introduce build-salt generated header kbuild: Link with generated build-salt header x86/vdso: Add build salt to the vDSO Makefile | 13 +++-- arch/x86/entry/vdso/vdso-layout.lds.S | 3 +++ init/Kconfig | 8 scripts/.gitignore| 1 + scripts/Makefile | 2 +- scripts/build-salt.lds.S | 5 + scripts/gensalt | 21 + scripts/link-vmlinux.sh | 3 ++- 8 files changed, 52 insertions(+), 4 deletions(-) create mode 100644 scripts/build-salt.lds.S create mode 100755 scripts/gensalt -- 2.16.2 -- To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFCv2 PATCH 0/3] Salted build ids via linker sections
2018-03-30 3:01 GMT+09:00 Laura Abbott : > Hi, > > This is v2 of my proposal to allow unique build-ids in the kernel. from > last time: > > "" > In Fedora, the debug information is packaged separately (foo-debuginfo) and > can be installed separately. There's been a long standing issue where only one > version of a debuginfo info package can be installed at a time. Mark Wielaard > made an effort for Fedora 27 to allow parallel installation of debuginfo (see > https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo for > more details) > > Part of the requirement to allow this to work is that build ids are > unique between builds. The existing upstream rpm implementation ensures > this by re-calculating the build-id using the version and release as a > seed. This doesn't work 100% for the kernel because of the vDSO which is > its own binary and doesn't get updated. After poking holes in a few of my > ideas, there was a discussion with some people from the binutils team about > adding --build-id-salt to let ld do the calculation debugedit is doing. There > was a counter proposal made about adding some extra information via a .comment > which will affect the build id calculation but just get stripped out. > "" I think you already know '--build-id=uuid' linker option. Doesn't this solve your problem? The disadvantage of this option is, we will lose reproducible building because --build-id=uuid adds every time random salt. The advantage is, the implementation is even simpler, and easier to migrate to --build-id-salt once it is supported in the future. > This v2 cleans up the naming to be consistent and also switches to a > config option vs. an environment variable. I've seen some sporadic > failures about missing the generated header so I think I'm still missing > a dependency somewhere. Right. There is no dependency between 'prepare' and 'scripts' in the top Makefile. Therefore, Kbuild can run them simultaneously, which would cause a race in parallel building. I'm still mostly looking for feedback whether > this would be acceptable for merging or if we should just persue a > --build-id-salt in binutils. > > Thanks, > Laura > > > Laura Abbott (3): > kbuild: Introduce build-salt generated header > kbuild: Link with generated build-salt header > x86/vdso: Add build salt to the vDSO > > Makefile | 13 +++-- > arch/x86/entry/vdso/vdso-layout.lds.S | 3 +++ > init/Kconfig | 8 > scripts/.gitignore| 1 + > scripts/Makefile | 2 +- > scripts/build-salt.lds.S | 5 + > scripts/gensalt | 21 + > scripts/link-vmlinux.sh | 3 ++- > 8 files changed, 52 insertions(+), 4 deletions(-) > create mode 100644 scripts/build-salt.lds.S > create mode 100755 scripts/gensalt > > -- > 2.16.2 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best Regards Masahiro Yamada
Re: [RFCv2 PATCH 0/3] Salted build ids via linker sections
2018-03-30 21:40 GMT+09:00 Mark Wielaard : > On Thu, 2018-03-29 at 11:01 -0700, Laura Abbott wrote: >> I'm still mostly looking for feedback whether >> this would be acceptable for merging or if we should just persue a >> --build-id-salt in binutils. > > Personally I would go with this approach. It seems simple and it might > take years before a new linker option is available everywhere. Indeed. This series is easier than --build-id-salt. If you do not see any better solution, I can accept this. BTW, when I read https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo I thought "we could reverse the symlink direction from debug file to build-id file)" sensible (but I understand it is not easy to change this way). If two packages share an identical image, one package can borrow the image from the other, then the storage space will be saved. So, having identical ID should be advantage, but we actually see only disadvantage... > To simplify things I think you could just always add the extra vdso > .comment initialized to something like KERNELRELEASE. Which distros > seem to update anyway to include their build number, so they wouldn't > need to do anything special to "update the build salt". > -- Best Regards Masahiro Yamada
Re: [RFCv2 PATCH 0/3] Salted build ids via linker sections
2018-05-15 6:02 GMT+09:00 Laura Abbott : > On 05/06/2018 11:28 PM, Masahiro Yamada wrote: >> >> 2018-03-30 3:01 GMT+09:00 Laura Abbott : >>> >>> Hi, >>> >>> This is v2 of my proposal to allow unique build-ids in the kernel. from >>> last time: >>> >>> "" >>> In Fedora, the debug information is packaged separately (foo-debuginfo) >>> and >>> can be installed separately. There's been a long standing issue where >>> only one >>> version of a debuginfo info package can be installed at a time. Mark >>> Wielaard >>> made an effort for Fedora 27 to allow parallel installation of debuginfo >>> (see >>> https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo for >>> more details) >>> >>> Part of the requirement to allow this to work is that build ids are >>> unique between builds. The existing upstream rpm implementation ensures >>> this by re-calculating the build-id using the version and release as a >>> seed. This doesn't work 100% for the kernel because of the vDSO which is >>> its own binary and doesn't get updated. After poking holes in a few of my >>> ideas, there was a discussion with some people from the binutils team >>> about >>> adding --build-id-salt to let ld do the calculation debugedit is doing. >>> There >>> was a counter proposal made about adding some extra information via a >>> .comment >>> which will affect the build id calculation but just get stripped out. >>> "" >> >> >> >> I think you already know '--build-id=uuid' linker option. >> >> Doesn't this solve your problem? >> >> The disadvantage of this option is, >> we will lose reproducible building because --build-id=uuid >> adds every time random salt. >> >> The advantage is, the implementation is even simpler, >> and easier to migrate to --build-id-salt once it is supported >> in the future. >> >> > > It could, theoretically. The reproducibility is nice though and > I'd like to keep the kernel close to matching what other packages > do though. Okay. I left some comments in v2 about improvement, and cause of the sporadic build failures. Will wait for v3. -- Best Regards Masahiro Yamada
Re: [RFCv2 PATCH 0/3] Salted build ids via linker sections
On Sun, May 6, 2018 at 11:59 PM Masahiro Yamada < yamada.masah...@socionext.com> wrote: > 2018-03-30 21:40 GMT+09:00 Mark Wielaard : > > On Thu, 2018-03-29 at 11:01 -0700, Laura Abbott wrote: > >> I'm still mostly looking for feedback whether > >> this would be acceptable for merging or if we should just persue a > >> --build-id-salt in binutils. > > > > Personally I would go with this approach. It seems simple and it might > > take years before a new linker option is available everywhere. > Indeed. This series is easier than --build-id-salt. > If you do not see any better solution, I can accept this. > BTW, when I read > https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo > I thought "we could reverse the symlink direction from debug file to > build-id file)" > sensible (but I understand it is not easy to change this way). > If two packages share an identical image, > one package can borrow the image from the other, > then the storage space will be saved. > So, having identical ID should be advantage, > but we actually see only disadvantage... > > To simplify things I think you could just always add the extra vdso > > .comment initialized to something like KERNELRELEASE. Which distros > > seem to update anyway to include their build number, so they wouldn't > > need to do anything special to "update the build salt". > > That's what I was thinking, too. Would that solve Fedora's problem?
Re: [RFCv2 PATCH 0/3] Salted build ids via linker sections
On Thu, 2018-03-29 at 11:01 -0700, Laura Abbott wrote: > I'm still mostly looking for feedback whether > this would be acceptable for merging or if we should just persue a > --build-id-salt in binutils. Personally I would go with this approach. It seems simple and it might take years before a new linker option is available everywhere. To simplify things I think you could just always add the extra vdso .comment initialized to something like KERNELRELEASE. Which distros seem to update anyway to include their build number, so they wouldn't need to do anything special to "update the build salt". Cheers, Mark
