On Wed, Apr 20, 2022 at 3:23 PM Jimmy Assarsson <jimmyassars...@gmail.com> wrote:
> On 2022-03-13 04:37, Bruce Ashfield wrote: > > On Sat, Mar 12, 2022 at 2:12 PM Jimmy Assarsson < > jimmyassars...@gmail.com <mailto:jimmyassars...@gmail.com>> wrote: > > > > On 2022-03-09 16:47, Bruce Ashfield wrote: > > > On Sat, Feb 26, 2022 at 2:17 AM Jimmy Assarsson > > > <jimmyassars...@gmail.com <mailto:jimmyassars...@gmail.com>> > wrote: > > >> On 2022-02-24 16:41, Bruce Ashfield wrote: > > >>> On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson > > >>> <jimmyassars...@gmail.com <mailto:jimmyassars...@gmail.com>> > wrote: > > >>>> On 2022-02-21 05:13, Bruce Ashfield wrote: > > >>>>> on 19/02/2022 Jimmy Assarsson wrote: > > >>>>>> On 2022-02-19 20:42, Bruce Ashfield wrote: > > >>>>>>> On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson > > >>>>>>> <jimmyassars...@gmail.com <mailto:jimmyassars...@gmail.com>> > wrote: > > >>>>>>>> > > >>>>>>>> When patching a kernel git repository, in a recipe > inheriting kernel-yocto.bbclass, > > >>>>>>>> the resulting commit hash will become different every time > the source is unpacked and patched. > > >>>>>>>> > > >>>>>>>> This is a problem that causes non-reproducible builds, > when the commit hash is built > > >>>>>>>> into the kernel (CONFIG_LOCALVERSION_AUTO=y). > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Currently it is not a problem in linux-yocto, since an > empty .scmversion is > > >>>>>>>> created in kernel_do_configure [1]. This is preventing the > kernel build from > > >>>>>>>> generating its own .scmversion. > > >>>>>>>> > > >>>>>>>> If removing this commit, setting > CONFIG_LOCALVERSION_AUTO=y and applying any > > >>>>>>>> patch in the linux-yocto recipe, this will result in a > non-reproducible build. > > >>>>>> > > >>>>>> ... > > >>>>>> > > >>>>>>>> diff --git a/tools/kgit-s2q b/tools/kgit-s2q > > >>>>>>>> index 706783e..b46a138 100755 > > >>>>>>>> --- a/tools/kgit-s2q > > >>>>>>>> +++ b/tools/kgit-s2q > > >>>>>>>> @@ -558,7 +558,7 @@ do > > >>>>>>>> echo "($APPLIED/$COUNT) `basename $i`" > > >>>>>>>> fi > > >>>>>>>> > > >>>>>>>> - git am --keep-non-patch $DIR/$i > /dev/null 2>&1 > > >>>>>>>> + git am --keep-non-patch > --committer-date-is-author-date $DIR/$i > /dev/null 2>&1 > > >>>>>>>> if [ $? != 0 ];then > > >>>>>>>> git am --abort > /dev/null 2>&1 > > >>>>>>>> echo "[INFO]: check of $DIR/$i with > \"git am\" did not pass, trying reduced context." > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> I'm not sure if this is a proper solution to fix the > problem or what side effects it may introduce? > > >>>>>>> > > >>>>>>> There's nothing fundamentally wrong with that solution, but > there are > > >>>>>>> modes for the kernel builds where we absolutely do not want > the > > >>>>>>> reproducible build (and older dates) to be in effect (see > my comments > > >>>>>>> on the OE core mailing list when the ability to disable > reproducible > > >>>>>>> builds was almost removed). > > >>>>>> > > >>>>>> Thanks, now I've looked it up. > > >>>>>> > > >>>>>>> We could add an option to the patch queue management that > was enabled > > >>>>>>> when reproducible builds are in play, so git am could > operate like > > >>>>>>> that. > > >>>>>>> > > >>>>>>> Alternatively, you could bbappend the other kernel's > do_configure and > > >>>>>>> do the same .scmversion trick to make sure that everything > is > > >>>>>>> consistent. > > >>>>>> > > >>>>>> I don't follow, can you be more specific? > > >>>>>> We encountered this problem when building linux-imx (from > meta-freescale). > > >>>>>> Preferably the .scmversion workaround in [1] should be > dropped, so that > > >>>>>> there is no need to reinvent/mimic the output of > > >>>>>> linux/scripts/setlocalversion, in a bitbake recipe. > > >>>>> > > >>>>> I see no compelling reason to drop the .scmversion in the > main kernel.bbclass > > >>>>> (at least not in the short term). It has been there for a > significant amount > > >>>>> of time, and removing it now would very likely cause some > ripple effects > > >>>>> througout the ecosystem (it is there for a few different > issues). While not > > >>>>> elegant, it is an available and workable solution to the > problem it is > > >>>>> solving. > > >>>> > > >>>> I agree. And obviously this is not a big issue for others. > > >>>> There have been at least two failed attempts on removing this > [4][5][6]. > > >>>> > > >>>>> I was suggesting that anyone can bbappend any kernel recipe > to do the similar > > >>>>> .scmversion set (assuming they aren't using the core kernel > bbclass > > >>>>> configure()). > > >>>>> > > >>>>> But from your next statement, that isn't going to work as the > sole solution, > > >>>>> since yes, that will only restore where we keep the git rev > out of the > > >>>>> localversion. > > >>>>> > > >>>>>> > > >>>>>> To be clear, we want the result of > CONFIG_LOCALVERSION_AUTO=y, but > > >>>>>> with a reproducible output. > > >>>>> > > >>>>> And that's actually part of the reason the .scmversion is > set, the > > >>>>> setlocalversion script was appending values in some > configurations and > > >>>>> combinations of patching, etc, such that issues were popping > up with version > > >>>>> matching. > > >>>>> > > >>>>> Part of getting that to work, definitely could be to apply > patches with the > > >>>>> author date, but that has to be triggered on reproducible > builds being > > >>>>> enabled (which is of course the default) and also preserving > the use of the > > >>>>> .scmversion for the baseline / basic kernel configuration > routines. > > >>>>> > > >>>>> .. so one config to say "don't set scmversion" and another to > say "apply > > >>>>> patches with author date" (this one toggling based on > reproducible builds). > > >>>>> And then both scenarios can be supported. > > >>>> > > >>>> Currently I'm only interested in fixing the "apply patches with > > >>>> author date", to get reproducible builds. > > >>>> > > >>>> So we will end up with a new config in kernel-yocto.bbclass, > since this > > >>>> is the only location where kgit-s2q is used. > > >>>> And we will get a new option in kgit-s2q, that will result in > > >>>> "git am --committer-date-is-author-date" being used. > > >>>> > > >>>> Any name suggestions for the config and option? > > >>>> KERNEL_PATCH_WITH_AUTHOR_DATE > > >>>> KERNEL_PATCH_REPRODUCIBLE_HASH > > >>>> KERNEL_PATCH_REPRODUCIBLE_COMMIT > > >>> > > >>> It is better to abstract it behind the reproducible feature > > >>> description, since there are other kernel reproducibility > components > > >>> that exist, or may appear in the future. > > >> > > >> Sure. > > >> > > >>> If you want, I can have a go at the patch, which will allow me > to run > > >>> my local tests and put it in my next consolidated pull request. > > >> > > >> That would be great :) > > >> Thanks! > > > > > > It took me a bit longer than expected, but the lightly tested v1 > is > > > here: > https://git.yoctoproject.org/poky-contrib/commit/?h=zedd/kernel&id=cfbc3bf788ba1fc7d5f4de1a95c72a66243562e6 > < > https://git.yoctoproject.org/poky-contrib/commit/?h=zedd/kernel&id=cfbc3bf788ba1fc7d5f4de1a95c72a66243562e6 > > > > > > > > And by lightly tested, I mean I can still patch the kernel and > things > > > didn't explode in flames. i haven't checked if the commit dates, > etc, > > > do what you were looking for. > > > > Thanks Bruce! > > > > Now I've tested it. > > > > The changes in kernel-yocto.bbclass looks good to me. > > > > In kgit-s2q, the check of $commit_sha_type is missing square > brackets: > > - if "$commit_sha_type" == "author"; then > > + if [ "$commit_sha_type" == "author" ]; then > > Once fixed, the commit dates are as expected and > test_reproducible_builds > > passes. > > > > > > oops. Yes, I had fixed that as well, but didn't push it, as I was > running out the door for a week of vacation. > > > > > > The commit-sha option and arguments looks good. > > However the help description of the 'date' option argument, is not > > completely clear, at least not to me: > > > > --commit-sha type of commit sha's to create: 'author date' or > 'commit date' (author or date) > > default is 'commit date' > > > > If it wasn't for 'author date', I would have assumed 'commit date', > gives > > the same result. 'applied date' or similar, makes more sense to me. > > > > > > I'll clarify the help a bit when I push the final version and send it to > OE core next week. > > > > Thanks for the testing! > > > > Bruce > > Gentle ping :) > Have you had time to look into this? > Yep, everything is fixed up, but it was too late to get into the upcoming release, so it is in my pending queue. That queue will be out in the next few days. We can ask for a backport to the upcoming LTS release, if there's demand, but I wasn't willing to take any chances with some unexpected breakage so close to the release builds. Bruce > > Best regards, > jimmy > > > > Best regards, > > jimmy > > > > > I'm heading out on vacation for the next week, so I won't send > this > > > for merging into OE core yet, but feel free to try it out, send > > > feedback and I'll do a v2 when I return (after March 19th). > > > > > > Bruce > > > > > >> > > >> Best regards, > > >> jimmy > > >> > > >>> Bruce > > >>> > > >>>> --patch-with-author-date > > >>>> --apply-patch-with-author-date > > >>>> --committer-date-is-author-date > > >>>> > > >>>> > > >>>> [4] https://patchwork.openembedded.org/patch/138333 < > https://patchwork.openembedded.org/patch/138333> > > >>>> [5] https://patchwork.openembedded.org/patch/171479 < > https://patchwork.openembedded.org/patch/171479> > > >>>> [6] https://patchwork.openembedded.org/patch/171513 < > https://patchwork.openembedded.org/patch/171513> > > >>>> > > >>>> > > >>>> Best regards, > > >>>> jimmy > > >>>> > > >>>> > > >>>>> Bruce > > >>>>> > > >>>>>> > > >>>>>> Best regards, > > >>>>>> jimmy > > >>>>>> > > >>>>>> > > >>>>>>> Bruce > > >>>>>>> > > >>>>>>>> > > >>>>>>>> [1] > https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589 > < > https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589 > > > > >>>>>>>> [2] > https://git.yoctoproject.org/meta-freescale/tree/classes/fsl-kernel-localversion.bbclass?h=honister > < > https://git.yoctoproject.org/meta-freescale/tree/classes/fsl-kernel-localversion.bbclass?h=honister > > > > >>>>>>>> [3] > https://github.com/OE4T/meta-tegra/blob/honister/recipes-kernel/linux/linux-tegra_4.9.bb > < > https://github.com/OE4T/meta-tegra/blob/honister/recipes-kernel/linux/linux-tegra_4.9.bb > > > > > > > > > > -- > > - Thou shalt not follow the NULL pointer, for chaos and madness await > thee at its end > > - "Use the force Harry" - Gandalf, Star Trek II > > > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11214): https://lists.yoctoproject.org/g/linux-yocto/message/11214 Mute This Topic: https://lists.yoctoproject.org/mt/89251386/21656 Group Owner: linux-yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
