On Sat, Mar 12, 2022 at 2:12 PM Jimmy Assarsson <jimmyassars...@gmail.com> wrote:
> On 2022-03-09 16:47, Bruce Ashfield wrote: > > On Sat, Feb 26, 2022 at 2:17 AM Jimmy Assarsson > > <jimmyassars...@gmail.com> wrote: > >> On 2022-02-24 16:41, Bruce Ashfield wrote: > >>> On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson > >>> <jimmyassars...@gmail.com> wrote: > >>>> On 2022-02-21 05:13, Bruce Ashfield wrote: > >>>>> on 19/02/2022 Jimmy Assarsson wrote: > >>>>>> On 2022-02-19 20:42, Bruce Ashfield wrote: > >>>>>>> On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson > >>>>>>> <jimmyassars...@gmail.com> wrote: > >>>>>>>> > >>>>>>>> When patching a kernel git repository, in a recipe inheriting > kernel-yocto.bbclass, > >>>>>>>> the resulting commit hash will become different every time the > source is unpacked and patched. > >>>>>>>> > >>>>>>>> This is a problem that causes non-reproducible builds, when the > commit hash is built > >>>>>>>> into the kernel (CONFIG_LOCALVERSION_AUTO=y). > >>>>>>>> > >>>>>>>> > >>>>>>>> Currently it is not a problem in linux-yocto, since an empty > .scmversion is > >>>>>>>> created in kernel_do_configure [1]. This is preventing the kernel > build from > >>>>>>>> generating its own .scmversion. > >>>>>>>> > >>>>>>>> If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and > applying any > >>>>>>>> patch in the linux-yocto recipe, this will result in a > non-reproducible build. > >>>>>> > >>>>>> ... > >>>>>> > >>>>>>>> diff --git a/tools/kgit-s2q b/tools/kgit-s2q > >>>>>>>> index 706783e..b46a138 100755 > >>>>>>>> --- a/tools/kgit-s2q > >>>>>>>> +++ b/tools/kgit-s2q > >>>>>>>> @@ -558,7 +558,7 @@ do > >>>>>>>> echo "($APPLIED/$COUNT) `basename $i`" > >>>>>>>> fi > >>>>>>>> > >>>>>>>> - git am --keep-non-patch $DIR/$i > /dev/null 2>&1 > >>>>>>>> + git am --keep-non-patch --committer-date-is-author-date > $DIR/$i > /dev/null 2>&1 > >>>>>>>> if [ $? != 0 ];then > >>>>>>>> git am --abort > /dev/null 2>&1 > >>>>>>>> echo "[INFO]: check of $DIR/$i with \"git > am\" did not pass, trying reduced context." > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> I'm not sure if this is a proper solution to fix the problem or > what side effects it may introduce? > >>>>>>> > >>>>>>> There's nothing fundamentally wrong with that solution, but there > are > >>>>>>> modes for the kernel builds where we absolutely do not want the > >>>>>>> reproducible build (and older dates) to be in effect (see my > comments > >>>>>>> on the OE core mailing list when the ability to disable > reproducible > >>>>>>> builds was almost removed). > >>>>>> > >>>>>> Thanks, now I've looked it up. > >>>>>> > >>>>>>> We could add an option to the patch queue management that was > enabled > >>>>>>> when reproducible builds are in play, so git am could operate like > >>>>>>> that. > >>>>>>> > >>>>>>> Alternatively, you could bbappend the other kernel's do_configure > and > >>>>>>> do the same .scmversion trick to make sure that everything is > >>>>>>> consistent. > >>>>>> > >>>>>> I don't follow, can you be more specific? > >>>>>> We encountered this problem when building linux-imx (from > meta-freescale). > >>>>>> Preferably the .scmversion workaround in [1] should be dropped, so > that > >>>>>> there is no need to reinvent/mimic the output of > >>>>>> linux/scripts/setlocalversion, in a bitbake recipe. > >>>>> > >>>>> I see no compelling reason to drop the .scmversion in the main > kernel.bbclass > >>>>> (at least not in the short term). It has been there for a > significant amount > >>>>> of time, and removing it now would very likely cause some ripple > effects > >>>>> througout the ecosystem (it is there for a few different issues). > While not > >>>>> elegant, it is an available and workable solution to the problem it > is > >>>>> solving. > >>>> > >>>> I agree. And obviously this is not a big issue for others. > >>>> There have been at least two failed attempts on removing this > [4][5][6]. > >>>> > >>>>> I was suggesting that anyone can bbappend any kernel recipe to do > the similar > >>>>> .scmversion set (assuming they aren't using the core kernel bbclass > >>>>> configure()). > >>>>> > >>>>> But from your next statement, that isn't going to work as the sole > solution, > >>>>> since yes, that will only restore where we keep the git rev out of > the > >>>>> localversion. > >>>>> > >>>>>> > >>>>>> To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but > >>>>>> with a reproducible output. > >>>>> > >>>>> And that's actually part of the reason the .scmversion is set, the > >>>>> setlocalversion script was appending values in some configurations > and > >>>>> combinations of patching, etc, such that issues were popping up with > version > >>>>> matching. > >>>>> > >>>>> Part of getting that to work, definitely could be to apply patches > with the > >>>>> author date, but that has to be triggered on reproducible builds > being > >>>>> enabled (which is of course the default) and also preserving the use > of the > >>>>> .scmversion for the baseline / basic kernel configuration routines. > >>>>> > >>>>> .. so one config to say "don't set scmversion" and another to say > "apply > >>>>> patches with author date" (this one toggling based on reproducible > builds). > >>>>> And then both scenarios can be supported. > >>>> > >>>> Currently I'm only interested in fixing the "apply patches with > >>>> author date", to get reproducible builds. > >>>> > >>>> So we will end up with a new config in kernel-yocto.bbclass, since > this > >>>> is the only location where kgit-s2q is used. > >>>> And we will get a new option in kgit-s2q, that will result in > >>>> "git am --committer-date-is-author-date" being used. > >>>> > >>>> Any name suggestions for the config and option? > >>>> KERNEL_PATCH_WITH_AUTHOR_DATE > >>>> KERNEL_PATCH_REPRODUCIBLE_HASH > >>>> KERNEL_PATCH_REPRODUCIBLE_COMMIT > >>> > >>> It is better to abstract it behind the reproducible feature > >>> description, since there are other kernel reproducibility components > >>> that exist, or may appear in the future. > >> > >> Sure. > >> > >>> If you want, I can have a go at the patch, which will allow me to run > >>> my local tests and put it in my next consolidated pull request. > >> > >> That would be great :) > >> Thanks! > > > > It took me a bit longer than expected, but the lightly tested v1 is > > here: > https://git.yoctoproject.org/poky-contrib/commit/?h=zedd/kernel&id=cfbc3bf788ba1fc7d5f4de1a95c72a66243562e6 > > > > And by lightly tested, I mean I can still patch the kernel and things > > didn't explode in flames. i haven't checked if the commit dates, etc, > > do what you were looking for. > > Thanks Bruce! > > Now I've tested it. > > The changes in kernel-yocto.bbclass looks good to me. > > In kgit-s2q, the check of $commit_sha_type is missing square brackets: > - if "$commit_sha_type" == "author"; then > + if [ "$commit_sha_type" == "author" ]; then > Once fixed, the commit dates are as expected and test_reproducible_builds > passes. > oops. Yes, I had fixed that as well, but didn't push it, as I was running out the door for a week of vacation. > > The commit-sha option and arguments looks good. > However the help description of the 'date' option argument, is not > completely clear, at least not to me: > > --commit-sha type of commit sha's to create: 'author date' or 'commit > date' (author or date) > default is 'commit date' > > If it wasn't for 'author date', I would have assumed 'commit date', gives > the same result. 'applied date' or similar, makes more sense to me. > > I'll clarify the help a bit when I push the final version and send it to OE core next week. Thanks for the testing! Bruce > Best regards, > jimmy > > > I'm heading out on vacation for the next week, so I won't send this > > for merging into OE core yet, but feel free to try it out, send > > feedback and I'll do a v2 when I return (after March 19th). > > > > Bruce > > > >> > >> Best regards, > >> jimmy > >> > >>> Bruce > >>> > >>>> --patch-with-author-date > >>>> --apply-patch-with-author-date > >>>> --committer-date-is-author-date > >>>> > >>>> > >>>> [4] https://patchwork.openembedded.org/patch/138333 > >>>> [5] https://patchwork.openembedded.org/patch/171479 > >>>> [6] https://patchwork.openembedded.org/patch/171513 > >>>> > >>>> > >>>> Best regards, > >>>> jimmy > >>>> > >>>> > >>>>> Bruce > >>>>> > >>>>>> > >>>>>> Best regards, > >>>>>> jimmy > >>>>>> > >>>>>> > >>>>>>> Bruce > >>>>>>> > >>>>>>>> > >>>>>>>> [1] > https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589 > >>>>>>>> [2] > https://git.yoctoproject.org/meta-freescale/tree/classes/fsl-kernel-localversion.bbclass?h=honister > >>>>>>>> [3] > https://github.com/OE4T/meta-tegra/blob/honister/recipes-kernel/linux/linux-tegra_4.9.bb > -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11049): https://lists.yoctoproject.org/g/linux-yocto/message/11049 Mute This Topic: https://lists.yoctoproject.org/mt/89251386/21656 Group Owner: linux-yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
