Any feedback on this or does it have to wait for 3.1?
At least the last patch is a fix for regression already included in 3.0.
On Fri, Aug 23, 2019 at 8:01 AM Martin Jansa wrote:
> Let me explain a bit what these changes do for us in LGE.
>
> We have jenkins jobs for CI as well for official releases.
>
> All built artifacts are moved from jenkins builder to fileserver after
> the build.
>
> Each jobs have some identifier which is then included in the filenames
> of all relevant build artifacts, e.g. CI jobs will add e.g. lgsvl-verf-12
> to show where it was created and what build created it (12 is
> BUILD_NUMBER from jenkins, verf is type of build, lgsvl is location).
>
> To do this you can already use IMAGE_VERSION_SUFFIX variable and add
> this as a suffix to current artifact names. But that has some bad
> limitations.
>
> A) If you keep IMAGE_VERSION_SUFFIX in do_deploy signatures, then
> the artifacts will be rebuilt even when the deploy sstate archive in
> cache is identical except tha filename. We had this for a while, but
> all CI jobs were slow, because of rebuilding kernel every single time.
>
> B) If you vardepexclude IMAGE_VERSION_SUFFIX from the tasks which use
> it, you get faster CI builds, but with inconsistent artifact names when
> kernel deploy sstate is reused, e.g. image will have lgsvl-verf-12
> but all kernel artifacts will end with lgsvl-verf-11, when the kernel
> do_deploy was reused from previously built sstate-cache
>
> It gets even worse with B) when you have some other tooling (like
> runtime testing farm) which is tasked to flash image and kernel from
> lgsvl-verf-12 and it fails to find kernel image to flash, because it's
> named differently.
>
> C) Using version-less artifacts and just storing them in different
> directories might work better, but then it would make sense to include
> IMAGE_VERSION_SUFFIX in DEPLOY_DIR_IMAGE and remove it from the actual
> files inside (with version-less symlinks pointing to them). But this is
> a bit problematic when the individual images are usually downloaded by
> BFUs over http (and they end with various identically named files for
> which they don't remember from where they came).
>
> So this was the motivation why we have this in webOS.
>
> The difference for you (most people shouldn't even notice):
>
> 1) hard links instead of symlinks in DEPLOY_DIR_IMAGE, because now the
>version is in the *_LINK_* variables and you don't want symlink with
>version release-1 pointing to file created with release-2 build.
>
> 2) do_deploy, do_rootfs can still be reused from sstate, it will restore
>the version-less artifact and then just quicky add another hardlink with
>new filename (do_deploy_links task).
>
> 3) we're using this for couple years (badly hacked into OE, because we
> didn't
>want to overlay all relevant .bbclasses, but still needed to inject
> do_deploy_links
>task dependency in multiple places) and the only issue I've noticed was
> with
>one our proprietary IMAGE_FSTYPE format which was appending to image
> file in
>DEPLOY_DIR_IMAGE if it existed before, instead of overwritting it from
> scratch -
>to fix that I've just changed fstype function to remove the file before
> creating
>it again.
>
> 4) Examples:
>All show "ls -laih DEPLOY_DIR_IMAGE | sort" to show the symlinks and
> hardlinks
>
>TMPDIR is removed between each example, except the e) and f), but
> sstate-cache
>is kept in all cases and reused where possible
>
>MODULE_TARBALL_DEPLOY = "1" is added to local.conf to have more than
> just 1
>kernel image as artifact from kernel (e.g. rpi MACHINEs have a lot of
> them with
>all the dtds).
>
> a) Current master with default values:
>
> $ ls -laih qemux86-64-master-default/ | sort
> 47054849 drwxr-xr-x 13 bitbake bitbake 4.0K Aug 16 16:37 ..
> 50735788 -rw-r--r-- 1 bitbake bitbake 612K Aug 15 17:16
> grub-efi-bootx64.efi
> 50735796 drwxr-xr-x 2 bitbake bitbake 4.0K Aug 16 14:23 .
> 50741892 -rwxr-xr-x 1 bitbake bitbake 95K Aug 15 17:16
> systemd-bootx64.efi
> 52067796 -rw-r--r-- 1 bitbake bitbake 7.6M Aug 16 14:22
> bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin
> 52067797 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22
> bzImage-qemux86-64.bin ->
> bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin
> 52067798 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22 bzImage ->
> bzImage--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.bin
> 52067799 -rw-r--r-- 1 bitbake bitbake 7.0M Aug 16 14:22
> modules--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.tgz
> 52068116 lrwxrwxrwx 1 bitbake bitbake 78 Aug 16 14:22
> modules-qemux86-64.tgz ->
> modules--5.0.19+git2+c2e34d9ab2_00638cdd8f-r0.17-qemux86-64-20190816141126.tgz
> 52068127 -rw-r--r-- 1 bitbake bitbake 1.3K Aug 16 14:23
> core-image-minimal-qemux86-64-20190816141126.qemuboot.conf
> 52068128 lrwxrwxrwx 1 bitbake bi
