Re: [OE-core] [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig
On Feb 21, 2019, at 18:51, Richard Purdie wrote: > > Multiconfig is meant to support this workflow. Unfortunately there are > open bugs and people haven't the time to work on it so its stalled. > That said, the key pieces are already there. > > A picture is worth a thousand words. I thought a demo might interest > people and "prove" this can work. > > To make it work you need four patches: > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=b896b2238c100d486e7c43992b49001150749d04 > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=e5ef6b008e85f42b5e4ffd72fae7036dd68b2ea5 > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=5d2bd43d14315ef3212e0d09a16446bd74305a2f > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=abdb83576b699448b76293294809aaa04b8e853b > > The first two are fixes from Alejandro which haven't merged as they're > not right. The third patch was me hitting the fixes and bitbake to make > the demo work. That proves they're not right and hints at where we have > some problems in bitbake too. We can get those issues sorted fairly > easily and I'm going to block 2.7 M3 on them. > > The final patch is the interesting one. > > It adds a layer, meta-multiconfig-example layer with a README about how > to configure it. Basically you create two multiconfig configurations, > "musl" and "tiny". > > The demo is to run "bitbake core-image-full-cmdline". > > The result is a core-image-full-cmdline image with a tiny qemux86 image > and a musl based qemux86-64 image installed into /var/lib/machines/ > > The way it does this is creative, having a recipe called image-packer > which creates virtclass variants of itself which depend on the > appropriate multiconfig. It pulls in the images and creates a package > containing them which the bbappend to core-image-full-cmdline can then > install. It'd be very easy to turn this into a list driven piece of > code rather than the two values currently listed. > > Its late here, some of the above code is horrible. I have rushed this > but hopefully it shows this is possible! :) > > I've love to turn this into an oe-selftest, that is my ultimate > intention for this code. > > As things stand today you can't query one multiconfig from another. As > this demo shows, that doesn't need to stop you making interesting > things though. We will do something with such querying eventually to > make things easier but until then, much is still possible. For search engines: the multiconfig demo from this thread is now is an oe-selftest: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-selftest/recipes-test/multiconfig/multiconfig-image-packager_0.1.bb https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/multiconfig.py Are there other public examples of combining artifacts from different machines into a single image via multiconfig? We can include them in a doc update for Dunfell. A list of use cases could motivate more people to try multiconfig. For developer workflows, multiconfig can accelerate multi-image builds (more parallelism) but slow single-package builds (more recipe parsing). As a workaround, BBMULTICONFIG="" restores the speed of building a target for a single MACHINE+DISTRO config. Is there a better way to disable multiconfig for a single bitbake invocation? Bitbake currently requires a MACHINE to be specified for multiconfig builds. It also accepts non-multiconfig target names, even when BBMULTICONFIG is defined. From a usability point of view, this CLI syntax could distinguish the two workflows: standard build: MACHINE=foo bitbake singleconfig-image multiconfig build: bitbake mc::multiconfig-image Rich-- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig
On Thu, 21 Mar 2019 at 06:48, Rich Persaud wrote: > Is it possible to avoid duplicate builds of packages which are used in > multiple VM images? Could multiconfig replace a dozen serial instances of > bitbake with a single parsing instance, or would it use a dozen parallel > instances? If the latter, is this a constraint in bitbake architecture, > recipe DSL, or time/resources? If the packages are the same and sstate cache would be used, then you can build one on its own first to populate the cache and then build the rest using the sstate. Currently if the sstate isn't present at startup bitbake schedules the same build twice, which obviously isn't ideal but also *should* be fixable with enough effort. Ross -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig
On Thu, 2019-02-21 at 20:09 -0500, Rich Persaud wrote: > > On Feb 21, 2019, at 18:51, Richard Purdie < > > richard.pur...@linuxfoundation.org> wrote: > > > > Multiconfig is meant to support this workflow. Unfortunately there > > are > > open bugs and people haven't the time to work on it so its stalled. > > That said, the key pieces are already there. > > > > A picture is worth a thousand words. I thought a demo might > > interest > > people and "prove" this can work. > > Thank you for the patches and PoC. We will test and are motivated to > see this work. Just to follow up, I've cleaned up the patches, they're now just fixes to bitbake and are done generically so we shouldn't have to patch the various BB_TASKDEPDATA users. They're on the bitbake-devel list and queued in master-next. This should mean multiconfig looks better in 2.7. I'm going to try and get a patch to add some tests/example too. Could anyone who's previously reported issues (Armin?) retest with these and confirm the issues seen have been fixed please? Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig
On Thu, 2019-02-21 at 20:09 -0500, Rich Persaud wrote: > > On Feb 21, 2019, at 18:51, Richard Purdie < > > richard.pur...@linuxfoundation.org> wrote: > > > > Multiconfig is meant to support this workflow. Unfortunately there > > are > > open bugs and people haven't the time to work on it so its stalled. > > That said, the key pieces are already there. > > > > A picture is worth a thousand words. I thought a demo might > > interest > > people and "prove" this can work. > > Thank you for the patches and PoC. We will test and are motivated to > see this work. I've been thinking about what Alejandro and I discussed and I think the correct fix for the multiconfig code is to filter the BB_TASKDEPDATA at source in bitbake to remove the multiconfig entries for other multiconfigs. I'll try and work up a patch for that in the next day or two as the hacks on my branch that do this are fragile right now. > I will also be experimenting with the generation of "immutable > infrastructure" from source code, to learn whether a single Packer.io > template can be used to both: > > (a) modify binary images, in the traditional Packer model, e.g. start > a bitbake-generated image in a VM or container, make runtime > configuration changes, then repack the machine image > > (b) perform automated customization of _inputs_ to bitbake recipes > and package source code, such that a machine image built by bitbake + > Packer template is equivalent to the image generated by Packer at > runtime > > If feasible, this would combine the agility of image customizations > with the security of bitbake's reproducible builds and supply chain > integrity. (a) definitely sounds easier than (b). I've not used packer.io but I'd be curious to see how that works out. People do find our image customisation hard to get to grips with :( Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig
Multiconfig is meant to support this workflow. Unfortunately there are open bugs and people haven't the time to work on it so its stalled. That said, the key pieces are already there. A picture is worth a thousand words. I thought a demo might interest people and "prove" this can work. To make it work you need four patches: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=b896b2238c100d486e7c43992b49001150749d04 http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=e5ef6b008e85f42b5e4ffd72fae7036dd68b2ea5 http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=5d2bd43d14315ef3212e0d09a16446bd74305a2f http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222=abdb83576b699448b76293294809aaa04b8e853b The first two are fixes from Alejandro which haven't merged as they're not right. The third patch was me hitting the fixes and bitbake to make the demo work. That proves they're not right and hints at where we have some problems in bitbake too. We can get those issues sorted fairly easily and I'm going to block 2.7 M3 on them. The final patch is the interesting one. It adds a layer, meta-multiconfig-example layer with a README about how to configure it. Basically you create two multiconfig configurations, "musl" and "tiny". The demo is to run "bitbake core-image-full-cmdline". The result is a core-image-full-cmdline image with a tiny qemux86 image and a musl based qemux86-64 image installed into /var/lib/machines/ The way it does this is creative, having a recipe called image-packer which creates virtclass variants of itself which depend on the appropriate multiconfig. It pulls in the images and creates a package containing them which the bbappend to core-image-full-cmdline can then install. It'd be very easy to turn this into a list driven piece of code rather than the two values currently listed. Its late here, some of the above code is horrible. I have rushed this but hopefully it shows this is possible! :) I've love to turn this into an oe-selftest, that is my ultimate intention for this code. As things stand today you can't query one multiconfig from another. As this demo shows, that doesn't need to stop you making interesting things though. We will do something with such querying eventually to make things easier but until then, much is still possible. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig
On 2/21/19 9:44 AM, Burton, Ross wrote: > So I'd like to use multiconfig for a fairly typical host/container use > case. I've a class which can be used to build packages from images in > a fairly simple manner, essential just taking files from DEPLOY_DIR > when creating a package: > > core-image-minimal-package.bb: > do_install[depends] += "core-image-minimal:do_image_complete > virtual/kernel:do_deploy" > do_install () { > install -d ${D}/var/lib/machines/ > install ${DEPLOY_DIR_IMAGE}/core-image-minimal-${MACHINE}.ext4 > ${D}/var/lib/machines/core-image-minimal.ext4 > install ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE} > ${D}/var/lib/machines/${IMAGE_NAME}.${KERNEL_IMAGETYPE} > } > > This part works fine, but the problem is straddling multiconfig boundaries... > > I can't just add "core-image-minimal-package" to IMAGE_INSTALL because > it will build that for the current configuration. I can't just add > "multiconfig:uos:core-image-minimal-package" because that will look > for a recipe with that name. I've a rather ugly hack to call opkg > directly in a post-rootfs hook which sort of works. Linking in bits > of the other configuration's deploy directory in a pre-rootfs hook and > extending IMAGE_INSTALL there almost works, apart from the rootfs code > then looks at the task dependencies to generate a pruned deploy dir, > which is missing the pieces I linked in. > > I suspect this is now breaking new ground: the multiconfiguration > support lets you do builds but so far the artefact sharing hasn't > really been explored. Has anyone dug further into this? I don't think anyone has. But as we continue moving toward containers/micro services environments -- this exactly type of thing is going to be more and more needed. One 'build', with various configurations that end up constructing a single device image with a container runtime (booting OS), and a number of container micro services should be able to be created with a single call to bitbake and (a potentially complex) config. I think the first question is, can one multiconfig require a dependency out of another yet? I thought this was still unimplemented. Once this can be done, then it should be much easier to have a 'configuration' whose job it is to assemble the various components while waiting for everything to finish being built. --Mark > Ross > ___ > Openembedded-architecture mailing list > openembedded-architect...@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-architecture > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core