Re: [yocto] [OE-core] RFC: Improving the developer workflow
Hi Nicolas, Apologies for the delayed reply. On Friday 08 August 2014 10:04:19 Nicolas Dechesne wrote: On Thu, Aug 7, 2014 at 11:10 AM, Paul Eggleton paul.eggle...@linux.intel.com wrote: Example workflow I won't give a workflow for every possible usage, but just to give a basic example - let's assume you want to build a new piece of software for which you have your own source tree on your machine. The rough set of steps required would be something like this (rough, e.g. the command names given shouldn't be read as final): 1. Install the SDK 2. Run a setup script to make the SDK tools available 3. Add a new recipe with sdktool add recipename - interactive process. The tool records that recipename is being worked on, creates a recipe that can be used to build the software using your external source tree, and places the recipe where it will be used automatically by other steps. 4. Build the recipe with sdktool build recipename. This probably only goes as far as do_install or possibly do_package_qa; in any case the QA process would be less stringent than with the standard build system however in order to avoid putting too many barriers in the way of testing on the target. 5. Fix any failures and repeat from the previous step as necessary. 6. Deploy changes to target with sdktool deploy-target ip address assuming SSH is available on the target. Alternatively sdktool build-image imagename can be used to regenerate an image with the changes in it; sdktool runqemu could do that (if necessary) and then run the result within QEMU with the appropriate options set. coincidentally, i was giving an OE workshop this week, and when I explained about the OE SDK, someone immediately brought up that it was quite limited because: 1- SDK cannot be used to generate deployable packages, e.g. using the SDK to create ipk/deb/rpm that can be delivered to targets/clients, not just for debugging, but also for production when production system has package management support. 2- SDK cannot be used to regenerate updated images. e.g. Company A delivers a SDK + board, Company B is making a product using the SDK (adding content) and wants to be able to make new images with the new content in order to sell/deploy it. I have included solutions for these two in my proposal. However, I am a bit concerned about it being used in the way you describe above. The main aim is to provide tools for developers to test their changes on the target easily; if people start using it to produce images that they then send to other people (particularly customers or other organisations) then the potential end-game is an image used in production is one produced from a single developer's machine, at which point you may have some reproducibility issues - where's the source for package xyz? Erm, not sure, that developer left six months ago... Of course there's nothing we'll really be able to do to stop people from using the tools in this way, so it's mainly going to be an exercise in documenting the right way to use them. It's worth thinking about these problems though especially if you want to encourage best practices. 3- SDK itself cannot be upgraded when the 'base image' and SDK are updated 4- SDK users cannot add content to the SDK. e.g. I am a SDK user I create a library and I want that library to be in the SDK now. We're certainly aiming to take care of these last two - indeed #4 is one of the main drivers behind the proposed new SDK. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [OE-core] RFC: Improving the developer workflow
Hi, Another approach I would like to suggest is the one that CoreOS has in place, that they call fastpatch. https://coreos.com/using-coreos/updates/ Essentially two root partitions, and boot to switch between the new image while leaving the old one intact as the last known good. It avoids the complexity of an overlay, and it fits with the current yocto filesystem image based approach. With this approach, forward/backward application configuration compatibility can be maintained with an extra bbclass to mix in to the recipes of local application configurations based on OS release version, so that they can be updated by the init scripts. Tim. -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Alex J Lennon Sent: vrijdag 8 augustus 2014 17:58 To: Paul Eggleton Cc: yocto@yoctoproject.org; openembedded-c...@lists.openembedded.org Subject: Re: [OE-core] [yocto] RFC: Improving the developer workflow Hi Paul, Personally with how fragile package management can end up being, I'm convinced that full-image updates are the way to go for a lot of cases, but ideally with some intelligence so that you only ship the changes (at a filesystem level rather than a package or file level). This ensures that an upgraded image on one device ends up exactly identical to any other device including a newly deployed one. Of course it does assume that you have a read-only rootfs and keep your configuration data / logs / other writeable data on a separate partition or storage medium. However, beyond improvements to support for having a read-only rootfs we haven't really achieved anything in terms of out- of-the-box support for this, mainly due to lack of resources. However, whilst I haven't had a chance to look at it closely, there has been some work on this within the community: http://sbabic.github.io/swupdate/swupdate.html https://github.com/sbabic/swupdate https://github.com/sbabic/meta-swupdate/ I had a quick look at this. It's interesting. If I am reading this correctly it's based on the old - Bootloader runs Partition A - Update Partition B, set Bootloader to run Partition B - On failure stay on partition A and retry update. - Bootloader runs Partition B - Update Partition A, set Bootloader to run Partition A etc. We've done this type of thing before and it works well. Of course the drawback is the amount of flash you need to achieve it but it is a good robust system. I'd be interested to see how this could work with filesystem deltas say. I don't _think_ that is documented here? ... Thinking a little further what would also really interest me would be to consider using the transactionality of the underlying file-system or block-management layer for the update process. Given nowadays journalling and log-structure file-systems are already designed to fail-back when file/meta-data modifications are interrupted surely we should be able to start a macro-transaction point at the start of the partition update, and if that update doesn't complete with a macro-commit then the f/s layer should be able to automatically roll itself back? Perhaps the same could be done at a block management layer? Cheers, Alex -- ___ Openembedded-core mailing list openembedded-c...@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [OE-core] RFC: Improving the developer workflow
On Thu, Aug 7, 2014 at 11:10 AM, Paul Eggleton paul.eggle...@linux.intel.com wrote: Example workflow I won't give a workflow for every possible usage, but just to give a basic example - let's assume you want to build a new piece of software for which you have your own source tree on your machine. The rough set of steps required would be something like this (rough, e.g. the command names given shouldn't be read as final): 1. Install the SDK 2. Run a setup script to make the SDK tools available 3. Add a new recipe with sdktool add recipename - interactive process. The tool records that recipename is being worked on, creates a recipe that can be used to build the software using your external source tree, and places the recipe where it will be used automatically by other steps. 4. Build the recipe with sdktool build recipename. This probably only goes as far as do_install or possibly do_package_qa; in any case the QA process would be less stringent than with the standard build system however in order to avoid putting too many barriers in the way of testing on the target. 5. Fix any failures and repeat from the previous step as necessary. 6. Deploy changes to target with sdktool deploy-target ip address assuming SSH is available on the target. Alternatively sdktool build-image imagename can be used to regenerate an image with the changes in it; sdktool runqemu could do that (if necessary) and then run the result within QEMU with the appropriate options set. coincidentally, i was giving an OE workshop this week, and when I explained about the OE SDK, someone immediately brought up that it was quite limited because: 1- SDK cannot be used to generate deployable packages, e.g. using the SDK to create ipk/deb/rpm that can be delivered to targets/clients, not just for debugging, but also for production when production system has package management support. 2- SDK cannot be used to regenerate updated images. e.g. Company A delivers a SDK + board, Company B is making a product using the SDK (adding content) and wants to be able to make new images with the new content in order to sell/deploy it. 3- SDK itself cannot be upgraded when the 'base image' and SDK are updated 4- SDK users cannot add content to the SDK. e.g. I am a SDK user I create a library and I want that library to be in the SDK now. -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
