Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
czw., 12 mar 2020 o 19:13 Khem Raj napisał(a): > > > > > > it depends what you mean here. > > > > > > 1. Hosting on git.yoctoproject.org is mostly for layers supported by YP > > > membership, these layers are under the responsibility of the YP TSC [1] > > > 2. Hosting on git.openembedded.org is under the responsibility of the OE > > > TSC [2] > > > > > > > I'm not sure what that means exactly in terms of making a layer > > official. I can only guess that non-members can't make their layers > > part of the project for political reasons. > > I would suggest to send the proposal to oe-tsc mailing list > t...@lists.openembedded.org > Thanks. I'll do this as soon as I have more things to show in this layer. Bartosz -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
On Thu, Mar 12, 2020 at 10:51 AM Bartosz Golaszewski wrote: > > śr., 11 mar 2020 o 14:28 Nicolas Dechesne > napisał(a): > > > > hi Bartosz, > > > > On Wed, Mar 11, 2020 at 9:04 AM Bartosz Golaszewski wrote: > >> > >> wt., 3 mar 2020 o 14:56 Bartosz Golaszewski napisał(a): > >> > > >> > If this is not something that should be part of meta-openembedded - is > >> > there any way to have it hosted at https://git.yoctoproject.org/cgit/ > >> > as an official yocto layer? I'm sorry if this is a dumb question, I > >> > don't know exactly what the policy is for that. > >> > > >> > >> Hi Khem, > >> > >> gentle ping on this. What is the procedure of hosting a layer at OE? > > > > Hi Nicolas, > > thanks for the response. > > > > > it depends what you mean here. > > > > 1. Hosting on git.yoctoproject.org is mostly for layers supported by YP > > membership, these layers are under the responsibility of the YP TSC [1] > > 2. Hosting on git.openembedded.org is under the responsibility of the OE > > TSC [2] > > > > I'm not sure what that means exactly in terms of making a layer > official. I can only guess that non-members can't make their layers > part of the project for political reasons. I would suggest to send the proposal to oe-tsc mailing list t...@lists.openembedded.org > > > We recently started an initiative using gitlab, to host layers: > > https://gitlab.com/openembedded/community, though it's not picking up much > > for now. But that would be another option, Paul B. or myself could give you > > access openembedded gitlab presence as an organization is new and being experimented with eventually it might or might not become more prominent. > > > > Yes, that is one way, but I just don't like githubs, gitlabs and other > bitbuckets. I prefer regular mailing list flow + cgit whenever > possible. > > > You might want to host the layer on your own (github, gitlab or anywhere > > else). I understand why you think that hosting on yoctoproject.org would > > make it 'more' official, however hundreds of layers are 'self hosted' and > > it's usually not a big concern. > > > > They are also scattered all over and significant part of them are not > being actively maintained. Hosting it at some official git would at > least mean that if I ever stop working on it, someone else could take > over without moving the git tree. It would also make it more visible > and make it possible for the development to happen on the official > mailing list. > > Anyway: is the material I described in this thread something that > could be committed to various existing parts of meta-openembedded > without creating a new layer? > > > In any case you should register your layer in the OE layer index. > > > > Sure. > > > > > [1] https://wiki.yoctoproject.org/wiki/TSC > > [2] https://www.openembedded.org/wiki/TSC > > Thanks, > Bartosz Golaszewski -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
śr., 11 mar 2020 o 14:28 Nicolas Dechesne napisał(a): > > hi Bartosz, > > On Wed, Mar 11, 2020 at 9:04 AM Bartosz Golaszewski wrote: >> >> wt., 3 mar 2020 o 14:56 Bartosz Golaszewski napisał(a): >> > >> > If this is not something that should be part of meta-openembedded - is >> > there any way to have it hosted at https://git.yoctoproject.org/cgit/ >> > as an official yocto layer? I'm sorry if this is a dumb question, I >> > don't know exactly what the policy is for that. >> > >> >> Hi Khem, >> >> gentle ping on this. What is the procedure of hosting a layer at OE? > Hi Nicolas, thanks for the response. > > it depends what you mean here. > > 1. Hosting on git.yoctoproject.org is mostly for layers supported by YP > membership, these layers are under the responsibility of the YP TSC [1] > 2. Hosting on git.openembedded.org is under the responsibility of the OE TSC > [2] > I'm not sure what that means exactly in terms of making a layer official. I can only guess that non-members can't make their layers part of the project for political reasons. > We recently started an initiative using gitlab, to host layers: > https://gitlab.com/openembedded/community, though it's not picking up much > for now. But that would be another option, Paul B. or myself could give you > access > Yes, that is one way, but I just don't like githubs, gitlabs and other bitbuckets. I prefer regular mailing list flow + cgit whenever possible. > You might want to host the layer on your own (github, gitlab or anywhere > else). I understand why you think that hosting on yoctoproject.org would make > it 'more' official, however hundreds of layers are 'self hosted' and it's > usually not a big concern. > They are also scattered all over and significant part of them are not being actively maintained. Hosting it at some official git would at least mean that if I ever stop working on it, someone else could take over without moving the git tree. It would also make it more visible and make it possible for the development to happen on the official mailing list. Anyway: is the material I described in this thread something that could be committed to various existing parts of meta-openembedded without creating a new layer? > In any case you should register your layer in the OE layer index. > Sure. > > [1] https://wiki.yoctoproject.org/wiki/TSC > [2] https://www.openembedded.org/wiki/TSC Thanks, Bartosz Golaszewski -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
On Wed, Mar 11, 2020 at 10:28 AM Nicolas Dechesne wrote: > On Wed, Mar 11, 2020 at 9:04 AM Bartosz Golaszewski wrote: >> wt., 3 mar 2020 o 14:56 Bartosz Golaszewski napisał(a): >> > >> > If this is not something that should be part of meta-openembedded - is >> > there any way to have it hosted at https://git.yoctoproject.org/cgit/ >> > as an official yocto layer? I'm sorry if this is a dumb question, I >> > don't know exactly what the policy is for that. >> gentle ping on this. What is the procedure of hosting a layer at OE? > > it depends what you mean here. > > 1. Hosting on git.yoctoproject.org is mostly for layers supported by YP > membership, these layers are under the responsibility of the YP TSC [1] > 2. Hosting on git.openembedded.org is under the responsibility of the OE TSC > [2] > > We recently started an initiative using gitlab, to host layers: > https://gitlab.com/openembedded/community, though it's not picking up much > for now. But that would be another option, Paul B. or myself could give you > access > > You might want to host the layer on your own (github, gitlab or anywhere > else). I understand why you think that hosting on yoctoproject.org would make > it 'more' official, however hundreds of layers are 'self hosted' and it's > usually not a big concern. > > In any case you should register your layer in the OE layer index. > > > [1] https://wiki.yoctoproject.org/wiki/TSC > [2] https://www.openembedded.org/wiki/TSC I've been preferring github over gitlab but either works. I also think we ought to put it under one of those platforms and rely on those for future. -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750 -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
hi Bartosz, On Wed, Mar 11, 2020 at 9:04 AM Bartosz Golaszewski wrote: > wt., 3 mar 2020 o 14:56 Bartosz Golaszewski napisał(a): > > > > If this is not something that should be part of meta-openembedded - is > > there any way to have it hosted at https://git.yoctoproject.org/cgit/ > > as an official yocto layer? I'm sorry if this is a dumb question, I > > don't know exactly what the policy is for that. > > > > Hi Khem, > > gentle ping on this. What is the procedure of hosting a layer at OE? > it depends what you mean here. 1. Hosting on git.yoctoproject.org is mostly for layers supported by YP membership, these layers are under the responsibility of the YP TSC [1] 2. Hosting on git.openembedded.org is under the responsibility of the OE TSC [2] We recently started an initiative using gitlab, to host layers: https://gitlab.com/openembedded/community, though it's not picking up much for now. But that would be another option, Paul B. or myself could give you access You might want to host the layer on your own (github, gitlab or anywhere else). I understand why you think that hosting on yoctoproject.org would make it 'more' official, however hundreds of layers are 'self hosted' and it's usually not a big concern. In any case you should register your layer in the OE layer index. [1] https://wiki.yoctoproject.org/wiki/TSC [2] https://www.openembedded.org/wiki/TSC > Bartosz > -- > ___ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-devel > -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
wt., 3 mar 2020 o 14:56 Bartosz Golaszewski napisał(a): > > If this is not something that should be part of meta-openembedded - is > there any way to have it hosted at https://git.yoctoproject.org/cgit/ > as an official yocto layer? I'm sorry if this is a dumb question, I > don't know exactly what the policy is for that. > Hi Khem, gentle ping on this. What is the procedure of hosting a layer at OE? Bartosz -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
pon., 2 mar 2020 o 19:25 Khem Raj napisał(a):
>
> >
> > Khem, Armin: any thoughts?
>
> there are many ota layers on OE, most of them are self-contained, so a
> question arises, how is this different, somethings here say it could be
> a base layer for all OTAs, which actually seems quite valuable, but it
> has to be such that the existing OTA layers start using pieces from this
No, it's actually the other way around. The existing OTA layers are
focused on supporting specific tools and they usually provide full
support for some reference platforms. However in every custom project
I worked on we needed to extend those layers and those extensions
became quite repetitive, hence the idea for a layer that gathers
common elements but that is independent from project-specific layers.
If anything the goal is to use this *in conjunction* with specialized
OTA layers.
> layer, Other part seems to be that its yet another OTA using binary
> delta update techniques, so in such a case, it should be thought of as
> another OTA and perhaps maintained independendently, if there are
> features which are common across all OTAs we can host them in core or
> meta-oe,
>
No, this is not an OTA on its own and the generation of binary delta
patches is just one of the features.
Let me give you more context on why I started to do this. We've had a
downstream layer for a client's project compatible with thud with a
complete verified boot chain of trust including dm-verity protected
rootfs mounted by an initramfs stored in signed fitImage + OTA
support. Generating this image required us to alter a couple
inter-task dependencies (fitImage needs to wait for initramfs, but
initramfs needs the dm-verity meta-data which needs the rootfs
partition, and then we need to sign the fitImage etc. etc.).
Then some time during warrior development commit 67628ea66b7d
("uboot-sign.bbclass: fix signature and deployment") was merged in
OE-core which caused us a lot of trouble with our dependencies when
trying to update the layer. In order to not make the same mistake
twice - we thought it's best to upstream our development too for
others to use and to be able to object when someone breaks it for us
(I'm mostly a kernel developer and this is how it works in linux, I'm
not sure if it's the same for yocto).
BTW I'm also preparing a series of patches for meta-security with
dm-verity images as part of this effort.
If this is not something that should be part of meta-openembedded - is
there any way to have it hosted at https://git.yoctoproject.org/cgit/
as an official yocto layer? I'm sorry if this is a dumb question, I
don't know exactly what the policy is for that.
Best regards,
Bartosz
--
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
On 3/2/20 9:39 AM, Bartosz Golaszewski wrote: pon., 2 mar 2020 o 12:25 Otavio Salvador napisał(a): On Mon, Mar 2, 2020 at 4:37 AM Bartosz Golaszewski wrote: niedz., 1 mar 2020 o 14:43 Otavio Salvador napisał(a): This single class surely doesn't justify a new layer but I have a bunch of other stuff lined up for upstreaming if this is accepted. This is thematically separate from most of the recipes in meta-oe too. So please give us an idea of what are your plans, so we can understand it better. Sure. Next steps would be: - Adding a class for generating provisioning partition images. Basically allowing to split parts of the rootfs into separate ext4 (or other) image similar to what meta-mender does in its mender-dataimg class for /data but generalized for configurable directories. - Adding an image recipe for a factory reset system, where we would store the provisioning rootfs on a read-only partition together with an initramfs the role of which would be to reflash the A/B partitions to bring the device to a known state, this is something we do a lot in our consulting work. - Adding standardized target-side scripts for applying binary-delta images. This uses the fact that many OTA frameworks support extensions to their client programs. For instance the same script could be used for applying the vcdiff and rsync patches both as a mender update module and a rauc handler (with a thin compatibility layer in their respective OE layers). It still doesn't exhaust the subject but I think this really makes sense in a separate layer than being sprinkled all-over meta-oe. Khem, Armin: any thoughts? there are many ota layers on OE, most of them are self-contained, so a question arises, how is this different, somethings here say it could be a base layer for all OTAs, which actually seems quite valuable, but it has to be such that the existing OTA layers start using pieces from this layer, Other part seems to be that its yet another OTA using binary delta update techniques, so in such a case, it should be thought of as another OTA and perhaps maintained independendently, if there are features which are common across all OTAs we can host them in core or meta-oe, Bart -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
Hello Bartosz, On Mon, Mar 2, 2020 at 2:39 PM Bartosz Golaszewski wrote: > pon., 2 mar 2020 o 12:25 Otavio Salvador > napisał(a): > > > > On Mon, Mar 2, 2020 at 4:37 AM Bartosz Golaszewski wrote: > > > niedz., 1 mar 2020 o 14:43 Otavio Salvador > > > napisał(a): > Khem, Armin: any thoughts? All this sounds great, I just don't see it fitting on meta-oe. I'd like to propose we create an orga for it, on github, and host it there. -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750 -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
pon., 2 mar 2020 o 12:25 Otavio Salvador napisał(a): > > On Mon, Mar 2, 2020 at 4:37 AM Bartosz Golaszewski wrote: > > niedz., 1 mar 2020 o 14:43 Otavio Salvador > > napisał(a): > > This single class surely doesn't justify a new layer but I have a > > bunch of other stuff lined up for upstreaming if this is accepted. > > This is thematically separate from most of the recipes in meta-oe too. > > So please give us an idea of what are your plans, so we can understand > it better. > Sure. Next steps would be: - Adding a class for generating provisioning partition images. Basically allowing to split parts of the rootfs into separate ext4 (or other) image similar to what meta-mender does in its mender-dataimg class for /data but generalized for configurable directories. - Adding an image recipe for a factory reset system, where we would store the provisioning rootfs on a read-only partition together with an initramfs the role of which would be to reflash the A/B partitions to bring the device to a known state, this is something we do a lot in our consulting work. - Adding standardized target-side scripts for applying binary-delta images. This uses the fact that many OTA frameworks support extensions to their client programs. For instance the same script could be used for applying the vcdiff and rsync patches both as a mender update module and a rauc handler (with a thin compatibility layer in their respective OE layers). It still doesn't exhaust the subject but I think this really makes sense in a separate layer than being sprinkled all-over meta-oe. Khem, Armin: any thoughts? Bart -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
On Mon, Mar 2, 2020 at 4:37 AM Bartosz Golaszewski wrote: > niedz., 1 mar 2020 o 14:43 Otavio Salvador > napisał(a): > This single class surely doesn't justify a new layer but I have a > bunch of other stuff lined up for upstreaming if this is accepted. > This is thematically separate from most of the recipes in meta-oe too. So please give us an idea of what are your plans, so we can understand it better. -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750 -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
niedz., 1 mar 2020 o 14:43 Otavio Salvador napisał(a): > > Hello, > > On Fri, Feb 28, 2020 at 12:03 PM Bartosz Golaszewski wrote: > ... > > Over-The-Air updates are a crucial part of IoT systems based on linux. > > There are several OTA update frameworks available and many offer some > > sort of support in yocto (e.g. meta-mender, meta-rauc). There are certain > > operations that are common to all of them such as: generating binary > > delta patches, system recovery, creating provisioning images etc. > > > > This patch proposes to add a new layer in meta-openembedded dedicated to > > OTA. As the first functionality it adds a bbclass for generating binary > > delta images using two popular algorithms - vcdiff and rsync. > > > > Such images can then be easily packaged in update artifacts for different > > OTA frameworks. > > > > Signed-off-by: Bartosz Golaszewski > > I see the value of this, as we are also doing OTA update framework > development, however I wonder if it is worth a new layer for this. For > now, I'd say to put it inside meta-oe directly. > This single class surely doesn't justify a new layer but I have a bunch of other stuff lined up for upstreaming if this is accepted. This is thematically separate from most of the recipes in meta-oe too. Bartosz -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
Re: [oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer
Hello, On Fri, Feb 28, 2020 at 12:03 PM Bartosz Golaszewski wrote: ... > Over-The-Air updates are a crucial part of IoT systems based on linux. > There are several OTA update frameworks available and many offer some > sort of support in yocto (e.g. meta-mender, meta-rauc). There are certain > operations that are common to all of them such as: generating binary > delta patches, system recovery, creating provisioning images etc. > > This patch proposes to add a new layer in meta-openembedded dedicated to > OTA. As the first functionality it adds a bbclass for generating binary > delta images using two popular algorithms - vcdiff and rsync. > > Such images can then be easily packaged in update artifacts for different > OTA frameworks. > > Signed-off-by: Bartosz Golaszewski I see the value of this, as we are also doing OTA update framework development, however I wonder if it is worth a new layer for this. For now, I'd say to put it inside meta-oe directly. -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750 -- ___ Openembedded-devel mailing list Openembedded-devel@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-devel
