Re: [OE-core] Automatic prebuilt management
On Fri, 2019-08-30 at 18:27 +0200, Loic Poulain wrote:
> HI Andre,
>
> On Fri, 23 Aug 2019 at 20:11, Andre McCurdy
> wrote:
> > On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <
> > loic.poul...@linaro.org> wrote:
> > > Say a company works with an OE internal tree, with open-source
> > and closed-source packages. The company wants to release the tree
> > so that its customers can fully customize the images/distro. The
> > closed sources being only available internally, the company has to
> > generate and handle prebuilt binaries for proprietary packages.
> > Ideally, with a unified public/internal OE tree (same recipes for
> > internal/public build).
> > >
> > > This question is actually similar to an older thread:
> > > https://marc.info/?l=openembedded-core=146779329804683
> >
> > That very old thread was also referenced recently in a follow up
> > where
> > I shared a later solution, see:
> >
> >
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html
>
> Thanks, I missed that one.
>
> >
> > > I started to work on this and added a 'generate-prebuilt' class
> > which generates a tarball of ${D} in deploy/prebuilts after
> > do_install task. Symmetrically, It's also possible to create an
> > 'install-prebuilt' class which bypasses do_fetch, do_unpack, ...,
> > do_install with noexec flag and instead uncompresses a previously
> > generated prebuilt tarball into ${D} and continue normally. But I
> > would like something smarter, e.g. first trying to check if the
> > SRC_URI is available, if not switching on using the prebuilt
> > package if available (e.g. in a DIR_PREBUILT)...
> > >
> > > Before going further is there already an existing solution for
> > that? do you have any recommendation on the easier/best way to
> > achieve this?
> >
> > I did initially try the approach of having a single recipe which
> > can
> > automatically support both building from source and extracting a
> > prebuilts tar file, but that (for me anyway) turned out to be a
> > dead
> > end. Building from source requires build dependencies and config
> > options but extracting a prebuilt tar file does not, so the two end
> > up
> > sharing very little... so 90% of the recipe ends up being
> > conditional
> > on which mode it's running in. The solution I ended up with (see
> > link
> > above) was for the class which creates the prebuilt tar file to
> > also
> > create a dedicated mini recipe to extract it.
> > From my experience however an equally hard part of the problem is
> > the
> > distribution of prebuilts. It starts off easy (you share a tar file
> > via email or an ftp site and the receiver manually copies to their
> > downloads directory...) but that doesn't scale if you need to make
> > regular updates.
>
> Very good point. maybe having binaries also in a repo could help a
> bit (e.g. using git LFS).
>
> > My solution is discussed a little more in the thread
> > linked to above.
>
> Your solution is interesting and I'll probably jump on the thread
> with more questions.
> Having an upstream solution part of oe-core would be nice though.
We'd be happy to have one, it will take someone to propose something
suitably general purpose that others can use along with tests.
Cheers,
Richard
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Automatic prebuilt management
HI Andre,
On Fri, 23 Aug 2019 at 20:11, Andre McCurdy wrote:
> On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain
> wrote:
> > Say a company works with an OE internal tree, with open-source and
> closed-source packages. The company wants to release the tree so that its
> customers can fully customize the images/distro. The closed sources being
> only available internally, the company has to generate and handle prebuilt
> binaries for proprietary packages. Ideally, with a unified public/internal
> OE tree (same recipes for internal/public build).
> >
> > This question is actually similar to an older thread:
> > https://marc.info/?l=openembedded-core=146779329804683
>
> That very old thread was also referenced recently in a follow up where
> I shared a later solution, see:
>
>
> http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html
Thanks, I missed that one.
>
> > I started to work on this and added a 'generate-prebuilt' class which
> generates a tarball of ${D} in deploy/prebuilts after do_install task.
> Symmetrically, It's also possible to create an 'install-prebuilt' class
> which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and
> instead uncompresses a previously generated prebuilt tarball into ${D} and
> continue normally. But I would like something smarter, e.g. first trying to
> check if the SRC_URI is available, if not switching on using the prebuilt
> package if available (e.g. in a DIR_PREBUILT)...
> >
> > Before going further is there already an existing solution for that? do
> you have any recommendation on the easier/best way to achieve this?
>
> I did initially try the approach of having a single recipe which can
> automatically support both building from source and extracting a
> prebuilts tar file, but that (for me anyway) turned out to be a dead
> end. Building from source requires build dependencies and config
> options but extracting a prebuilt tar file does not, so the two end up
> sharing very little... so 90% of the recipe ends up being conditional
> on which mode it's running in. The solution I ended up with (see link
> above) was for the class which creates the prebuilt tar file to also
> create a dedicated mini recipe to extract it.
>
> From my experience however an equally hard part of the problem is the
> distribution of prebuilts. It starts off easy (you share a tar file
> via email or an ftp site and the receiver manually copies to their
> downloads directory...) but that doesn't scale if you need to make
> regular updates.
Very good point. maybe having binaries also in a repo could help a bit
(e.g. using git LFS).
> My solution is discussed a little more in the thread
> linked to above.
>
Your solution is interesting and I'll probably jump on the thread with more
questions.
Having an upstream solution part of oe-core would be nice though.
Regards,
Loic
That's a very good point
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Automatic prebuilt management
On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain wrote:
> Say a company works with an OE internal tree, with open-source and
> closed-source packages. The company wants to release the tree so that its
> customers can fully customize the images/distro. The closed sources being
> only available internally, the company has to generate and handle prebuilt
> binaries for proprietary packages. Ideally, with a unified public/internal OE
> tree (same recipes for internal/public build).
>
> This question is actually similar to an older thread:
> https://marc.info/?l=openembedded-core=146779329804683
That very old thread was also referenced recently in a follow up where
I shared a later solution, see:
http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html
> I started to work on this and added a 'generate-prebuilt' class which
> generates a tarball of ${D} in deploy/prebuilts after do_install task.
> Symmetrically, It's also possible to create an 'install-prebuilt' class which
> bypasses do_fetch, do_unpack, ..., do_install with noexec flag and instead
> uncompresses a previously generated prebuilt tarball into ${D} and continue
> normally. But I would like something smarter, e.g. first trying to check if
> the SRC_URI is available, if not switching on using the prebuilt package if
> available (e.g. in a DIR_PREBUILT)...
>
> Before going further is there already an existing solution for that? do you
> have any recommendation on the easier/best way to achieve this?
I did initially try the approach of having a single recipe which can
automatically support both building from source and extracting a
prebuilts tar file, but that (for me anyway) turned out to be a dead
end. Building from source requires build dependencies and config
options but extracting a prebuilt tar file does not, so the two end up
sharing very little... so 90% of the recipe ends up being conditional
on which mode it's running in. The solution I ended up with (see link
above) was for the class which creates the prebuilt tar file to also
create a dedicated mini recipe to extract it.
>From my experience however an equally hard part of the problem is the
distribution of prebuilts. It starts off easy (you share a tar file
via email or an ftp site and the receiver manually copies to their
downloads directory...) but that doesn't scale if you need to make
regular updates. My solution is discussed a little more in the thread
linked to above.
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Automatic prebuilt management
On Mon, 5 Aug 2019 at 18:39, chris.lapla...@agilent.com <
chris.lapla...@agilent.com> wrote:
> > But I would like something smarter, e.g. first trying to check if the
> SRC_URI is available, if not switching on using the prebuilt package if
> available (e.g.
> > in a DIR_PREBUILT)...
> >
> > Before going further is there already an existing solution for that? do
> you have any recommendation on the easier/best way to
> > achieve this?
>
> I don't have a "best practice" recommendation to offer. But coincidentally
> I suggested something similar for upstreaming a few weeks ago. It was
> (rightfully) rejected as too "hacky" for inclusion in core code.
> Regardless, here's a bbclass we currently use:
>
> addhandler skip_eventhandler
> python skip_eventhandler() {
> try:
> bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d)
> except bb.fetch2.FetchError:
> raise bb.parse.SkipRecipe("skip-inaccessible: could not access
> upstream repo; check SRC_URI, access rights, and network availability")
> }
> skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise"
>
>
> As was pointed out to me, this is a pretty heavy-handed approach - it can
> fail for any number of reasons, not just the URI actually being
> unavailable. So you may consider a more surgical check, e.g. actually
> trying to fetch the SRC_URI directly using raw GET requests.
>
> To implement the handover, you'd have separate recipes for the
> build-from-source and the prebuilt case which PROVIDE the same package. You
> probably also want to set PREFERRED_PROVIDER in your layer conf to prefer
> the build-from-source recipe.
>
> Alternatively, you can do it all in a single recipe by rewriting the
> SRC_URI from within the event handler. But I think separate recipes is
> nicer.
>
Well, I ended up with the attached class, which generates or install
prebuilt tarball. It's only a draft and I'll come back with a polished
version, but the idea is to look for the related prebuilt package into a
'PREBUILT_SRC_DIR' (if defined) and install it instead of fetching and
compiling from source. This allows having only one recipe. Then a company
just needs to provide closed-source software as a set of prebuilt tarballs
and customer need to point to the prebuilt dir (e.g. in local.conf).
Regards,
Loic
prebuilt.bbclass
Description: Binary data
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Automatic prebuilt management
> But I would like something smarter, e.g. first trying to check if the SRC_URI
> is available, if not switching on using the prebuilt package if available
> (e.g.
> in a DIR_PREBUILT)...
>
> Before going further is there already an existing solution for that? do you
> have any recommendation on the easier/best way to
> achieve this?
I don't have a "best practice" recommendation to offer. But coincidentally I
suggested something similar for upstreaming a few weeks ago. It was
(rightfully) rejected as too "hacky" for inclusion in core code. Regardless,
here's a bbclass we currently use:
addhandler skip_eventhandler
python skip_eventhandler() {
try:
bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d)
except bb.fetch2.FetchError:
raise bb.parse.SkipRecipe("skip-inaccessible: could not access upstream
repo; check SRC_URI, access rights, and network availability")
}
skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise"
As was pointed out to me, this is a pretty heavy-handed approach - it can fail
for any number of reasons, not just the URI actually being unavailable. So you
may consider a more surgical check, e.g. actually trying to fetch the SRC_URI
directly using raw GET requests.
To implement the handover, you'd have separate recipes for the
build-from-source and the prebuilt case which PROVIDE the same package. You
probably also want to set PREFERRED_PROVIDER in your layer conf to prefer the
build-from-source recipe.
Alternatively, you can do it all in a single recipe by rewriting the SRC_URI
from within the event handler. But I think separate recipes is nicer.
Chris
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] Automatic prebuilt management
Hello,
Say a company works with an OE internal tree, with open-source and
closed-source packages. The company wants to release the tree so that its
customers can fully customize the images/distro. The closed sources being
only available internally, the company has to generate and handle prebuilt
binaries for proprietary packages. Ideally, with a unified public/internal
OE tree (same recipes for internal/public build).
This question is actually similar to an older thread:
https://marc.info/?l=openembedded-core=146779329804683
I started to work on this and added a 'generate-prebuilt' class which
generates a tarball of ${D} in deploy/prebuilts after do_install task.
Symmetrically, It's also possible to create an 'install-prebuilt' class
which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and
instead uncompresses a previously generated prebuilt tarball into ${D} and
continue normally. But I would like something smarter, e.g. first trying to
check if the SRC_URI is available, if not switching on using the prebuilt
package if available (e.g. in a DIR_PREBUILT)...
Before going further is there already an existing solution for that? do you
have any recommendation on the easier/best way to achieve this?
Regards,
Loic
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
