On Sun, Apr 19, 2020 at 12:04 PM Richard Purdie
<[email protected]> wrote:
>
> I'm a bit worried about some of the mulitlib issues we've seen
> recently. In particular we have a bug where license exclusion doesn't
> work properly in multilib builds, which is fairly serious.
>
> To dive into the details of what I found, as things stand today, the
> code sequence is:
>
> a) bb.event.RecipePreFinalise
> b) bb.data.expandKeys()
> c) base.bbclass anonymous python
> d) multilib.bbclass anonymous python
>
> The mutlilib code does half the work in a) but delays half to d) since
> it has to happen after python key expansion, there is no way to avoid
> that.
>
> The issue is that there is code in base.bbclass's anonymous python
> (step c) that assumes the package information in the datastore is
> correct, yet it isn't under after d).
>
> For example if you set:
>
> PACKAGES += "X-bar"
> LICENSE_${PN}-bar = "Y"
> (and PN is X)
>
> then base.bbclass can't see the LICENSE value by iterating PACKAGES.
> Yes, you could argue PACKAGES should match the keys used but that is
> hard to enforce and really shouldn't be necessary.
>
> Having debugged this I decided an "easy" fix would be to add a new
> event, bb.event.RecipePostKeyExpansion which happens after key
> expansion and before anonymous python. This updates the flow to:
>
> a) bb.event.RecipePreFinalise
> b) bb.data.expandKeys()
> c) multilib bb.event.RecipePostKeyExpansion
> d) base.bbclass anonymous python
>
> However things break :(.
>
> There is anonymous python that assumes it runs before multilib, e.g.
> mesa so we have to smatter in MLPREFIX references to make the code work
> again as after this change, all anonymous python runs after multilib so
> must use MLPREFIX in changes. Ok, fine, this at least gives us a
> consistent API.
>
> Sadly, things still break. For example:
>
> DEPENDS += "${@bb.utils.contains('GI_DATA_ENABLED', 'True', 'libxslt-native',
> '', d)}"
>
> since this is expanded by the multilib RecipePostKeyExpansion event,
> this happens before DISTRO_FEATURES is manipulated by base.bbclass and
> the value is lost.
>
> PACKAGES += "${@bb.utils.contains('PACKAGECONFIG', 'pulseaudio',
> 'alsa-plugins-pulseaudio-conf', '', d)}"
>
> which also suffered from effectively the same issue.
>
> There were also problems from:
>
> RECOMMENDS_${PN} += "${BOOST_PACKAGES}"
>
> Since BOOST_PACKAGES is unset until after anonymous python runs and the
> new multilib code tried to create packages like:
>
> "lib32-${BOOST_PACKAGES}" which translated to "lib32-lib32-boost-A
> lib32-boost-B".
>
> I'm struggling to decide how best we can fix much of this. One approach
> is to avoid expanding PACKAGES and DEPENDS, instead we can wrap them,
> saving the unexpanded value to some hidden variable and creating a
> function which dynamically expands them on the fly. There are patches
> to do this in master-next. Its rather ugly magic but does appear to
> work, I'm testing on the autobuilder atm just to see if it does fix
> issues.
>
> Its possible we could create a new bitbake API for this with variables
> having "filter" functions, called when the value is returned, I just
> worry about performance overheads.
Continuing the discussion from the OE TSC meeting, now that I've had a
little time to digest the idea: I don't think the filter functions are
necessarily a bad idea. I think the real problem is how to prevent
them from becoming API maintenance and debugging problems, and as you
stated, performance issues. I think there might be a few ways to
address this, and most of it comes down to strictly defining the API.
Firstly, I don't think we want to allow (at least at this point)
completely arbitrary python functions to be called as a filter.
Hopefully, we can figure out the few filter manipulations that are
required and design the API such that those can be used without having
to open it all the way up. Secondly, I think we should assume that any
filter function is "pure" [1], meaning it only takes in the input
variable value string, possibly some other manipulation arguments, and
outputs the new variable value string. Notable, it should *not* take
in the datastore object, since this is inherently global data that can
change at any time. Hopefully, this will allow optimizations and
prevent performance issues because bitbake can keep a cache of filter
function inputs mapped to outputs. Since the function outputs *only*
depend on the inputs, the filter can avoid being called multiple times
if the input set has been seen before.
I can dream up a few ways this might look. In this example, I'm
defining a filter function that simply adds a defined prefix to every
value in a space separated variable.
First, the filter function itself:
def filter_prefix(in, prefix):
return ' '.join(prefix + v for v in in.split())
As for how to express this in a recipe, I can think of a few ways.
This first one is a variable flag. The base flag is "filter" and the
suffix indicates which filter is applied (in this case, the "prefix"
function). The value are the extra arguments passed to the function.
DEPENDS[filter:prefix] = "${MLPREFIX}"
Another option would be to encode the function and arguments in the
flag value, which might also allow filters to be chained (although, I
would hope that's a rare requirement!):
DEPENDS[filter] = "prefix ${MLPREFIX}"
Chaining might look something like:
DEPENDS[filter] = "prefix ${MLPREFIX}; suffix ${MY_SUFFIX}"
The great part about pure functions here is that the property is
transitive, so the entire result of chain with both operations (e.g.
suffix(prefix(${DEPENDS}))) can be cached if desired.
There's probably a few other ways to express this, but the idea is
that it's more tightly controlled about what can be applied.
[1]: https://en.wikipedia.org/wiki/Pure_function
>
> Does anyone have any other ideas or opinions on this?
>
> Cheers,
>
> Richard
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1054):
https://lists.openembedded.org/g/openembedded-architecture/message/1054
Mute This Topic: https://lists.openembedded.org/mt/73131657/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
