Hi everyone,
I've got a proposal for some new functionality in BuildStream. Forgive
me for the long email... I'm told I need to email here before I try
writing any code. So, I'd like to first motivate the new feature, and
then explain what I think it should look like.
# Background
One of my strategic goals for carbonOS (and now GNOME OS) has been to
support device-specific images wherever appropriate. Device-specific
images are, by nature, all very similar to each other: just some extra
packages here, some partition table tweaks there, and so on. Build tools
that deal with this (for example: Android's[1][2], ChromeOS's[3][4], and
postmarketOS's[5][6]) have all converged on similar solutions. They all
have a unified and highly parameterized image build pipeline, and
device-specific configuration files that control the pipeline's numerous
settings.
Buildstream, as it currently is, isn't well suited to create this
optimal image-build pipeline. But why not? Well, composing a collection
of buildstream elements into a single OS image generally requires more
than one element definition: you need at least one `stack` to group up
the packages that'll go into your image, then you need a
`collect-manifest` to generate your SBOM, maybe a couple of `compose`
elements to build your initrd and root directory trees, and finally some
`script`s to shove everything into the final disk images. This is
perfectly fine if you're building one image, but if you want to build a
second device-specific variant you need to copy and then edit these
elements. This quickly becomes unmaintainable. There's no mechanism in
Buildstream to parameterize across element boundaries like we'd need to
here.
My solution for this in carbonOS was a workaround. I added a "code
generation" script [7], that parses a device-specific configuration file
[8] and generates the necessary per-device elements for me. The bash
script was definitely crude, but it did give me the solution I wanted:
For each device, I could write a config file along with some
device-specific package definitions, while reusing the behavior of a
unified image build pipeline.
# Proposed Solution
I would like to introduce a new type of plugin to BuildStream, which I'm
tentatively calling a "generator". Much like an element, each instance
of a generator plugin corresponds to a generator definition file
(.bstg). Much like an element, we decide which plugin to instantiate
from the `kind:` field found in a generator definition file. A
generator's job would be to transform the contents of its definition
file into a dict of `Node`s that can then be parsed into BuildStream
elements as if they were read from disk.
Each entry in the dict maps a filename to the root `Node` of the
corresponding generated element. This allows other elements, from
outside of the generator, to refer to elements created by the generator.
For example, let's say there's a generator defined at
elements/something/demo.bstg, which generates two elements: foo.bst and
bar.bst. Elements anywhere in the project can refer to these generated
elements via something/demo.bstg/foo.bst and something/demo.bstg/bar.bst
respectively. From the perspective of other elements, a generator
appears like a directory that contains normal element definitions inside.
I think the easiest way to explain how this generator system would work
in practice would be via examples. So let's do that. Here's what the
simplest generator plugin might look like (pseudocode):
```
# plugins/generators/basic.py
class BasicGenerator(Generator):
def __init__(self, def: Node):
self.elements = def["elements"]
def generate(self) -> Dict[str, Node]:
return self.elements
```
This generator just lets you define multiple elements in one file. It
does nothing special other than that. A corresponding definition file
might look like this:
```
# elements/demo.bstg
kind: basic
elements:
foo.bst:
kind: meson
[...]
foo-filtered.bst:
kind: filter
build-depends:
- demo.bstg/foo.bst
[...]
```
Of course, the point of these generator plugins would be to do something
more interesting. So, here's an example of what a generator implementing
our device-specific image build pipeline would look like:
```
# elements/boards/microsoft-surface/board.bstg
kind: device
description: Board definition for the Microsoft Surface series of devices
supported_arches:
- x86_64 # Most devices before 2024
- aarch64 # Devices after 2024 switched to Arm
kernel: boards/microsoft-surface/linux-surface.bst
extra_packages:
- boards/microsoft-surface/webcam-firmware.bst
- boards/microsoft-surface/extra-power-profiles.bst
- boards/microsoft-surface/detatchable-keyboard-daemon.bst
- [...]
append_cmdline: >-
intel.iommu=force
qcom.iommu=enable
[...]
```
```
# plugins/generators/device.py
class Device(Generator):
def __init__(self, def: Node):
# Here we read and parse all the settings from the elements
self.arches = def["supported_arches"]
self.kernel = def["kernel"]
self.extra_cmdline = def["append_cmdline"]
self.baseline_packages = def["baseline_packages"] # This gets set
project-wide
self.extra_packages = def["extra_packages"] # This gets set per-image
[...]
def generate(self) -> Dict[str, Node]:
arch_check = [f"arch == {supported}" for supported in
self.arches].join(" || ");
# Here we return all of the elements that I had the "codegen"
script generating before
return {
# The `stack` that pulls in all the dependencies
'pkgs.bst': Node({
'kind': 'stack',
'(?)': [ { arch_check: { '(!)': f"This device family only
supports {self.arches}" } } ]
'depends': self.baseline_packages + self.extra_packages,
}),
# The `compose` that generates /usr, using the stack defined above
'compose.bst': Node({
'kind': 'compose',
'dependencies': [ self._path_to_element('pkgs.bst') ],
[...]
}),
# Kernel, UKI, SBOM, final image build, etc etc etc
[...]
}
```
Here's another interesting example, that's not part of the motivating
use case but is still enabled by this architecture. Right now, it's
pretty painful to split subpackages out of a package (trying to keep
lists dependencies up-to-date, etc). We could use a generator to make
life easier:
```
# elements/sdk-deps/systemd.bstg
kind: subpackage
base:
kind: meson
sources: [...]
[...]
# Dependencies for all subpackages
depends:
- freedesktop-sdk.bst:bootstrap/glibc.bst
# Dependencies needed to build the package. Auto-extended by the
# generator with the runtime dependencies of all subpackages
build-depends:
- [...]
subpackages:
libs:
files:
- '%{usr}/lib/libsystemd.so'
- '%{usr}/lib/libsystemd-shared.so'
bootable:
files:
- '%{usr}/lib/systemd/systemd'
- [...]
depends:
- sdk-deps/systemd.bstg/libs.bst
- freedesktop-sdk.bst:components/kmod.bst
- [...]
initrd: [...]
```
```
# plugins/generators/subpackage.py
class Subpackage(Generator):
def __init__(self, def: Node):
self.base = def["base"]
self.subpackages = def["subpackages"]
def generate(self) -> Dict[str, Node]:
# Extend the split_rules of the base with a new rule for each
subpackage
rules = self.base["public"]["bst"]["split_rules"]
for name, opts in self.subpackages:
rules["__subpackage_" + name] = opts["files"]
self.base["public"]["bst"]["split_rules"] = rules
# Extend the build dependencies of the base w/ runtime deps of
subpackages
deps = self.base["build-depends"]
for name, opts in self.subpackages:
deps += opts["depends"]
# Filter out dependencies on our own subpackages
deps.filter(lambda dep: !dep.starts_with(self._path_to_generator()))
self.base["build-depends"] = deps
out = { 'base.bst': self.base }
for name, opts in self.subpackages:
out[subpackage + '.bst'] = Node({
'kind': 'filter',
'depends': opts["depends"] + self.base["depends"],
'build-depends': [ self._path_to_element('base.bst') ],
'config': {
'include': [ '__subpackage_' + name ],
}
})
return out
```
Anyway, I hope that paints a clear enough picture of how the feature
would work. Please let me know what you think, and what kind of things
I'd need to do before attempting an implementation of this.
Best,
Adrian
---
[1]: https://source.android.com/docs/setup/create/new-device#build-a-product
[2]: (An example Android definition for a real device)
https://android.googlesource.com/device/google/akita/+/refs/heads/main/device-akita.mk
[3]:
https://www.chromium.org/chromium-os/developer-library/guides/chromiumos-board-porting-guide/
[4]: (An example for a real ChromeOS device)
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/refs/heads/main/baseboard-geralt/
and
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/refs/heads/main/overlay-geralt/
[5]: https://wiki.postmarketos.org/wiki/Deviceinfo_reference
[6]: (An example postmarketOS device)
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/tree/master/device/community/device-oneplus-enchilada
[7]: https://gitlab.com/carbonOS/build-meta/-/blob/main/tools/codegen
[8]: https://gitlab.com/carbonOS/build-meta/-/blob/main/boards/README.md
P.S.: I'm sorry if this shows up multiple times on the mailing list.
I've been struggling to get this email posted onto there. I couldn't
find any documentation about the requirements for posting to this list;
this is my third attempt to send, after subscribing and also confirming
my subscription.
