Hi Adrian,
I've recently been experimenting with a similar thing, but with slightly
different goals, but worth covering to highlight differences.
Our usecase is that we have a number of manifests describing a bunch of
artifacts that needed importing, they are in large number of different
formats from other teams and projects, using other build tools. We'd
like to migrate them all over to Buildstream eventually but currently
not possible. In the meantime we want to pull in this information from
all of these different formats.
Our approach to the problem has been to leverage the `kind: junction`
plugin and writing custom buildstream source plugins to parse each of
the formats and translate them to Buildstream elements, essentially
generating a Buildsteam project from it, leveraging the existing
manifest structure. In our case the elements are mainly import ones, but
no reason they couldn't generate buildable elements.
So for example:
Some `manifest.yaml`:
```yaml
packages:
- name: package1
url: https://some_url/package_1.tar.gz
- name: package2
url: https://some_url/package_2.tar.gz
```
subproject.bst
```yaml
kind: junction
sources:
- kind: manifest_yaml_project_generator
url: https://some_url/manifest.yaml
ref: some_sha256_of_the_file
```
The `manifest_project_generator` source plugin would output a
Buildstream project with a simple project.conf and a few files:
packages.bst
```yaml
kind: stack
depends:
- package1.bst
- package2.bst
```
package1.bst
```yaml
kind: import
sources:
- kind: tar
url: https://some_url/package_1.tar.gz
ref: some_sha256_of_the_file
```
package2.bst
```yaml
kind: import
sources:
- kind: tar
url: https://some_url/package_2.tar.gz
ref: some_sha256_of_the_file
```
Then it can be used in other elements:
e.g. Depending on all packages from the subproject:
some_other_element_2.bst
```yaml
kind: manual
build-depends:
- subproject.bst:packages.bst
*snip*
```
or depending on a specific package from the subproject.
some_other_element.bst
```yaml
kind: manual
build-depends:
- subproject.bst:package1.bst
*snip*
```
Something like this approach might provide the functionality you are
after within the existing Buildstream framework?
board.bst
```
kind: junction
sources:
- kind: device
supported_arches:
- x86_64 # Most devices before 2024
- aarch64
kernel: boards/microsoft-surface/linux-surface.bst
extra_packages:
- boards/microsoft-surface/webcam-firmware.bst
- boards/microsoft-surface/extra-power-profiles.bst
- boards/microsoft-surface/detatchable-keyboard-daemon.bst
- [...]
append_cmdline: >-
intel.iommu=force
qcom.iommu=enable
[...]
```
I think the main downside of using junctions for your use-case is the
requirement to depend on elements in the parent project (e.g. the
extra_packages and kernel elements in the example), which would need
some weird parent junction in the subproject. So a more tightly coupled
`generator` plugin element type might be helpful here.
thanks,
Nathan
