Hi all,
After some thought, here is a drastically different proposal.
The rationale for this 180, is that it is a "must have" to support
mirroring of sources which are not under your control, i.e. one must be
able to reliably mirror sources for your subprojects.
As such, we cannot ask for element YAML participation in URI
resolution, as upstream project maintainers have no knowledge or
requirement to adhere to your downstream mirroring solution.
Provided below, is a completely new proposal which takes this into
account.
Cheers,
-Tristan
Extended project.conf format
----------------------------
The project.conf currently allows specifying mirrors only for the
current project, here the plan is to extend it in two ways:
o When specifying a mirror, we add the ability for that mirror to
apply to URI aliases from subprojects.
The mirrors defined in parent projects would take precedence over
mirrors defined in subprojects.
o When specifying a mirror, we add the ability to specify an optional
plugin for the mirror.
o In the case that a plugin is specified, it may be consulted even
if there is no mapping URI specified for a given project/alias
tuple.
This renders all of the projects/alias mappings optional, and
may support sparse configuration.
The current behavior is to be implemented as the default plugin, and
will not produce any results for any (project_name, source_alias)
tuples which are not accounted for in the mirror data.
Example:
mirrors:
- name: ponyland
# [ADDED] Optional plugin & config
kind: custom
config:
animal: pony
# [ADDED] Optional project / alias mappings
projects:
project-name:
aliases:
foo:
- https://www.ponyland.com/foo/1
- https://www.ponyland.com/foo/2
bar:
- https://www.ponyland.com/bar/1
- https://www.ponyland.com/bar/2
# Aliases to use for this project
aliases:
foo:
- http://www.ponyland.com/foo/1
- http://www.ponyland.com/foo/2
bar:
- http://www.ponyland.com/bar/1
- http://www.ponyland.com/bar/2
New kind of plugin
------------------
This proposal would add a new `SourceMirror` kind of plugin, whos only
purpose is to produce results for `Source.translate_url()`[0]
This new plugin would derive from the base `Plugin`, and support all
the same mechanics for plugin loading (local, pip or junction)[1].
Plugin level configuration, similar to elements, can be provided in the
`config` dictionary of the mirror declaration in project.conf.
This new plugin would respond to the following API:
# translate_url()
#
# Provide a URI translation for the provided URL.
#
# Note that the "alias_url" may be None in the case that no
# mapping was specified for the given URL.
#
# Args:
# url: The aliased URL provided by the Source
# project_name: The name of the project of the calling Source
# alias_name: The leading alias name in the url
# alias_url: The alternative URL string provided in the mirror
# configuration
#
# Returns:
# The fully qualified URL, or None if one could not be resolved.
#
def translate_url(
self,
url: str,
project_name: str,
alias_name: str,
alias_url: Optional[str],
) -> Optional[str]
Extended buildstream.conf format
--------------------------------
In this proposal so far, the project.conf can by itself extend mirrors
of subprojects, possibly using plugin backed mirrors, as such it does
not completely address policy.
As such I suggest we add a new option to the fetch and track sources[2]
[3] which define policy around which sources are trusted to perform
fetch and track operations.
The current values for the fetch/track `source` configuration include:
o all: Fetch from mirrors defined in user configuration, project
configuration, and default alias URIs.
o aliases: Only allow fetching from project defined default alias
URIs.
o mirrors: Only allow fetching from mirrors defined in user
configuration and project configuration
o user: Only allow fetching from mirrors defined in user
configuration
Among these, the "user" policy is the most strict, as it allows the
user to constrain BuildStream to only ever consult the URIs which the
user has explicitly provided.
Since this proposal does not provide for using SourceMirror plugins in
user configuration, I propose we a new value here:
o toplevel-mirrors: Only allow fetching from mirrors defined in the
toplevel project configuration
SourceMirror users must own projects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As far as I can see, this approach has the main limitation that
SourceMirror plugins cannot be used without creating your own project.
Supporting plugin loading in buildstream.conf without a supporting
project would be very awkward and I don't think this is a worthwhile
endeavor.
This however seems to be a minor concern, given that:
o If you are mirroring the sources of an upstream BuildStream
project, it is a reasonable expectation that you already have a
toplevel project which is junctioning that upstream project.
o If for some reason you are not maintaining your own BuildStream
project and you still want to mirror sources, the cost of creating
a skeleton project.conf with a junction to that external project
is very low.
[0]:
https://docs.buildstream.build/2.0/buildstream.source.html#buildstream.source.Source.translate_url
[1]: https://docs.buildstream.build/2.0/format_project.html#loading-plugins
[2]: https://docs.buildstream.build/2.0/using_config.html#fetch-controls
[3]: https://docs.buildstream.build/2.0/using_config.html#track-controls
[4]: https://docs.buildstream.build/2.0/format_project.html#duplicate-junctions
[5]: https://docs.buildstream.build/2.0/format_project.html#internal-junctions
