Hi all, TLDR: please review https://github.com/apache/buildstream/pull/1896
I've hit this recently in trying to get mirroring to work, but managed to work around it. Now, trying to use Tristan's SourceMirrors work, my workaround is coming back to bite me. So I'm trying to get this resolved once and for all. BuildStream currently imposes that Source.translate_url() or Source.mark_download_url() is called for every URL that the source is going to use at configure time. It will raise an assertion error if either of these methods is called later with a URL that hasn't been seen at configure time. However, we have a few plugins that have a part of the URL in their ref (such as the cargo plugin from buildstream-plugins, and the cpan and pypi plugins from bst-plugins-experimental). For these plugins the URL could change when tracking, which isn't allowed under the current rules. The workaround I used was to only call translate_url on the configured part of the URL then append the ref part of the URL. This worked fine until SourceMirrors, where the source mirror plugin assumes it is getting the whole URL to be able to translate it correctly. So after discussing this with Tristan, we reached the conclusion it would be better to drop the check, but keep the rules in the documentation. I tried to do that in https://github.com/apache/buildstream/pull/1896, please take a look and let me know what you think. The changes that this allows in the plugins are in * https://github.com/apache/buildstream-plugins/pull/60 (cargo) * https://gitlab.com/BuildStream/bst-plugins-experimental/-/merge_requests/282 (pypi and cpan) The plugin MRs are still draft as they still need to be cleaned up, but wanted to get this to you soon. Cheers, Abderrahim
