On Wed, Mar 13, 2019 at 06:39 Javier Pena wrote: > Wow, this looks great! I've added some comments to one of the reviews, to > clarify some DLRN internal details. A couple more comments inline: > > ----- Original Message ----- >> >> Hello, >> >> This sprint we worked on a new set of distro-jobs to investigate a new >> rpm build pipeline. The goal is to unify the review.rdoproject.org rdoinfo, >> softwarefactory sfinfo and dci dci-config jobs. All these jobs do the >> same thing: build rpm on new change, enable integration tests, and >> publish them in the gate pipeline. >> >> Here is the results of this investigation: >> >> # Add a new distroinfo configuration endpoint in managesf resources >> >> The first idea is to remove duplication of information and keeps package >> information in the existing resources definition. To do that we plan on >> adding a "distroinfo" resource type: >> >> ``` >> resources: >> distroinfos: >> # the distro name >> tdpw: >> projects: >> # the project to collect package information >> - tdpw >> distroinfo: >> releases: >> - name: master >> branch: rpm-master >> >> # custom templates for 'source-repositories' configuration >> package-configs: >> internal-package: >> name: "%(project)s" >> upstream: >> https://softwarefactory-project.io/r/tdpw/%(project)s.git >> distgit: >> https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git >> internal-package-with-spec-included: >> name: "%(project)s" >> upstream: >> https://softwarefactory-project.io/r/tdpw/%(project)s.git >> distgit: >> https://softwarefactory-project.io/r/tdpw/%(project)s.git >> external-package-openstack: >> name: "%(project)s" >> upstream: https://git.openstack.org/openstack/%(project)s.git >> distgit: >> https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git >> ``` >> >> Then in the project source-repository list we could have a 'package-config' >> option so that the endpoint would be able to generate the distroinfo >> file with all the package automatically included, similarly to what we >> do for zuul tenant config. >> > > If I got this correctly, we're splitting the distroinfo information between > some general configuration, and per-package configuration in each package > resource, then generating the distroinfo repo on the fly. Is it correct? >
Yes that's the proposal. The idea is that you would register either an
upstream project, or a distgit project in the resource once with the
correct package-config, and then you could get the complete
distroinfo.yml from managesf.
For example:
```
resources:
project:
rdo:
source-repository:
- openstack/nova-distgit:
package-config: rpmfactory-core
repos:
- openstack/nova:
acl: ...
```
Though we may be able to support external flat-file. In fact we do that
for zuul configuration which is composed from resources/*.yaml and zuul/*.yaml:
https://review.rdoproject.org/manage/v2/configurations/zuul
>>
>> # dnrl-preprare job provided by sfconfig
>>
>> After deployment, sfconfig configures a dlrn-base abstract job that will
>> use the dlrn-prepare role with these variables:
>> ```
>> dlrn_url: https://sftests.com/dlrn/
>> dlrn_api_url: https://sftests.com/dlrn-worker-api/
>> dlrn_config_dir: sftests.com/config/dlrn
>> dlrn_config_url: https://sftests.com/manage/v2/configurations/dlrn
>> ```
>>
>> The goal of dlrn-prepare is to setup the distroinfo.yml and dlrn.ini on
>> the test instances.
>>
>>
>> # dlrn-build and dlrn-publish job pipeline
>>
>> The distro-jobs repository contains a couple of abstract jobs. The
>> dlrn-build definition looks like this:
>>
>> ```
>> - job:
>> name: dlrn-build
>> description: Build project's package.
>> parent: dlrn-base
>> run: playbooks/dlrn/build.yaml
>> post-run: playbooks/dlrn/fetch.yaml
>> provides: dlrn-repo
>> requires: dlrn-repo
>> ```
>>
>> It uses the new provides/requires zuul artifact logic. The idea is that
>> in the case of depends-on, we want to be able to use the rpm built by
>> the change ahead without rebuild them in each queue item. The enabling
>> task is:
>>
>> ```
>> - name: Add intermediary repositories from changes ahead
>> blockinfile:
>> path: "{{ dlrn_data }}/{{ dlrn_project }}.cfg"
>> marker: ''
>> insertafter: '# SPECULATIVE REPO HERE'
>> content: |
>> {% for artifact in zuul.artifacts|default([]) %}{% if artifact.name ==
>> 'dlrn-repo' %}
>> [{{ artifact.project }}-{{ artifact.change }}-{{ artifact.patchset }}]
>> name={{ artifact.project }}-{{ artifact.change }}-{{ artifact.patchset
>> }}
>> enabled=1
>> baseurl={{ artifact.url }}/current
>> gpgcheck=0
>>
>> {% endif %}{% endfor %}
>> ```
>>
>> # the project interface
>>
>> The internal details of distro-jobs and configuration are still being
>> explored, but the end user interface should looks like this:
>>
>> * Defines the distro information in the resources
>> * Add dlrn-build and dlrn-publish job to project's pipelines.
>>
>> Here is an example distribution pipeline configurations:
>>
>> ```
>> ---
>> # Distro jobs implementation
>> - job:
>> name: tdpw-dlrn-build
>> parent: dlrn-build
>> vars:
>> dlrn_project: tdpw
>>
>> - job:
>> name: tdpw-dlrn-publish
>> parent: dlrn-publish
>> vars:
>> dlrn_project: tdpw
>> dlrn_user:
>> username: tdpw
>> password: secret
>>
>> # Distro ci jobs
>> - job:
>> name: tdpw-ci-functional
>> requires: dlrn-repo
>> run: playbooks/functional.yaml
>>
>> # Ci pipelines
>> - project-template:
>> name: tdpw-jobs
>> check:
>> jobs:
>> - tdpw-dlrn-build
>> - tdpw-ci-functional:
>> dependencies:
>> - tdpw-dlrn-build
>> gate:
>> queue: tdpw
>> jobs:
>> - tdpw-dlrn-build
>> - tdpw-ci-functional:
>> dependencies:
>> - tdpw-dlrn-build
>> - wait-for-changes-ahead:
>> dependencies:
>> - tdpw-ci-functional
>> - tdpw-dlrn-publish:
>> dependencies:
>> - wait-for-changes-ahead
>> release:
>> jobs:
>> - tdpw-dlrn-publish
>> ```
>>
>> # Automatic dependencies bump
>>
>> Time ran out, but it seems like we should also be able to leverage the
>> git-drivers or the gerrit-drivers to also run the DLRN service inside a
>> zuul jobs. For example, when a new tag is created in one the upstream
>> project, a dlrn-propose job could propose a review to bump the version
>> number automatically.
>>
>
> This bump should happen on the distroinfo information (so the resources
> part). The DLRN job could be used to test it, then a post-merge job would be
> in charge of the final build.
>
The use-case we had in mind was for common requirements where we want to
actually bump the distgit based on new tags (with restriction. For
example, the ansible requirements for zuul would be any tag < 2.16).
>>
>> # Example implementations
>>
>> We prototyped those changes with the tdpw project
>> ([t]est [d]istribution [p]ackaging [w]orkflow):
>>
>> The dlrn-prepare job: https://softwarefactory-project.io/r/15206
>> The distro-jobs: https://softwarefactory-project.io/r/15207
>> The tdpw ci pipelines: https://softwarefactory-project.io/r/15208
>>
>>
>> Todos are tracked in the README of distro-jobs.
>>
>> Regards,
>> -Tristan
>>
>
> Great! Feel free to add me to the loop if you need more info on the DLRN
> internals.
>
That's good to hear, we actually might need some help to finalize this
properly :)
Regards,
-Tristan
signature.asc
Description: PGP signature
_______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
