Hello, This sprint we worked on a new set of distro-jobs to investigate a new rpm build pipeline. The goal is to unify the review.rdoproject.org rdoinfo, softwarefactory sfinfo and dci dci-config jobs. All these jobs do the same thing: build rpm on new change, enable integration tests, and publish them in the gate pipeline.
Here is the results of this investigation:
# Add a new distroinfo configuration endpoint in managesf resources
The first idea is to remove duplication of information and keeps package
information in the existing resources definition. To do that we plan on
adding a "distroinfo" resource type:
```
resources:
distroinfos:
# the distro name
tdpw:
projects:
# the project to collect package information
- tdpw
distroinfo:
releases:
- name: master
branch: rpm-master
# custom templates for 'source-repositories' configuration
package-configs:
internal-package:
name: "%(project)s"
upstream: https://softwarefactory-project.io/r/tdpw/%(project)s.git
distgit:
https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git
internal-package-with-spec-included:
name: "%(project)s"
upstream: https://softwarefactory-project.io/r/tdpw/%(project)s.git
distgit: https://softwarefactory-project.io/r/tdpw/%(project)s.git
external-package-openstack:
name: "%(project)s"
upstream: https://git.openstack.org/openstack/%(project)s.git
distgit:
https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git
```
Then in the project source-repository list we could have a 'package-config'
option so that the endpoint would be able to generate the distroinfo
file with all the package automatically included, similarly to what we
do for zuul tenant config.
# dnrl-preprare job provided by sfconfig
After deployment, sfconfig configures a dlrn-base abstract job that will
use the dlrn-prepare role with these variables:
```
dlrn_url: https://sftests.com/dlrn/
dlrn_api_url: https://sftests.com/dlrn-worker-api/
dlrn_config_dir: sftests.com/config/dlrn
dlrn_config_url: https://sftests.com/manage/v2/configurations/dlrn
```
The goal of dlrn-prepare is to setup the distroinfo.yml and dlrn.ini on
the test instances.
# dlrn-build and dlrn-publish job pipeline
The distro-jobs repository contains a couple of abstract jobs. The
dlrn-build definition looks like this:
```
- job:
name: dlrn-build
description: Build project's package.
parent: dlrn-base
run: playbooks/dlrn/build.yaml
post-run: playbooks/dlrn/fetch.yaml
provides: dlrn-repo
requires: dlrn-repo
```
It uses the new provides/requires zuul artifact logic. The idea is that
in the case of depends-on, we want to be able to use the rpm built by
the change ahead without rebuild them in each queue item. The enabling
task is:
```
- name: Add intermediary repositories from changes ahead
blockinfile:
path: "{{ dlrn_data }}/{{ dlrn_project }}.cfg"
marker: ''
insertafter: '# SPECULATIVE REPO HERE'
content: |
{% for artifact in zuul.artifacts|default([]) %}{% if artifact.name ==
'dlrn-repo' %}
[{{ artifact.project }}-{{ artifact.change }}-{{ artifact.patchset }}]
name={{ artifact.project }}-{{ artifact.change }}-{{ artifact.patchset }}
enabled=1
baseurl={{ artifact.url }}/current
gpgcheck=0
{% endif %}{% endfor %}
```
# the project interface
The internal details of distro-jobs and configuration are still being
explored, but the end user interface should looks like this:
* Defines the distro information in the resources
* Add dlrn-build and dlrn-publish job to project's pipelines.
Here is an example distribution pipeline configurations:
```
---
# Distro jobs implementation
- job:
name: tdpw-dlrn-build
parent: dlrn-build
vars:
dlrn_project: tdpw
- job:
name: tdpw-dlrn-publish
parent: dlrn-publish
vars:
dlrn_project: tdpw
dlrn_user:
username: tdpw
password: secret
# Distro ci jobs
- job:
name: tdpw-ci-functional
requires: dlrn-repo
run: playbooks/functional.yaml
# Ci pipelines
- project-template:
name: tdpw-jobs
check:
jobs:
- tdpw-dlrn-build
- tdpw-ci-functional:
dependencies:
- tdpw-dlrn-build
gate:
queue: tdpw
jobs:
- tdpw-dlrn-build
- tdpw-ci-functional:
dependencies:
- tdpw-dlrn-build
- wait-for-changes-ahead:
dependencies:
- tdpw-ci-functional
- tdpw-dlrn-publish:
dependencies:
- wait-for-changes-ahead
release:
jobs:
- tdpw-dlrn-publish
```
# Automatic dependencies bump
Time ran out, but it seems like we should also be able to leverage the
git-drivers or the gerrit-drivers to also run the DLRN service inside a
zuul jobs. For example, when a new tag is created in one the upstream
project, a dlrn-propose job could propose a review to bump the version
number automatically.
# Example implementations
We prototyped those changes with the tdpw project
([t]est [d]istribution [p]ackaging [w]orkflow):
The dlrn-prepare job: https://softwarefactory-project.io/r/15206
The distro-jobs: https://softwarefactory-project.io/r/15207
The tdpw ci pipelines: https://softwarefactory-project.io/r/15208
Todos are tracked in the README of distro-jobs.
Regards,
-Tristan
signature.asc
Description: PGP signature
_______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
