Hi Tristan, Thanks for this great recap. I updated the related Taiga epic today according to the README TODO list. Then we now have 4 more stories to groom in : https://tree.taiga.io/project/morucci-software-factory/epic/2077
Cheers, Fabien On Wed, Mar 13, 2019 at 8:30 AM Tristan Cacqueray <[email protected]> wrote: > > Hello, > > This sprint we worked on a new set of distro-jobs to investigate a new > rpm build pipeline. The goal is to unify the review.rdoproject.org > rdoinfo, > softwarefactory sfinfo and dci dci-config jobs. All these jobs do the > same thing: build rpm on new change, enable integration tests, and > publish them in the gate pipeline. > > Here is the results of this investigation: > > # Add a new distroinfo configuration endpoint in managesf resources > > The first idea is to remove duplication of information and keeps package > information in the existing resources definition. To do that we plan on > adding a "distroinfo" resource type: > > ``` > resources: > distroinfos: > # the distro name > tdpw: > projects: > # the project to collect package information > - tdpw > distroinfo: > releases: > - name: master > branch: rpm-master > > # custom templates for 'source-repositories' configuration > package-configs: > internal-package: > name: "%(project)s" > upstream: > https://softwarefactory-project.io/r/tdpw/%(project)s.git > distgit: > https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git > internal-package-with-spec-included: > name: "%(project)s" > upstream: > https://softwarefactory-project.io/r/tdpw/%(project)s.git > distgit: > https://softwarefactory-project.io/r/tdpw/%(project)s.git > external-package-openstack: > name: "%(project)s" > upstream: https://git.openstack.org/openstack/%(project)s.git > distgit: > https://softwarefactory-project.io/r/tdpw/%(project)s-distgit.git > ``` > > Then in the project source-repository list we could have a 'package-config' > option so that the endpoint would be able to generate the distroinfo > file with all the package automatically included, similarly to what we > do for zuul tenant config. > > > # dnrl-preprare job provided by sfconfig > > After deployment, sfconfig configures a dlrn-base abstract job that will > use the dlrn-prepare role with these variables: > ``` > dlrn_url: https://sftests.com/dlrn/ > dlrn_api_url: https://sftests.com/dlrn-worker-api/ > dlrn_config_dir: sftests.com/config/dlrn > dlrn_config_url: https://sftests.com/manage/v2/configurations/dlrn > ``` > > The goal of dlrn-prepare is to setup the distroinfo.yml and dlrn.ini on > the test instances. > > > # dlrn-build and dlrn-publish job pipeline > > The distro-jobs repository contains a couple of abstract jobs. The > dlrn-build definition looks like this: > > ``` > - job: > name: dlrn-build > description: Build project's package. > parent: dlrn-base > run: playbooks/dlrn/build.yaml > post-run: playbooks/dlrn/fetch.yaml > provides: dlrn-repo > requires: dlrn-repo > ``` > > It uses the new provides/requires zuul artifact logic. The idea is that > in the case of depends-on, we want to be able to use the rpm built by > the change ahead without rebuild them in each queue item. The enabling > task is: > > ``` > - name: Add intermediary repositories from changes ahead > blockinfile: > path: "{{ dlrn_data }}/{{ dlrn_project }}.cfg" > marker: '' > insertafter: '# SPECULATIVE REPO HERE' > content: | > {% for artifact in zuul.artifacts|default([]) %}{% if artifact.name > == 'dlrn-repo' %} > [{{ artifact.project }}-{{ artifact.change }}-{{ artifact.patchset > }}] > name={{ artifact.project }}-{{ artifact.change }}-{{ > artifact.patchset }} > enabled=1 > baseurl={{ artifact.url }}/current > gpgcheck=0 > > {% endif %}{% endfor %} > ``` > > # the project interface > > The internal details of distro-jobs and configuration are still being > explored, but the end user interface should looks like this: > > * Defines the distro information in the resources > * Add dlrn-build and dlrn-publish job to project's pipelines. > > Here is an example distribution pipeline configurations: > > ``` > --- > # Distro jobs implementation > - job: > name: tdpw-dlrn-build > parent: dlrn-build > vars: > dlrn_project: tdpw > > - job: > name: tdpw-dlrn-publish > parent: dlrn-publish > vars: > dlrn_project: tdpw > dlrn_user: > username: tdpw > password: secret > > # Distro ci jobs > - job: > name: tdpw-ci-functional > requires: dlrn-repo > run: playbooks/functional.yaml > > # Ci pipelines > - project-template: > name: tdpw-jobs > check: > jobs: > - tdpw-dlrn-build > - tdpw-ci-functional: > dependencies: > - tdpw-dlrn-build > gate: > queue: tdpw > jobs: > - tdpw-dlrn-build > - tdpw-ci-functional: > dependencies: > - tdpw-dlrn-build > - wait-for-changes-ahead: > dependencies: > - tdpw-ci-functional > - tdpw-dlrn-publish: > dependencies: > - wait-for-changes-ahead > release: > jobs: > - tdpw-dlrn-publish > ``` > > # Automatic dependencies bump > > Time ran out, but it seems like we should also be able to leverage the > git-drivers or the gerrit-drivers to also run the DLRN service inside a > zuul jobs. For example, when a new tag is created in one the upstream > project, a dlrn-propose job could propose a review to bump the version > number automatically. > > > # Example implementations > > We prototyped those changes with the tdpw project > ([t]est [d]istribution [p]ackaging [w]orkflow): > > The dlrn-prepare job: https://softwarefactory-project.io/r/15206 > The distro-jobs: https://softwarefactory-project.io/r/15207 > The tdpw ci pipelines: https://softwarefactory-project.io/r/15208 > > > Todos are tracked in the README of distro-jobs. > > Regards, > -Tristan > -----BEGIN PGP SIGNATURE----- > > iQEzBAEBCAAdFiEE6xA96LXmnmMcb/F5IrmgXJJcxdgFAlyIsZIACgkQIrmgXJJc > xdg9Zwf/TIzaizv8Of1iEpzul+plR5e6+bzvn+73Bg2B0zL4GewqH4Xix5yCm322 > 8Hnym6GITT2XqoFaEeg5Vvq69AXGJ29YN0p+LaSbKD0txuIA1jUNJ17zEbISV1eC > TSLF7HxcMCH1OvLg46Njudfd71/RjEKkrNQyTUqsEnTUJpIFXhkhHLgZwiApgtvX > OL0uzldgMxL5DfM8mSgE1SsvtKybyJjLba445rXyYIj+KE4Y8aAJSy8kbeV2sPfV > ki9H0attqiMv2WD2I88ODg/J0MNTA7vnLfoQwnxA43fHUw8bctE0qxNSlkJy0/Hh > iLVaK0qHbbSpl3tIFFN8tpKqnd2l/g== > =pDmG > -----END PGP SIGNATURE-----_______________________________________________ > Softwarefactory-dev mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/softwarefactory-dev >
_______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
