On Tue, Sep 29, 2020 at 2:26 PM Dennis Kliban <dkli...@redhat.com> wrote:
> On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans <maart...@inuits.eu> > wrote: > >> On 2020-09-29 18:22, Dennis Kliban wrote: >> >> On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans <maart...@inuits.eu> >> wrote: >> >>> We are trying to automate our Pulp3 setup the same way as our Pulp2 >>> setup. >>> >>> We have the following use cases in our Pulp2 setup: >>> >>> a) We create mirrors and repositories automatically for every >>> environment (using puppet and the API). The repositories are defined in a >>> yaml configuration file (Puppet Hiera). >>> - We have several problems for doing this in Pulp3. It's very complex >>> to automatically create mirrors (ex. centos-8-appstream) for the >>> development environment (that are in sync with the upstream repositories). >>> A possible solution for this problem would be to add a 'latest' flag to >>> publications and distributions so that they are always pointing to the >>> latest version of the repository. >>> - It is possible to create the repositories with scripts, but that's >>> specific and quite hacky. It's also not possible to automatically update >>> the publications and distributions this way. We would need to keep a local >>> copy of what distribution maps to what repository, but that's not a good >>> idea. Why keep a local copy of the mapping if that's present in pulp. >>> >> >> We are already planning to improve this workflow. A single API call will >> create a task that will sync a repo, create a new publication, and update >> the distribution. We are first going to introduce the change in pulp_file >> and then other plugins will follow. https://pulp.plan.io/issues/7469 >> >> This issue seems to be solving my issue. >> >> >> >>> b) We upload packages from one repository (that is not served to >>> clients) to another repository based on lists in yaml (cherry picking), or >>> straight from a (jenkins) Pipeline. This is done by the pulp-admin commands. >>> - cherry picking could be done by copying content from one repository >>> to another using the copy API call, but we want to automatically expose >>> those changes to the development environment (promotion can be done as in >>> c). >>> - Uploading content isn't an issue because this is possible by >>> converting the pulp-admin commands to api calls. >>> >> >> You want to be able to upload a package and have that operation create a >> new publication and update the distribution. This would be similar to the >> story that I linked above for syncing - except for the upload use case. Am >> I understanding correctly? >> >> Yes, that's what we want. If we create an rpm itself, we want that's it >> directly available in the (development) repository so we don't have to >> create a new publication and update the distribution. For adding it to >> production, we would promote the repository as described in c. Same when >> copying packages between repositories. >> > > I definitely see the utility in making the content available to the users > right away. Please file a story at https://pulp.plan.io/issues/new/ so we > have this use captured. This use case should be considered also when > implementing #7469. > > >> >> >>> c) We want to be able to 'promote' a set of repositories from one >>> environment to another. We are using a script that is generated by puppet >>> when creating the repositories in a). >>> For 'promoting' a set of repositories to one environment to another >>> environment, this is explained >>> https://docs.pulpproject.org/pulpcore/workflows/promotion.html. >>> >>> >> I don't understand what challenge you are describing in C. >> >> Now we have different repositories for development and production. We >> want to be able to easily promote repositories from development to >> production (with archiving, the production repository keeps X older >> versions of packages), this may be done with one command in the cli >> interface or different commands that can be scripted. This is somewhat >> explained in the docs, but it's very high level. Maybe it's a good idea to >> add an example in the rpm docs (with the promoting of an CentOS repository)? >> > > I agree that we need more docs around this. Can you please file an issue > about this also? > > There are two ways to accomplish this. > > 1) Create a distribution for each environment. In this case whenever you > are ready to promote a publication from the Dev distribution to the > Production distribution you just simply look at which publication is > currently assigned to Dev and assign it to Production. > > When you want to roll back to a previous publication you need to query for > a publication that is associated with a specific repository version (or > publication date of creation) that you are interested in. Then you need to > update the distribution with that publication. > > This workflow gets tricky when you need to release a hotfix to production > and skip all the other environments. Since the repository version history > is probably much further along than Production is right now, you need to > find the repository version that is associated with the current publication > that is associated with the Production distribution. Then you need to use > the 'modify' API[0] with the Production repository version as the > base_version and add/remove any packages that you need for Production. Then > you need to create a new publication and associate it with the Production > distribution. Then you will need to create another repository version using > the previous repository version as the base_version. This will get you back > to where you were before in your Development environment. > > 2) Create a repository and distribution for each environment. Each time > you want to promote a repository to the Production environment you need to > use the repository 'modify' API[0] to create a new Production repository > version. The base_version would be the Dev repository version which you > want to promote. You would then create a publication for that new > Production repository version and associate that publication with the > Production distribution. > > The rollback process is similar to the above, but the advantage of this > approach is that you have separate repository version history for each > environment. When you need to skip environments and go straight to > Production, you can simply add/remove packages to the latest Production > repository version without having to figure out which version your > Production repository is at. > > > I just read through our docs and now I am not sure if my explanation is any better than what we already have there. Please reply with any questions about the workflow. I am hoping this dialogue will help us improve the docs. > [0] > https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify > > >> >> >>> Thanks for all the work >>> >>> Maarten >>> _______________________________________________ >>> Pulp-list mailing list >>> Pulp-list@redhat.com >>> https://www.redhat.com/mailman/listinfo/pulp-list >> >> -- >> (o- Maarten Beeckmans >> //\ Open Source Consultant >> V_/_ Inuits - https://www.inuits.eu >> >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-list > >
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list