On 2020-09-29 21:44, Dennis Kliban wrote:
On Tue, Sep 29, 2020 at 2:26 PM Dennis Kliban <[email protected]
<mailto:[email protected]>> wrote:
On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans
<[email protected] <mailto:[email protected]>> wrote:
On 2020-09-29 18:22, Dennis Kliban wrote:
On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans
<[email protected] <mailto:[email protected]>> wrote:
We are trying to automate our Pulp3 setup the same way as
our Pulp2 setup.
We have the following use cases in our Pulp2 setup:
a) We create mirrors and repositories automatically for
every environment (using puppet and the API). The
repositories are defined in a yaml configuration file
(Puppet Hiera).
- We have several problems for doing this in Pulp3.
It's very complex to automatically create mirrors (ex.
centos-8-appstream) for the development environment (that
are in sync with the upstream repositories). A possible
solution for this problem would be to add a 'latest' flag
to publications and distributions so that they are always
pointing to the latest version of the repository.
- It is possible to create the repositories with
scripts, but that's specific and quite hacky. It's also
not possible to automatically update the publications and
distributions this way. We would need to keep a local
copy of what distribution maps to what repository, but
that's not a good idea. Why keep a local copy of the
mapping if that's present in pulp.
We are already planning to improve this workflow. A single
API call will create a task that will sync a repo, create a
new publication, and update the distribution. We are first
going to introduce the change in pulp_file and then other
plugins will follow. https://pulp.plan.io/issues/7469
This issue seems to be solving my issue.
b) We upload packages from one repository (that is not
served to clients) to another repository based on lists
in yaml (cherry picking), or straight from a (jenkins)
Pipeline. This is done by the pulp-admin commands.
- cherry picking could be done by copying content from
one repository to another using the copy API call, but we
want to automatically expose those changes to the
development environment (promotion can be done as in c).
- Uploading content isn't an issue because this is
possible by converting the pulp-admin commands to api calls.
You want to be able to upload a package and have that
operation create a new publication and update the
distribution. This would be similar to the story that I
linked above for syncing - except for the upload use case.
Am I understanding correctly?
Yes, that's what we want. If we create an rpm itself, we want
that's it directly available in the (development) repository
so we don't have to create a new publication and update the
distribution. For adding it to production, we would promote
the repository as described in c. Same when copying packages
between repositories.
I definitely see the utility in making the content available to
the users right away. Please file a story at
https://pulp.plan.io/issues/new/ so we have this use captured.
This use case should be considered also when implementing #7469.
I've described this problem in the following issue:
https://pulp.plan.io/issues/7626
c) We want to be able to 'promote' a set of repositories
from one environment to another. We are using a script
that is generated by puppet when creating the
repositories in a).
For 'promoting' a set of repositories to one
environment to another environment, this is explained
https://docs.pulpproject.org/pulpcore/workflows/promotion.html.
I don't understand what challenge you are describing in C.
Now we have different repositories for development and
production. We want to be able to easily promote repositories
from development to production (with archiving, the production
repository keeps X older versions of packages), this may be
done with one command in the cli interface or different
commands that can be scripted. This is somewhat explained in
the docs, but it's very high level. Maybe it's a good idea to
add an example in the rpm docs (with the promoting of an
CentOS repository)?
I agree that we need more docs around this. Can you please file an
issue about this also?
There are two ways to accomplish this.
1) Create a distribution for each environment. In this case
whenever you are ready to promote a publication from the Dev
distribution to the Production distribution you just simply look
at which publication is currently assigned to Dev and assign it to
Production.
When you want to roll back to a previous publication you need to
query for a publication that is associated with a specific
repository version (or publication date of creation) that you are
interested in. Then you need to update the distribution with that
publication.
This workflow gets tricky when you need to release a hotfix to
production and skip all the other environments. Since the
repository version history is probably much further along than
Production is right now, you need to find the repository version
that is associated with the current publication that is associated
with the Production distribution. Then you need to use the
'modify' API[0] with the Production repository version as the
base_version and add/remove any packages that you need for
Production. Then you need to create a new publication and
associate it with the Production distribution. Then you will need
to create another repository version using the previous repository
version as the base_version. This will get you back to where you
were before in your Development environment.
2) Create a repository and distribution for each environment. Each
time you want to promote a repository to the Production
environment you need to use the repository 'modify' API[0] to
create a new Production repository version. The base_version would
be the Dev repository version which you want to promote. You would
then create a publication for that new Production repository
version and associate that publication with the Production
distribution.
The rollback process is similar to the above, but the advantage of
this approach is that you have separate repository version history
for each environment. When you need to skip environments and go
straight to Production, you can simply add/remove packages to the
latest Production repository version without having to figure out
which version your Production repository is at.
I just read through our docs and now I am not sure if my explanation
is any better than what we already have there. Please reply with any
questions about the workflow. I am hoping this dialogue will help us
improve the docs.
Your explanation here is much more clear to me than how it's described
in the docs. I think it would be a better idea to add the advantages and
disadvantages to the documentation and maybe add some examples as in the
other workflows.
[0]
https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify
Thanks for all the work
Maarten
_______________________________________________
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
