[Pulp-dev] pulpcore 3.9.1 is generally available

[Dennis Kliban]

Pulpcore 3.9.1 has been released to PyPI[0]. Client libraries are available
on PyPI[1] and rubygems.org[2]. This release addresses a single bug that
was encountered by users that set a custom CHUNKED_UPLOAD_DIR setting. The
value for this setting must now be a relative path inside MEDIA_ROOT[3].

# Installation and Upgrade

Users should use the 3.9.1 release of pulp_installer[4] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.9. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

[0] https://pypi.org/project/pulpcore/3.9.1/
[1] https://pypi.org/project/pulpcore-client/3.9.1/
[2] https://rubygems.org/gems/pulpcore_client/versions/3.9.1
[3] https://docs.pulpproject.org/pulpcore/en/3.9.1/changes.html#id1
[4] https://galaxy.ansible.com/pulp/pulp_installer
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Backports and LTS

[Dennis Kliban]

+1 to fully automating the release pipeline. This would allow us to do as
many releases as needed to satisfy all stakeholders.

On Tue, Jan 12, 2021 at 4:11 PM David Davis  wrote:

> There are some great features that have been added to Github Actions--one
> of them being manual triggers for workflows (before we weren't sure how we
> could trigger the entire release process). So I think we're in a good
> position now that we're on Github Actions to automate the rest of the
> release process.
>
> The problem is finding the time. I think we're all spread thin and we're
> talking about a week or two...? I would love to see this happen though as I
> think it would pay for itself after a couple releases.
>
> David
>
>
> On Tue, Jan 12, 2021 at 3:48 PM Brian Bouterse 
> wrote:
>
>> Stakeholder's won't agree on a single "LTS" (agreed daniel that's not a
>> great name for it) so I think option 3 is a non-starter. Option 2 is what
>> we do today and I agree it has the issues already mentioned.
>>
>> So +1 to option 1, but as already mentioned, practically speaking, I
>> believe we can't release any more than we are without fully automating the
>> process. Can we fully automate the release process first, and then
>> re-discuss a policy change after?
>>
>>
>>
>> On Wed, Jan 6, 2021 at 12:12 PM Daniel Alley  wrote:
>>
>>> Matt, that is a good observation and meanwhile with pulp2 we had such a
 policy, we cannot adopt it with pulp3. Release fast and release often is
 something we are trying to stick to.

>>>
>>> I don't think Matt was suggesting in any way that we not release fast
>>> and often, it's just that we have to pick a combination of cadance and # of
>>> supported releases that works for everyone.  This is basically Option 1
>>> with a pre-determined number of supported releases (rather than "backport
>>> to whatever version the stakeholder is using + all the others in between"
>>> which is how I interpret Option 1).
>>>
>>> I totally agree with David that it would be a pain to manage without
>>> automation though, same as Option 1.
>>>
>>> I think what would most likely happen under that plan is that we would
>>> see each stakeholder will take a release, stick to it until it is about to
>>> be dropped and then upgrade to the newest one, which is roughly the same as
>>> the LTS plan except that each stakeholder can choose their own "LTS"
>>> version and they could do an mid-cycle upgrade if there was ever a good
>>> reason to.
>>>
>>> Whatever option we choose, I'm kind of negative about actually using the
>>> term "LTS" given that I don't think we'd be supporting any version for more
>>> than 5-7 months.  The timeframe is a bit short for how the LTS moniker is
>>> typically used.
>>>
>>> On Wed, Jan 6, 2021 at 8:58 AM Ina Panova  wrote:
>>>
 Matt, that is a good observation and meanwhile with pulp2 we had such a
 policy, we cannot adopt it with pulp3. Release fast and release often is
 something we are trying to stick to.

 It would be best to agree on the LTS version that would make all the
 stakeholders happy but as it was pointed out not sure how we'd make this
 possible, everyone has a different pace and release cycle.
 Having that said, we should probably stick to option1 which puts more
 burden on us but in return users and stakeholders are happy and the
 project's upgrade path is stable.
 One thing we could possibly do about the backport requests is to really
 be thorough about which ones to accept by assessing the impact of the
 stakeholder who has created the request and their possibility to upgrade if
 any.
 
 Regards,

 Ina Panova
 Senior Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Wed, Jan 6, 2021 at 2:18 PM Matt Pusateri 
 wrote:

> Another option is Current Version minus X (N-1, N-2, etc.) If for
> instance you say we support N-1, and current version is 3.9, then you'll
> back port to latest 3.8.x.  N-2 at current version 3.9, would backport to
> 3.8.x, 3.7.x, etc.  The advantages here are that you already set the
> expectation for everyone, you limit the versions you support, you force
> people to stay current to get fixes.  The downside is that people have to
> upgrade and or it could impact downstream schedules.  The impact with 
> going
> this route is affected by the release velocity.  So if you're rapidly
> releasing major versions, then it's more likely people won't keep up, or
> that they'll have to upgrade and are not able to for some reason.
>
> Matt P.
>
>
> On Wed, Jan 6, 2021 at 6:05 AM Matthias Dellweg 
> wrote:
>
>> Thanks for putting all the options together.
>> I would say that option 2 is a recipe for very bad user experience,
>> and i'd vote strongly against it.
>> I am 

[Pulp-dev] daily testing of released plugins with next Y release of pulpcore

[Dennis Kliban]

The plugin-template can now be used to enable a cron job that tests the
latest release of a plugin on PyPI against the master branch of pulpcore.
This job will identify when there is a compatibility issue with a future
release of pulpcore.

The 'test_released_plugin_with_next_pulpcore_release' parameter in the
template_config.yaml enables this workflow. All plugins are encouraged to
enable it.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] moving plugin docs to docs.pulpproject.org

[Dennis Kliban]

The docs are published as part of the release workflow and also as a cron
job. The cron job publishes to
docs.pulpproject.org///nightly/.
e.g.: https://docs.pulpproject.org/pulpcore/en/master/nightly/

On Fri, Dec 11, 2020 at 4:13 AM Tanya Tereshchenko 
wrote:

> Thanks!
>
> How do I trigger the docs job?
> Is it a part of the release workflow only or is there a way for me to
> publish docs before a release?
>
> Tanya
>
> On Thu, Dec 10, 2020 at 8:45 PM Dennis Kliban  wrote:
>
>> Each plugin repository in the Pulp organization on GitHub has been
>> configured with a PULP_DOCS_KEY secret that enables publishing docs to
>> docs.pulpproject.org/.
>>
>> Each plugin should enable publishing docs to docs.pulpproject.org by
>> setting 'publish_docs_to_pulpprojectdotorg' to true in the
>> template_config.yaml and re-running the 'plugin-template --github' command.
>>
>> Once your plugin has been configured to publish to docs.pulpproject.org,
>> please update the README for the git repository as well as
>> pulpproject.org with the new links for documentation[0]. After this, the
>> readthedocs.org site should be disabled.
>>
>> [0] https://pulpproject.org/content-plugins/
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] moving plugin docs to docs.pulpproject.org

[Dennis Kliban]

Each plugin repository in the Pulp organization on GitHub has been
configured with a PULP_DOCS_KEY secret that enables publishing docs to
docs.pulpproject.org/.

Each plugin should enable publishing docs to docs.pulpproject.org by
setting 'publish_docs_to_pulpprojectdotorg' to true in the
template_config.yaml and re-running the 'plugin-template --github' command.

Once your plugin has been configured to publish to docs.pulpproject.org,
please update the README for the git repository as well as pulpproject.org
with the new links for documentation[0]. After this, the readthedocs.org
site should be disabled.

[0] https://pulpproject.org/content-plugins/
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] check-manifest

[Dennis Kliban]

Plugins will also need to add 'check-manifest' to their
dev_requirements.txt.

On Fri, Oct 9, 2020 at 11:38 AM David Davis  wrote:

> Due to problems with files missing from Pulp 3 packages, the build team
> has asked us to use a tool called check-manifest[0] to ensure that files
> are not being omitted from our packages. I've added this to our
> plugin_template[1].
>
> The next time plugins update their Travis files, they will need to update
> their manifests or explicitly ignore any files they don't want shipped in
> their packages. Here are examples from pulpcore and pulp_file:
>
> https://github.com/pulp/pulpcore/pull/957
> https://github.com/pulp/pulp_file/pull/437
>
> Alternatively, you can set 'check_manifest' to false in your
> template_config.yml if you'd prefer not to run this check. You can run this
> tool manually by installing the package with pip and running
> 'check-manifest'.
>
> [0] https://pypi.org/project/check-manifest/
> [1] https://github.com/pulp/plugin_template/pull/274
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] namespacing base_path of Distributions

[Dennis Kliban]

The container plugin is introducing a 'namespace' resource that will be
associated with Distributions. The name of the namespace will be used as
the first part of the base_path for the ContainerDistribution. Role based
access control will allow users to create content under their own namespace
and any other namespace that they have permission to push container images
to.

Would a similar feature be useful in any other plugins?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] sphinx theme for docs.pulpproject.org

[Dennis Kliban]

Looks like dynaconf is no longer using Sphinx for the docs. Unfortunately
that is not an option for us at this time.

On Thu, Sep 24, 2020 at 8:57 AM Fabricio Aguiar 
wrote:

> I like the new dynaconf theme: https://www.dynaconf.com/
>
> Best regards,
> Fabricio Aguiar
> Software Engineer, Pulp Project
> Red Hat Brazil - Latam <https://www.redhat.com/>
> +55 11 999652368
>
>
> On Thu, Sep 24, 2020 at 9:48 AM Dennis Kliban  wrote:
>
>> We are slowly rolling out plugin docs to docs.pulpproject.org. At this
>> time pulp_file and pulp_ansible are the only plugins publishing their docs
>> there. What I noticed is that they have completely different themes. We
>> need to pick a single theme to be used by all plugins. Does anyone have
>> suggestions for which theme that should be?
>>
>> https://docs.pulpproject.org/pulp_ansible/
>> https://docs.pulpproject.org/pulp_file/
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] sphinx theme for docs.pulpproject.org

[Dennis Kliban]

On Thu, Sep 24, 2020 at 9:01 AM Quirin Pamp  wrote:

> I don't have a strong opinion, but I feel like if we don't come up with
> anything better we could just  fall back to what the pulpcore docs are
> currently using.
>
> https://docs.pulpproject.org/pulpcore/
>
> https://github.com/pulp/pulpcore/blob/d31eb22453cd2d56b43ed884ec193fa166cafe62/docs/conf.py#L116
>
>
+1 to making everything consistent with pulpcore.


> quba42
> --
> *From:* pulp-dev-boun...@redhat.com  on
> behalf of Dennis Kliban 
> *Sent:* 24 September 2020 14:47
> *To:* Pulp-dev 
> *Subject:* [Pulp-dev] sphinx theme for docs.pulpproject.org
>
> We are slowly rolling out plugin docs to docs.pulpproject.org. At this
> time pulp_file and pulp_ansible are the only plugins publishing their docs
> there. What I noticed is that they have completely different themes. We
> need to pick a single theme to be used by all plugins. Does anyone have
> suggestions for which theme that should be?
>
> https://docs.pulpproject.org/pulp_ansible/
> https://docs.pulpproject.org/pulp_file/
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] sphinx theme for docs.pulpproject.org

[Dennis Kliban]

On Thu, Sep 24, 2020 at 8:59 AM Melanie Corr  wrote:

> Hi Dennis,
>
> This reminds me that Pulp got feedback related to the difference in style
> between the main website and the docs from the OSPO audit last year:
>
> "● Create consistent styles between main and docs domain"
>
> Would this be hard to achieve?
>

We would need to find a Jekyll theme (pulpproject.org) that is paired with
a Sphinx theme (docs.pulpproject.org). This would most likely require us
creating something custom for both. Seems like a big effort.


>
> Ar Déar 24 MFómh 2020 ag 13:48, scríobh Dennis Kliban  >:
>
>> We are slowly rolling out plugin docs to docs.pulpproject.org. At this
>> time pulp_file and pulp_ansible are the only plugins publishing their docs
>> there. What I noticed is that they have completely different themes. We
>> need to pick a single theme to be used by all plugins. Does anyone have
>> suggestions for which theme that should be?
>>
>> https://docs.pulpproject.org/pulp_ansible/
>> https://docs.pulpproject.org/pulp_file/
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
>
> Melanie Corr, RHCE
>
> Community Manager
>
> Red Hat <https://www.redhat.com>
>
> Remote, Ireland
>
> mc...@redhat.com
> M: +353857774436 IM: mcorr
> <https://www.redhat.com>
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] sphinx theme for docs.pulpproject.org

[Dennis Kliban]

We are slowly rolling out plugin docs to docs.pulpproject.org. At this time
pulp_file and pulp_ansible are the only plugins publishing their docs
there. What I noticed is that they have completely different themes. We
need to pick a single theme to be used by all plugins. Does anyone have
suggestions for which theme that should be?

https://docs.pulpproject.org/pulp_ansible/
https://docs.pulpproject.org/pulp_file/
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] python-nectar 1.6.2 needed

[Dennis Kliban]

The Jenkins job[0] that tested the 2.21 staging repo showed that all the
Docker related tests pass. However, there was a Puppet test that failed.
The failure was related to puppet plugin syncing from a bad URL. The test
has an assertion that an exception will be present in the Task status and
that exception is not there.

Ina, could this be related to the Nectar changes?

I am going to re-run this job now.

[0]
https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/pulp-2.21-dev-rhel7/15/

On Mon, Sep 21, 2020 at 8:58 AM Lai Tran  wrote:

> Hey Ina,
>
> I'm still working on it.  There was some db issues I've encountered with
> some machines and not others.  I'll let you know once I'm done testing.
>
> Lai
>
> On Mon, Sep 21, 2020 at 6:41 AM Ina Panova  wrote:
>
>> Hi,
>> Still waiting on the test results.
>>
>> I will keep you posted. Thanks!
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Mon, Sep 21, 2020 at 8:46 AM Evgeni Golov  wrote:
>>
>>> Moin,
>>>
>>> is there a verdict about the builds?
>>>
>>> Evgeni
>>>
>>> On Thu, Sep 17, 2020 at 2:43 PM Evgeni Golov  wrote:
>>> >
>>> > and packaging is in https://github.com/pulp/pulp-packaging/pull/114
>>> >
>>> > the result is now available in the 2.21 stage repo, including the
>>> > fresh nectar from earlier this week.
>>> >
>>> > I kicked off
>>> https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/pulp-2.21-dev-rhel7/
>>> > and
>>> https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/pulp-2.21-dev-rhel7-fips/
>>> ,
>>> > enjoy!
>>> >
>>> > On Thu, Sep 17, 2020 at 1:02 PM Evgeni Golov 
>>> wrote:
>>> > >
>>> > > Yes I will ;-)
>>> > >
>>> > > pick PR for 3.2-release is up at
>>> https://github.com/pulp/pulp_docker/pull/462
>>> > >
>>> > > On Wed, Sep 16, 2020 at 4:56 PM Ina Panova 
>>> wrote:
>>> > > >
>>> > > > Hi Evgeni,
>>> > > > today it has been identified that pulp_docker also needs a release
>>> due to this related commit
>>> https://github.com/pulp/pulp_docker/commit/859ae5589b7bd5152814f001f4566d4a270cd3e8
>>> > > > Will you be able to work on the pulp_docker z stream
>>> release(3.2.7)?
>>> > > >
>>> > > > Thank you.
>>> > > > 
>>> > > > Regards,
>>> > > >
>>> > > > Ina Panova
>>> > > > Senior Software Engineer| Pulp| Red Hat Inc.
>>> > > >
>>> > > > "Do not go where the path may lead,
>>> > > >  go instead where there is no path and leave a trail."
>>> > > >
>>> > > >
>>> > > > On Tue, Sep 15, 2020 at 4:50 PM Ina Panova 
>>> wrote:
>>> > > >>
>>> > > >> Hey,
>>> > > >> we should do some testing first.
>>> > > >> Please keep the build for now in stage, once it is confirmed that
>>> the patches fix the original issue we can push to stable.
>>> > > >>
>>> > > >> Thank you!
>>> > > >> 
>>> > > >> Regards,
>>> > > >>
>>> > > >> Ina Panova
>>> > > >> Senior Software Engineer| Pulp| Red Hat Inc.
>>> > > >>
>>> > > >> "Do not go where the path may lead,
>>> > > >>  go instead where there is no path and leave a trail."
>>> > > >>
>>> > > >>
>>> > > >> On Tue, Sep 15, 2020 at 8:19 AM Evgeni Golov 
>>> wrote:
>>> > > >>>
>>> > > >>> I've released nectar 1.6.2 RPMs to
>>> > > >>>
>>> https://repos.fedorapeople.org/repos/pulp/pulp/testing/automation/2-master/stage/
>>> > > >>> Do you want to do any testing, or should I push the same RPMs
>>> directly
>>> > > >>> to the stable 2.21 repo too (or first to staging 2.21)?
>>> > > >>>
>>> > > >>> On Mon, Sep 14, 2020 at 12:18 PM Evgeni Golov 
>>> wrote

Re: [Pulp-dev] [CentOS-devel] repo_gpgcheck for centos repos?

[Dennis Kliban]

At this time Pulp always regenerates the repo metadata. In Pulp 3, we plan
to add the ability to mirror the metadata. However, the development of that
feature has not started yet.

On Tue, Sep 8, 2020 at 3:02 PM James Cassell 
wrote:

>
> On Tue, Sep 8, 2020, at 11:12 AM, Neal Gompa wrote:
> > On Fri, Sep 4, 2020 at 1:10 PM Brian Stinson  wrote:
> > >
> > > While we want signed repodata to be *available* to folks who want to
> enable it, We don’t want it necessarily to be the default for all users. We
> want it to be a decision that folks make for their own sites.
> > >
> >
> > This is a very bizarre stance to take. Enabling repo_gpgcheck for
> > the CentOS provided repos in their repo files should not harm anything
> > else, and only further ensures the integrity of the repository
> > content.
> >
> > Is there a compelling reason to *not* change the defaults? Because
> > from my perspective, I don't see any.
> >
>
> The only reason might be to prevent breaking folks who regenerate the
> repomd locally. Not sure whether pulp preserves the original md or
> regenerates its own. (I always use exactly the upstream repomd for
> precisely this reason of avoiding breaking repo_gpgcheck, which is often on
> "security hardening" checklists.)
>
> V/r,
> James Cassell
>
>
> >
> >
> > --
> > 真実はいつも一つ！/ Always, there's only one truth!
> > ___
> > CentOS-devel mailing list
> > centos-de...@centos.org
> > https://lists.centos.org/mailman/listinfo/centos-devel
> >
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.6.3 is generally available

[Dennis Kliban]

pulpcore 3.6.3 is available on PyPI[0]. This release fixes a packaging
bug[1].

# Installation and Upgrade

Users should use the 3.6.3 release of pulp_installer[2] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.6. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

# Plugin API

There are no plugin API changes.

# Client libraries

Both the Python[3] and Ruby[4] clients are available also.

[0] https://pypi.org/project/pulpcore/3.6.3/
[1] https://docs.pulpproject.org/pulpcore/en/3.6.3/changes.html#id1
[2] https://galaxy.ansible.com/pulp/pulp_installer
[3] https://pypi.org/project/pulpcore-client/3.6.3/
[4] https://rubygems.org/gems/pulpcore_client/versions/3.6.3
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulpcore 3.6.1 and 3.6.2 are Generally Available

[Dennis Kliban]

Yes, we can do that.

On Thu, Sep 3, 2020 at 12:04 PM Justin Sherrill  wrote:

> I discovered this issue with 3.6.2 (and prior) that prevents proper
> operation: https://pulp.plan.io/issues/7450
>
> Would it be possible to get a 3.6.3 i near term?
>
> Justin
> On 9/2/20 5:13 PM, Dennis Kliban wrote:
>
> Pulpcore 3.6.1 was released yesterday, but a bug with the OpenAPI schema
> was discovered before this announcement could be sent out. This bug has
> been fixed and pulpcore 3.6.2 is now available also[0]. These releases
> contain three bug fixes and improvements to documentation. For a list of
> all changes, please check the changelog for pulpcore[1].
>
> # Installation and Upgrade
>
> Users should use the 3.6.2 release of pulp_installer[2] to install or
> upgrade their installations. This version of the installer will check
> compatibility of all installed plugins with pulpcore 3.6. The installer
> will abort if any plugin is incompatible.
>
> The pulp_installer collection can be installed from Ansible Galaxy with
> the following command:
>
> ansible-galaxy collection  install --force pulp.pulp_installer
>
> The --force flag will upgrade the collection if you had a previous version
> installed.
>
> # Plugin API
>
> The plugin API changes were all related to problems with the OpenAPI
> schema. These bugs were all introduced in 3.6.0 when pulpcore upgraded to
> OpenAPI version 3. The OpenAPI schema improvements are also evident in the
> both the Python[3] and Ruby[4] clients. For the full list of changes,
> please check the plugin API changelog[5].
>
> [0] https://pypi.org/project/pulpcore/3.6.2/
> [1] https://docs.pulpproject.org/pulpcore/en/3.6.2/changes.html#id1
> [2] https://galaxy.ansible.com/pulp/pulp_installer
> [3] https://pypi.org/project/pulpcore-client/3.6.2/
> [4] https://rubygems.org/gems/pulpcore_client/versions/3.6.2
> [5] https://docs.pulpproject.org/en/3.6.2/changes.html#plugin-api
>
> ___
> Pulp-dev mailing 
> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.6.1 and 3.6.2 are Generally Available

[Dennis Kliban]

Pulpcore 3.6.1 was released yesterday, but a bug with the OpenAPI schema
was discovered before this announcement could be sent out. This bug has
been fixed and pulpcore 3.6.2 is now available also[0]. These releases
contain three bug fixes and improvements to documentation. For a list of
all changes, please check the changelog for pulpcore[1].

# Installation and Upgrade

Users should use the 3.6.2 release of pulp_installer[2] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.6. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

# Plugin API

The plugin API changes were all related to problems with the OpenAPI
schema. These bugs were all introduced in 3.6.0 when pulpcore upgraded to
OpenAPI version 3. The OpenAPI schema improvements are also evident in the
both the Python[3] and Ruby[4] clients. For the full list of changes,
please check the plugin API changelog[5].

[0] https://pypi.org/project/pulpcore/3.6.2/
[1] https://docs.pulpproject.org/pulpcore/en/3.6.2/changes.html#id1
[2] https://galaxy.ansible.com/pulp/pulp_installer
[3] https://pypi.org/project/pulpcore-client/3.6.2/
[4] https://rubygems.org/gems/pulpcore_client/versions/3.6.2
[5] https://docs.pulpproject.org/en/3.6.2/changes.html#plugin-api
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp_deb plugin maintenance meeting

[Dennis Kliban]

I would like to participate. Can we move the meeting time to 14:00 o'clock
CEST ?

On Tue, Jul 21, 2020 at 10:11 AM Quirin Pamp  wrote:

> Hi everyone,
>
> I would like to propose a meeting on handling pulp_deb plugin maintenance.
>
> I will contact several people individually, that I think should be there,
> but I also wanted to announce on the mailing list in case anyone is
> interested who is not on my radar.
>
> I propose Monday, July 27th, 13:00 o'clock CET as a preliminary date, but
> this will likely still change, since I have not yet checked with anyone's
> schedule.
>
> Feel free to propose a different time, or amend the Agenda here:
>
> [1] https://hackmd.io/kGjFFGtWTJSGuIDlVlgqZw
>
> thanks,
> Quirin Pamp
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API

[Dennis Kliban]

On Tue, Jul 21, 2020 at 11:22 AM Brian Bouterse  wrote:

> I'm concerned if we don't make a change, here's the user experience I'm
> worried about.
>
> 1. User A creates repo 'rhel7'
> 2. user B can't see repo 'rhel7' because of queryset scoping
> 3. user B goes to create 'rhel7'
> 4. user B is told 'rhel7' already exists
>
> Users should be able to use simple names. I don't know what the answer is
> to the import/export implementation conflict, but let's brainstorm some.
> For the benefit of our users, I don't think that implementation should
> interfere with this basic use.
>
>
I agree that this is a usability problem for our users.

With regard to import/export, the ideal solution would use the same UUID in
both the system that's exporting and the system that's importing. Is my
understanding correct?



> Side note: from early on in Pulp3, pk's not names have been the primary
> identifier. I'm unclear on how we got away from that.
>
>
>
> On Tue, Jul 21, 2020 at 9:03 AM Matthias Dellweg 
> wrote:
>
>> I always understood the "lifting the uniqueness" as allowing to have
>> the same name used for different resource types. So the new
>> natrual_key (aka unique_together) would be ["name", "type"].
>>
>> On Tue, Jul 21, 2020 at 2:55 PM David Davis 
>> wrote:
>> >
>> > Agreed.
>> >
>> > David
>> >
>> >
>> > On Tue, Jul 21, 2020 at 8:42 AM Grant Gainey 
>> wrote:
>> >>
>> >> On Tue, Jul 21, 2020 at 8:14 AM Dennis Kliban 
>> wrote:
>> >>>
>> >>> Does anyone else have an opinion? If not, I am going to start by
>> writing a task to remove this name uniqueness constraint for repositories.
>> >>
>> >>
>> >> Import/export relies on non-pulp_id-uniqueness to identify Things. I
>> was assuming we were talking about adding pulp_type to the Repository
>> uniqueness-constraint, so that a given name/type would be unique (which
>> would require a single change to RepositoryResource)
>> >>
>> >> If we're talking about just removing the uniqueness-constraint
>> altogether, then life gets a lot harder.
>> >>
>> >> G
>> >> --
>> >> Grant Gainey
>> >> Principal Software Engineer, Red Hat System Management Engineering
>> >> ___
>> >> Pulp-dev mailing list
>> >> Pulp-dev@redhat.com
>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >
>> > ___
>> > Pulp-dev mailing list
>> > Pulp-dev@redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API

[Dennis Kliban]

Does anyone else have an opinion? If not, I am going to start by writing a
task to remove this name uniqueness constraint for repositories.

On Wed, Jul 1, 2020 at 9:21 AM Ina Panova  wrote:

> This name uniqueness problem is extended to remotes and distributions, and
> probably some other objects.
>
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Tue, Jun 30, 2020 at 11:14 PM Grant Gainey  wrote:
>
>> On Tue, Jun 30, 2020 at 4:56 PM Daniel Alley  wrote:
>>
>>> Why would it cause pain (to relax the restriction)?
>>>
>>
>> Oh, good point - I was thinking backwards.  Carry on, carry on... :)
>>
>> G
>> --
>> Grant Gainey
>> Principal Software Engineer, Red Hat System Management Engineering
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] AccessPolicy Model Advice ... input needed

[Dennis Kliban]

Pulp 2 users would definitely be familiar with describing policies in terms
of urls. The Pulp 3 REST API already uses the pulp_href field as the
primary identifier. You should implement idea 1.

On Mon, Jul 13, 2020, 5:45 PM Brian Bouterse  wrote:

> I need some design input. To store AccessPolicy data in the DB I think we
> want one Model where each instance is the access policy for a given
> viewset. I think this would be better than one Model per Viewset which
> would generate N tables for N viewsets with 1 instance of each which would
> be very strange and inefficient.
>
> So let's assume we have a simple definition like the one below. Each
> instance stores the policy for one viewset.
>
> class AccessPolicy(BaseModel):
> data = JSONField()
>
> So what second field can I add to this that would allow me to relate an
> instance of this model to a viewset. For example the FileRemoteViewset
> here:
> https://github.com/pulp/pulp_file/blob/de638519fc02d588f403db4c5cfcca7552543c50/pulp_file/app/viewsets.py#L116
>
> Idea 1: Add a viewset = CharField(). Have it store values as URLs, e.g.
> /pulp/api/v3/remotes/file/file/.
> Idea 2: Add a viewset = CharField(), and have it store values as
> classpaths, e.g. 'pulp_file.app.viewsets.RemoteFileViewset'.
>
> I think Idea 1 makes the most sense because that's how our users think of
> it. I can't think of a good alternative. What do you think makes the most
> sense? What alternative ideas should we consider?
>
> If you have feedback please share it. I need to start into something to
> get it going tomorrow even if it's just Idea 1 for lack of an alternate
> proposal.
>
> Thanks,
> Brian
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp Container meeting minutes

[Dennis Kliban]

13 july, 2020

Pulp 3:

Token certificate generation needs to be added to the installer and
operator

   -

   Installer ticket already written
   -

   Dkliban to write issue for pulp-operator



https://pulp.plan.io/issues/7124 - anything that modifies the token should
cause the registry to return a Unauthorized response

   -

   Lmjachky to review the PR for resolving this

https://pulp.plan.io/issues/7125 - if the schema converter can’t convert
the V2 manifest, return 404 to the older client

   -

   Lmjachky will open a PR to resolve this issue


Merging master branch into 2.0

   -

   https://github.com/pulp/pulp_container/pull/117 - Need to disable issue
   checking for this PR and then submit a follow up PR to re-enable it. This
   will allow us to see all tests passing before merging.
   -

  We decided to postpone this effort, because there is nothing we
  desperately need in the feature preview that is the 2.0 branch
  -

   2.0.0 GA milestone
   -

  There are 2 most important issues to resolve
  -

 https://pulp.plan.io/issues/7012
 -

 https://pulp.plan.io/issues/7013 - lmjachky will work on this
 -

  Pulpcore upload model re-use is a nice to have, but if we do decide
  to use it, we should do it with the 2.0.0 release and not after.
  -

 https://pulp.plan.io/issues/7025
 -

  The two redirects being converted to one can be done with the 2.1
  release.
  -

 https://pulp.plan.io/issues/6826
 -

   AI Dkliban to release 2.0 beta
   -

   AI Mdellweg to release 1.4.2
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp Container meeting minutes

[Dennis Kliban]

6 july, 2020

Pulp 3:

   -

   With rbac we might want to have namespaces as objects to assign
   permissions to it
   -

  Should we validate the relative_path of a distribution to have
  exactly on “/”?
  -

 Dkliban to test that you can docker and podman pull from a
 distribution that has multiple / in the base_path
 -

  Mdellweg to write a story to allow creation of namespaces via Pulp
  REST API.
  -

   Distribution Base_path is unique across pulp
   -

  Existing Rpm distribution with base_path foo/test will not allow
  creation of container distribution
  -

 Dkliban to write proposal for how to allow plugin Distributions to
 opt out of cross-plugin uniqueness.
 -

  Path overlap and relative path validation does not make sense to keep
  for container distribution
  -


 
https://github.com/pulp/pulpcore/blob/master/pulpcore/app/serializers/publication.py#L124
 -


 
https://github.com/pulp/pulpcore/blob/master/pulpcore/app/serializers/base.py#L53
 -

   Performance progress/results
   -

  Still trying to figure out how to refactor the method
  relate_manifest_tag():
  
https://github.com/pulp/pulp_container/blob/master/pulp_container/app/tasks/sync_stages.py#L502
  -


  
https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/plugin/docker/migrator.py#L143
  -

   Problematic language https://pulp.plan.io/issues/7070
   -

  Will discuss language consistency between plugins at open floor
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulpcore 3.5 release is scheduled for July 7

[Dennis Kliban]

We identified a release blocker during triage[0]. The release is postponed
until tomorrow, July 8th.


[0] https://pulp.plan.io/issues/7087

On Fri, Jun 26, 2020 at 11:53 AM Dennis Kliban  wrote:

> The current plan is to release pulpcore 3.5 on July 7th, 2020. If you want
> to have your feature or bug fix included, please make sure it gets merged
> before that day.
>
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Name uniqueness problem in Pulp 3 REST API

[Dennis Kliban]

The REST APIs for repositories in Pulp 3 do not allow a name to be reused
between repository types. e.g.: A File Repository with name 'test' prevents
a user from creating an RPM Repository with name 'test', even though these
are two separate APIs.

Do you agree that the namespace of one API should not interfere with the
namespace of another API?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.5 release is scheduled for July 7

[Dennis Kliban]

The current plan is to release pulpcore 3.5 on July 7th, 2020. If you want
to have your feature or bug fix included, please make sure it gets merged
before that day.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] upgrade testing meeting minutes

[Dennis Kliban]

Today we continued the discussion about upgrade testing[0]. The main ideas
we discussed revolved around creating a set of tests that each use unique
fixture repositories and don't use orphan cleanup between tests. These
tests would be run in our regular PR build jobs and would allow us to
re-use these tests to populate Pulp with data before performing an upgrade
and then running a second set of post-upgrade tests that make assumptions
about data being present in Pulp. Some shorter term goals included the
following:

# Ideas from June 24

* Requirements:
  * Create a set of tests that are run before an upgrade for pulp_file
  * Create a set of unique fixtures for pulp_file tests involved during
upgrade testing
  * Remove the use of orphan cleanup for tests that are used by
post_upgrade tests
  * Create a set of post-upgrade tests that rely on data generated by
pre-upgrade tests


[0] https://hackmd.io/IwhWRn6OS9ekKJNfQrGYFw
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Fwd: Status of pulp-python

[Dennis Kliban]

We recently had an opportunity to devote more resources to the Python
plugin. Beta 9 was released earlier today[0]. We are hoping to make a GA
release in the near future.

[0] https://www.redhat.com/archives/pulp-list/2020-June/msg1.html


On Tue, May 5, 2020 at 3:45 PM Robin Chan  wrote:

> Hello Christoph,
>
> The Python plugin isn't abandoned, but we have not been able to maintain
> it due to other higher priority issues for other plugins. Our best case
> scenario would be to find a user or users who use the Python plugin and are
> interested in helping maintain this plugin.  We would definitely be able to
> support those efforts including PR reviews and help with coming up to speed
> on how to contribute.
>
> I'd suggest a first step of working with Daniel (Python plugin maintainer)
> to come up with a gap analysis on what you would need from the Python
> plugin for your use cases. We could commit to PR reviews on bug fixes but
> no feature development in the near future (this next calendar quarter
> already planned out commitments.) To be realistic, I would recommend
> someone from your organization know how to send PR for any future
> discovered critical bugs to make this a feasible option. I'm anticipating
> together you may discover some additional efforts like CI/test fixes or
> releasing a new version that may be needed as well since it's probably a
> bit behind on that. Those aren't off the table but I'd support Daniel
> spending some time to figure out where we are with this. Does that sound
> like something you would be interested in?
>
> -Robin
>
> Robin Chan
>
> She/Her/Hers
>
> Satellite Software Engineering Manager - Pulp
>
> Red Hat 
>
> IRC: rchan
> 
>
>
>
> On Tue, May 5, 2020 at 4:18 AM Christoph Höger <
> christoph.hoe...@celeraone.com> wrote:
>
>> Dear all,
>>
>> I am currently in the process of investigating the usage of pulp for our
>> company's repository management and think I can fulfill all the
>> requirements except for a pypi mirror. While this is exactly what
>> pulp-python seems to provides it has a showstopper bug:
>>
>> https://pulp.plan.io/issues/5452
>>
>> which appears to be the same as:
>>
>> https://pulp.plan.io/issues/5387
>>
>> I assume that I could get one of our python guys working on the matter,
>> if I can refer them to a) someone to talk to about the project and b) get
>> some assurance that this plugin is not dead already.
>>
>> So do you know what the status of pulp-python is? Is it just not very
>> high on the priorities or is it already more or less abandoned? I am asking
>> this, because pulpcore and some other plugins seem to have a relatively
>> fast pace. If pulp-python is abandoned we might just go on with deb/rpm
>> support and use something else to mirror pypi.
>>
>> thank you very much,
>>
>> Christoph
>>
>> --
>> Christoph Höger
>>
>> Email: christoph.hoe...@celeraone.com 
>> Web: www.celeraone.com
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulpcore 3.4.0 release scheduled for May 27th

[Dennis Kliban]

The pulpcore 3.4.0 release will be published tomorrow to PyPI.

On Tue, May 19, 2020 at 8:55 PM Dennis Kliban  wrote:

> Pulpcore 3.4.0 is scheduled to be released on May 27th. There are
> currently 3 issues that have been proposed as blockers for this release[0].
> Please respond to this thread with any other issues that should potentially
> be addressed in this release.
>
> [0] https://pulp.plan.io/versions/88
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] upgrade testing meeting minutes

[Dennis Kliban]

Earlier today we discussed ideas for upgrade testing. The notes were kept
here [0]. The summary of the meeting minutes is below.

* from which version to which version?
* from the latest x.y.z release of pulpcore and plugin being to tested
to the master branch of pulpcore and master branch of plugin
* which database tables should we test?
* all the content types, all remote types, all exporters, ALL resources
* each plugin team should decide it
* we want to test:
* migrations
* REST API
* have each plugin upgrade test be a separate job
* stretch goal - test all katello plugins together
* add a check to regular CI that ensures no new migrations are generated
when makemigrations is run
* https://pulp.plan.io/issues/4984
* https://pulp.plan.io/issues/6637 - the epic for improving functional test
* Should we combine 6637 effort with the upgrade testing?
* if we don't combine, we will end up with 2 sets of tests
* need to handle cases where a feature is not present in version
N-1 and added in version N
* tests for each plugin are already specific to the version
they sit within
* as a first step in this giant effort we should simply test that we can
install a previous version of pulpcore + plugin and then run migrations for
master branch of pulpcore + plugin.



[0] https://hackmd.io/IwhWRn6OS9ekKJNfQrGYFw
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Unified CI/CD pipeline

[Dennis Kliban]

Each release of pulpcore requires a separate release of pulp_installer. In
the near future we will also release container images with each new release
of pulpcore. Since all of these assets are always released together, they
need to be released as part of a single pipeline that only releases all
three when all three have been tested.

The CI team has put together a proposal for such a pipeline[0]. Please take
a look and provide feedback. We'd like to formally write up this effort in
pulp.plan.io issues in a week from today.

[0] https://hackmd.io/z8HZw9oyQ7-MAV6W5lse4g
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] CI/CD meeting notes

[Dennis Kliban]

May 20, 2020


   -

   Quay.io was down again  on 05-19
   -

  Normal docker client won’t let you use quay.io unless you specify it
  in image name.
  -

  We want to use DockerHub for reliability instead, we are upstream.
  -

   CI status check
   -

   Pulp_installer and pulp_rpm_prerequisites to have a github action to
   publish on galaxy
   -


  
https://github.com/theforeman/foreman-ansible-modules/blob/master/.github/workflows/release.yml
  -

  We have to use an individual user account’s API key though.
   Which one?
  -

 Use pulpbot github
 -

  Fao89 will file an issue
  -

 https://pulp.plan.io/issues/6779
 -

 https://pulp.plan.io/issues/6780
 -

   Unified release pipeline vision -
   https://hackmd.io/z8HZw9oyQ7-MAV6W5lse4g
   -

  Redmine epic
  -

  Dkliban will send an email on pulp-dev
  -

   Not all plugins are testing docs scripts
   -

  Worked on pulp_rpm - https://github.com/pulp/pulp_rpm/pull/1716
  -

  Started to work on pulp_ansible -
  https://github.com/pulp/pulp_ansible/pull/315
  -

   Pulp_container
   -

  Token_auth
  -

 Smash config - added token
 -

 TOKEN_AUTH_DISABLED, changed from True to False
 -

 Mdellweg - file a task to activate it in plugin_template
 -

  Problem with pulpcore imports
  -

 Fao89 - file an issue and mark as a blocker for 3.4.0 (milestone)
 -

https://pulp.plan.io/issues/6781
-

   Data migration tests
   -

  Dkliban will schedule a meeting
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.4.0 release scheduled for May 27th

[Dennis Kliban]

Pulpcore 3.4.0 is scheduled to be released on May 27th. There are currently
3 issues that have been proposed as blockers for this release[0]. Please
respond to this thread with any other issues that should potentially be
addressed in this release.

[0] https://pulp.plan.io/versions/88
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Fwd: Naming/Tagging-Schema for container images

[Dennis Kliban]

On Mon, May 18, 2020 at 10:37 AM Matthias Dellweg 
wrote:

> In the long run, i want to publish a ci image based on centos and
> another one on fedora? Would you rather put the os_name in the tag? Or
> would you only include the os_name if it's not centos8?
> How would you see the transition to centos9?
>
> What is the purpose of publishing images that are based on different OSes?
I am genuinely curious.



> As i see it, we have three information that we need to encode:
> 1. Purpose of the container: pulp, pulp-ci, pulp-molecule, ...
> 2. Base OS: centos8, centos9 (eventually), fedora31, debian10, ...
> 3. (Y-stream) Version of pulpcore involved: master, devel, latest, 3.2, ...
> [ 4. Build number of the image ]
>
> With respect to 4., I am unsure how much value there is to keep older
> builds lying around. Is that a common practice?
>
> Older images allow users to re-deploy the exact same thing that they had
deployed somewhere else.


> I guess, we could skip "centos8" as the default value (but it should
> not hurt to tag the same image with the fully qualifying name anyway).
>
> The (harder) question is, which of these information should make up
> the (docker-/quay-)repository name and which encode the tag?
> e.g.
>   - The fedora and alpine repository have one tag for each
> (pre-)released version.
>   - Debian has tags for each version and again for the version with
> added '-slim' or '-backports'.
>   - Python uses python version with debian or alpine release as tags.
>   * [3.8.3-slim-buster, 3.8-slim-buster, 3-slim-buster,
> slim-buster, 3.8.3-slim, 3.8-slim, 3-slim, slim] all refer to the same
> container image.
>
> It seems quite common to have simple repository names and combine a
> lot of very different images with an elaborate tagging schema. Also
> certain images tend to have several tags.
>
> I agree that it is more common to include just a name in the repository
name. Pulp is different from most applications because it ships a variable
number of plugins.

We could create tags that include the name of all the plugins inside the
container. So the user would be able to run a command such as

podman run pulp/pulp:3.3.1-ansible-container-file-maven-rpm

or

podman run
pulp/pulp:3.3.1-ansible0.2.0b12-container1.3.0-file0.3.0-maven0.1.0-rpm3.3.2


This tag can get very long.

> On Mon, May 18, 2020 at 2:46 PM Dennis Kliban  wrote:
> >
> > Long term, I would like to stop publishing container images based on
> Fedora. Images for production use should be built on top of CentOS 8
> stream[0]. The name of the image repository should not contain the OS name.
> >
> > Each 3.y release of pulpcore should live in its own repository called
> pulp/pulp-3-y. The initial release should be tagged as both 'latest' and
> '0'. Each time a compatible plugin is released, this image should be
> updated and the tag should be incremented by 1. The project website should
> contain a table that is automatically generated. The table should list what
> versions of plugins are included in each of the tags.
> >
> > What do others think?
> >
> > [0] https://pulp.plan.io/issues/6676
> >
> >
> > On Thu, May 14, 2020 at 12:54 PM Matthias Dellweg 
> wrote:
> >>
> >> We have recently started a new repository calles pulp-oci-images that
> >> should emit according to its name OCI compatible images with pulp
> >> installed.
> >> In the first go, this includes the single-container promoted though
> >> this blog post [0].
> >> Soon to be added is the base container image that shall speed up our CI
> [1].
> >> In the future, i envision a similar single-container solution based on
> >> centos instead of fedora,
> >> as well as ci base images based on centos having python3.6 installed.
> >> Does anyone think, we even need different ci-images for pulp release
> branches?
> >>
> >> The big question now is: How are we going to name and tag those images?
> >>
> >> The one from [0] is called "pulp/pulp-fedora31:latest".
> >> We could go with that and add names like:
> >> - "pulp/pulp-centos8:3.2"
> >>   installation of core version 3.2 with all compatible plugins on
> centos8
> >> - "pulp/pulp-ci-fedora32:latest"
> >> - "pulp/pulp-ci-centos8:latest"
> >>
> >> BTW, the ci-base images can be built by using the same Conteinerfile
> >> interrupted early.
> >> (with --target in a multistage build)
> >>
> >> What do you think?
> >>
> >> [0] https://pulpproject.org/2020/03/15/pulp-fedora31-single-container/
> >> [1]
> https://github.com/pulp/pulpcore/blob/master/.travis/Containerfile.ci_base
> >>
> >> ___
> >> Pulp-dev mailing list
> >> Pulp-dev@redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Bug triage is moving to #pulp-meeting on Freenode

[Dennis Kliban]

The decision to do this was actually made a while ago when we noticed that
holding meetings in #pulp-dev interrupted discussions that were happening
in that channel around the same time as the meeting. However, I did not
follow through to actually announce this decision and update the website.
@bmbouter brought this up again on Friday during open floor.

On Mon, May 18, 2020 at 8:35 AM David Davis  wrote:

> Any chance we can get some background on how, why, where this decision was
> made? I'm not opposed to it but having some more information in this
> announcement would be helpful.
>
> David
>
>
> On Fri, May 15, 2020 at 11:21 AM Dennis Kliban  wrote:
>
>> Starting on May 19th, bug triage will be held in #pulp-meeting on
>> Freenode[0].
>>
>> [0] https://pulpproject.org/get_involved/#meetings
>> ___
>> Pulp-list mailing list
>> pulp-l...@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Naming/Tagging-Schema for container images

[Dennis Kliban]

Long term, I would like to stop publishing container images based on
Fedora. Images for production use should be built on top of CentOS 8
stream[0]. The name of the image repository should not contain the OS name.

Each 3.y release of pulpcore should live in its own repository called
pulp/pulp-3-y. The initial release should be tagged as both 'latest' and
'0'. Each time a compatible plugin is released, this image should be
updated and the tag should be incremented by 1. The project website should
contain a table that is automatically generated. The table should list what
versions of plugins are included in each of the tags.

What do others think?

[0] https://pulp.plan.io/issues/6676


On Thu, May 14, 2020 at 12:54 PM Matthias Dellweg 
wrote:

> We have recently started a new repository calles pulp-oci-images that
> should emit according to its name OCI compatible images with pulp
> installed.
> In the first go, this includes the single-container promoted though
> this blog post [0].
> Soon to be added is the base container image that shall speed up our CI
> [1].
> In the future, i envision a similar single-container solution based on
> centos instead of fedora,
> as well as ci base images based on centos having python3.6 installed.
> Does anyone think, we even need different ci-images for pulp release
> branches?
>
> The big question now is: How are we going to name and tag those images?
>
> The one from [0] is called "pulp/pulp-fedora31:latest".
> We could go with that and add names like:
> - "pulp/pulp-centos8:3.2"
>   installation of core version 3.2 with all compatible plugins on centos8
> - "pulp/pulp-ci-fedora32:latest"
> - "pulp/pulp-ci-centos8:latest"
>
> BTW, the ci-base images can be built by using the same Conteinerfile
> interrupted early.
> (with --target in a multistage build)
>
> What do you think?
>
> [0] https://pulpproject.org/2020/03/15/pulp-fedora31-single-container/
> [1]
> https://github.com/pulp/pulpcore/blob/master/.travis/Containerfile.ci_base
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Bug triage is moving to #pulp-meeting on Freenode

[Dennis Kliban]

Starting on May 19th, bug triage will be held in #pulp-meeting on
Freenode[0].

[0] https://pulpproject.org/get_involved/#meetings
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Installer project

[Dennis Kliban]

I am +1 to creating a new redmine project.

I want to point out that we plan to move installer documentation into the
main pulpcore docs. Our CI checks for issues associated with the
appropriate projects. Do that mean we will need to create docs issues in
the Pulp project when the docs need to be updated for the installer?

On Tue, May 5, 2020 at 11:25 AM David Davis  wrote:

> Cool. If we go down this path, would this plan make sense?
>
> 1. Create an installer project in redmine
> 2. Move over all issues (opened and closed) to this project that are
> tagged "Pulp 3 installer"
> 3. Remove the "Pulp 3 installer" tag
>
> David
>
>
> On Tue, May 5, 2020 at 11:10 AM Fabricio Aguiar <
> fabricio.agu...@redhat.com> wrote:
>
>> +1 for creating a new project,
>> Installer query: https://pulp.plan.io/projects/pulp/issues?query_id=154
>>
>> Best regards,
>> Fabricio Aguiar
>> Software Engineer, Pulp Project
>> Red Hat Brazil - Latam 
>> +55 11 999652368
>>
>>
>> On Tue, May 5, 2020 at 11:48 AM David Davis 
>> wrote:
>>
>>> During triage today, a majority of the issues that came up were for the
>>> installer but only a handful of people could really speak to these issues.
>>> I proposed that we skip these issues during triage and let the installer
>>> team triage them.
>>>
>>> In order to do so, we'd have to create a separate project in redmine. My
>>> initial feeling is that this makes sense since the installer work has
>>> really grown into its own project and is no longer in the purview of
>>> pulpcore.
>>>
>>> Thoughts?
>>>
>>> I'll put a deadline on this discussion for May 8 unless we don't reach a
>>> consensus by then.
>>>
>>> David
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 CLI MVP

[Dennis Kliban]

On Thu, May 7, 2020 at 7:13 AM Tatiana Tereshchenko 
wrote:

> +1 to `pulp` command.
>
> I think for me as a user, the most logical would be to have a plugin name
> first and then follow the URL pattern.
> The majority of commands are plugin specific. If I work with multiple
> plugins, it also makes it easy for me as a user to just change the plugin
> name in front for the commands which have the same structure in different
> plugins.
> It also makes it visually clear that I work with a specific plugin, in
> comparison to when the plugin name is somewhere in the 3rd/4th place.
> It will also allow not to guess in commands like the `pulp repositories
> rpm rpm`  which one is the plugin name and which one is a repo type.
>
> I agree that this would make much more clear to the user which 'rpm' is
the plugin type and which 'rpm' is the resource type.


> +1 for
> pulp rpm content packages
> pulp rpm repositories rpm
> pulp rpm repositories mirror
> ...
>
> On Wed, May 6, 2020 at 7:58 PM Dennis Kliban  wrote:
>
>> On Wed, May 6, 2020 at 12:30 PM David Davis 
>> wrote:
>>
>>> Matthias and I met today to go over some plans for a prototype. I wrote
>>> some notes[0] down. As part of the prototype, we'd propose two deliverables
>>> (one this week and one next week):
>>>
>>> 1. A set of ~2-3 click commands that use the bindings to interact with
>>> Pulp
>>> 2. Some openapi-generator templates that will be able to generate such
>>> commands from the schema
>>>
>>> There is a question we had about how the commands for typed resources
>>> will be structured in the CLI. To illustrate with two endpoints:
>>>
>>>
>> We should call the command 'pulp' instead of pulp-cli.
>>
>>
>>> # rpm.package content (/pulp/api/v3/content/rpm/packages/):
>>> - pulp-cli rpm content packages ...
>>> - pulp-cli content rpm packages ...
>>> - pulp-cli rpm packages content ...
>>> - ???
>>>
>>
>> I was thinking we should structure the commands similar to the URLs in
>> the REST API.
>>
>> pulp content rpm packages
>>
>>
>>>
>>> # file.file repositories (/pulp/api/v3/repositories/file/file/):
>>> - pulp-cli file repositories file ...
>>> - pulp-cli repositories file file ...
>>> - pulp-cli file file repositories ...
>>> - ???
>>>
>>> pulp repositories file
>>
>> Plugins that provide multiple types of the same resource would need to be
>> more descriptive. Though I can see a practical reason for requiring all
>> resources to be that descriptive.
>>
>> pulp repositories rpm rpm
>> pulp repositories rpm mirror
>>
>>
>>
>>> [0] https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Prototype
>>>
>>> David
>>>
>>>
>>> On Thu, Apr 30, 2020 at 1:42 PM David Davis 
>>> wrote:
>>>
>>>> Today we met to discuss some ideas for a technical design for how the
>>>> CLI would work. Here's a copy of our notes:
>>>>
>>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Technical-discussion
>>>>
>>>> And there is a rough design in the document as well:
>>>>
>>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Design
>>>>
>>>> I have also entered the CLI user stories from our meeting last week
>>>> into redmine under the Pulp CLI project:
>>>>
>>>> https://pulp.plan.io/versions/93
>>>>
>>>> And I've filed a user story that we talked about today that would
>>>> handle sync, publish, and distribution of repos. Feedback welcome:
>>>>
>>>> https://pulp.plan.io/issues/6626
>>>>
>>>> Matthias and I are planning to meet next week to look at creating a
>>>> proof of concept that would provide 2-3 commands. If anyone is interested
>>>> in joining us, please let me know and I can add you.
>>>>
>>>> David
>>>>
>>>>
>>>> On Tue, Apr 28, 2020 at 8:06 AM David Davis 
>>>> wrote:
>>>>
>>>>> I've also started working on some questions about how the CLI will
>>>>> work. Feel free to add some of your own:
>>>>>
>>>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Technical-discussion
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Tue, Apr 28, 2020 at 8:05 AM David Davis 
>>>>> wrote:
>>>>>
>&g

Re: [Pulp-dev] signing service interface

[Dennis Kliban]

On Wed, May 6, 2020 at 4:07 AM Matthias Dellweg  wrote:

> As i see it, it is up to the subclass (e.g. AptReleaseSigningService,
> YumMetadataSigninigService, ...) to provide a stable interface. And the way
> it is implemented, the script for an AptReleaseSigninigService is required
> to report the filenames of both created signatures. And that is verified by
> the service before saving to the database.
>

In case of the RPM plugin, the content handler needs to be able to know
what the public key file is named without actually executing the sign() or
validate() method. I don't see anything in the AptReleaseSigninigService[0]
that provides that functionality.

The implementation of the AsciiArmoredDetachedSigningService[1] could
provide a method for retrieving the public key file name and the validate()
method would have to enforce it. Would this be more valuable to implement
at the base class (SigningService) level[2]?

[0]
https://github.com/pulp/pulp_deb/blob/master/pulp_deb/app/models/signing_service.py#L12
[1]
https://github.com/pulp/pulpcore/blob/3.3/pulpcore/app/models/content.py#L447
[2]
https://github.com/pulp/pulpcore/blob/3.3/pulpcore/app/models/content.py#L377


>
> On Tue, May 5, 2020 at 11:51 PM Dennis Kliban  wrote:
>
>> On Tue, May 5, 2020 at 3:39 AM Quirin Pamp  wrote:
>>
>>> Could you explain the reasoning for a 'public.key' file?
>>>
>>
>> The public.key file is the file that a yum/dnf client can use to verify
>> that the metadata in an RPM repository was signed by the signing service
>> associated with the repository. The name of the file can be anything - the
>> path to it needs to be specified in the repository config on the client.
>>
>>
>>> In the case of the AptReleaseSigningService we built for pulp_deb we saw
>>> zero need for this file and consequently did not add it in.
>>>
>>> (It would not be hard to add it just to satisfy the interface, it just
>>> would not serve any useful purpose.)
>>>
>>
>> It is definitely up to each plugin if it wants to provide the public key
>> as part of the publication. It is currently impossible for the plugin to
>> know exactly what files are produced by the signing service. This is where
>> I would like to see an improvement in the API. Pupcore should provide a
>> guarantee to plugin writers that a signing service configured by an
>> administrator is functioning in a predictable way. One possible way to do
>> that is with an interface that lets a plugin writer inspect a signing
>> service without executing it. Though I am looking for other ideas in this
>> area.
>>
>>
>>>
>>> Since we are on the topic of signing services, a colleague has had a PR
>>> relating to them just sitting their waiting for a review for quite a while
>>> now ;-):
>>> https://github.com/pulp/pulpcore/pull/659
>>>
>>>
>>> It would be great if you (or somebody else) could have a look at it. I
>>> believe it is mostly ready, but probably needs the eyes of an experienced
>>> pulp core developer to look over it and suggest style consistency changes
>>> and where and whether to add documentation. ;-)
>>>
>>
>> I'll take a look at this PR.
>>
>>
>>
>>>
>>> Quirin
>>> --
>>> *From:* pulp-dev-boun...@redhat.com  on
>>> behalf of Dennis Kliban 
>>> *Sent:* 04 May 2020 22:50:54
>>> *To:* Pulp-dev 
>>> *Subject:* [Pulp-dev] signing service interface
>>>
>>> The Plugin API of Signing Services in Pulp 3 is too vague. I came to
>>> this conclusion while working with @lieter on an RPM plugin feature that
>>> allows users to download a repo config file from a distribution[0]. As a
>>> result, we decided to document that the signing service needs to produce a
>>> public key file named 'public.key'[1].
>>>
>>> We should revisit the design of the signing service API to ensure that
>>> we enforce this naming convention.
>>>
>>> [0] https://pulp.plan.io/issues/5356
>>> [1]
>>> https://github.com/pulp/pulp_rpm/pull/1687/files#diff-c91893c1f4e7afe73e414d1a76162463R30
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] the "relative path" problem

[Dennis Kliban]

I'd like to provide a little bit more context for my previous email by
going back to the original problem statement:

On Wed, Apr 1, 2020 at 9:23 AM Daniel Alley  wrote:

> Problem:
>
> Currently, a relative_path is tied to content in Pulp. This means that if
> a content unit exists in two places within a repository or across
> repositories, it has to be stored as two separate content units. This
> creates redundant data and potential confusion for users.
>
> As a specific example, we need to support mirroring content in pulp_rpm
> <https://pulp.plan.io/issues/6353>. Currently, for each location at which
> a single package is stored, we’ll need to create a content unit. We could
> end up with several records representing a single package. Users may be
> confused about why they see multiple records for a package and they may
> have trouble for example deciding which content unit to copy.
>
Users need to be able to store the same content unit at different relative
paths in different repository versions. This problem is not unique to the
RPM plugin. Do we agree about that?

I've been working on a potential solution that solves this problem in a
document[0]. It is a complicated change and the document does not fully
capture the plan yet. Feedback and help on the design is welcome.

[0] https://hackmd.io/02KBjCD3Q0WP7p4ALwzhJw?edit


On Mon, May 4, 2020 at 4:11 PM Dennis Kliban  wrote:

> I've reached two conclusions while trying to formulate a solution:
>
> This problem needs to be solved at the repository version level.
> Repository membership needs to be tracked at the artifact level, and not
> content level as it is now.
>
> On Thu, Apr 30, 2020 at 1:11 PM Daniel Alley  wrote:
>
>> Cool, so the only difference is whether to try to store the relationship
>> in the DB, or leverage the fact that we already have the metadata and just
>> re-parse it.
>>
>> I know the latter approach has yet to be written up, but my concern there
>> is that adding another layer of indirection between "repository version"
>> and "content" is going to have an adverse impact on performance, since it
>> is already the most complex and demanding query we issue to the DB and one
>> of the most common and important.
>>
>> On Thu, Apr 30, 2020 at 12:50 PM David Davis 
>> wrote:
>>
>>> Yes but I was imagining the mapping would be stored not as Content but
>>> as a separate object. So we wouldn't use filename for the mapping (rather
>>> we'd use ContentArtifact pk) and  we wouldn't need to change
>>> ContentArtifact's relative_path at all. That said, I think your solution
>>> captures the idea though and is better in some ways.
>>>
>>> Changing the RepositoryContent model to point to ContentArtifacts and
>>> store relative_paths is probably the best and most correct solution in
>>> theory. However, it's going to be painful to implement for both core and
>>> plugins.
>>>
>>> David
>>>
>>>
>>> On Thu, Apr 30, 2020 at 12:33 PM Daniel Alley  wrote:
>>>
>>>> @David Davis   so this proposal would go
>>>> something like this, correct?:
>>>>
>>>> * For the signed metadata / exact mirror use-case we need to store the
>>>> repository metadata itself as a content unit inside the RepositoryVersion
>>>> anyway (because the hash must be equal)
>>>> * Because we have this metadata lying around, we can reference it at
>>>> publish time to discover the appropriate PublishedArtifact.relative_path
>>>>* Create a map of "filename" -> "location_href" and look up the
>>>> filename of each RPM package to find the appropriate path
>>>>* This should be pretty fast for the RPM plugin since createrepo_c
>>>> is doing all the hard work
>>>> * Data migration to ensure ContentArtifact.relative_path is only
>>>> storing the filename (and I would suggest we also change the name to
>>>> "filename")
>>>> * If metadata isn't present in the RepositoryVersion, then just tweak
>>>> the PublishedArtifact.relative_path so that it uses whichever our default
>>>> repo layout is
>>>>
>>>> On Tue, Apr 28, 2020 at 11:41 AM David Davis 
>>>> wrote:
>>>>
>>>>> Yes, that's correct. During our meeting we discussed two options: the
>>>>> first was to extend RepositoryContent to store relative path per
>>>>> ContentArtifact as storing a relative_path per Content won't work for
>>>>> multi-Artifact Content units.
>>&g

Re: [Pulp-dev] Pulp 3 CLI MVP

[Dennis Kliban]

On Wed, May 6, 2020 at 12:30 PM David Davis  wrote:

> Matthias and I met today to go over some plans for a prototype. I wrote
> some notes[0] down. As part of the prototype, we'd propose two deliverables
> (one this week and one next week):
>
> 1. A set of ~2-3 click commands that use the bindings to interact with Pulp
> 2. Some openapi-generator templates that will be able to generate such
> commands from the schema
>
> There is a question we had about how the commands for typed resources will
> be structured in the CLI. To illustrate with two endpoints:
>
>
We should call the command 'pulp' instead of pulp-cli.


> # rpm.package content (/pulp/api/v3/content/rpm/packages/):
> - pulp-cli rpm content packages ...
> - pulp-cli content rpm packages ...
> - pulp-cli rpm packages content ...
> - ???
>

I was thinking we should structure the commands similar to the URLs in the
REST API.

pulp content rpm packages


>
> # file.file repositories (/pulp/api/v3/repositories/file/file/):
> - pulp-cli file repositories file ...
> - pulp-cli repositories file file ...
> - pulp-cli file file repositories ...
> - ???
>
> pulp repositories file

Plugins that provide multiple types of the same resource would need to be
more descriptive. Though I can see a practical reason for requiring all
resources to be that descriptive.

pulp repositories rpm rpm
pulp repositories rpm mirror



> [0] https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Prototype
>
> David
>
>
> On Thu, Apr 30, 2020 at 1:42 PM David Davis  wrote:
>
>> Today we met to discuss some ideas for a technical design for how the CLI
>> would work. Here's a copy of our notes:
>>
>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Technical-discussion
>>
>> And there is a rough design in the document as well:
>>
>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig#Design
>>
>> I have also entered the CLI user stories from our meeting last week into
>> redmine under the Pulp CLI project:
>>
>> https://pulp.plan.io/versions/93
>>
>> And I've filed a user story that we talked about today that would handle
>> sync, publish, and distribution of repos. Feedback welcome:
>>
>> https://pulp.plan.io/issues/6626
>>
>> Matthias and I are planning to meet next week to look at creating a proof
>> of concept that would provide 2-3 commands. If anyone is interested in
>> joining us, please let me know and I can add you.
>>
>> David
>>
>>
>> On Tue, Apr 28, 2020 at 8:06 AM David Davis 
>> wrote:
>>
>>> I've also started working on some questions about how the CLI will work.
>>> Feel free to add some of your own:
>>>
>>> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig?view#Technical-discussion
>>>
>>> David
>>>
>>>
>>> On Tue, Apr 28, 2020 at 8:05 AM David Davis 
>>> wrote:
>>>
 I have set up a meeting to discuss the CLI technical design. Below are
 the details. I think a video conference might be easier for technical
 discussion but am open to consider meeting on #pulp-meeting again.

 URL: https://meet.google.com/vgx-bzbb-wnh
 Date/time: April 30, 2020 at 9:00am ET (1pm UTC)

 David


 On Fri, Apr 24, 2020 at 10:29 AM David Davis 
 wrote:

> Today we met in #pulp-meeting on freenode to discuss the user stories
> for a Pulp 3 CLI MVP. The document with the user stories is available
> below. I'd like to ask for any feedback from users or plugin writers.
>
> The goal of the CLI MVP is to cover the pulp_file happy path (sync,
> publish, distribute) and make it possible for plugin writers to generate
> and write their own commands. I'm imagining that plugins will release 
> their
> own sets of CLI commands after we complete the initial MVP.
>
> https://hackmd.io/aH9RqAS_TrGyxoi1UGRgig
>
> Feedback is welcome. I plan to enter these user stories into redmine
> next week.
>
> David
>
 ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] signing service interface

[Dennis Kliban]

On Tue, May 5, 2020 at 3:39 AM Quirin Pamp  wrote:

> Could you explain the reasoning for a 'public.key' file?
>

The public.key file is the file that a yum/dnf client can use to verify
that the metadata in an RPM repository was signed by the signing service
associated with the repository. The name of the file can be anything - the
path to it needs to be specified in the repository config on the client.


> In the case of the AptReleaseSigningService we built for pulp_deb we saw
> zero need for this file and consequently did not add it in.
>
> (It would not be hard to add it just to satisfy the interface, it just
> would not serve any useful purpose.)
>

It is definitely up to each plugin if it wants to provide the public key as
part of the publication. It is currently impossible for the plugin to know
exactly what files are produced by the signing service. This is where I
would like to see an improvement in the API. Pupcore should provide a
guarantee to plugin writers that a signing service configured by an
administrator is functioning in a predictable way. One possible way to do
that is with an interface that lets a plugin writer inspect a signing
service without executing it. Though I am looking for other ideas in this
area.


>
> Since we are on the topic of signing services, a colleague has had a PR
> relating to them just sitting their waiting for a review for quite a while
> now ;-):
> https://github.com/pulp/pulpcore/pull/659
>
>
> It would be great if you (or somebody else) could have a look at it. I
> believe it is mostly ready, but probably needs the eyes of an experienced
> pulp core developer to look over it and suggest style consistency changes
> and where and whether to add documentation. ;-)
>

I'll take a look at this PR.



>
> Quirin
> ------
> *From:* pulp-dev-boun...@redhat.com  on
> behalf of Dennis Kliban 
> *Sent:* 04 May 2020 22:50:54
> *To:* Pulp-dev 
> *Subject:* [Pulp-dev] signing service interface
>
> The Plugin API of Signing Services in Pulp 3 is too vague. I came to this
> conclusion while working with @lieter on an RPM plugin feature that allows
> users to download a repo config file from a distribution[0]. As a result,
> we decided to document that the signing service needs to produce a public
> key file named 'public.key'[1].
>
> We should revisit the design of the signing service API to ensure that we
> enforce this naming convention.
>
> [0] https://pulp.plan.io/issues/5356
> [1]
> https://github.com/pulp/pulp_rpm/pull/1687/files#diff-c91893c1f4e7afe73e414d1a76162463R30
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] signing service interface

[Dennis Kliban]

The Plugin API of Signing Services in Pulp 3 is too vague. I came to this
conclusion while working with @lieter on an RPM plugin feature that allows
users to download a repo config file from a distribution[0]. As a result,
we decided to document that the signing service needs to produce a public
key file named 'public.key'[1].

We should revisit the design of the signing service API to ensure that we
enforce this naming convention.

[0] https://pulp.plan.io/issues/5356
[1]
https://github.com/pulp/pulp_rpm/pull/1687/files#diff-c91893c1f4e7afe73e414d1a76162463R30
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] improving functional tests

[Dennis Kliban]

I've written an issue to track improvements to functional tests[0]. Please
provide feedback on this thread or on the issue directly.


[0] https://pulp.plan.io/issues/6637
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] the "relative path" problem

[Dennis Kliban]

I've reached two conclusions while trying to formulate a solution:

This problem needs to be solved at the repository version level. Repository
membership needs to be tracked at the artifact level, and not content level
as it is now.

On Thu, Apr 30, 2020 at 1:11 PM Daniel Alley  wrote:

> Cool, so the only difference is whether to try to store the relationship
> in the DB, or leverage the fact that we already have the metadata and just
> re-parse it.
>
> I know the latter approach has yet to be written up, but my concern there
> is that adding another layer of indirection between "repository version"
> and "content" is going to have an adverse impact on performance, since it
> is already the most complex and demanding query we issue to the DB and one
> of the most common and important.
>
> On Thu, Apr 30, 2020 at 12:50 PM David Davis 
> wrote:
>
>> Yes but I was imagining the mapping would be stored not as Content but as
>> a separate object. So we wouldn't use filename for the mapping (rather we'd
>> use ContentArtifact pk) and  we wouldn't need to change ContentArtifact's
>> relative_path at all. That said, I think your solution captures the idea
>> though and is better in some ways.
>>
>> Changing the RepositoryContent model to point to ContentArtifacts and
>> store relative_paths is probably the best and most correct solution in
>> theory. However, it's going to be painful to implement for both core and
>> plugins.
>>
>> David
>>
>>
>> On Thu, Apr 30, 2020 at 12:33 PM Daniel Alley  wrote:
>>
>>> @David Davis   so this proposal would go
>>> something like this, correct?:
>>>
>>> * For the signed metadata / exact mirror use-case we need to store the
>>> repository metadata itself as a content unit inside the RepositoryVersion
>>> anyway (because the hash must be equal)
>>> * Because we have this metadata lying around, we can reference it at
>>> publish time to discover the appropriate PublishedArtifact.relative_path
>>>* Create a map of "filename" -> "location_href" and look up the
>>> filename of each RPM package to find the appropriate path
>>>* This should be pretty fast for the RPM plugin since createrepo_c is
>>> doing all the hard work
>>> * Data migration to ensure ContentArtifact.relative_path is only storing
>>> the filename (and I would suggest we also change the name to "filename")
>>> * If metadata isn't present in the RepositoryVersion, then just tweak
>>> the PublishedArtifact.relative_path so that it uses whichever our default
>>> repo layout is
>>>
>>> On Tue, Apr 28, 2020 at 11:41 AM David Davis 
>>> wrote:
>>>
 Yes, that's correct. During our meeting we discussed two options: the
 first was to extend RepositoryContent to store relative path per
 ContentArtifact as storing a relative_path per Content won't work for
 multi-Artifact Content units.

 An alternative that I pitched was to have plugins (or maybe even core
 someday) store this information outside RepositoryContent and then use this
 information during publishing to set relative_path on PublishedArtifacts.
 We'd have to modify the content app if we wanted to support pass through
 publications but I think asking plugins to use published artifacts in this
 case is warranted. That said, I don't think anyone else was keen on this
 idea though.

 David


 On Tue, Apr 28, 2020 at 10:30 AM Matthias Dellweg 
 wrote:

> That is only used for passthrough publication afaik. If you publish
> each content unit "by hand", you create a new relative path for each
> published artifact. That is, why it can be empty and still the content can
> be published.
>
> On Tue, Apr 28, 2020 at 4:09 PM Daniel Alley 
> wrote:
>
>> We realized in our discussion that the original proposal described in
>> my email will not work, because "relative_path" ultimately describes the
>> path of the published *artifacts* (not content), and for content
>> types with multiple artifacts, storing this information in a field on
>> RepositoryContent would not be possible.
>>
>> On Mon, Apr 27, 2020 at 6:08 PM Daniel Alley 
>> wrote:
>>
>>> There is a video call scheduled to discuss this issue tomorrow
>>> (Tuesday April 28th) at 13:30 UTC (please convert to your local time).
>>> https://meet.google.com/scy-csbx-qiu
>>>
>>> On Sat, Apr 25, 2020 at 7:02 AM David Davis 
>>> wrote:
>>>
 I had a chance to think about this some more yesterday and wanted
 to email out my thoughts. I also think that this change sounds scary 
 and
 will have a big impact on plugin writers so I thought of a couple
 alternatives:

 First, we could add a relative_path field to RepositoryContent
 instead of moving it there. This would be an optional field. It would 
 be up
 to plugins to manage this field and they would still need to populate 

Re: [Pulp-dev] serializers in pulp 3 can't use write_only fields

[Dennis Kliban]

This discussion grew out of concern for the users of bindings. These users
don't usually instantiate models when performing GET requests. Instead,
they pass in a pulp_href to an api client object and receive an
instantiated model. On the other hand, when performing POST/PATCH
operations, the user does instantiate a model. So another solution to the
problem is to have the openapi generator automatically split the
serializers into two when write_only fields are present. However, only
rename the serializer used for GET operations. In this case, users will be
guaranteed that models that they do instantiate themselves, will never
change in name. If there is code that performs some kind of type checking
on the model returned by a GET operation, the user would need to update
that code.

I prefer this solution.

On Mon, Apr 27, 2020 at 9:11 AM Daniel Alley  wrote:

> I just want to point out that we have a somewhat similar kind of feature
> implemented, "minimal serializers", which uses the 2-serializer approach.
> It is a very tiny amount of extra code/work.
>
> e.g.
>
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/serializers/task.py#L119
>
> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/serializers.py#L290
>
> the implementation
>
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/viewsets/base.py#L114-L139
>
> I lean towards option 2 as well for consistency.
>
> On Mon, Apr 27, 2020 at 5:20 AM Ina Panova  wrote:
>
>> I like option number 2 more, even if there is more work behind the scene
>> it is more explicit and does not leave opened questions like ' why certain
>> fields are null'
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Fri, Apr 24, 2020 at 5:24 PM Dennis Kliban  wrote:
>>
>>> Today during open floor we concluded that write_only fields cannot be
>>> used in serializers because any automated handling of them during OpenAPI
>>> schema generation will cause mutations to the schema which will result in
>>> backwards incompatible changes to our bindings users.
>>>
>>> This change should only affect the Content serializers that have a
>>> 'file' and 'repository' write_only fields that can be used for uploading
>>> content into a repository in a single request. Two solutions were proposed:
>>>
>>> 1) Use a single serializer for GET and POST and leave 'file' and
>>> 'repository' null for GET requests. This solution is the simplest for
>>> plugin writers because they don't have to do anything extra. However, it
>>> does cause users to wonder why those fields are present and null on GET
>>> requests.
>>>
>>> 2) Require plugin writers to provide 2 serializers for Content - one for
>>> GET and another for POST. This is more work for plugin writers, but will
>>> make it clear to users what fields are expected in each case.
>>>
>>> Even though I initially proposed 1, I am in favor of adopting 2. What
>>> does everyone else think?
>>>
>>> We will need to provide validation at startup that will check
>>> serializers for presence of write_only fields. The application should fail
>>> to start if any are present. This way plugin writers will know not to use
>>> the fields. We will also need to provide plugin writer docs that clearly
>>> state the guidelines around write_only fields and suggestions for getting
>>> by without them.
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] unit test runners disabled for Pulp 2 repositories

[Dennis Kliban]

Due to challenges with maintaining infrastructure used to run unit tests
for Pulp 2 Pull Requests on GitHub, I've disabled the requirement for these
checks to pass before merging. I do not anticipate any challenges with
manually testing PRs due to the low volume of changes in those repos.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] serializers in pulp 3 can't use write_only fields

[Dennis Kliban]

Today during open floor we concluded that write_only fields cannot be used
in serializers because any automated handling of them during OpenAPI schema
generation will cause mutations to the schema which will result in
backwards incompatible changes to our bindings users.

This change should only affect the Content serializers that have a 'file'
and 'repository' write_only fields that can be used for uploading content
into a repository in a single request. Two solutions were proposed:

1) Use a single serializer for GET and POST and leave 'file' and
'repository' null for GET requests. This solution is the simplest for
plugin writers because they don't have to do anything extra. However, it
does cause users to wonder why those fields are present and null on GET
requests.

2) Require plugin writers to provide 2 serializers for Content - one for
GET and another for POST. This is more work for plugin writers, but will
make it clear to users what fields are expected in each case.

Even though I initially proposed 1, I am in favor of adopting 2. What does
everyone else think?

We will need to provide validation at startup that will check serializers
for presence of write_only fields. The application should fail to start if
any are present. This way plugin writers will know not to use the fields.
We will also need to provide plugin writer docs that clearly state the
guidelines around write_only fields and suggestions for getting by without
them.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] plugin API improvement for custom content handlers

[Dennis Kliban]

I've been working with @lieter on adding a new feature to the RPM
plugin[0]. We've identified that an improvement in pulpcore would make for
a better user experience. I've written up the pulpcore improvement in a
separate story[1]. I am looking for feedback on both features.

[0] https://pulp.plan.io/issues/5356
[1] https://pulp.plan.io/issues/6570
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Turning release notes into user content

[Dennis Kliban]

On Mon, Apr 20, 2020 at 2:38 PM Melanie Corr  wrote:

> Hi all,
>
> As part of the effort to broaden awareness of Pulp and increase
> interactions with both current and future community members, I am looking
> into increasing frequency of blog posts on the Pulp site. This will also
> help with a social media strategy also.
>
> Currently, I can see that there is a high level of transparency on the
> site, with release note update information being regularly posted, but with
> some effort this could be slightly more digestible for the reader.
>
> I would like to try and take this content that is gathered with the
> towncrier tool to see if I could develop the output into user content, for
> example, why this incremental update benefits the user, how does fixing x
> bug make the user's life easier, and so on.
>
> To do this, I would need to collect the output of towncrier myself, or the
> draft output if I wanted to prepare a post in advance of a release.
>
> Can you advise how I would go about collecting this?
> If you have any further suggestions, I would love to hear them.
>
>
You will need to install 'towncrier' from PyPI and then run it from the
root of the pulpcore repository. Here are the commands I use for that.

pip install towncrier
git clone https://github.com/pulp/pulpcore
cd pulpcore
towncrier --version x.y.z --draft


Thanks,
>
> --
>
> Melanie Corr, RHCE
>
> Community Manager
>
> Red Hat 
>
> Remote, Ireland
>
> mc...@redhat.com
> M: +353857774436 IM: mcorr
> 
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 in One Container

[Dennis Kliban]

I've moved the git repository to the pulp org[0]. I've also pushed the
image to the pulp org on docker hub[1]. I also updated the blog post with
the new locations. I also changed the commands slightly to make them more
user friendly. The launched container now has the name 'pulp' so the final
command for changing the password no longer uses a checksum to refer to the
container.


[0] https://github.com/pulp/pulp-fedora31
[1] https://hub.docker.com/repository/docker/pulp/pulp-fedora31
[2] https://pulpproject.org/2020/03/15/pulp-fedora31-single-container/

On Thu, Apr 16, 2020 at 2:02 PM David Davis  wrote:

> +1 from me. @mcorr, thank you for raising this.
>
> David
>
>
> On Thu, Apr 16, 2020 at 1:46 PM Dennis Kliban  wrote:
>
>> That sounds like a great idea. I'll move the git repository under 'pulp'
>> organization on GitHub and the same for Docker Hub image repository. I'll
>> update this thread when this has been done.
>>
>> On Thu, Apr 16, 2020 at 10:26 AM Melanie Corr  wrote:
>>
>>> Hi all,
>>>
>>> I was reading Dennis Kilban's blog post [1] that talks about using Pulp
>>> 3 in a container. Although this is not production-ready as of yet, it is a
>>> valuable tool for users.
>>>
>>> I would like to take the content of Dennis's blog and create a 'Try Pulp
>>> 3 in a Container' page so that it is easy for new users to find, with the
>>> caveat that this is not production ready.
>>>
>>> One thing that I think could improve is the location of the container.
>>> Currently it is located in Dennis's GitHub space. It would be better, if we
>>> were to store it somewhere within the main Pulp project space.
>>>
>>> Do you think that would be possible?
>>>
>>> Thanks,
>>>
>>> Melanie
>>>
>>> [1]
>>> https://pulpproject.org/2020/03/15/pulp-fedora31-single-container/
>>>
>>> --
>>>
>>> Melanie Corr, RHCE
>>>
>>> Community Manager
>>>
>>> Red Hat <https://www.redhat.com>
>>>
>>> Remote, Ireland
>>>
>>> mc...@redhat.com
>>> M: +353857774436 IM: mcorr
>>> <https://www.redhat.com>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 in One Container

[Dennis Kliban]

That sounds like a great idea. I'll move the git repository under 'pulp'
organization on GitHub and the same for Docker Hub image repository. I'll
update this thread when this has been done.

On Thu, Apr 16, 2020 at 10:26 AM Melanie Corr  wrote:

> Hi all,
>
> I was reading Dennis Kilban's blog post [1] that talks about using Pulp 3
> in a container. Although this is not production-ready as of yet, it is a
> valuable tool for users.
>
> I would like to take the content of Dennis's blog and create a 'Try Pulp 3
> in a Container' page so that it is easy for new users to find, with the
> caveat that this is not production ready.
>
> One thing that I think could improve is the location of the container.
> Currently it is located in Dennis's GitHub space. It would be better, if we
> were to store it somewhere within the main Pulp project space.
>
> Do you think that would be possible?
>
> Thanks,
>
> Melanie
>
> [1]
> https://pulpproject.org/2020/03/15/pulp-fedora31-single-container/
>
> --
>
> Melanie Corr, RHCE
>
> Community Manager
>
> Red Hat 
>
> Remote, Ireland
>
> mc...@redhat.com
> M: +353857774436 IM: mcorr
> 
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] SUSE repositories in Pulp

[Dennis Kliban]

During last week's RPM team meeting, a concern was raised about using the
same repository type for both Red Hat and SUSE repositories. Since that
meeting I have only been able to identify a single difference between the
two repositories. SUSE repos can contain the same package in two different
locations in the same repository. Even though I just referred to this as a
difference, I don't actually believe that to be true. All RPM repositories
should be able to support this.

I propose that we not add a separate repository type for SUSE and simply
add the 'location' attribute of an RPM to it's uniqueness constraint.  What
do you all think?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Should signing service be associated with Publication or Repository?

[Dennis Kliban]

On Fri, Mar 20, 2020 at 8:35 AM Neal Gompa  wrote:

> On Thu, Mar 19, 2020 at 11:14 PM Dennis Kliban  wrote:
> >
> > RPM plugin allows users to define a signing service per repository. All
> publications created from repository versions of that repository are signed
> with that signing service.
> >
> > The Debian plugin requires the user to specify the signing service each
> time a publication is created. The signing service foreign key is stored
> with each publication.
> >
> > Even though the implementation in Debian requires the user to provide
> the service href each time a publication is created, it seems like a
> stronger model. The signing service associated with a repository can change
> thus making it challenging to keep track of which signing service was used
> to create a publication.
> >
> > We should change the behavior in the RPM plugin before we release this
> feature.
>
> Isn't the reason for the difference that Debian repos only have
> repodata signed and not packages?
>
> I guess technically we could use different GPG keys for each
> repository publish, but that would lead to multiple copies of the same
> RPM with different data, since the expectation is that both RPMs and
> the repodata should be signed for RPM repositories.
>
> The RPM plugin does not currently provide the ability to sign packages.
This discussion is only about singing the metadata.



>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Should signing service be associated with Publication or Repository?

[Dennis Kliban]

On Fri, Mar 20, 2020 at 5:45 AM Matthias Dellweg  wrote:

> Actually Quirin and I have also seen the difference, and we discussed,
> we should implement a combination, where you can specify on the
> repo-level but override the signing service with each publication.
> It is a little more work for a lot more convenience, imho.
> And of course it might be nice to see it handled as similarly as
> possible in both plugins.
>
>
I like this idea. In both cases though, the signing service foreign key
should always be stored with a publication. Is that what you had in mind
also?


> On Thu, 19 Mar 2020 23:13:23 -0400
> Dennis Kliban  wrote:
>
> > RPM plugin allows users to define a signing service per repository.
> > All publications created from repository versions of that repository
> > are signed with that signing service.
> >
> > The Debian plugin requires the user to specify the signing service
> > each time a publication is created. The signing service foreign key
> > is stored with each publication.
> >
> > Even though the implementation in Debian requires the user to provide
> > the service href each time a publication is created, it seems like a
> > stronger model. The signing service associated with a repository can
> > change thus making it challenging to keep track of which signing
> > service was used to create a publication.
> >
> > We should change the behavior in the RPM plugin before we release this
> > feature.
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Should signing service be associated with Publication or Repository?

[Dennis Kliban]

RPM plugin allows users to define a signing service per repository. All
publications created from repository versions of that repository are signed
with that signing service.

The Debian plugin requires the user to specify the signing service each
time a publication is created. The signing service foreign key is stored
with each publication.

Even though the implementation in Debian requires the user to provide the
service href each time a publication is created, it seems like a stronger
model. The signing service associated with a repository can change thus
making it challenging to keep track of which signing service was used to
create a publication.

We should change the behavior in the RPM plugin before we release this
feature.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Changelog conventions

[Dennis Kliban]

On Wed, Mar 11, 2020 at 3:33 PM David Davis  wrote:

> I want to reopen this thread for discussion as this topic has come up
> again this week. I don't think we need very strict or detailed requirements
> for changelog messages. But rather, some guidance around things like which
> tense to use would be helpful to standardize the changelog and not cause us
> to debate formatting in our PRs.
>
> Regarding tense, I think either past simple or present makes sense. I also
> believe that the changelog message should describe the fix and not the
> problem (unlike the associated redmine issue).
>
> +1 ... This works for me.



> David
>
>
> On Wed, Mar 4, 2020 at 9:23 AM Robin Chan  wrote:
>
>> Thanks for closing out this thread/topic, Lubos. That is a helpful
>> practice.
>>
>> Given Ina's input, I see that as a concern, because best practice for
>> bugs is to have a title or description of the behavior that is a problem
>> and NOT the solution so that they can easily be found by others
>> experiencing the same bug. Whereas with features, it is best to have a
>> title or description of new behavior being added. So I can see that this
>> would make it hard to standardize practices here.
>>
>> Robin Chan
>>
>> She/Her/Hers
>>
>> Satellite Software Engineering Manager - Pulp
>>
>> Red Hat 
>>
>> IRC: rchan
>> 
>>
>>
>>
>> On Fri, Feb 28, 2020 at 6:04 AM Lubos Mjachky 
>> wrote:
>>
>>> No inputs have been received for a week. Therefore, the changelog
>>> conventions should remain the same. That is, basically, no rules, and no
>>> conventions at all. I will continue using past simple on my own.
>>>
>>> On Mon, Feb 17, 2020 at 5:31 PM Ina Panova  wrote:
>>>
 Thank you for starting this thread.

 The format of the changelog is different because the usual practice is
 to re-use the title of the issue/story/task/refactor.

 If we want to create some standards than maybe we should start with the
 titles from redmine.

 
 Regards,

 Ina Panova
 Senior Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Mon, Feb 17, 2020 at 4:55 PM Lubos Mjachky 
 wrote:

> Dear colleagues,
>
> I have recently noticed that our changelog often contains non-uniform
> messages informing about particular changes.
>
> For instance, take a look at the changelogs in pulp_container (
> https://pulp-container.readthedocs.io/en/latest/changes.html) or
> pulpcore (https://docs.pulpproject.org/en/master/nightly/changes.html
> ):
> 1. As a user I can manage images in OCI format.
> 2. Let users fetch the list of all distributed repositories via the
> _catalog endpoint.
> 3. Adds ability to build OCI images from Containerfiles.
> 4. Added v2s2 to v2s1 converter.
> 5. Allow administrators to add a signing service.
> 6. Files stored on S3 and Azure now download with the correct filename.
>
> As you can see, we are using there random tenses and sentence
> structures. I am in favor of establishing one feasible convention for all
> changelog messages. We should strive to use only one tense, e.g. past
> simple. Then, the messages would rather look like this [0]:
> 1. Removed the filter for the field 'digest'.
> 2. Added support for mirror mode.
>
> Please feel free to support this idea or raise any concerns. If we
> reach a viable consensus I will create a PR which will add a note about
> acceptable changelog messages to the corresponding section here
> https://docs.pulpproject.org/en/master/nightly/contributing/git.html#changelog-update.
> Thank you in advance.
>
> [0] https://keepachangelog.com/en/1.0.0/#how
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] pulpcore 3.2.0 and pulp_file 0.2.0 releases

[Dennis Kliban]

pulp_container 1.2.0 has been released[0]. It is compatible with pulpcore
3.2. For a list of all changes please check the changelog[1].

[0]: https://pypi.org/project/pulp-container/1.2.0/
[1] https://pulp-container.readthedocs.io/en/latest/changes.html#id1

On Thu, Feb 27, 2020 at 4:02 PM David Davis  wrote:

> pulpcore 3.2.0[0] and pulp_file 0.2.0[1] have been released. For a list of
> all changes, please check the changelogs for pulpcore[2] and pulp_file[3].
>
> # Installation and Upgrade
>
> Users should use the 3.2.0 release of ansible-pulp installer[4] to install
> or upgrade their installations. This version of the installer will check
> compatibility of all installed plugins with pulpcore 3.2. The installer
> will abort if any plugin is incompatible.
>
> # Plugin API
>
> Plugin writers can see the plugin API changelog here (
> https://docs.pulpproject.org/en/3.2.0/changes.html#plugin-api ). There
> was only one backwards incompatible change[5], and in keeping with the
> recommended strategy to pin plugins to a 3.y version of pulpcore, plugins
> should release compatibility releases with 3.2 as soon as they can. We
> recommend using "pulpcore>=3.0,<3.3".
>
> The installer also has a backwards incompatible change in 3.2.0 where it
> no longer attempts to handle custom URLs for plugins automatically. If your
> plugin uses URLs outside of /pulp/api/v3/ or /pulp/content/ you will need
> to add a snippet. See these docs[6] for more.
>
> [0]: https://pypi.org/project/pulpcore/3.2.0/
> [1]: https://pypi.org/project/pulp-file/0.2.0/
> [2]: https://docs.pulpproject.org/changes.html#id1
> [3]: https://pulp-file.readthedocs.io/en/0.2.0/changes.html#id1
> [4]: https://github.com/pulp/ansible-pulp/releases/tag/3.2.0
> [5]: https://github.com/pulp/pulpcore/pull/555
> [6]: https://pulp.plan.io/issues/6057
>
> David
> ___
> Pulp-list mailing list
> pulp-l...@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importers/Exporters

[Dennis Kliban]

We can't provide any data from the Katello database, but we can provide
enough data for the archive to contain all the published metadata and
distributions needed to not require the user any extra steps to make the
content available after import.

We could definitely limit which resources are allowed to be specified for
this API. The user would never have to specify pulp_href for an artifact.
Content would only be exported using repositories, repository versions, or
publications. If a user chooses to export a repository, all the repository
versions for that repository would be exported along with the content and
artifacts that belong to those repo versions. When individual repository
versions are specified, only those repository versions are exported.
Publications would work the same way.

My main goal is to make the import process as simple as possible for the
user.

On Fri, Feb 21, 2020 at 7:53 AM David Davis  wrote:

> A couple comments. I'm not sure how pulp will be able to export all the
> extra metadata that comes from Katello as some of it relates to content
> views. Also, I'm hesitant to have the user export a generic list of pulp
> hrefs. I think this could be confusing (do users have to supply artifact
> hrefs to get artifacts?). I'd rather have a list of params users can
> specifically export (eg repository_versions, publications, etc). I think
> Pulp will have decide what you get when you for example export a repository
> version (likely Content, Artifacts, ContentArtifacts).
>
> David
>
>
> On Thu, Feb 20, 2020 at 7:13 PM Dennis Kliban  wrote:
>
>> Thanks for all the details. I would like to provide Pulp 3 users with a
>> similar feature. In order to do that, the archive produced by Pulp will
>> need to include all that extra metadata that comes from Katello right now.
>> Pulp should support 2 use cases:
>>
>>   - As a user, I can generate an archive by specifying a list of
>> pulp_hrefs.
>>   - As a user, I can import an archive that was generated on another pulp.
>>
>> The archive would contain database migrations needed to restore all the
>> resources. It would also have all the files needed to back the artifacts.
>>
>> Users could then provide a list of repository versions, publications, and
>> distributions when creating an artchive. Once the archive is imported, Pulp
>> is serving the content without having to republish.
>>
>> On Thu, Feb 20, 2020 at 9:53 AM Justin Sherrill 
>> wrote:
>>
>>> There are two different forms of export today in katello:
>>>
>>> Legacy version:
>>>
>>>   * Uses pulp2's export functionality
>>>
>>>   * Takes the tarball as is
>>>
>>> "New" Version
>>>
>>>* Just copies published repository as is (following symlinks)
>>>
>>>* Adds own 'katello' metadata to existing tarball
>>>
>>>
>>> I would imagine that with pulp3 we would somewhat combine these two
>>> approaches and take the pulp3 generated export file and add in a metadata
>>> file of some sort.
>>>
>>> Justin
>>> On 2/19/20 2:28 PM, Dennis Kliban wrote:
>>>
>>> Thank you for the details. More questions inline.
>>>
>>> On Wed, Feb 19, 2020 at 2:04 PM Justin Sherrill 
>>> wrote:
>>>
>>>> the goal from our side is to have a very similar experience to the
>>>> user.  Today the user would:
>>>>
>>>> * run a command (for example, something similar to hammer content-view
>>>> version export --content-view-name=foobar --version=1.0)
>>>>
>>>> * this creates a tarball on disk
>>>>
>>> What all is in the tarball? Is this just a repository export created by
>>> Pulp or is there extra information from the Katello db?
>>>
>>>> * they copy the tarball to external media
>>>>
>>>> * they move the external media to the disconnected katello
>>>>
>>>> * they run 'hammer content-view version import
>>>> --export-tar=/path/to/tarball
>>>>
>>> Does katello untar this archive, create a repository in pulp, sync from
>>> the directory containing the unarchive, and then publish?
>>>
>>>> I don't see this changing much for the user, anything additional that
>>>> needs to be done in pulp can be done behind the cli/api in katello.  
>>>> Thanks!
>>>>
>>>
>>>
>>>
>>>> Justin
>>>> On 2/19/20 12:52 PM, Dennis Kliban wrote:
>>>>
>>>> In Katello that uses Pulp 2, what steps does

Re: [Pulp-dev] Importers/Exporters

[Dennis Kliban]

Thanks for all the details. I would like to provide Pulp 3 users with a
similar feature. In order to do that, the archive produced by Pulp will
need to include all that extra metadata that comes from Katello right now.
Pulp should support 2 use cases:

  - As a user, I can generate an archive by specifying a list of
pulp_hrefs.
  - As a user, I can import an archive that was generated on another pulp.

The archive would contain database migrations needed to restore all the
resources. It would also have all the files needed to back the artifacts.

Users could then provide a list of repository versions, publications, and
distributions when creating an artchive. Once the archive is imported, Pulp
is serving the content without having to republish.

On Thu, Feb 20, 2020 at 9:53 AM Justin Sherrill  wrote:

> There are two different forms of export today in katello:
>
> Legacy version:
>
>   * Uses pulp2's export functionality
>
>   * Takes the tarball as is
>
> "New" Version
>
>* Just copies published repository as is (following symlinks)
>
>* Adds own 'katello' metadata to existing tarball
>
>
> I would imagine that with pulp3 we would somewhat combine these two
> approaches and take the pulp3 generated export file and add in a metadata
> file of some sort.
>
> Justin
> On 2/19/20 2:28 PM, Dennis Kliban wrote:
>
> Thank you for the details. More questions inline.
>
> On Wed, Feb 19, 2020 at 2:04 PM Justin Sherrill 
> wrote:
>
>> the goal from our side is to have a very similar experience to the user.
>> Today the user would:
>>
>> * run a command (for example, something similar to hammer content-view
>> version export --content-view-name=foobar --version=1.0)
>>
>> * this creates a tarball on disk
>>
> What all is in the tarball? Is this just a repository export created by
> Pulp or is there extra information from the Katello db?
>
>> * they copy the tarball to external media
>>
>> * they move the external media to the disconnected katello
>>
>> * they run 'hammer content-view version import
>> --export-tar=/path/to/tarball
>>
> Does katello untar this archive, create a repository in pulp, sync from
> the directory containing the unarchive, and then publish?
>
>> I don't see this changing much for the user, anything additional that
>> needs to be done in pulp can be done behind the cli/api in katello.  Thanks!
>>
>
>
>
>> Justin
>> On 2/19/20 12:52 PM, Dennis Kliban wrote:
>>
>> In Katello that uses Pulp 2, what steps does the user need to take when
>> importing an export into an air gapped environment? I am concerned about
>> making the process more complicated than what the user is already used to.
>>
>> On Wed, Feb 19, 2020 at 11:20 AM David Davis 
>> wrote:
>>
>>> Thanks for the responses so far. I think we could export publications
>>> along with the repo version by exporting any publication that points to a
>>> repo version.
>>>
>>> My concern with exporting repositories is that users will probably get a
>>> bunch of content they don't care about if they want to export a single repo
>>> version. That said, if users do want to export entire repos, we could add
>>> this feature later I think?
>>>
>>> David
>>>
>>>
>>> On Wed, Feb 19, 2020 at 10:30 AM Justin Sherrill 
>>> wrote:
>>>
>>>>
>>>> On 2/14/20 1:09 PM, David Davis wrote:
>>>>
>>>> Grant and I met today to discuss importers and exporters[0] and we'd
>>>> like some feedback before we proceed with the design. To sum up this
>>>> feature briefly: users can export a repository version from one Pulp
>>>> instance and import it to another.
>>>>
>>>> # Master/Detail vs Core
>>>>
>>>> So one fundamental question is whether we should use a Master/Detail
>>>> approach or just have core control the flow but call out to plugins to get
>>>> export formats.
>>>>
>>>> To give some background: we currently define Exporters (ie
>>>> FileSystemExporter) in core as Master models. Plugins extend this model
>>>> which allows them to configure or customize the Exporter. This was
>>>> necessary because some plugins need to export Publications (along with
>>>> repository metadata) while other plugins who don't have Publications or
>>>> metadata export RepositoryVersions.
>>>>
>>>> The other option is to have core handle the workflow. The user would
>>>> call a core endpoint and p

Re: [Pulp-dev] Importers/Exporters

[Dennis Kliban]

Thank you for the details. More questions inline.

On Wed, Feb 19, 2020 at 2:04 PM Justin Sherrill  wrote:

> the goal from our side is to have a very similar experience to the user.
> Today the user would:
>
> * run a command (for example, something similar to hammer content-view
> version export --content-view-name=foobar --version=1.0)
>
> * this creates a tarball on disk
>
What all is in the tarball? Is this just a repository export created by
Pulp or is there extra information from the Katello db?

> * they copy the tarball to external media
>
> * they move the external media to the disconnected katello
>
> * they run 'hammer content-view version import
> --export-tar=/path/to/tarball
>
Does katello untar this archive, create a repository in pulp, sync from the
directory containing the unarchive, and then publish?

> I don't see this changing much for the user, anything additional that
> needs to be done in pulp can be done behind the cli/api in katello.  Thanks!
>



> Justin
> On 2/19/20 12:52 PM, Dennis Kliban wrote:
>
> In Katello that uses Pulp 2, what steps does the user need to take when
> importing an export into an air gapped environment? I am concerned about
> making the process more complicated than what the user is already used to.
>
> On Wed, Feb 19, 2020 at 11:20 AM David Davis 
> wrote:
>
>> Thanks for the responses so far. I think we could export publications
>> along with the repo version by exporting any publication that points to a
>> repo version.
>>
>> My concern with exporting repositories is that users will probably get a
>> bunch of content they don't care about if they want to export a single repo
>> version. That said, if users do want to export entire repos, we could add
>> this feature later I think?
>>
>> David
>>
>>
>> On Wed, Feb 19, 2020 at 10:30 AM Justin Sherrill 
>> wrote:
>>
>>>
>>> On 2/14/20 1:09 PM, David Davis wrote:
>>>
>>> Grant and I met today to discuss importers and exporters[0] and we'd
>>> like some feedback before we proceed with the design. To sum up this
>>> feature briefly: users can export a repository version from one Pulp
>>> instance and import it to another.
>>>
>>> # Master/Detail vs Core
>>>
>>> So one fundamental question is whether we should use a Master/Detail
>>> approach or just have core control the flow but call out to plugins to get
>>> export formats.
>>>
>>> To give some background: we currently define Exporters (ie
>>> FileSystemExporter) in core as Master models. Plugins extend this model
>>> which allows them to configure or customize the Exporter. This was
>>> necessary because some plugins need to export Publications (along with
>>> repository metadata) while other plugins who don't have Publications or
>>> metadata export RepositoryVersions.
>>>
>>> The other option is to have core handle the workflow. The user would
>>> call a core endpoint and provide a RepositoryVersion. This would work
>>> because for importing/exporting, you wouldn't ever use Publications because
>>> metadata won't be used for importing back into Pulp. If needed, core could
>>> provide a way for plugin writers to write custom handlers/exporters for
>>> content types.
>>>
>>> If we go with the second option, the question then becomes whether we
>>> should divorce the concept of Exporters and import/export. Or do we also
>>> switch Exporters from Master/Detail to core only?
>>>
>>> # Foreign Keys
>>>
>>> Content can be distributed across multiple tables (eg UpdateRecord has
>>> UpdateCollection, etc). In our export, we could either use primary keys
>>> (UUIDs) or natural keys to relate records. The former assumes that UUIDs
>>> are unique across Pulp instances. The safer but more complex alternative is
>>> to use natural keys. This would involve storing a set of fields on a record
>>> that would be used to identify a related record.
>>>
>>> # Incremental Exports
>>>
>>> There are two big pieces of data contained in an export: the dataset of
>>> Content from the database and the artifact files. An incremental export
>>> cuts down on the size of an export by only exporting the differences.
>>> However, when performing an incremental export, we could still export the
>>> complete dataset instead of just a set of differences
>>> (additions/removals/updates). This approach would be simpler and it would
>>> allow us to ensure that the new repo version 

Re: [Pulp-dev] Importers/Exporters

[Dennis Kliban]

In Katello that uses Pulp 2, what steps does the user need to take when
importing an export into an air gapped environment? I am concerned about
making the process more complicated than what the user is already used to.

On Wed, Feb 19, 2020 at 11:20 AM David Davis  wrote:

> Thanks for the responses so far. I think we could export publications
> along with the repo version by exporting any publication that points to a
> repo version.
>
> My concern with exporting repositories is that users will probably get a
> bunch of content they don't care about if they want to export a single repo
> version. That said, if users do want to export entire repos, we could add
> this feature later I think?
>
> David
>
>
> On Wed, Feb 19, 2020 at 10:30 AM Justin Sherrill 
> wrote:
>
>>
>> On 2/14/20 1:09 PM, David Davis wrote:
>>
>> Grant and I met today to discuss importers and exporters[0] and we'd like
>> some feedback before we proceed with the design. To sum up this feature
>> briefly: users can export a repository version from one Pulp instance and
>> import it to another.
>>
>> # Master/Detail vs Core
>>
>> So one fundamental question is whether we should use a Master/Detail
>> approach or just have core control the flow but call out to plugins to get
>> export formats.
>>
>> To give some background: we currently define Exporters (ie
>> FileSystemExporter) in core as Master models. Plugins extend this model
>> which allows them to configure or customize the Exporter. This was
>> necessary because some plugins need to export Publications (along with
>> repository metadata) while other plugins who don't have Publications or
>> metadata export RepositoryVersions.
>>
>> The other option is to have core handle the workflow. The user would call
>> a core endpoint and provide a RepositoryVersion. This would work because
>> for importing/exporting, you wouldn't ever use Publications because
>> metadata won't be used for importing back into Pulp. If needed, core could
>> provide a way for plugin writers to write custom handlers/exporters for
>> content types.
>>
>> If we go with the second option, the question then becomes whether we
>> should divorce the concept of Exporters and import/export. Or do we also
>> switch Exporters from Master/Detail to core only?
>>
>> # Foreign Keys
>>
>> Content can be distributed across multiple tables (eg UpdateRecord has
>> UpdateCollection, etc). In our export, we could either use primary keys
>> (UUIDs) or natural keys to relate records. The former assumes that UUIDs
>> are unique across Pulp instances. The safer but more complex alternative is
>> to use natural keys. This would involve storing a set of fields on a record
>> that would be used to identify a related record.
>>
>> # Incremental Exports
>>
>> There are two big pieces of data contained in an export: the dataset of
>> Content from the database and the artifact files. An incremental export
>> cuts down on the size of an export by only exporting the differences.
>> However, when performing an incremental export, we could still export the
>> complete dataset instead of just a set of differences
>> (additions/removals/updates). This approach would be simpler and it would
>> allow us to ensure that the new repo version matches the exported repo
>> version exactly. It would however increase the export size but not by much
>> I think--probably some number of megabytes at most.
>>
>> If its simper, i would go with that.  Saving even ~100-200 MB isn't that
>> big of a deal IMO.  the biggest savings is in the RPM content.
>>
>>
>>
>> [0] https://pulp.plan.io/issues/6134
>>
>> David
>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] switch from Python to JSON settngs file

[Dennis Kliban]

On Wed, Feb 19, 2020 at 9:39 AM Dennis Kliban  wrote:

> On Wed, Feb 19, 2020 at 9:22 AM Bruno Rocha 
> wrote:
>
>> From the issue:
>>
>> Writing aws_default_acl: None in the pulp_settings variable gets
>>> rendered to AWS_DEFAULT_ACL = "None", not AWS_DEFAULT_ACL = None
>>>
>>
>> As TOML does not allow `null` values dynaconf has a special syntax for
>> expressing that:
>> https://dynaconf.readthedocs.io/en/latest/guides/environment_variables.html#precedence-and-type-casting
>>
>> aws_default_acl: "@none None" might be a workaround.
>>
>>
> Thank you Bruno! Could you please comment on the issue?
>
> This should resolve the issue that I linked in my original email.
> However,there is still a problem when the setting value is a list or a
> dictionary. The template needs to be much more robust to handle that. It
> still makes sense to switch to JSON.
>

The other option is to make no change at all and simply add documentation
that says you can provide values to ansible-pulp in the same format as you
would using environment variables. In that case everything can be treated
as a string and Dynaconf interprets it into Python at runtime. I am leaning
toward this option.


>
>
>> Also there is an issue opened regarding the YAML null values:
>> https://github.com/rochacbruno/dynaconf/issues/288
>>
>>
>> On Wed, Feb 19, 2020 at 10:57 AM Dennis Kliban 
>> wrote:
>>
>>> ansible-pulp currently tries to generate a Python settings file.from
>>> settings provided by the user in the playbook[0]. However, the generated
>>> Python code is not always syntactically correct[1].
>>>
>>> The settings file is generated using a Jinja template. Jinja provides a
>>> to_json that can be used to generate the settings as JSON. I propose that
>>> we change the behavior of ansible-pulp to generate settings.json instead of
>>> settings.py.
>>>
>>> Are there any objections or other ideas?
>>>
>>> [0]
>>> https://github.com/pulp/ansible-pulp/blob/master/roles/pulp/templates/settings.py.j2
>>> [1] https://pulp.plan.io/issues/5687
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] switch from Python to JSON settngs file

[Dennis Kliban]

On Wed, Feb 19, 2020 at 9:22 AM Bruno Rocha  wrote:

> From the issue:
>
> Writing aws_default_acl: None in the pulp_settings variable gets rendered
>> to AWS_DEFAULT_ACL = "None", not AWS_DEFAULT_ACL = None
>>
>
> As TOML does not allow `null` values dynaconf has a special syntax for
> expressing that:
> https://dynaconf.readthedocs.io/en/latest/guides/environment_variables.html#precedence-and-type-casting
>
> aws_default_acl: "@none None" might be a workaround.
>
>
Thank you Bruno! Could you please comment on the issue?

This should resolve the issue that I linked in my original email.
However,there is still a problem when the setting value is a list or a
dictionary. The template needs to be much more robust to handle that. It
still makes sense to switch to JSON.


> Also there is an issue opened regarding the YAML null values:
> https://github.com/rochacbruno/dynaconf/issues/288
>
>
> On Wed, Feb 19, 2020 at 10:57 AM Dennis Kliban  wrote:
>
>> ansible-pulp currently tries to generate a Python settings file.from
>> settings provided by the user in the playbook[0]. However, the generated
>> Python code is not always syntactically correct[1].
>>
>> The settings file is generated using a Jinja template. Jinja provides a
>> to_json that can be used to generate the settings as JSON. I propose that
>> we change the behavior of ansible-pulp to generate settings.json instead of
>> settings.py.
>>
>> Are there any objections or other ideas?
>>
>> [0]
>> https://github.com/pulp/ansible-pulp/blob/master/roles/pulp/templates/settings.py.j2
>> [1] https://pulp.plan.io/issues/5687
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] ansible-pulp pre-install check of plugin version conflicts

[Dennis Kliban]

Users are able to specify pulpcore and plugin versions that they wish
ansible-pulp to install. The plugin versions specified are not always
guaranteed to work with the version of pulpcore. As a result, users can end
up with broken installations. I've written a story that describes the
current plan for preventing this from happening to our users[0]. I'd like
to have this implemented before we release pulpcore 3.2.0. Please provide
feedback on the issue.


[0] https://pulp.plan.io/issues/6189
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] switch from Python to JSON settngs file

[Dennis Kliban]

ansible-pulp currently tries to generate a Python settings file.from
settings provided by the user in the playbook[0]. However, the generated
Python code is not always syntactically correct[1].

The settings file is generated using a Jinja template. Jinja provides a
to_json that can be used to generate the settings as JSON. I propose that
we change the behavior of ansible-pulp to generate settings.json instead of
settings.py.

Are there any objections or other ideas?

[0]
https://github.com/pulp/ansible-pulp/blob/master/roles/pulp/templates/settings.py.j2
[1] https://pulp.plan.io/issues/5687
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importers/Exporters

[Dennis Kliban]

On Fri, Feb 14, 2020 at 1:11 PM David Davis  wrote:

> Grant and I met today to discuss importers and exporters[0] and we'd like
> some feedback before we proceed with the design. To sum up this feature
> briefly: users can export a repository version from one Pulp instance and
> import it to another.
>

Exporting just repository versions is not sufficient for reproducing a Pulp
instance in an air gapped environment. Users need to be able to use the
"export" to populate a Pulp instance without needing to create any
publications and/or distributions afterwards. What about letting users
specify a repository to export and then have pulpcore figure out which
repository versions, publications, distributions, content, metadata, and
artifacts need to be exported?


>
> # Master/Detail vs Core
>
> So one fundamental question is whether we should use a Master/Detail
> approach or just have core control the flow but call out to plugins to get
> export formats.
>
> To give some background: we currently define Exporters (ie
> FileSystemExporter) in core as Master models. Plugins extend this model
> which allows them to configure or customize the Exporter. This was
> necessary because some plugins need to export Publications (along with
> repository metadata) while other plugins who don't have Publications or
> metadata export RepositoryVersions.
>
> The other option is to have core handle the workflow. The user would call
> a core endpoint and provide a RepositoryVersion. This would work because
> for importing/exporting, you wouldn't ever use Publications because
> metadata won't be used for importing back into Pulp. If needed, core could
> provide a way for plugin writers to write custom handlers/exporters for
> content types.
>
> If we go with the second option, the question then becomes whether we
> should divorce the concept of Exporters and import/export. Or do we also
> switch Exporters from Master/Detail to core only?
>
> # Foreign Keys
>
> Content can be distributed across multiple tables (eg UpdateRecord has
> UpdateCollection, etc). In our export, we could either use primary keys
> (UUIDs) or natural keys to relate records. The former assumes that UUIDs
> are unique across Pulp instances. The safer but more complex alternative is
> to use natural keys. This would involve storing a set of fields on a record
> that would be used to identify a related record.
>
> # Incremental Exports
>
> There are two big pieces of data contained in an export: the dataset of
> Content from the database and the artifact files. An incremental export
> cuts down on the size of an export by only exporting the differences.
> However, when performing an incremental export, we could still export the
> complete dataset instead of just a set of differences
> (additions/removals/updates). This approach would be simpler and it would
> allow us to ensure that the new repo version matches the exported repo
> version exactly. It would however increase the export size but not by much
> I think--probably some number of megabytes at most.
>
> [0] https://pulp.plan.io/issues/6134
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Bindings' limitations

[Dennis Kliban]

On Tue, Feb 11, 2020 at 6:04 PM Lubos Mjachky  wrote:

> Dear colleagues,
>
> functional tests are currently being refactored to use bindings in various
> plugins. This will eventually allow us to query Pulp in a more pythonic way
> instead of using raw REST API calls, or pulp_smash utilities.
>
> While refactoring the functional tests in pulp_container, I noticed that
> sometimes it is necessary to send or receive data which surprisingly do not
> satisfy all the declared requirements in serializers because we want, for
> example, to test whether a feature was correctly implemented or not and
> whether errors are properly handled or not.
>
> However, in the bindings, it is not always possible to accomplish that in
> a graceful way. To create a request with invalid data, one should not do
> this:
>
> distribution_data =
> ContainerContainerDistribution(**gen_distribution(foo="bar")) # raised an
> exception; got an unexpected keyword argument 'foo'
> with self.assertRaises(ApiException) as exc:
> self.distribution_api.create(distribution)
>
> But should do rather this:
>
> distribution_data = gen_distribution(foo="bar") # a simple dictionary
> with self.assertRaises(ApiException) as exc:
> self.distributions_api.create(distribution_data)
>
> To disable validation in data classes, one can pass a Configuration object
> with the attribute client_side_validation=False to the method __init__ of a
> corresponding data class, like so:
>
> configuration = Configuration()
> configuration.client_side_validation = False
> ContainerManifest(**manifest_without_required_field,
> local_vars_configuration=configuration)
>
> Then we may use the first approach without any problems. This, in fact,
> disables the implicit validation for required fields which is turned on by
> default for every single data class.
>
> Another problem might be observed in generated methods for api calls (e.g.
> list, read, ...) which return data classes, such as ContainerManifest,
> where a Configuration object that was declared globally for ApiClient is
> ignored. This is based on my assumptions but I really could not find a way
> how to pass an existing configuration to these api calls. Due to that, a
> new Configuration is created separately in each data class and the
> validation is enabled again. For instance, the following call will fail,
> because in pulp_container, a manifest list does not have config_blob:
>
> ml = manifests_api.read(ml_href) # raised an exception; invalid value for
> `config_blob`, must not be `None`
>
>
The problem here is that config_blob shouldn't be a required field. Let's
fix that in the serializer.


> Instead, you should do the following:
>
> response = manifests_api.read(ml_href, _preload_content=False)
> ml_dict = json.loads(response.data)
> ml = ContainerManifest(**ml_dict,
> local_vars_configuration=api_client.configuration)
>
> The issue here is that in the serializer, the field config_blob is
> required, but there is also declared the additional field allow_null which
> is set to True. Yet, the api.json schema does not take that into
> consideration. Note that bindings are generated from api.json schema. The
> reason is presumably stated in the comment #7 here
> https://pulp.plan.io/issues/6069#note-7.
>
> And as you can see, the initial idea was to stray away from REST calls and
> to get rid of working with raw responses. However, it is inevitable even
> now in some use cases.
>
> I would love to see any follow-up discussion here because I believe that
> these little hacks I proposed can be problematic in the future. Still, we
> can wait for OpenAPI v3 where the issue with the field allow_null is
> resolved.
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp3 Applicability Design thoughts (and Katello)

[Dennis Kliban]

On Fri, Jan 17, 2020 at 8:31 AM Justin Sherrill  wrote:

> There have been some design discussions going on around applicability
> (https://hackmd.io/ydvHuzXNRA6T9eXx6cqy5A) in pulp3.
>
> There are some big changes compared to pulp2, including:
>
> * Package profile, module profile,and repository list have to be
> uploaded on every applicability computation
>
> * Calls for applicability are not asynchronous (which makes sense as
> they are one profile at a time).
>
> Also keep in mind that due to the complexity of all the information
> needed, Katello has been the primary (and sometimes the only?) user of
> the service.
>
> For an example of what this might looks like, consider a repository that
> syncs some new packages.  For 10,000 clients, it has to send the full
> package profiles for all 10,000 clients, as well as the other
> information in 10,000 api calls.  In Addition our task workers will have
> to wait around for all 10,000 clients to be calculated.  One last point
> is that katello already knows all the NVREA's for the rpms, which rpms
> are in which repositories, which artifacts are in which modules, and
> which packages are in what errata.
>
> Given all this, does it make sense for pulp to calculate the
> applicability?  Or does it make sense for katello to?
>
>
It does not make sense for Pulp 3 to calculate applicability.


> This assumes that no one else wants to use applicability in pulp3 (and
> given the barrier to entry is even higher than it was in pulp2, that may
> be possible).
>

People want it, but applicability calculation without any notion of
registered clients simply falls outside of Pulp's domain.


>
> Justin
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

I've written 4 stories to start the implementation on the use cases we have
come up with so far. Do these accurately capture what was discussed?

[0] https://pulp.plan.io/issues/5943
[1] https://pulp.plan.io/issues/5944
[2] https://pulp.plan.io/issues/5945
[3] https://pulp.plan.io/issues/5946

On Mon, Dec 9, 2019 at 4:40 PM Dennis Kliban  wrote:

> I've updated the document[0] with the meeting minutes from December 5. The
> full chat logs are here[1]. I would like to host another meeting at 9:30 AM
> EST (in your time zone
> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-10=9.5-10.5>)
> on Tuesday, December 10, 2019 (tomorrow). During this meeting we will focus
> on the details of the following 2 use cases:
>
>
>-
>
>As an administrator, I can use django-admin to add/remove a Signing
>Service.
>-
>
>As a REST API user (repo admin), I can assign a Signing Service to a
>repository and provide a key signature as additional configuration.
>
>
>
> [0]
> https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
> [1]
> https://pulpadmin.fedorapeople.org/triage/pulp-meeting/2019/pulp-meeting.2019-12-05-14.00.log.html
>
>
> On Tue, Nov 26, 2019 at 4:44 PM Dennis Kliban  wrote:
>
>> Thank you everyone who participated in the discussion earlier today. I
>> have moved the content to a google doc[0]. You can find the summary of
>> today's discussion below the use cases. The full log of the discussion is
>> here[1].
>>
>> I would like to continue the discussion at 9 AM EST (in your time zone
>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-5=9-10>)
>> on Thursday, December 5, 2019 in #pulp-meeting on IRC. Your
>>
>> [0]
>> https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
>> [1]
>> https://pulpadmin.fedorapeople.org/triage/pulp-dev/2019/pulp-dev.2019-11-26-14.30.log.html
>>
>> On Thu, Nov 21, 2019 at 8:27 AM Dennis Kliban  wrote:
>>
>>> On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban 
>>> wrote:
>>>
>>>> I will be hosting a chat meeting to discuss use cases for signing in
>>>> Pulp 3.y.
>>>> The meeting details and agenda link are below. Both users and plugin
>>>> developers are invited. Please join if you're interested!
>>>>
>>>> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
>>>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
>>>> .
>>>>
>>>
>>> Correction: this will take place on Tuesday. The link is correct.
>>>
>>>
>>>> Where: #pulp-dev on freenode
>>>> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>>>>
>>>> Questions and feedback are also welcome ahead of time.
>>>>
>>>> Thanks!
>>>>
>>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

I've updated the document[0] with the meeting minutes from December 5. The
full chat logs are here[1]. I would like to host another meeting at 9:30 AM
EST (in your time zone
<https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-10=9.5-10.5>)
on Tuesday, December 10, 2019 (tomorrow). During this meeting we will focus
on the details of the following 2 use cases:


   -

   As an administrator, I can use django-admin to add/remove a Signing
   Service.
   -

   As a REST API user (repo admin), I can assign a Signing Service to a
   repository and provide a key signature as additional configuration.



[0]
https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
[1]
https://pulpadmin.fedorapeople.org/triage/pulp-meeting/2019/pulp-meeting.2019-12-05-14.00.log.html


On Tue, Nov 26, 2019 at 4:44 PM Dennis Kliban  wrote:

> Thank you everyone who participated in the discussion earlier today. I
> have moved the content to a google doc[0]. You can find the summary of
> today's discussion below the use cases. The full log of the discussion is
> here[1].
>
> I would like to continue the discussion at 9 AM EST (in your time zone
> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-5=9-10>)
> on Thursday, December 5, 2019 in #pulp-meeting on IRC. Your
>
> [0]
> https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
> [1]
> https://pulpadmin.fedorapeople.org/triage/pulp-dev/2019/pulp-dev.2019-11-26-14.30.log.html
>
> On Thu, Nov 21, 2019 at 8:27 AM Dennis Kliban  wrote:
>
>> On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban  wrote:
>>
>>> I will be hosting a chat meeting to discuss use cases for signing in
>>> Pulp 3.y.
>>> The meeting details and agenda link are below. Both users and plugin
>>> developers are invited. Please join if you're interested!
>>>
>>> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
>>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
>>> .
>>>
>>
>> Correction: this will take place on Tuesday. The link is correct.
>>
>>
>>> Where: #pulp-dev on freenode
>>> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>>>
>>> Questions and feedback are also welcome ahead of time.
>>>
>>> Thanks!
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

Thank you everyone who participated in the discussion earlier today. I have
moved the content to a google doc[0]. You can find the summary of today's
discussion below the use cases. The full log of the discussion is here[1].

I would like to continue the discussion at 9 AM EST (in your time zone
<https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-12-5=9-10>)
on Thursday, December 5, 2019 in #pulp-meeting on IRC. Your

[0]
https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
[1]
https://pulpadmin.fedorapeople.org/triage/pulp-dev/2019/pulp-dev.2019-11-26-14.30.log.html

On Thu, Nov 21, 2019 at 8:27 AM Dennis Kliban  wrote:

> On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban  wrote:
>
>> I will be hosting a chat meeting to discuss use cases for signing in Pulp
>> 3.y.
>> The meeting details and agenda link are below. Both users and plugin
>> developers are invited. Please join if you're interested!
>>
>> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
>> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
>> .
>>
>
> Correction: this will take place on Tuesday. The link is correct.
>
>
>> Where: #pulp-dev on freenode
>> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>>
>> Questions and feedback are also welcome ahead of time.
>>
>> Thanks!
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban  wrote:

> I will be hosting a chat meeting to discuss use cases for signing in Pulp
> 3.y.
> The meeting details and agenda link are below. Both users and plugin
> developers are invited. Please join if you're interested!
>
> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
> <https://www.worldtimebuddy.com/?qm=1=4460243,100,3448439,3078610=4460243=2019-11-26=9.5-10.5>
> .
>

Correction: this will take place on Tuesday. The link is correct.


> Where: #pulp-dev on freenode
> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>
> Questions and feedback are also welcome ahead of time.
>
> Thanks!
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

I will be hosting a chat meeting to discuss use cases for signing in Pulp
3.y.
The meeting details and agenda link are below. Both users and plugin
developers are invited. Please join if you're interested!

When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone

.
Where: #pulp-dev on freenode
Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases

Questions and feedback are also welcome ahead of time.

Thanks!
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Solving the "callback problem" ... aka how pulpcore will stop finalizing RepositoryVersion

[Dennis Kliban]

On Tue, Nov 5, 2019 at 5:01 PM Brian Bouterse  wrote:

> Thank you for writing!
>
> On Tue, Nov 5, 2019 at 4:29 PM Matthias Dellweg  wrote:
>
>> Hi Brian,
>> i like the the change in the code flow, but since the
>> DeclarativeVersion (in your example) does not create the repository
>> version anymore, i think it should be renamed.
>> Maybe it's the SyncPipeline and we call perform on it instead of create.
>>
> I see what you're saying. What about the perspective that the
> new_repository_version is the one DeclarativeVersion is acting upon? The
> challenge w/ a name like SyncPipeline is that the Stages API aren't always
> used for sync, for example the migration tool uses it for migrating
> content. These are probably small semantic differences in the names, but
> then again we're having a naming convo.
>
>
>> Also i do not see the benefit of making the RelativePathFixer a context
>> manager instead of a simple function to be called after the pipeline.
>> It can even go wrong badly, if __exit__ is called after an exeption
>> broke the pipeline.
>>
> I agree fully with your concerns and reasoning. I've shared examples w/
> the "context_manager" approach, but since it's never involved in error
> handling, let's plan that the actual implementation all over would be using
> the style like this gist had:
> https://github.com/dralley/pulp_file/compare/typed-repositories...bmbouter:plugin-finalize-no-context-manager
>
>
The plugin writer's guide will recommend that every time the plugin writer
creates a repository version they should call into some code that
validates/fixes the new repository version. It's up to the plugin writer to
keep this DRY. This sounds good to me.

The 'modify' endpoint provided by core seems to be an exception to this
rule though. I don't really see much of a difference between the plugin
writer passing in a context manager or core calling into a plugin provided
hook. I am not opposed to this pattern, but if we are trying to avoid it,
then core most likely can't provide the repository version "modify"
operation. In that case the rule from above applies to this operation also.
The downside of that would be for the user who wouldn't be guaranteed to
have this interface for every type of repository. That's ok with me. We can
encourage plugin writers to implement certain interfaces.

As for the SingleArtifactUploadSerializer, it can be treated the same way.
The plugin writer has to implement their own 'create' method that performs
the validation of the repository version.

This all seems like a lot of work for plugin writers, but I just don't see
how core can provide more without limiting what the plugin can provide.


>
>> On Tue, 5 Nov 2019 14:46:18 -0500
>> Brian Bouterse  wrote:
>>
>> > As a followup to the chat discussion from triage/open-floor today,
>> > here is the POC on top of typed repositories. It's actually a very
>> > small change, the *only* significant difference is that the stages
>> > API no longer uses the RepositoryVersion context manager. Thus, the
>> > plugin writer must finalize it, but they can do that using
>> > core-provided facilities. The links below are diffs on top of
>> > @dalley's unmerged PRs so the links are long:
>> >
>> >
>> https://github.com/dralley/pulpcore/compare/typed-repositories...bmbouter:plugin-context-manager
>> >
>> https://github.com/dralley/pulp_file/compare/typed-repositories...bmbouter:plugin-context-manager
>> >
>> > I'm able to run the pulp-smash test with these changes so I think it's
>> > working:
>> > django-admin test
>> > pulp_file.tests.functional.api.test_sync.BasicFileSyncTestCase.test_sync
>> >
>> > Note that the context manager is only syntactic sugar. The pulp_file
>> > sync code could also just as easily be as shown below. This is
>> > incomplete, but I think you'll get the idea.
>> >
>> >
>> https://github.com/dralley/pulp_file/compare/typed-repositories...bmbouter:plugin-finalize-no-context-manager
>> >
>> > With this plugins can even do what they want in terms of style
>> > (context manager or not). Also they can not use it at all and the
>> > only extra responsibility would be to finalize the RepositoryVersion
>> > with its context manager (core provided).
>> >
>> >
>> > I'd like to ask for feedback on this design asap. Questions are
>> > concerns ... please send 'em.
>> >
>> > An extensive description was given at open floor, but those logs
>> > aren't up yet. The gist is that content modification/validation will
>> > require user options, the plugin already knows that, so let's stop
>> > having the core finalize the RepositoryVersion.
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Repo versions with no changes

[Dennis Kliban]

+1 to making this change

On Mon, Nov 4, 2019 at 3:51 PM David Davis  wrote:

> Currently in pulp, syncs always create repository versions regardless of
> whether or not any content changed. One of the tasks[0] for 3.0 GA is to
> document this behavior. However, I've heard several complaints about this
> from users so I wonder if it's worth reconsidering.
>
> Here are some reasons against always creating repo versions:
> - They were meant to serve as a historical record but this information is
> available by looking at the tasks api
> - It creates additional, unnecessary versions and bumps the latest version
> number of the repo
> - If we ever have a feature to retain only the latest X repo versions,
> it'll be less useful since some repo versions may not have any changes
>
> Any thoughts? I'd like to get this on the sprint by Wednesday so it can be
> changed before the dev freeze date of Nov 12.
>
> [0] https://pulp.plan.io/issues/3308
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Repo version validation

[Dennis Kliban]

The plan outlined in 3541 solves the problem in a way that gives plugin
writers a lot of control. +1 to implementing it.

On Thu, Oct 3, 2019 at 12:23 PM David Davis  wrote:

> We have a blocker for Pulp 3.0 GA[0] that we need to address soon in order
> to let plugins leverage it in their upcoming GA releases. It involves
> allowing plugin writers to validate content in a repo version. It's
> somewhat related to validating uniqueness in a repo version[1] except there
> are cases other than uniqueness that plugins might want to handle. One
> example might be a case where we want to prevent a user from adding a
> docker tag that points to a manifest outside a repo from getting added to
> the repo. I'm not sure if this is an actual example but it gives you an
> idea that there might be other non-unique validation plugin writers might
> want to add.
>
> Brian proposed a solution on 3541 that I think solves the problem[2]. I
> was hoping to maybe get some feedback on it so we could proceed by October
> 9.
>
> [0] https://pulp.plan.io/issues/3541
> [1] https://pulp.plan.io/issues/5008
> [2] https://pulp.plan.io/issues/3541
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [BREAKING] PublishedMetadata

[Dennis Kliban]

The changes to pulpcore, pulpcore-plugin, and pulp_file have been merged.

On Mon, Sep 23, 2019 at 5:23 PM Dennis Kliban  wrote:

> On September 25th due to issue #5304[0] we are going to convert
> PublishedMetadata into a type of Content.
>
> The following PRs need review before then:
>
> https://github.com/pulp/pulpcore-plugin/pull/133
> https://github.com/gmbnomis/pulp_cookbook/pull/45
> https://github.com/pulp/pulp_file/pull/278/
> https://github.com/pulp/pulp_rpm/pull/1450/
> https://github.com/pulp/pulp_deb/pull/118/
> https://github.com/pulp/pulp_python/pull/256/
>
>
>
> [0] https://pulp.plan.io/issues/5304
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp-2to3-migrate name changing to pulp-2to3-migration

[Dennis Kliban]

I renamed the repository on GitHub and made a PR[0] to change the name
everywhere in the repo.

[0] https://github.com/pulp/pulp-2to3-migration/pull/20

On Tue, Sep 24, 2019 at 10:14 AM Dennis Kliban  wrote:

> The current name for the migration plugin[0] was chosen when the tooling
> was going to be a CLI. Now that this is a pulp 3 plugin, it is more
> appropriate to name it pulp-2to3-migration.
>
> I'll be making this change later today. I will then work on publishing
> both Ruby and Python bindings for it[1].
>
>
> [0] https://github.com/pulp/pulp-2to3-migrate
> [1] https://pulp.plan.io/issues/5492
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulp-2to3-migrate name changing to pulp-2to3-migration

[Dennis Kliban]

The current name for the migration plugin[0] was chosen when the tooling
was going to be a CLI. Now that this is a pulp 3 plugin, it is more
appropriate to name it pulp-2to3-migration.

I'll be making this change later today. I will then work on publishing both
Ruby and Python bindings for it[1].


[0] https://github.com/pulp/pulp-2to3-migrate
[1] https://pulp.plan.io/issues/5492
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] [BREAKING] PublishedMetadata

[Dennis Kliban]

On September 25th due to issue #5304[0] we are going to convert
PublishedMetadata into a type of Content.

The following PRs need review before then:

https://github.com/pulp/pulpcore-plugin/pull/133
https://github.com/gmbnomis/pulp_cookbook/pull/45
https://github.com/pulp/pulp_file/pull/278/
https://github.com/pulp/pulp_rpm/pull/1450/
https://github.com/pulp/pulp_deb/pull/118/
https://github.com/pulp/pulp_python/pull/256/



[0] https://pulp.plan.io/issues/5304
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

[Dennis Kliban]

On Wed, Sep 4, 2019 at 11:12 AM Brian Bouterse  wrote:

> I want to bring back a variation on upload needs that has come out of
> various discussions w/ several plugin developers. I wrote it up here in
> more detail and I'll sumarize below:  https://pulp.plan.io/issues/5403
>
> 1. Make all uploads of a specific content type live at POST
> /pulp/api/v3/content///
>

+1 ... However, what are we going to provide in the plugin template for
this? Right now we provide a ModelViewSet. I am picturing this would be a
default implementation from pulpcore-plugin that would not extract any
information from the file, but would simply create an instance of the
*Content model and a ContentArtifact with a relative path.

Is that what you had in mind?


> 2. Have it accept either binary data (to create an Artifact from before
> the Content unit) OR a reference to an existing Artifact (allowing the
> chunked upload API to be used) but not both.
>
> What do you think?
>
>
>
> On Wed, Aug 14, 2019 at 12:32 PM Ina Panova  wrote:
>
>> There have been ongoing couple of more discussions offline regarding
>> copy/remove and it kinda boiled down to:
>>
>> v3///action/  which will allow filtering
>> v3///  will not allow filtering
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Thu, Aug 1, 2019 at 8:31 PM Daniel Alley  wrote:
>>
>>> Ina and I discussed this earlier today and she made good points -- she
>>> pointed out that what I suggested wouldn't work for content types that have
>>> relations on non-content models, as several content types in the Docker and
>>> Python plugins do. So, I no longer think it's a good idea to have an
>>> endpoint with "types=[]" in core.
>>>
>>> However, I still think that the "types=[]" pattern doesn't compose very
>>> well with other features like filtering so I'm not sure if we want to adopt
>>> that yet.
>>>
>>> On Thu, Aug 1, 2019 at 8:54 AM Ina Panova  wrote:
>>>
 For the one shot upload usecase I tend to lean towards
 /v3///upload/ and for content_types that require
 special treatment we can define separate endpoints.
 If talking about modulemd or modulemd_defaults, it could be
 /v3/rpm/modules/upload/.

 
 Regards,

 Ina Panova
 Senior Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Wed, Jul 31, 2019 at 1:04 PM Tatiana Tereshchenko <
 ttere...@redhat.com> wrote:

> If the goal is to make endpoints unified across all actions, then I
> think we can only do
> POST /pulp/api/v3//plugin/action/ types=[]
>
> Having plugin/content_type/upload would be nice, however I'm not sure
> if it covers enough use cases.
> E.g. For pulp_rpm,  it makes sense for packages or advisories to have
> a dedicated endpoint each, however it doesn't make much sense for modulemd
> or modulemd_defaults, because usually they are in the same file and
> uploaded in bulk (maybe a separate endpoint is needed for this case).
>
> For the copy case, it's common to copy more than one type, I think, so
> probably 'plugin/copy/ types=[]' makes more sense.
>
> It would be great to here from more people and other plugins.
>
>
>
> On Mon, Jul 29, 2019 at 5:46 PM Pavel Picka  wrote:
>
>> +1 for discuss this to keep some standard as I have already opened
>> PRs for rpm modulemd[-defaults].
>> I like idea of /upload in the end.
>> But also think it can work without as it will be differ by POST/GET
>> methods.
>>
>> On Mon, Jul 29, 2019 at 4:49 PM Dana Walker 
>> wrote:
>>
>>> Just to provide an added data point, I'll be merging the one-shot PR
>>> for pulp_python soon and it currently uses /api/v3/python/upload/
>>>
>>> I wanted to keep it simple as well, and so would be happy to change
>>> it for consistency based on whatever we decide.
>>>
>>> --Dana
>>>
>>> Dana Walker
>>>
>>> She / Her / Hers
>>>
>>> Software Engineer, Pulp Project
>>>
>>> Red Hat 
>>>
>>> dawal...@redhat.com
>>> 
>>>
>>>
>>>
>>> On Mon, Jul 29, 2019 at 10:42 AM Ina Panova 
>>> wrote:
>>>
 Hi all,
 As of today, plugins have the freedom to define whichever endpoints
 they want ( to some extent).
 This leads to the question - shall we namespace one-shot upload and
 copy endpoints for some consistency?

 POST /api/v3/content/rpm/packages/upload/
 POST /api/v3/content/rpm/packages/copy/

 or

 POST /api/v3/content/rpm/upload/ type =package
 POST /api/v3/content/rpm/copy/ type = [package, 

Re: [Pulp-dev] Announcing the plugin-template change to use containers for CI

[Dennis Kliban]

On Wed, Sep 4, 2019 at 11:11 AM Mike DePaulo  wrote:

> *Why?*
> pulp_rpm, and likely other plugins in the future, have C dependencies.
> They are often difficult to satisfy on Ubuntu's LTS releases that Travis
> provides. Since we are developing containers and a Kubernetes operator
> anyway, we decided to migrate plugin-template's Travis CI scripts to using
> them. These containers are currently based on Fedora 30, but will likely
> later be based on CentOS 7.7 or 8.0, which contain python 3.6.
> This is task #5004 .
>
> *Overview:*
> 1. During install.sh, the container image named like "pulp_file" is built
> according to pulpcore/containers/
> 2. During install.sh, k3s, the lightweight kubernetes implementation, is
> installed & configured on the Travis Ubuntu VM, which already has docker.
> 3. During install.sh, pulp-operator is deployed (up.sh), which in turn
> deploys the postgres database, redis, and the pulp_* image you built. It
> spins the single image up 4 separate containers (services/processes):
> pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
> 4. During script.sh, we install some commonly needed testing tools into
> the pulp-api container by prefixing the install command with $CMD_PREFIX .
> So long as this container never gets restarted, they will remain installed
> in it. These tools are not available to the other containers.
> 5. We use $CMD_PREFIX to run the unit tests
> 6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
> docs, or use bindings, etc.
> 7. Note that many other pieces of the puzzle were modified, including
> pulp-smash, to make this integration all work.
>
> *Changes you may need to make:*
> (in addition to the usual plugin-template commands):
> 1. Ensure that in template_config.yml, "plugin_name" is set to a value
> like "pulp_file" rather than "file"
> 2. If you write out ~/.netrc , the pulp application password has been
> changed from "admin" to "password" (which other pulp test / development
> code uses.)
>

You only need to make this change if you are continuing to use ansible-pulp
to deploy pulp on Travis. Otherwise, the containers built using the latest
Travis config will have the right password.


> 3. git rm .travis/{playbook,postgres,mariadb}.yml
>
> *Planned improvements include:*
> 1. Speeding up in the image build, largely through 3.
> 2. Publishing images to quay.io. #5062 
> 3. Only building your pulp_* image from scratch if you have a required PR
> for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
> will be layered on top of the "pulpcore" image. #5062
> 
> 4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python
> 3.7 based)
> 5. re-introducing codecov
> 6. (possible / long-term): Improving how testing packages are installed. #5404
> 
>
> *Feedback wanted on:*
> 1. Any confusing/unclear output.
> 2. Any reliability issues.
>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat 
>
> IM: mikedep333
>
> GPG: 51745404
> 
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] status of the ansible postgres role

[Dennis Kliban]

Combination of 1 and 2: we need to find a way to collaborate with the
author more closely to get outstanding PRs merged.

On Tue, Sep 3, 2019 at 8:28 AM Matthias Dellweg  wrote:

> Hi all,
> in the ansible-pulp role (that is meant to install pulp) we use a role
> to install the postgresql db-server from galaxy named
> ansible-role-postgres.
> Sadly the upstream version of this role is missing fedora30 support,
> and the PR for this has not been merged for a long time.
> This leads to ansible-pulp using a clone of this role, which is
> hosted on github in a personal namespace and is missing debian10 support
> respectively.
> This sounds to me like a kind of short term workaround, but it is in
> place for almost half a year now.
>
> I see several ways to move this forward:
>
> 1)Leave it as is, wait for upstream.
>   pros: nothing to do (now)
>   cons: no good debian support
> 2)Use upstream role and add fedora30 config like debian10 config [0]
>   pros: no need to maintain a clone of the role
>   cons: ugly workaround
> 3)Use upstream, and drop fedora30 support for now
>   pros: no need to maintain a clone of the role
>   cons: seems quite obvious?
> 4)Maintain a clone of the role in the pulp namespace with a team of
>   committers
>   pros: most flexibility, fedora30 & debian10 support
>   cons: extra maintainance work
>
> (The order is random, and the numbers are only for future references. I
> do not want to express a personal preference this way.)
>
>   Matthias
>
> [0] https://github.com/pulp/pulplift/pull/45
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] proposal for 2.20.1 release

[Dennis Kliban]

No one has brought up any other issues. The spec file change will be the
only change that will be included in the 2.20.1 beta tomorrow.

On Mon, Aug 26, 2019 at 1:23 PM Robin Chan  wrote:

> Hi Dennis,
>
> I think you need to release this into a .z specifically because these
> changes would be best to go into a .z before the 2.21.0 next .y release.
> I'm calling this out since 2.21.0 announcement should go out soon and folks
> may wonder why this [0] doesn't just go into the next .y.
>
> Robin Chan
>
> She/Her/Hers
>
> Satellite Software Engineering Manager - Pulp
>
> Red Hat <https://www.redhat.com>
>
> IRC: rchan
> <https://www.redhat.com>
>
>
>
> On Sun, Aug 25, 2019 at 10:07 AM Dennis Kliban  wrote:
>
>> I would like to release 2.20.1 with at least 1 bug fix[0] in it. Are
>> there any other bug fixes we would like to include?
>>
>> [0] https://pulp.plan.io/issues/5220
>>
>>
>> - Dennis
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] proposal for 2.20.1 release

[Dennis Kliban]

I would like to release 2.20.1 with at least 1 bug fix[0] in it. Are there
any other bug fixes we would like to include?

[0] https://pulp.plan.io/issues/5220


- Dennis
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pagination Requirements and Defaults?

[Dennis Kliban]

On Tue, Aug 20, 2019 at 1:06 PM David Davis  wrote:

> #3801 only describes the move from CursorPagination to
> PageNumberPagination. I don't think we considered
> using LimitOffsetPagination.
>
> That's right David. The CursorPagination was picked randomly when we were
implementing the initial base classes for Pulp 3[0]. It was part of this
commit[1].

I am interested in users such as Katello providing feedback on whether or
not they could switch to using the LimitOffsetPagination[2].

[0] https://pulp.plan.io/issues/1873
[1]
https://github.com/pulp/pulpcore/commit/d03d73c9714aa4c21c296b3749c929f688ea2f89
[2]
https://www.django-rest-framework.org/api-guide/pagination/#limitoffsetpagination


> David
>
>
> On Tue, Aug 20, 2019 at 1:01 PM Dennis Kliban  wrote:
>
>> On Tue, Aug 20, 2019 at 12:17 PM Brian Bouterse 
>> wrote:
>>
>>> Recently with pulp_ansible, users were interested in using pagination
>>> with LimitOffsetPagination [0]. Pulp currently defaults to
>>> PageNumberPagination. I looked at our current DRF defaults, and I noticed
>>> two things.
>>>
>>> 1. We default to the not-as-common PageNumberPagination based on
>>> examples in the drf docs.
>>> 2. We customize it here [1] in various ways.
>>>
>>> Can someone help me remember why these pagination style choices were
>>> made or where the requirements came from?
>>>
>>
>> I believe the motivation is described here:
>> https://pulp.plan.io/issues/3801
>>
>>
>>> Would our bindings work with a LimitOffsetPagination style?
>>>
>>
>> Yes, the bindings will work with anything that uses query parameters for
>> pagination.
>>
>>
>>> What use cases drove the use and customization in this area?
>>>
>>> Also, @katello how would a pagination style change (like switching to
>>> LimitOffsetPagination) affect you?
>>>
>>> Thanks for any info you can provide. Maybe what we have right now is
>>> just what we need, but I'm not sure.
>>>
>>> -Brian
>>>
>>> [0]:
>>> https://www.django-rest-framework.org/api-guide/pagination/#setting-the-pagination-style
>>> [1]:
>>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/pagination.py
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pagination Requirements and Defaults?

[Dennis Kliban]

On Tue, Aug 20, 2019 at 12:17 PM Brian Bouterse  wrote:

> Recently with pulp_ansible, users were interested in using pagination with
> LimitOffsetPagination [0]. Pulp currently defaults to PageNumberPagination.
> I looked at our current DRF defaults, and I noticed two things.
>
> 1. We default to the not-as-common PageNumberPagination based on examples
> in the drf docs.
> 2. We customize it here [1] in various ways.
>
> Can someone help me remember why these pagination style choices were made
> or where the requirements came from?
>

I believe the motivation is described here: https://pulp.plan.io/issues/3801


> Would our bindings work with a LimitOffsetPagination style?
>

Yes, the bindings will work with anything that uses query parameters for
pagination.


> What use cases drove the use and customization in this area?
>
> Also, @katello how would a pagination style change (like switching to
> LimitOffsetPagination) affect you?
>
> Thanks for any info you can provide. Maybe what we have right now is just
> what we need, but I'm not sure.
>
> -Brian
>
> [0]:
> https://www.django-rest-framework.org/api-guide/pagination/#setting-the-pagination-style
> [1]:
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/pagination.py
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] plugin build failures on Travis

[Dennis Kliban]

The plugin_template PR is the correct fix. However, pulp_ansible uses it's
own playbook and as a result needed the change applied to that playbook.

I am hoping that after we improve the installer some more[0], we will be
able to manage all the plugin install playbooks using the plugin-template
script.

[0] https://pulp.plan.io/issues/4770

On Tue, Aug 6, 2019 at 5:06 PM Dennis Kliban  wrote:

> Plugins are experiencing build failures on Travis due to a change in the
> installer[0]. I have made a PR[1] to update the plugin template with the
> needed change. I've also made a PR against pulp_ansible to test the fix.[2]
>
> Travis is experiencing some problems right now[3] so it is difficult to
> validate the fix works. However, a similar change in pulpcore[4] resolved
> the problem there.
>
> I'll send out an update when the fix has been validated and can be applied
> to plugins.
>
>
>
> [0] https://github.com/pulp/ansible-pulp/pull/128
> [1] https://github.com/pulp/plugin_template/pull/92
> [2] https://github.com/pulp/pulp_ansible/pull/159
> [3] https://www.traviscistatus.com/
> [4] https://github.com/pulp/pulpcore/pull/256
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] plugin build failures on Travis

[Dennis Kliban]

Plugins are experiencing build failures on Travis due to a change in the
installer[0]. I have made a PR[1] to update the plugin template with the
needed change. I've also made a PR against pulp_ansible to test the fix.[2]

Travis is experiencing some problems right now[3] so it is difficult to
validate the fix works. However, a similar change in pulpcore[4] resolved
the problem there.

I'll send out an update when the fix has been validated and can be applied
to plugins.



[0] https://github.com/pulp/ansible-pulp/pull/128
[1] https://github.com/pulp/plugin_template/pull/92
[2] https://github.com/pulp/pulp_ansible/pull/159
[3] https://www.traviscistatus.com/
[4] https://github.com/pulp/pulpcore/pull/256
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp 2 to 3 upgrade experience concerns

[Dennis Kliban]

After closer examination of the filesystem of a Pulp 2 install, I have
determined that we do not need to change the permissions of
/var/lib/pulp/content because the 'other' users can already read the files
stored there. That means we don't need to worry about long running scripts
to prepare the filesystem for migration to Pulp 3.

I have already closed a related issue[0]

I am going to update my PR[1] to reflect this. I will remove the -R from
the 'chown' command here[2] so the change is not performed recursively.

[0] https://pulp.plan.io/issues/5154#note-4
[1] https://github.com/pulp/pulp-packaging/pull/102
[2]
https://github.com/pulp/pulp-packaging/blob/master/packages/pulp/pulp.spec#L513

On Wed, Jul 31, 2019 at 12:18 PM Brian Bouterse  wrote:

>
>
> On Wed, Jul 31, 2019 at 12:12 PM Tatiana Tereshchenko 
> wrote:
>
>>
>>
>> On Wed, Jul 31, 2019 at 5:15 PM Dennis Kliban  wrote:
>>
>>> On Wed, Jul 31, 2019 at 10:27 AM Brian Bouterse 
>>> wrote:
>>>
>>>> Thank you for raising this concern.
>>>>
>>>> On Wed, Jul 31, 2019 at 10:06 AM Dennis Kliban 
>>>> wrote:
>>>>
>>>>> Users of Pulp 2 that are upgrading to Pulp 3 need to make files stored
>>>>> in /var/lib/pulp/content readable by Pulp 3. There are two way to do that:
>>>>>
>>>>> 1. Install Pulp 3 with 'pulp_user' set to 'apache'. (Pulp will run as
>>>>> user 'apache')
>>>>>
>>>>> 2. Install Pulp 3 with any 'pulp_user' and any 'pulp_group'.
>>>>> Recursively change group ownership of /var/lib/pulp/content/ to
>>>>> the value of 'pulp_group' and change the mode to 775 so that group can 
>>>>> read
>>>>> the files.
>>>>>
>>>>> We have already started moving users toward the second approach by
>>>>> including the relabeling of /var/lib/pulp in the 2.20.0 release[0].
>>>>> However, I am concerned that this will lead to poor experiences for users
>>>>> that have a lot of content in their Pulp 2. Hours or days of upgrading
>>>>> their Pulp to 2.20.0+.
>>>>>
>>>> I hear this concern.
>>>>
>>>>
>>>>> I propose that we revert the change introduces in 2.20.0 and add an
>>>>> RPM scriptlet that changes everything back to 'apache:apache'. Then we can
>>>>> tell users that they have the two options from above when upgrading from
>>>>> Pulp 2.
>>>>>
>>>> The installer could be configured to work with these requirements It
>>>> takes those options currently AIUI. This proposal doesn't adjust the
>>>> installer's defaults does it?
>>>>
>>>>>
>>>>>
>>> The installer only supports 'pulp_user'. The 'pulp_group' is being
>>> introduced in an open PR[0].
>>>
>>> [0] https://github.com/pulp/ansible-pulp/pull/128/files
>>>
>> Great thank you.
>
>
>>>
>>>
>>>> What are your thoughts?
>>>>>
>>>> I agree it takes time to perform that permission updating. The
>>>> motivation was that we didn't want Pulp as an application running as user
>>>> apache, especially with apache becoming a reverse proxy for Pulp3. We can
>>>> rethink that, but that is the current prevailing logic that I think of. I'm
>>>> open to other ideas here. If this is the case, and the filesystem is large,
>>>> taking time to fix these permissions is unavoidable (to me). The best thing
>>>> we can do is ensure it can be done online.
>>>>
>>>> What if we had better alternatives to alleviate the long-running rpm
>>>> upgrade? We could make a script that adjusts the groups so that either pulp
>>>> or apache could be the owners, and the performs the relabeling while pulp2
>>>> 2.19.z is online before the 2.20 upgrade. This would move the workload out
>>>> of upgrade time during downtime and to a pre-migration uptime. What about
>>>> ideas like that?
>>>>
>>>>
>>> I like the idea of providing a script that will run while Pulp 2 is
>>> operating. However, we should probably still change the behavior of Pulp 2
>>> so that it starts creating new files with the correct permissions. This way
>>> we can guarantee that the filesystem relabeling will eventually finish.
>>> Pulp 2 workers receive these settings through systemd unit files. The
>>> desired user and group values will need to be specified i

Re: [Pulp-dev] Pinning dependencies in Pulp 3

[Dennis Kliban]

+1

I really like that there is automation to help us update the deps. If the
PR from dependabot passes CI, we can just merge. Otherwise we will file an
issue.

On Fri, Jul 26, 2019 at 11:38 AM David Davis  wrote:

> Recently, Pulp 3 package installs were broken by a new version of DRF
> which necessitated a new release of pulpcore (RC4)[0]. Our releases are
> fragile and unstable because they don't pin versions of dependencies.
>
> I was thinking of a new strategy whereby we pin pulpcore's dependencies to
> specific versions (either y or z releases) and we use something like
> dependabot[1] to notify us of new updates for pulpcore dependencies. It
> looks like it'll open new PRs when it detects a dependency is out of date.
>
> The one downside I do see is that dependabot PRs could be ignored.
> However, I think the stability of our releases outweighs this potential
> risk especially as we get closer to GA.
>
> Thoughts?
>
> [0] https://www.redhat.com/archives/pulp-dev/2019-July/msg00076.html
> [1] https://dependabot.com/
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Non-source Installs are Broken

[Dennis Kliban]

+1

On Wed, Jul 24, 2019 at 4:40 PM Mike DePaulo  wrote:

> +1
>
> On Wed, Jul 24, 2019 at 4:05 PM Dana Walker  wrote:
>
>> +1
>>
>> a) it's broken
>> b) release early, release often
>> c) *thumbs up* makes sense to me
>>
>> Dana Walker
>>
>> She / Her / Hers
>>
>> Software Engineer, Pulp Project
>>
>> Red Hat 
>>
>> dawal...@redhat.com
>> 
>>
>>
>>
>> On Wed, Jul 24, 2019 at 3:39 PM David Davis 
>> wrote:
>>
>>> This makes sense to me.
>>>
>>> David
>>>
>>>
>>> On Wed, Jul 24, 2019 at 3:28 PM Brian Bouterse 
>>> wrote:
>>>
 tl;dr We need to release pulpcore and pulpcore-plugin RC4's asap (like
 tomorrow).

 On July 15th djangorestframeowork released 3.10 which is incompatible
 with pulpcore, so any install of RC3 after July 15th would fail to start.
 This was unnoticed until today due to a Travis cron job not being
 configured, and that has been remedied. We noticed it from the failure in
 Travis builds like this one:
 https://travis-ci.org/pulp/ansible-pulp/jobs/563173633#L737 That is
 the error anyone trying to use RC3 in a non source install would get.

 Pulp's source has a temporary-fix in place for this already
 https://github.com/pulp/pulpcore/commit/adeb2a7f073b1a6cbb3c72c6b6a23d55e1e7386e
 so the best option I can think of to fix this is to release it.

 In order to unblock Travis merging high-prio installer changes, and to
 unblock any user who wants to test an RC, I believe we should to release
 RC4 asap. Our last release was Jun 28th, so we're roughly on the month
 cadence still.

 I hope to get some feedback from other devs here. I'm preparing the
 release now as branches, and if there are no -1's by tomorrow morning EST
 I'll plan to release.

 Please give feedback, options/ideas, or concerns.

 Thanks,
 Brian

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat 
>
> IM: mikedep333
>
> GPG: 51745404
> 
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [BREAKING] drf 3.10 has breaking changes

[Dennis Kliban]

drf_yasg 1.16.1 has been released[0]. We can unpin the version of DRF and
fix 5125.

[0] https://github.com/axnsan12/drf-yasg/releases/tag/1.16.1


On Mon, Jul 15, 2019 at 1:51 PM Austin Macdonald  wrote:

> CI breakage *resolved*.
>
> https://pulp.plan.io/issues/5125 has not been fixed yet, it is blocked
> upstream. In the meantime we have restricted drf version to < 3.10.
> https://github.com/pulp/pulpcore/pull/209/
>
>
>
> On Mon, Jul 15, 2019 at 12:59 PM Austin Macdonald 
> wrote:
>
>> This will probably be a CI blocker. (It does prevent pulplift from
>> completing.) I've taken https://pulp.plan.io/issues/5125 as assigned.
>>
>> I'll update the issue as I go, and respond here when things are working
>> again.
>>
>> On Mon, Jul 15, 2019 at 12:08 PM David Davis 
>> wrote:
>>
>>> A new version of DRF was released and it has some breaking changes. The
>>> first thing I saw was that 'detail_route' was removed (in favor of
>>> 'action'). I've filed this issue:
>>>
>>> https://pulp.plan.io/issues/5125
>>>
>>> David
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Database support in Pulp 3

[Dennis Kliban]

I filed an issue[0] to track the effort to remove MariaDB support in Pulp 3.


[0] https://pulp.plan.io/issues/5129

On Mon, Jul 15, 2019 at 1:01 PM Robin Chan  wrote:

> +1
> In May, the Foreman community warned us against trying to support many
> databases - saying the cost was too high.  Without any compelling reason to
> abandon earlier efforts to stay compatible, no action was taken at that
> time since the cost was nominal until now. With the new information, I
> agree the list of reasons in this thread justify the decision to drop
> MariaDB/MySQL.
>
> Mixed plugin support is the #1 reason for me.
>
> Robin Chan
> Satellite Software Engineering Manager - Pulp
>
>
>
> On Mon, Jul 15, 2019 at 11:59 AM Mike DePaulo 
> wrote:
>
>> +1 to drop
>>
>> My rationale:
>> At a previous job, MariaDB support would have helped us; we could use our
>> cluster.
>> But a non-clustered postgresql server + Pulp having a lot more features
>> (due to the limitations of MariaDB, and the saved developer time) would
>> have been overall more valuable.
>>
>> -Mike
>>
>> On Mon, Jul 15, 2019 at 11:06 AM Dana Walker  wrote:
>>
>>> +1 to drop MariaDB testing/support
>>>
>>> Dana Walker
>>>
>>> She / Her / Hers
>>>
>>> Software Engineer, Pulp Project
>>>
>>> Red Hat 
>>>
>>> dawal...@redhat.com
>>> 
>>>
>>>
>>>
>>> On Mon, Jul 15, 2019 at 10:41 AM Ina Panova  wrote:
>>>
 +1 to drop MariaDB support.


 
 Regards,

 Ina Panova
 Senior Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Sun, Jul 14, 2019 at 10:10 PM Brian Bouterse 
 wrote:

> I believe we have reached a point where Pulp (core and its plugins)
> can no longer support MariaDB due to technical problems. I've been an
> advocate for Pulp to support MariaDB because it's what our users want. The
> community survey has 16 respondents (IIRC) and 30% of them said they 
> wanted
> to use MariaDB. Also, anecdotally at conference booths, users want choice
> in their database. To not give them that, we need good, technical reasons.
> I didn't feel we had enough of them before, but now I feel we're at a 
> point
> where we aren't able to solve these issues; it's becoming a choice of
> giving users features they want versus db portability.
>
> Here are the main reasons I think about (mostly what others have also
> stated):
>
> * utf8 issues - 3-byte utf-8 issues   < is this fixable with just
> a schema change?
> * full text search - pulp_ansible recently implemented full text
> search. To my knowledge it's not possible with MySQL. full text search is
> amazing, and it's driving pulp_ansible to drop postgreSQL. I bet all
> plugins will want this. With MariaDB, there is no search.
> * mixed plugin support concerns - With some plugins dropping support
> for MariaDB, the pulp community will fracture based on what DB you choose
> to use because not all plugins can run in all places.
> * specific field support - plugin writers have expressed the desire to
> store json data when their plugins natively contain json data and perhaps
> you want to filter on it for example. Having mariaDB support prevents us
> from using the JSONField.
> * performance concerns - The performance testing showed that Pulp does
> run significantly slower
>
> If we drop MariaDB we should publish a blog post and drop it with RC4.
> To remove MariaDB testing from Travis I propose we remove it from the
> plugin_template and use the Travis CI tool from @dkliban to push that
> config out to all repositories.
>
> I'll be offline this week. I wanted to get this reply out there in the
> hope that you all can make and enact the final decision.
>
> Thanks,
> Brian
>
>
> On Thu, Jul 11, 2019 at 4:02 PM Daniel Alley 
> wrote:
>
>> One more note:  Not all MySQL / MariaDB installations support
>> transactions, which we use heavily (and rely on?)
>>
>>
>> https://docs.djangoproject.com/en/2.2/topics/db/transactions/#transactions-in-mysql
>>
>> On Thu, Jul 11, 2019 at 3:55 PM David Davis 
>> wrote:
>>
>>> Two plugins are currently running into issues trying to support
>>> mariadb/mysql. The pulp_ansible plugin is interested in adding full text
>>> search and JSONFields. Meanwhile, the pulp_python plugin is trying to 
>>> store
>>> emojis in text which mariadb/mysql doesn't handle well since it uses 
>>> 3-byte
>>> utf-8 by default[0]. Given such cases, I wonder if we'd be better 
>>> served by
>>> dropping mariadb/mysql support and going with Postgresql only. Gitlab
>>> recently came to a similar conclusion[1].
>>>
>>> I personally am hesitant to give up being 

[Pulp-dev] plugin_template changes

[Dennis Kliban]

What's new?
- bootstrap.py is now called plugin-template
- Plugins now use a template_config.yml to store all the options that
they want to use when interacting with plugin-template commands


What's gone?
   - the command line arguments that told bootstrap.py what options to use

Find out more in the README[0].

[0] https://github.com/pulp/plugin_template/blob/master/README.md
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Database support in Pulp 3

[Dennis Kliban]

+1 to drop support for mariadb

On Mon, Jul 15, 2019 at 10:04 AM Austin Macdonald 
wrote:

> +1 to remove.
>
> On Sun, Jul 14, 2019 at 4:10 PM Brian Bouterse 
> wrote:
>
>> I believe we have reached a point where Pulp (core and its plugins) can
>> no longer support MariaDB due to technical problems. I've been an advocate
>> for Pulp to support MariaDB because it's what our users want. The community
>> survey has 16 respondents (IIRC) and 30% of them said they wanted to use
>> MariaDB. Also, anecdotally at conference booths, users want choice in their
>> database. To not give them that, we need good, technical reasons. I didn't
>> feel we had enough of them before, but now I feel we're at a point where we
>> aren't able to solve these issues; it's becoming a choice of giving users
>> features they want versus db portability.
>>
>> Here are the main reasons I think about (mostly what others have also
>> stated):
>>
>> * utf8 issues - 3-byte utf-8 issues   < is this fixable with just a
>> schema change?
>> * full text search - pulp_ansible recently implemented full text search.
>> To my knowledge it's not possible with MySQL. full text search is amazing,
>> and it's driving pulp_ansible to drop postgreSQL. I bet all plugins will
>> want this. With MariaDB, there is no search.
>> * mixed plugin support concerns - With some plugins dropping support for
>> MariaDB, the pulp community will fracture based on what DB you choose to
>> use because not all plugins can run in all places.
>> * specific field support - plugin writers have expressed the desire to
>> store json data when their plugins natively contain json data and perhaps
>> you want to filter on it for example. Having mariaDB support prevents us
>> from using the JSONField.
>> * performance concerns - The performance testing showed that Pulp does
>> run significantly slower
>>
>> If we drop MariaDB we should publish a blog post and drop it with RC4. To
>> remove MariaDB testing from Travis I propose we remove it from the
>> plugin_template and use the Travis CI tool from @dkliban to push that
>> config out to all repositories.
>>
>> I'll be offline this week. I wanted to get this reply out there in the
>> hope that you all can make and enact the final decision.
>>
>> Thanks,
>> Brian
>>
>>
>> On Thu, Jul 11, 2019 at 4:02 PM Daniel Alley  wrote:
>>
>>> One more note:  Not all MySQL / MariaDB installations support
>>> transactions, which we use heavily (and rely on?)
>>>
>>>
>>> https://docs.djangoproject.com/en/2.2/topics/db/transactions/#transactions-in-mysql
>>>
>>> On Thu, Jul 11, 2019 at 3:55 PM David Davis 
>>> wrote:
>>>
 Two plugins are currently running into issues trying to support
 mariadb/mysql. The pulp_ansible plugin is interested in adding full text
 search and JSONFields. Meanwhile, the pulp_python plugin is trying to store
 emojis in text which mariadb/mysql doesn't handle well since it uses 3-byte
 utf-8 by default[0]. Given such cases, I wonder if we'd be better served by
 dropping mariadb/mysql support and going with Postgresql only. Gitlab
 recently came to a similar conclusion[1].

 I personally am hesitant to give up being database agnostic but we
 already don't support sqlite. Also, I see some advantages like utilizing
 several Postgresql-only features like extra field types, full text search,
 etc. Also, supporting multiple databases means we'll likely have to write
 db specific code in some places or have plugins that only work with certain
 database types.

 Thoughts?

 [0]
 https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434
  https://code.djangoproject.com/ticket/18392
 [1] https://about.gitlab.com/2019/06/27/removing-mysql-support/

 David
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] filtering tasks by reserved resources

[Dennis Kliban]

On Thu, Jul 11, 2019 at 4:21 PM David Davis  wrote:

> ReservedResources are unique[0] and the tasking code relies on this
> uniqueness to prevent race conditions[1]. The db transaction prevents two
> workers from acquiring a lock on the same resource. I read through 5120 but
> I am not sure how the design would prevent that from happening?
>
> Thanks for pointing this out. How about creating a separate model for
keeping track of historical CreatedResource - TaskCreatedResource? We can
worry about improving the tasking system performance in a separate story.



> [0]
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/task.py#L35
> [1]
> https://github.com/pulp/pulpcore/blob/ae6be0d89c5df26ac1cfdc876f3b131aa6e9bcf8/pulpcore/app/models/task.py#L229-L246
>
> David
>
>
> On Thu, Jul 11, 2019 at 4:04 PM Dennis Kliban  wrote:
>
>> I just wrote up a story to add an ability to filter Tasks by the
>> resources that they reserved. This is needed for the migration plan tasks
>> and will be just as useful for other tasks.
>>
>> The design requires storing ReservedResources permanently in the
>> database. The status of the task associated with the ReservedResource will
>> be used to determine if a ReservedResource is still active or not.
>> Additionally it will no longer be necessary to dispatch a task for removing
>> the ReservedResource. Pulp will no longer have to dispatch 2 tasks for
>> every single task with a reservation.
>>
>> What questions or concerns do you have?
>>
>> [0] https://pulp.plan.io/issues/5120
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp 2 + pulp 3 dev environment problem

[Dennis Kliban]

I am proposing that we provide additional support for deploying with httpd
as the reverse proxy.

On Wed, Jul 10, 2019 at 8:27 AM David Davis  wrote:

> To clarify, are you proposing we should drop support for nginx in
> ansible-pulp in favor of apache? Or are you proposing we add support for
> httpd? I think the latter makes sense.
>
> David
>
>
> On Tue, Jul 9, 2019 at 9:06 PM Dennis Kliban  wrote:
>
>> Our ansible role[0] for deploying Pulp 2 deploys httpd and has it bind to
>> ports 80 and 443. The ansible-pulp roles[1] deploy nginx and have it bind
>> to port 80 also. When Pulp 2 is already installed, nginx fails to start.
>>
>> We need to add support to ansible-pulp for deploying Pulp 3 with httpd
>> instead of nginx.
>>
>> Does anyone have other solutions?
>>
>>
>> [0] https://github.com/pulp/pulp-ci/tree/master/ci/ansible/roles/pulp
>> [1] https://github.com/pulp/ansible-pulp
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulp 2 + pulp 3 dev environment problem

[Dennis Kliban]

Our ansible role[0] for deploying Pulp 2 deploys httpd and has it bind to
ports 80 and 443. The ansible-pulp roles[1] deploy nginx and have it bind
to port 80 also. When Pulp 2 is already installed, nginx fails to start.

We need to add support to ansible-pulp for deploying Pulp 3 with httpd
instead of nginx.

Does anyone have other solutions?


[0] https://github.com/pulp/pulp-ci/tree/master/ci/ansible/roles/pulp
[1] https://github.com/pulp/ansible-pulp
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev