[Pulp-dev] Help us test the new tasking system please

[Brian Bouterse]

If you use the latest commit from pulp_installer which has this PR
 merged, you'll see the
line below in your `example.dev-config.yml`. Please add this to your
`local.dev-config.yml`, and re-vagrant up.

This enables the new-style tasking system in dev environments and we want
as many folks to test this as possible.

use_new_worker_type: true

Thanks!
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore==3.14 pre-release planning -- target date June 29th

[Brian Bouterse]

See the discussion details here:
https://github.com/pulp/community/discussions/22
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore meeting minutes

[Brian Bouterse]

See them posted to the discussions thread here:
https://github.com/pulp/community/discussions/8#discussioncomment-840117
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Github Discussions

[Brian Bouterse]

I did this also for the pulpcore meeting:
https://github.com/pulp/community/discussions/8

My format was a little different, but the same idea.

On Thu, Jun 3, 2021 at 10:13 AM Grant Gainey  wrote:

> On Thu, Jun 3, 2021 at 9:40 AM David Davis  wrote:
>
>> Based on feedback, I've moved discussions to its own repo:
>> https://github.com/pulp/community/discussions.
>>
>
> Brillliant!
>
> One discovery I made this week - for 'Meetings' threads that exist to have
> meeting-minutes posted, the first entry should be a description of what the
> meeting you're recording is for, and each set of minutes should be a
> comment. This lets the reader sort by "Newest" and get
> most-recent-minutes-first.The initial message in a discussion is always at
> the top, no matter how you sort - so if it's your first meeting-minutes,
> they'll always be first.
>
> I redid the katello/pulp and community/pulp integration discussion-threads
> (in their new location) in light of this, apologies to anyone who got some
> notification-spam as a result this morning.
>
>- https://github.com/pulp/community/discussions/7
>- https://github.com/pulp/community/discussions/4
>
> G
>
>>
>> David
>>
>>
>> On Tue, May 25, 2021 at 1:49 PM David Davis 
>> wrote:
>>
>>> We've heard from the community about the amount of friction involved in
>>> getting help with Pulp and one of the areas I think we could improve is
>>> user communications. We currently run two mailing lists: pulp-list and
>>> pulp-dev.
>>>
>>> At today's open floor meeting, we talked about using Github's new
>>> Discussions feature[0] to host these conversations instead. I've set up a
>>> Discussion against pulpcore[1] for us to try but here's also an example of
>>> a project that has a lot of threads[2].
>>>
>>> I think the consensus was that we'd just keep pulpcore as our one and
>>> only Github Discussions instance, which would serve as a replacement for
>>> pulp-list and pulp-dev. I'd propose that we try this out for a bit and
>>> eventually decommission our mailing lists.
>>>
>>> [0] https://docs.github.com/en/discussions
>>> [1] https://github.com/pulp/pulpcore/discussions
>>> [2] https://github.com/vercel/next.js/discussions
>>>
>>> David
>>>
>> ___
>> Pulp-list mailing list
>> pulp-l...@redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-list
>
>
>
> --
> Grant Gainey
> Principal Software Engineer, Red Hat System Management Engineering
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulpcore team meeting notes - May 25, 2021

[Brian Bouterse]

Hi Neal,

Apologies for the late reply. We've added the RBAC to various endpoints in
pulpcore and some plugins, but what I think would be helpful is if we had a
way to more clearly identify what has RBAC and what doesn't across pulpcore
and plugins. Would that be helpful?

Thank you,
Brian


On Tue, May 25, 2021 at 11:46 AM Neal Gompa  wrote:

> On Tue, May 25, 2021 at 11:23 AM Brian Bouterse 
> wrote:
> >
> > * Making top-level Authentication page in docs
> > * https://pulp.plan.io/issues/8799
>
> The docs currently mention that RBAC stuff is planned, is there
> something that shows the status of that? This is something that I'd
> love to see in Pulp sooner rather than later...
>
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore meeting - June 1, 2021

[Brian Bouterse]

Posted as discussion here: https://github.com/pulp/pulpcore/discussions/1379
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore team meeting notes - May 25, 2021

[Brian Bouterse]

## Topics
* Is this a bug or intentional?
* https://pulp.plan.io/issues/8798
* [outcome] probably not intentional
* we cache these results to avoid recalculating them constantly
* Non-blocking orphan cleanup https://pulp.plan.io/issues/7659
* Plugin code needs to update timestamp
* [proposal] add `timestamp_of_interest` via migration in 3.14, then in
3.15 have orphan_cleanup become non-blocking and use `timestamp_of_interest`
* Django upgrade to 3.2
* https://pulp.plan.io/issues/8488
* django-guardian now resolved
*
https://github.com/pulp/pulpcore/pull/1352#pullrequestreview-667076933
* PR resolves pulp's issues
* https://github.com/pulp/pulpcore/pull/1116
* [goal] attempt to switch to Django 3.2 soon, maybe with 3.14
* Challenges with Python 3.6
* dependencies are dropping support
* Making top-level Authentication page in docs
* https://pulp.plan.io/issues/8799
* Relate backport requests to the original issue
*
https://docs.pulpproject.org/pulpcore/bugs-features.html#for-backport-requests
* will we benefit from automation here?
* Danny's feedback

## Action Items
* [dkliban] to engage contributor @dannysauer and help with their PRs
* [bmbouter & dalley] add tasking env var to django 3.2 epic
* [david] email asking users about python 3.6 and django 3.2
* [dkliban] file issue to update CI to use python 3.7 and the single
container to use python 3.7
* [ipanova] reach out to Melanie w/r to Danny's feedback so we prepare it
in a format that can be shared/discussed with a wide group/community
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore team meeting notes - May 18, 2021

[Brian Bouterse]

## Topics
* Docs Backlog ( let's time box it 20mins)
* https://tinyurl.com/36hc2c9h
* 3.13.0 release moved to Monday May 24
* single container requires plugins to be compatible
*
https://github.com/pulp/pulp-oci-images/blob/latest/pulp/Containerfile#L14-L22
* 3.11.2 in-progress
* Logging suggestion/concern
* https://bugzilla.redhat.com/show_bug.cgi?id=1961508

## Action Items
* [ipanova] to invite pulp docs day since we identified issues
* [dkliban] to engage contributor @dannysauer and help with their PRs
* [dalley] to move back the 3.13 release to may 24th
* [bmbouter, dalley, gerrod] to help round up 3.13 bocking Prs
* [dkliban] to review upgrade test docs in plugin_template and merge if good
* [x9c4] to review what plugins need releases prior to 3.13 release and
bring up at open floor today
* [ttereshc] to file upstream logging bug related to BZ and invite
discussion about how to handle
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Feedback for pulp3 demo next week

[Brian Bouterse]

I've prepared some content for my talk next Thursday, May 20. The content
is here [0], please edit/add to it change it. I think it's easier than
explaining change to me and then me making them. I'll proof the content
later and fixup anything that doesn't make sense.

Here are some things I would particularly like to ask others for.
* some commands you would use to demo Python
* some commands you would use to demo Container
* some commands you would use to demo RPM

You'll see ^ all in sections at the bottom with TODOs. Please enter/give
feedback by Tuesday May 18th.

[0]: https://hackmd.io/yUb12OPyRw-0hbGCGFf0_A

Thanks,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore team meeting notes - May 11, 2021

[Brian Bouterse]

## Topics
* Hold https://pulp.plan.io/issues/8386 for 3.14
* could potentially break plugins (pulp_rpm) that want 3.13
* Possible scheme to "backport idempotent data migrations"
* We cannot backport migrations, but we could wrap the migrating code
into a post_migrate action on the backport only.
* This can only work for pure data migrations( no schema change).
* Being on the dot releases, the post_migrate code may be run
arbitrarily often before the real migration (coming with the minor release)
will run once. Therefore the code must be idempotent.
* The post_migrate code should not appear on master ever, because it
may be incompatible with later database schemas.
* Response to checksum migration
* https://github.com/pulp/pulpcore/pull/1313#issuecomment-838075001
* considered 3 options:
* option (1): make it a no-op
* option (2): make it continue even if there are errors
* option (3): leave it as is
* question: should the migration also download content
* overall folks believe no, mostly because the common case is
reducing the number of checksums not adding them
* decision: go with option (2) @daviddavis to update his PR and reply
to the concern
* Docs Backlog ( let's time box it 20mins)
* https://tinyurl.com/36hc2c9h
* Idea/question: should we add DRF token authentication to pulpcore, can
pulp-container use it?
* Backports for django CVE fix
* 3.9 - 3.12 for pulp_deb
* katello & galaxy_ng need: 3.7, 3.9, 3.11
* bmbouter to release backports

## Action Items
[fao] write a report about upgrade tests and send on pulp-dev mailing list
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] How to enable HTTPS for our tests in pulpcore and all plugins?

[Brian Bouterse]

On Fri, May 7, 2021 at 11:27 AM Robin Chan  wrote:

> Can someone enlighten me on the main motivation for making this change?
> I wasn't at the meeting and just curious what other context I'm missing. I
> definitely understand https > http from a security standpoint but wondering
> if there were other factors or motivations I'm missing.
>
It's a good question. I have two main ones, but none are especially
timeline driven:

* it's problematic for development today. The installer (which installs dev
envs also) default to https, but the tests are incompatible with that and
can only work with http. Even though we work with it everyday we regularly
have test failures and spend hours only to realize our local tests aren't
working because we forgot to "unconfigure https" manually. This happened to
me on Tuesday for example. Non-daily-developers would have no way of
knowing this.

* user security: When demoing pulp-ansible with the CLI and container
installs at fosdem for example, the first thing we have to do is instruct
users to disable security.

Maybe others have other reasons too, but those were my interests.


> -rchan
>
> On Fri, May 7, 2021 at 10:53 AM David Davis  wrote:
>
>> To confirm, the "latest" tag will continue to ship with http? I imagine
>> most users will end up with http then.
>>
>> Also, what (if anything) do we do about y release tags (e.g. the upcoming
>> 3.13 tag)? Do they continue to ship with http?
>>
>> David
>>
>>
>> On Fri, May 7, 2021 at 10:51 AM Brian Bouterse 
>> wrote:
>>
>>> a yis
>>>
>>> On Fri, May 7, 2021 at 10:46 AM Fabricio Aguiar 
>>> wrote:
>>>
>>>> I changed https://github.com/pulp/pulp-oci-images/pull/73 to ship both,
>>>> latest as is, and the new tag: https
>>>>
>>>> Best regards,
>>>> Fabricio Aguiar
>>>> Software Engineer, Pulp Project
>>>> Red Hat Brazil - Latam <https://www.redhat.com/>
>>>> +55 22 999000595
>>>>
>>>>
>>>>
>>>> On Fri, May 7, 2021 at 11:41 AM Brian Bouterse 
>>>> wrote:
>>>>
>>>>> +1 to this observation, we probably need to either ship both or make
>>>>> it configurable somehow. Shipping both is probably easier on users.
>>>>>
>>>>> On Fri, May 7, 2021 at 5:11 AM Matthias Dellweg 
>>>>> wrote:
>>>>>
>>>>>> This is a great piece of work!
>>>>>> The problem I see is that the SSL free container image may be used in
>>>>>> places we do not control. And having this http based container equipped
>>>>>> with an external https reverse proxy is imho a valid use case.
>>>>>> Therefore i would prefer, if we could provide both versions of the
>>>>>> image (with and without SSL) as different tags.
>>>>>> This would also give us the opportunity to switch the plugins one by
>>>>>> one to use the new container.
>>>>>> Ideally, the SSL container would be a thin OCI-layer on top of the
>>>>>> http version.
>>>>>>
>>>>>> On Thu, May 6, 2021 at 10:10 PM Fabricio Aguiar 
>>>>>> wrote:
>>>>>>
>>>>>>> I finally made pulp_container CI work with https,
>>>>>>> I also did some changes on pulp_installer, I believe these changes
>>>>>>> will make it possible to run functional tests on dev environment.
>>>>>>>
>>>>>>> I think now it is a matter of deciding when is the best time to
>>>>>>> merge the PR on the single container and if latest tag should be https 
>>>>>>> or
>>>>>>> not
>>>>>>>
>>>>>>> PRs:
>>>>>>> https://github.com/pulp/pulp-oci-images/pull/73
>>>>>>> https://github.com/pulp/pulp_installer/pull/614
>>>>>>> https://github.com/pulp/plugin_template/pull/379
>>>>>>> https://github.com/pulp/pulpcore/pull/1283
>>>>>>> https://github.com/pulp/pulp_container/pull/304
>>>>>>> https://github.com/pulp/pulp_rpm/pull/1977
>>>>>>> https://github.com/pulp/pulp_ansible/pull/572
>>>>>>> https://github.com/pulp/pulp-2to3-migration/pull/362
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Fabricio Aguiar
>>>>>>> Software Engineer, Pulp Proje

Re: [Pulp-dev] How to enable HTTPS for our tests in pulpcore and all plugins?

[Brian Bouterse]

a yis

On Fri, May 7, 2021 at 10:46 AM Fabricio Aguiar  wrote:

> I changed https://github.com/pulp/pulp-oci-images/pull/73 to ship both,
> latest as is, and the new tag: https
>
> Best regards,
> Fabricio Aguiar
> Software Engineer, Pulp Project
> Red Hat Brazil - Latam <https://www.redhat.com/>
> +55 22 999000595
>
>
>
> On Fri, May 7, 2021 at 11:41 AM Brian Bouterse 
> wrote:
>
>> +1 to this observation, we probably need to either ship both or make it
>> configurable somehow. Shipping both is probably easier on users.
>>
>> On Fri, May 7, 2021 at 5:11 AM Matthias Dellweg 
>> wrote:
>>
>>> This is a great piece of work!
>>> The problem I see is that the SSL free container image may be used in
>>> places we do not control. And having this http based container equipped
>>> with an external https reverse proxy is imho a valid use case.
>>> Therefore i would prefer, if we could provide both versions of the image
>>> (with and without SSL) as different tags.
>>> This would also give us the opportunity to switch the plugins one by one
>>> to use the new container.
>>> Ideally, the SSL container would be a thin OCI-layer on top of the http
>>> version.
>>>
>>> On Thu, May 6, 2021 at 10:10 PM Fabricio Aguiar 
>>> wrote:
>>>
>>>> I finally made pulp_container CI work with https,
>>>> I also did some changes on pulp_installer, I believe these changes will
>>>> make it possible to run functional tests on dev environment.
>>>>
>>>> I think now it is a matter of deciding when is the best time to merge
>>>> the PR on the single container and if latest tag should be https or not
>>>>
>>>> PRs:
>>>> https://github.com/pulp/pulp-oci-images/pull/73
>>>> https://github.com/pulp/pulp_installer/pull/614
>>>> https://github.com/pulp/plugin_template/pull/379
>>>> https://github.com/pulp/pulpcore/pull/1283
>>>> https://github.com/pulp/pulp_container/pull/304
>>>> https://github.com/pulp/pulp_rpm/pull/1977
>>>> https://github.com/pulp/pulp_ansible/pull/572
>>>> https://github.com/pulp/pulp-2to3-migration/pull/362
>>>>
>>>> Best regards,
>>>> Fabricio Aguiar
>>>> Software Engineer, Pulp Project
>>>> Red Hat Brazil - Latam <https://www.redhat.com/>
>>>> +55 22 999000595
>>>>
>>>>
>>>>
>>>> On Tue, Apr 27, 2021 at 5:35 PM Fabricio Aguiar 
>>>> wrote:
>>>>
>>>>> I created https branch:
>>>>> https://github.com/pulp/pulp-oci-images/tree/https
>>>>> and pushed the following images:
>>>>> - pulp/pulp-ci-centos:https
>>>>> - pulp/pulp:https
>>>>>
>>>>> Now we can test on the plugins,
>>>>> I followed your suggestion and did it on pulp_npm:
>>>>> https://github.com/pulp/pulp_npm/pull/89
>>>>>
>>>>> Best regards,
>>>>> Fabricio Aguiar
>>>>> Software Engineer, Pulp Project
>>>>> Red Hat Brazil - Latam <https://www.redhat.com/>
>>>>> +55 22 999000595
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 27, 2021 at 9:25 AM David Davis 
>>>>> wrote:
>>>>>
>>>>>> This is great. Thank you for working on it.
>>>>>>
>>>>>> As a next step, would it make sense to create a branch and then try
>>>>>> to deploy a new temporary tag from that branch? Then maybe we can test a
>>>>>> plugin (eg pulp_npm) against this new image and see what breaks.
>>>>>>
>>>>>> David
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 26, 2021 at 5:01 PM Fabricio Aguiar 
>>>>>> wrote:
>>>>>>
>>>>>>> I started this POC: https://github.com/pulp/pulp-oci-images/pull/73
>>>>>>> It enables https on the single container, once merged, the CI for
>>>>>>> every plugin will run the functional tests using https.
>>>>>>> Probably it would break the majority of the CIs, we need to discuss
>>>>>>> when is the best moment to merge this PR or discuss alternatives
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Fabricio Aguiar
>>>>>>> Software Engineer, Pu

Re: [Pulp-dev] How to enable HTTPS for our tests in pulpcore and all plugins?

[Brian Bouterse]

+1 to this observation, we probably need to either ship both or make it
configurable somehow. Shipping both is probably easier on users.

On Fri, May 7, 2021 at 5:11 AM Matthias Dellweg  wrote:

> This is a great piece of work!
> The problem I see is that the SSL free container image may be used in
> places we do not control. And having this http based container equipped
> with an external https reverse proxy is imho a valid use case.
> Therefore i would prefer, if we could provide both versions of the image
> (with and without SSL) as different tags.
> This would also give us the opportunity to switch the plugins one by one
> to use the new container.
> Ideally, the SSL container would be a thin OCI-layer on top of the http
> version.
>
> On Thu, May 6, 2021 at 10:10 PM Fabricio Aguiar 
> wrote:
>
>> I finally made pulp_container CI work with https,
>> I also did some changes on pulp_installer, I believe these changes will
>> make it possible to run functional tests on dev environment.
>>
>> I think now it is a matter of deciding when is the best time to merge the
>> PR on the single container and if latest tag should be https or not
>>
>> PRs:
>> https://github.com/pulp/pulp-oci-images/pull/73
>> https://github.com/pulp/pulp_installer/pull/614
>> https://github.com/pulp/plugin_template/pull/379
>> https://github.com/pulp/pulpcore/pull/1283
>> https://github.com/pulp/pulp_container/pull/304
>> https://github.com/pulp/pulp_rpm/pull/1977
>> https://github.com/pulp/pulp_ansible/pull/572
>> https://github.com/pulp/pulp-2to3-migration/pull/362
>>
>> Best regards,
>> Fabricio Aguiar
>> Software Engineer, Pulp Project
>> Red Hat Brazil - Latam <https://www.redhat.com/>
>> +55 22 999000595
>>
>>
>>
>> On Tue, Apr 27, 2021 at 5:35 PM Fabricio Aguiar 
>> wrote:
>>
>>> I created https branch:
>>> https://github.com/pulp/pulp-oci-images/tree/https
>>> and pushed the following images:
>>> - pulp/pulp-ci-centos:https
>>> - pulp/pulp:https
>>>
>>> Now we can test on the plugins,
>>> I followed your suggestion and did it on pulp_npm:
>>> https://github.com/pulp/pulp_npm/pull/89
>>>
>>> Best regards,
>>> Fabricio Aguiar
>>> Software Engineer, Pulp Project
>>> Red Hat Brazil - Latam <https://www.redhat.com/>
>>> +55 22 999000595
>>>
>>>
>>>
>>> On Tue, Apr 27, 2021 at 9:25 AM David Davis 
>>> wrote:
>>>
>>>> This is great. Thank you for working on it.
>>>>
>>>> As a next step, would it make sense to create a branch and then try to
>>>> deploy a new temporary tag from that branch? Then maybe we can test a
>>>> plugin (eg pulp_npm) against this new image and see what breaks.
>>>>
>>>> David
>>>>
>>>>
>>>> On Mon, Apr 26, 2021 at 5:01 PM Fabricio Aguiar 
>>>> wrote:
>>>>
>>>>> I started this POC: https://github.com/pulp/pulp-oci-images/pull/73
>>>>> It enables https on the single container, once merged, the CI for
>>>>> every plugin will run the functional tests using https.
>>>>> Probably it would break the majority of the CIs, we need to discuss
>>>>> when is the best moment to merge this PR or discuss alternatives
>>>>>
>>>>> Best regards,
>>>>> Fabricio Aguiar
>>>>> Software Engineer, Pulp Project
>>>>> Red Hat Brazil - Latam <https://www.redhat.com/>
>>>>> +55 22 999000595
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 9, 2021 at 10:55 AM Fabricio Aguiar 
>>>>> wrote:
>>>>>
>>>>>> Our nginx conf only supports http now:
>>>>>> https://github.com/pulp/pulp-oci-images/blob/latest/assets/nginx.conf#L15
>>>>>> For not breaking all plugins, I believe we can build a new CI image
>>>>>> that supports https.
>>>>>> Maybe a template_config parameter - test_https: true would switch the
>>>>>> images
>>>>>>
>>>>>> Best regards,
>>>>>> Fabricio Aguiar
>>>>>> Software Engineer, Pulp Project
>>>>>> Red Hat Brazil - Latam <https://www.redhat.com/>
>>>>>> +55 22 999000595
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 9, 2021 at 5:16 AM Matthias Dellweg 
>>&

Re: [Pulp-dev] Tasking System Changes and Feedback

[Brian Bouterse]

 then it will be much
>>>> easier to maintain consistent state.
>>>>
>>>
>>> Are there any benefits to improving RQ vs the invented here method? I'm
>>> just curious about the cost of maintaining a tasking system versus being
>>> part of a community built one. This feels like the kind of problem that
>>> many other applications should have in the Python world -- or are there
>>> elements of Pulp's deployment architecture that make it unique here?
>>>
>>>
>>>> * *Maybe*.  We're considering using Redis as a cache to improve
>>>> content serving performance (after all, caching is one of the primary uses
>>>> of Redis). If we do, then Redis would remain in the architecture, but it
>>>> could potentially be an optional component and would be easier to remove at
>>>> some point in the future.
>>>> * We'd just be adding a small amount of information to each task
>>>> record, and it wouldn't prevent cleanup later.
>>>>
>>>
>>> This is sort of an aside to this general change. Are Pulp tasks cleaned
>>> up from the database today?
>>>
>>>
>>>>
>>>>
>>>>
>>>> On Thu, Apr 8, 2021 at 4:42 PM Eric Helms  wrote:
>>>>
>>>>> A few initial questions that get a bit into the stack but will help
>>>>> the Foreman project think on the proposed changes:
>>>>>
>>>>>  * Does this move away from RQ entirely or just RQ workers?
>>>>>  * Do the new workers remove Pulp 3's use of Redis all together?
>>>>>  * Will using the database result in any additional build up of
>>>>> tasking information that can impact performance over time? (Or does all
>>>>> task data get cleaned up eventually?)
>>>>>
>>>>> Thanks for sending this along early.
>>>>>
>>>>> On Fri, Apr 2, 2021 at 4:43 PM Brian Bouterse 
>>>>> wrote:
>>>>>
>>>>>> FYI, @mdellweg and I have been collaborating on the tasking system
>>>>>> changes. This email is to share some info to transition the work to
>>>>>> @mdellweg while I'm out. With the new-style disabled by default I am 
>>>>>> hoping
>>>>>> it can go into 3.13.
>>>>>>
>>>>>> ## The PoC and ticket info
>>>>>>
>>>>>> The PoC is basically functional, but it's a PoC:
>>>>>> https://github.com/pulp/pulpcore/pull/1222/
>>>>>>
>>>>>> * The epic is being tracked here which recaps why we're doing this
>>>>>> and the high level approach. The sub-tasks capture the various detailed
>>>>>> changes. https://pulp.plan.io/issues/8495
>>>>>>
>>>>>> * This is totally separate from the RQ workers you use today, and
>>>>>> those will continue to be available for a while.
>>>>>>
>>>>>> ## Next Steps
>>>>>>
>>>>>> * @mdellweg will continue the work and hopefully merge the PoC while
>>>>>> I'm out
>>>>>>
>>>>>> * Once it's demo-able I've asked @mdellweg to give a 20 minute,
>>>>>> public (hopefully recorded) technical demo. While it is designed to be a
>>>>>> drop-in replacement from a user perspective, we think sharing the 
>>>>>> internals
>>>>>> will be helpful to get feedback and increase the list of those who
>>>>>> understand the work.
>>>>>>
>>>>>> All the best,
>>>>>> Brian
>>>>>>
>>>>>> ___
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev@redhat.com
>>>>>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Eric Helms
>>>>> Principal Software Engineer
>>>>> Satellite
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>
>>>
>>> --
>>> Eric Helms
>>> Principal Software Engineer
>>> Satellite
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Tracking issues for plugin_template

[Brian Bouterse]

I imagined for downstream Bugzilla for example, we would have the
downstream BZ link to the upstream using the tracker ggainey mentioned.
Then for the github issues we would have a 'Bugzilla' label, and the link
would be in the comments. I expect the scripts would ensure these were
present like they do today. So mostly we would need to update the scripts
(also like ggainey said). With this setup, searching in bugzilla would be
roughly the same to what it is today, and we could query on the label to
look at the big-picture and then find any bugzilla from any specific github
issue from the comments.

I don't think the external autolinks is that valuable since a regular link
is just as usable even if its longer.

On Fri, Apr 30, 2021 at 11:56 AM David Davis  wrote:

> Interesting. I did not know about this. Here's a link with more info:
>
>
> https://docs.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
>
> David
>
>
> On Fri, Apr 30, 2021 at 11:34 AM Ewoud Kohl van Wijngaarden <
> ewoud+p...@kohlvanwijngaarden.nl> wrote:
>
>> On Sat, Apr 24, 2021 at 09:21:05PM -0400, Daniel Alley wrote:
>> >On Fri, Apr 23, 2021 at 10:37 AM David Davis 
>> wrote:
>> >
>> >> We started working on a plan to move repos over to Github Issues after
>> >> PulpCon last year but I think it kind of fell by the wayside over the
>> past
>> >> few months due to how busy we've been with stakeholder work. It would
>> >> definitely be a requirement in my mind to sort out things like how to
>> link
>> >> BZs to issues before moving plugins and other repos over to Gihub
>> Issues
>> >> that might have issues that affect downstream.
>> >
>> >I doubt there's any "good" way to link external bugs from Github because
>> >Mozilla doesn't do so.  They just add an extra tag and you just have to
>> >find the link somewhere in the comments.
>> >
>> >e.g.
>> >
>> https://github.com/servo/webrender/issues?q=is%3Aopen+is%3Aissue+label%3Abugzilled
>> >
>>
>> If you're on a paid plan you can replace links to an external system. I
>> can't find exactly where since it's hell to search for, but an example
>> can be seen in https://github.com/puppetlabs/puppet where PUP-NNN links
>> to their JIRA instance.
>>
>> It looks like it doesn't replace it everywhere as can be seen in a
>> random PR: https://github.com/puppetlabs/facter/pull/2371
>>
>> Certainly this isn't great.
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Tasking System Changes and Feedback

[Brian Bouterse]

FYI, @mdellweg and I have been collaborating on the tasking system changes.
This email is to share some info to transition the work to @mdellweg while
I'm out. With the new-style disabled by default I am hoping it can go into
3.13.

## The PoC and ticket info

The PoC is basically functional, but it's a PoC:
https://github.com/pulp/pulpcore/pull/1222/

* The epic is being tracked here which recaps why we're doing this and the
high level approach. The sub-tasks capture the various detailed changes.
https://pulp.plan.io/issues/8495

* This is totally separate from the RQ workers you use today, and those
will continue to be available for a while.

## Next Steps

* @mdellweg will continue the work and hopefully merge the PoC while I'm out

* Once it's demo-able I've asked @mdellweg to give a 20 minute, public
(hopefully recorded) technical demo. While it is designed to be a drop-in
replacement from a user perspective, we think sharing the internals will be
helpful to get feedback and increase the list of those who understand the
work.

All the best,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] FYI: New logger for deprecation warnings

[Brian Bouterse]

tl;dr use the logger @daviddavis made
https://github.com/pulp/pulpcore/pull/1225/files

See this issue for the backstory:  https://pulp.plan.io/issues/8499
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Benchmarking Tasking System Results

[Brian Bouterse]

Ahead of the tasking system changes @mdellweg and I are collaborating on,
we wanted to benchmark the current tasking system throughput which I've
measured as a maximum 2.6 tasks / sec.

Check out the test plan, results, and analysis here:
https://hackmd.io/DV633ocwTnShsI8LdajshA

We'll be using this same benchmark on the new-style workers we're putting
together. Look for more info soon on that.

All the best,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] CI/CD meeting minutes

[Brian Bouterse]

On Wed, Mar 31, 2021 at 10:50 AM Matthias Dellweg 
wrote:

>
>
> On Wed, Mar 31, 2021 at 4:42 PM Brian Bouterse 
> wrote:
>
>>
>>
>> On Wed, Mar 31, 2021 at 10:34 AM Fabricio Aguiar 
>> wrote:
>>
>>> ## March 31, 2021
>>>
>>> * Github Actions
>>> * good overall
>>> * faster and more reliable so far than Travis
>>> * easier to debug with tmate action
>>> * downsides
>>> * virtualization
>>> * not possible to re-run individual jobs
>>> * https://github.com/actions/runner/issues/432
>>> * people need write permission to re-run CI
>>> * https://github.com/actions/runner/issues/841
>>> * workaround: `git commit --amend`, no changes, force push
>>> * Release automation https://pulp.plan.io/issues/8093
>>> * Availability
>>> * Mike after operator?
>>> * David has ~1 FTE
>>> * Fabricio? Not sure. Maybe 0.5 FTE.
>>> * Dkliban?
>>> * First goal? https://pulp.plan.io/issues/7868
>>> * [david] to write up design
>>> * Regular CI/CD meeting?
>>> * Every other Wednesday 9:30-9:55am (15:30 EST)
>>> * [david] to schedule
>>> * Convert CI/CD code to ansible playbook?
>>> * Benefits: idempotence, easier to run locally
>>> * *Big* rewrite
>>>
>> I'm a little worried about this even besides its effort. One of my
>> worries if we switch to Ansible is that not everyone knows it. If folks
>> have to learn a new technology that's another barrier to usage. I've seen
>> this happen with the installer already to some extent.
>>
>
> Def valid concern. OTOH this was (and it is missing from the benefits)
> also driven by the idea that our current CI is understood only by a handful
> of people. And that would vastly improve if you could run that locally and
> see where it breaks.
>
That's fair. When I think about why our CI isn't well understood (to me)
it's because of a lack of working with it versus a lack of understanding
the technology that builds it. I'd be more concerned about the latter.
Running locally would be nice tho.

>
>
>>> https://hackmd.io/@pulp/cicd
>>>
>>> Best regards,
>>> Fabricio Aguiar
>>> Software Engineer, Pulp Project
>>> Red Hat Brazil - Latam <https://www.redhat.com/>
>>> +55 22 999000595
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] CI/CD meeting minutes

[Brian Bouterse]

On Wed, Mar 31, 2021 at 10:34 AM Fabricio Aguiar 
wrote:

> ## March 31, 2021
>
> * Github Actions
> * good overall
> * faster and more reliable so far than Travis
> * easier to debug with tmate action
> * downsides
> * virtualization
> * not possible to re-run individual jobs
> * https://github.com/actions/runner/issues/432
> * people need write permission to re-run CI
> * https://github.com/actions/runner/issues/841
> * workaround: `git commit --amend`, no changes, force push
> * Release automation https://pulp.plan.io/issues/8093
> * Availability
> * Mike after operator?
> * David has ~1 FTE
> * Fabricio? Not sure. Maybe 0.5 FTE.
> * Dkliban?
> * First goal? https://pulp.plan.io/issues/7868
> * [david] to write up design
> * Regular CI/CD meeting?
> * Every other Wednesday 9:30-9:55am (15:30 EST)
> * [david] to schedule
> * Convert CI/CD code to ansible playbook?
> * Benefits: idempotence, easier to run locally
> * *Big* rewrite
>
I'm a little worried about this even besides its effort. One of my worries
if we switch to Ansible is that not everyone knows it. If folks have to
learn a new technology that's another barrier to usage. I've seen this
happen with the installer already to some extent.

>
> https://hackmd.io/@pulp/cicd
>
> Best regards,
> Fabricio Aguiar
> Software Engineer, Pulp Project
> Red Hat Brazil - Latam 
> +55 22 999000595
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Removing pulp-smash from PyPI

[Brian Bouterse]

I'm +1 to this plan and the removal of pulp-smash from PyPI.

On Fri, Mar 26, 2021 at 11:26 AM David Davis  wrote:

> Yes, that's the plan. Sorry I didn't mention that in my email.
>
> David
>
>
> On Fri, Mar 26, 2021 at 11:23 AM Mike DePaulo 
> wrote:
>
>> It's declared in many plugins' test dependencies without the git path,
>> such as pulp_file/functest_requirements.txt
>>
>> I often install those as part of my installer development, and support
>> for other teams.
>>
>> So I agree with this approach, but update the plugins too.
>>
>> -Mike
>>
>> On Fri, Mar 26, 2021 at 11:16 AM David Davis 
>> wrote:
>>
>>> It recently came up that the pulp-smash release PyPI is badly out of
>>> date. It's an extra burden to release it and it doesn't seem to have any
>>> benefit. So instead we're considering using the git repo directly (see [0])
>>> but I wanted to first check if that's a problem for anyone?
>>>
>>> [0] https://github.com/pulp/pulpcore/pull/1210
>>>
>>> David
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://listman.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> --
>>
>> Mike DePaulo
>>
>> He / Him / His
>>
>> Service Reliability Engineer, Pulp
>>
>> Red Hat 
>>
>> IM: mikedep333
>>
>> GPG: 51745404
>> 
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] PulpCon 2021

[Brian Bouterse]

On Mon, Mar 15, 2021 at 11:12 AM David Davis  wrote:

> The topic of PulpCon came up today as spring is usually the time we begin
> to plan PulpCon. The main question I think is whether we should hold
> PulpCon again virtually this year or not.
>
> Optimally, we'd like to meet in person but given the uncertainty of our
> current situation, I think we should consider going virtual again this
> year. I noticed that other conferences such as DevConf.us (September 2021)
> are already planning to be virtual this year. And also, if we meet
> virtually this year, we could do an in-person PulpCon in early 2022 perhaps.
>
+1 to planning on virtual again for fall 2021. I think it allowed for more
attendance. +1 to also having an in-person event in early 2022.


> As for time frame, we'd need at least 3 months to prepare if it's virtual.
> So we'd be looking at sometime between July and December.
>
> Thoughts?
>
> David
> ___
> Pulp-list mailing list
> pulp-l...@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Run pulpcore-manager check --deploy

[Brian Bouterse]

I certainly share your interest in this matter. Thank you for bringing this
up. I have some questions, and I also want to share some complicating
aspects that I'm not entirely so sure what to do about as of yet.

## Questions

What did you imagine going in SILENCED_SYSTEM_CHECKS in pulpore.git?

## Complications

We tried to expand the checks you contributed with the FIPS epic for 3.11
and we ran into several problems causing us to conclude we couldn't use the
django checks mechanism. Here's what we got stuck on:

1. The checks are not run at application start time. This is a problem
because the things we are checking for are settings which users can change
at any point, not just 'deploy' time. If we start running them with each
application start, the application itself should just run them.

2. While (1) is a solvable problem, getting every environment to "use the
checks" when the application itself doesn't run automatically is a constant
uphill battle. As of now we have: the katello installer, the pulp
installer, openshift one-off deployments, containers built by galaxy_ng,
the operator, official containers built by pulp, containers built by users.
For this reason having all the checks run at startup is what we went with
for the fips checks and if Pulp takes longer to startup that would be ok.

There are two downsides with having the checks run by the application at
startup:
* It's startup overhead. This is why the Django checks framework doesn't do
it. As of now though, our checks aren't expensive enough to make it
prohibitive so for Pulp this is ok.
* You have to start pulp to run the checks. So our ideal situation is where
the same checks are runnable by the django checks mechanism, and at
application start time.

We've been discussing this at the pulpcore meetings. What do you think
about all this?


On Thu, Mar 11, 2021 at 10:34 AM Ewoud Kohl van Wijngaarden <
ewoud+p...@kohlvanwijngaarden.nl> wrote:

> Hello everyone,
>
> Django has a checks framework[1] which is exposed via django-admin check
> and this means there's also pulpcore-manager check.
>
> When I introduced the new default deployment layout[2], I implemented a
> check for storage paths[3] to verify settings.MEDIA_ROOT and
> settings.FILE_UPLOAD_TEMP_DIR are on the same filesystem, following the
> general recommendation to make moves super cheap. This is an example of
> a deploy only check since it only makes sense on the actual system.
> Those are only run when --deploy is passed to pulpcore-manager check.
>
> Checks are great, but if nobody runs them they're useless. That means we
> should run them. To do so, I opened a PR to our puppet-pulpcore module
> to run it as part of the acceptance tests. This generates a list of
> errors/warnings[4] that I think are invalid. Checks can be silenced via
> a setting[5].
>
> When there is a built in check mechanism that is also extendable in
> plugins, it makes it easier to verify an installation works. This can be
> integrated into debug procedures and tools like sosreport.
>
> IMHO the default SILENCED_SYSTEM_CHECKS should live in pulpcore.git
> because it's the application itself that determines what's relevant. An
> installer (pulp_installer and foreman-installer) can then rely on this
> without having to duplicate it. Ideally Pulp's CI is also modified to
> run pulpcore-manager check. Depending on how it's tested either with or
> without --deploy.
>
> The concrete question now is, do people agree this is a sensible path?
> If so, can we ship a clean config with SILENCED_SYSTEM_CHECKS configured
> so that out of the box it's clean and CI to keep it clean?
>
> Regards,
> Ewoud Kohl van Wijngaarden
>
> [1] https://docs.djangoproject.com/en/2.2/topics/checks/
> [2]
> https://github.com/pulp/pulpcore/commit/f8a8c64bb28cbe3908720ea56f417312a4389862
> [3] https://github.com/pulp/pulpcore/blob/master/pulpcore/app/checks.py
> [4]
> https://github.com/theforeman/puppet-pulpcore/pull/155#issuecomment-741083088
> [5]
> https://docs.djangoproject.com/en/2.2/ref/settings/#std:setting-SILENCED_SYSTEM_CHECKS
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Removing MD5 and SHA-1 as default available checksums in 3.11

[Brian Bouterse]

hat one isn't typically as big a problem.
Sha1 you may want to think about prompting your users to think about via
docs.

Note that programmatically changing this setting can get tricky
(practically speaking) if multiple plugins start to try to modify it on
behalf of their users. Ultimately though it's up to you as a plugin writer
to decide for users, or have them decide. We've opted mostly for letting
users having to take action if they want md5 and sha1, but whatever you
decide is ok with me. Generally I think your options are a) document adding
them back, b) having a setting that can enable/disable checksums to
default-re-enable one or both checksums when pulp_deb is installed, or c)
doing nothing and defaulting to disincluding sha1 and md5.

I don't feel particularly happy about this since this ammounts to an
> indefinite recommendation against the default pulpcore configuration for
> all users who want their pulp_deb repositories to resemble official
> Debian/Ubuntu repositories.
> (And for a security relevant configuration at that.)
>
I can understand being uneasy about setting pulp_deb to have a default that
is not aligned with pulpcore; once it's put in place like that I don't
think it'll be easy or possible to practically change it. This is one
motivation for going the docs route overall, which keeps the defaults
aligned, but users on the hook to opt-back-in.

Note that Katello (I think) is re-enabling all of them by default on their
installs, so also consider the choice in front of pulp_deb in the context
of just independent pulp_deb users, not Katello users. Please confirm this
with katello also, but this is my understanding.

>
> Alternatively, I could store the unloved md5 and sha1 hashes in the
> content models themselves instead of the artifacts, but this is ugly for a
> whole host of reasons:
>
> 1) It is a lot more work
> 2) For artifacts pulpcore automatically does the work of actually checking
> the checksums against the artifacts.
> 3) It duplicates the source of truth for checksums (the horror!) for users
> that do not go along with the new default pulpcore configuration.
>
> So that approach is probably a non-starter.
>
I agree this is a non-starter. Our goal is to not have them available at
all if admins don't want them to be, so storing them to the side I wouldn't
recommend. As an alternative to this idea, I think it would be better for
your users to add back sha1 and md5 to ALLOWED_CONTENT_CHECKSUMS instead.

>
> From the point of view of pulp_deb it would be better if pulpcore did not
> so much refuse to handle md5 and sha1, but rather would guarantee that at
> least one strong checksum is also present and used for integrity checking.
> Which I believe is the case anyway since we absolutely require sha256 to
> be present, no?
>
The data model always require sha256, in fact Pulp will not start if this
is removed from ALLOWED_CONTENT_CHECKSUM. It probably would be easier on
users to do what you're describing, but we also have to consider the
organizations who choose to use Pulp and have different expectations. We
ended up with a pretty strict behavior of removing checksums that are no
longer trusted from the db because it was the simplest way of causing
plugin writers to have to honor the requirements of the organization
deploying Pulp who decide at the organization level what they feel is
trustworthy regarding package integrity checking. I understand this makes
the life of a plugin writer more difficult. I also know plugin writers
dealing with that can be a pain. We've been spending a lot of effort on
this recently, so for sure, we feel the pain also.

>
> To summarize: I am uncertain how best to proceed, but perhaps I am
> overthinking this and simply respecting ALLOWED_CONTENT_CHECKSUMS and
> letting users decide is best.
>
The question I'll ask to help answer yours is: how much does pulp_deb break
with 3.11's defaults? This would be good to know. Want to run a few tests
and let us know? Maybe we can help give more info with that.

Aside from that, my general advice is to expect that pulp_deb users will
change this setting, and to have the pulp_deb code work with the checksums
it has available and error when it cannot fulfill their request due to not
having the checksums it would need to do so.

>
> regards,
> Quirin
> --
> *From:* pulp-dev-boun...@redhat.com  on
> behalf of Brian Bouterse 
> *Sent:* 12 February 2021 21:13
> *To:* Pulp-dev ; pulp-list 
> *Subject:* [Pulp-dev] Removing MD5 and SHA-1 as default available
> checksums in 3.11
>
> tl;dr With pulpcore 3.11, the plan is to remove MD5 and SHA-1 from the
> list of default available checksums.  RPM and Migration plugin users will
> need to add this back in at 3.11 upgrade time for your systems to continue
> working. Please give on-list feedback on this change.
&g

[Pulp-dev] Pulpcore team meeting notes

[Brian Bouterse]

# March 9, 2021

## Previous AIs
* [mdellweg] look for any docs issues that we might need for 3.11
* don't have time to complete for 3.11, so we skip and will revisit for
3.12
* [dalley] get more info about what happens remote gets deleted to revisit
next week
* deleting a remote, then recreating the remote, then resyncing it,
everything is fixed

## Topics
* Deleting a remote breaks content artifacts:
https://pulp.plan.io/issues/8305
* add a validation for remote removal 1)check whether it is used in any
of the RA or 2) linked to any repo 3) add the force flag for force removal
* deleting and rejecting
* https://pulp.plan.io/issues/7659 run orphan clean up in parallel
* retain N repository versions
* https://pulp.plan.io/issues/8368

## Action Items
[mdellweg] summarize "safety in deletion" of remotes feature onto
https://pulp.plan.io/issues/8305 [done]
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] https://fixtures.pulpproject.org/ downtime

[Brian Bouterse]

Thank you!

On Thu, Feb 25, 2021, 5:47 AM David Davis  wrote:

> The site has been updated and should be working now.
>
> On Tuesday, February 23, 2021, David Davis  wrote:
>
>> We're switching over to openshift for fixtures.pulpproject.org on
>> Thursday February 25 at 10:00 UTC (5am ET). It may take some time for the
>> DNS to update so we expect about an hour or so of downtime.
>>
>> David
>>
>
>
> --
>
> David
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] CI status board

[Brian Bouterse]

Thank you!

On Fri, Feb 19, 2021 at 12:44 PM Matthias Dellweg 
wrote:

> We have put up a collection of CI status badges across (hopefully soon)
> all our projects.
>
> https://github.com/pulp/pulp-ci/blob/master/README.md
>
> Please take a look regularly, because especially the nightlies are
> supposed to tell us something.
> In case you are missing a project, please just add them via a pr.
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Removing MD5 and SHA-1 as default available checksums in 3.11

[Brian Bouterse]

tl;dr With pulpcore 3.11, the plan is to remove MD5 and SHA-1 from the list
of default available checksums.  RPM and Migration plugin users will need
to add this back in at 3.11 upgrade time for your systems to continue
working. Please give on-list feedback on this change.

## Background

Pulp has the ALLOWED_CONTENT_CHECKSUMS setting [0] which, by default,
currently includes md5, sha-1, sha-224, sha-256, sha-384, and sha-512. Pulp
code is restricted to only using hashers from this list. This feature gives
admins the ability to prohibit hashers they do not trust. Pulp uses these
checksums for package integrity verification purposes when syncing and
publishing content.

## Motivation

We need to make Pulp secure by default. MD5 is known to be insecure, and
therefore it is unsafe for Pulp to allow its use for calculating package
integrity by default. SHA-1 is widely believed to be insecure, or will be
soon, and should not be allowed by default for the same reason.

## Proposal

Pulpcore 3.11 would remove md5 and sha-1 from the default list of allowed
checksums, leaving sha-224..sha-512. Specifically this change is occuring
in the `ALLOWED_CONTENT_CHECKSUMS` setting [0]. This is only a change to
the default settings; any specific system can be configured as desired.
Nothing is "being taken away".

## Required User Action with 3.11

We believe both RPM plugin users and Migration plugin users will be
impacted by this and mostly from the SHA-1 removal. SHA-1 is still used on
a variety of CDNs including Red Hat's. Also as data is migrated from Pulp2
systems, this also likely uses SHA-1 and MD5 as the migration plugin runs.

If users are using the defaults for `ALLOWED_CONTENT_CHECKSUMS` and want to
continue using SHA-1, they will need to update `ALLOWED_CONTENT_CHECKSUMS`
in their settings file. Alternatively, users will need to run
`pulpcore-manager handle-artifact-checksums` after upgrade to update any
existing artifacts after upgrading.

## Why not automate this?

We do not take manual user action at upgrade time lightly. However, this is
a security change, and we believe we need each Pulp system to opt-in for
themselves.

[0]:
https://docs.pulpproject.org/pulpcore/settings.html#allowed-content-checksums

Thanks!
The Pulpcore Team
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] How to enable HTTPS for our tests in pulpcore and all plugins?

[Brian Bouterse]

I believe all of our plugins (and CI) require HTTP and do not work with
HTTPS. I'm not well versed in what needs to be done to fix this, but I
think we should fix it.

Can the CI group have a 30 min call to talk over what needs to be done? Or
maybe share some info here?

The main issue I'm aware of is that the tests are not prepared to trust an
https certificate that is self-signed. I'm not exactly sure where we can
change that in one place either.

Thanks!
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] FYI: Use get_model() in migrations. Maybe audit your plugin too?

[Brian Bouterse]

We ran into an issue  (I originally
created ) where migrations were importing a Model like:

from my_plugin.apps.models import MyModel

Instead Django wants us to use `apps.get_model('app_name', 'MyModel')`
which reconstructs a historical model from the DB.

I opened a PR to fix in pulpcore
, and audited pulp_file and
pulp_ansible. Consider auditing your plugins too?

Cheers,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Feature Proposal: Virus Scanning Downloaded Content

[Brian Bouterse]

A user brought up this story/need in the dev channel, and they expressed a
goal to contribute it. Through discussion with the pulpcore team, @x9c4 and
I are going to try to guide the contribution. Anyone is welcome to help,
but it gives the contributor a clear person(s) to ask questions to.

Here's a revised writeup with input from the 3 of us:
https://pulp.plan.io/issues/8088

Any feedback or discussion (on issue please) is appreciated. We don't have
a timeline for implementation; it'll be up to the contributor.

Cheers,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Backports and LTS

[Brian Bouterse]

Stakeholder's won't agree on a single "LTS" (agreed daniel that's not a
great name for it) so I think option 3 is a non-starter. Option 2 is what
we do today and I agree it has the issues already mentioned.

So +1 to option 1, but as already mentioned, practically speaking, I
believe we can't release any more than we are without fully automating the
process. Can we fully automate the release process first, and then
re-discuss a policy change after?



On Wed, Jan 6, 2021 at 12:12 PM Daniel Alley  wrote:

> Matt, that is a good observation and meanwhile with pulp2 we had such a
>> policy, we cannot adopt it with pulp3. Release fast and release often is
>> something we are trying to stick to.
>>
>
> I don't think Matt was suggesting in any way that we not release fast and
> often, it's just that we have to pick a combination of cadance and # of
> supported releases that works for everyone.  This is basically Option 1
> with a pre-determined number of supported releases (rather than "backport
> to whatever version the stakeholder is using + all the others in between"
> which is how I interpret Option 1).
>
> I totally agree with David that it would be a pain to manage without
> automation though, same as Option 1.
>
> I think what would most likely happen under that plan is that we would see
> each stakeholder will take a release, stick to it until it is about to be
> dropped and then upgrade to the newest one, which is roughly the same as
> the LTS plan except that each stakeholder can choose their own "LTS"
> version and they could do an mid-cycle upgrade if there was ever a good
> reason to.
>
> Whatever option we choose, I'm kind of negative about actually using the
> term "LTS" given that I don't think we'd be supporting any version for more
> than 5-7 months.  The timeframe is a bit short for how the LTS moniker is
> typically used.
>
> On Wed, Jan 6, 2021 at 8:58 AM Ina Panova  wrote:
>
>> Matt, that is a good observation and meanwhile with pulp2 we had such a
>> policy, we cannot adopt it with pulp3. Release fast and release often is
>> something we are trying to stick to.
>>
>> It would be best to agree on the LTS version that would make all the
>> stakeholders happy but as it was pointed out not sure how we'd make this
>> possible, everyone has a different pace and release cycle.
>> Having that said, we should probably stick to option1 which puts more
>> burden on us but in return users and stakeholders are happy and the
>> project's upgrade path is stable.
>> One thing we could possibly do about the backport requests is to really
>> be thorough about which ones to accept by assessing the impact of the
>> stakeholder who has created the request and their possibility to upgrade if
>> any.
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Wed, Jan 6, 2021 at 2:18 PM Matt Pusateri  wrote:
>>
>>> Another option is Current Version minus X (N-1, N-2, etc.) If for
>>> instance you say we support N-1, and current version is 3.9, then you'll
>>> back port to latest 3.8.x.  N-2 at current version 3.9, would backport to
>>> 3.8.x, 3.7.x, etc.  The advantages here are that you already set the
>>> expectation for everyone, you limit the versions you support, you force
>>> people to stay current to get fixes.  The downside is that people have to
>>> upgrade and or it could impact downstream schedules.  The impact with going
>>> this route is affected by the release velocity.  So if you're rapidly
>>> releasing major versions, then it's more likely people won't keep up, or
>>> that they'll have to upgrade and are not able to for some reason.
>>>
>>> Matt P.
>>>
>>>
>>> On Wed, Jan 6, 2021 at 6:05 AM Matthias Dellweg 
>>> wrote:
>>>
 Thanks for putting all the options together.
 I would say that option 2 is a recipe for very bad user experience, and
 i'd vote strongly against it.
 I am ok with option 1, but I would add that the backport to the
 intermediate minor releases _must_ be performed (as in released) counting
 down, to always meet that upgrade expectation. Remember releasing can be
 delayed unexpectedly due to reasons.
 If we go for option 3, I think it's advisable to try to sync up
 stakeholders, because we don't want to maintain consecutive minor versions
 as LTS, and again, backporting a fix must traverse all maintained LTS in
 backward order. We should add expectations to how long we can keep an LTS
 version alive, and how often we bless one.

 On Tue, Jan 5, 2021 at 6:34 PM Tanya Tereshchenko 
 wrote:

> With more and further away backport requests coming in, there is more
> need for a clear policy/docs to set expectations for users and to define
> requirements for those performing a backport.
>
> The question which hasn't been answered yet (in documented way) is:
>

[Pulp-dev] Upcoming pulpcore==3.10 plugin requirement

[Brian Bouterse]

Starting with pulpcore==3.10 plugins will need to have the `versions`
attribute on the PulpPluginAppConfig subclass. If you want to release a
plugin that is forward compatible with both pulpcore 3.9 *and* 3.10 you'll
need to make this change now. The 3.10 breaking change
 is not yet made on master.

Since forward compatibility changes are new-ish I'm sending a friendly FYI,
but the usual process where plugin writers read the deprecations section of
the pulpcore changelog should suffice over time. Here's the 3.9 plugin
deprecations
 for
example.

Regarding this change, make a PR to your plugin like this pulp_file one
. It's basically a redeploy of
the latest plugin_tempalte to your plugin + some manual changes that are in
the plugin template but not files the plugin template controls. See the
pulp_file commit message for a recap of those manual changes (docs/conf.py,
__version__ removal, bumpersion config, and the `version` change itself).

Any questions or feedback are welcome.

All the best,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Tasking system improvements

[Brian Bouterse]

Recently at triage we discussed tasking system improvements. I have mostly
good news to share regarding the resolution of those.

Through postmortem analysis of the system we identified one specific
improvement and that got merged today. See the issue description for full
details on the failure scenario and the PR that fixed it:
https://pulp.plan.io/issues/7907

We also identified an opportunity for Pulp to avoid these types of problems
with less human intervention, and I wrote up that additional "health check
and recovery" bugfix here: https://pulp.plan.io/issues/7912

@dalley I'm hoping maybe you could consider implementing 7912 if there are
no objections or improvements from others.

The not so great news is we also identified a variety of race conditions
stemming from spreading our correctness across two data systems without
transactional support "across" them, i.e. postgresql and redis. While ^
specific fixes are great, I believe we will need to work longer term to
eliminate both redis and the resource-manager from the architecture to
fully close the door on these issues. I will write up a motivation for that
separately.

Any feedback, ideas, or concerns are welcome.

Cheers,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] OSTree Guesstimates of a Basic Implementation

[Brian Bouterse]

I was asked to look to write down a guess of what it would take to
bootstrap a basic-functionality ostree plugin. I wrote my guesses here:
https://hackmd.io/KT6AZGMBScuP7z6m9316Tg Basically 7.5 FTE weeks is my
claim.

Any feedback is appreciated, either here or on the document. Also it's just
an estimate; I think we need to be really thoughtful about when the right
time is to start this plugin. It's up to the developer community and
stakeholders to determine what (and when) we do next on this.

All the best,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Feature Proposal: Alternate Content Sources

[Brian Bouterse]

I wrote up an "Alternate Content Sources" feature proposal for Pulp3. This
would be similar to the Pulp2 version. It includes background on the Pulp2
feature, use cases I've heard motivating it then (and now), and a proposed
implementation.

https://pulp.plan.io/issues/7832

Please send or post on the issue, any feedback or questions, or ideas. This
feature would likely be developed over the sprint '21 timeframe, to be used
by Katello.

Thanks,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Travis pricing

[Brian Bouterse]

On Mon, Nov 9, 2020 at 10:27 AM David Davis  wrote:

> I looked a little bit more into Gitlab CI this morning. If my
> understanding is correct, their cap is 50,000 minutes. Right now, we're
> using about 44,000 minutes in Travis[0]. Some of our jobs (eg
> pulp_installer, pulp-cli, etc) are on GHA already so in theory if we moved
> everything over to Gitlab CI, we could potentially hit their cap?
>
This is my main concern. If we're going to spend the effort to switch, we
need to not end up in the same situation.


> [0] https://travis-ci.com/github/pulp?tab=insights
>
> David
>
>
> On Wed, Nov 4, 2020 at 8:18 AM David Davis  wrote:
>
>> We looked at Fedora's Zuul instance before and decided against using it
>> for two reasons: (A) it'd be a lot of work/maintaince (eg we'd have to
>> write our own zuul jobs, bring our own compute resource, etc), and (B) we
>> were worried about the support since it's not a paid (or freemium) option.
>>
>> That said, it might be worth considering again if nothing else for
>> testing on specific environments such as selinux which is difficult to do
>> on hosted CI providers.
>>
>> David
>>
>>
>> On Tue, Nov 3, 2020 at 3:51 PM Neal Gompa  wrote:
>>
>>> On Tue, Nov 3, 2020 at 3:35 PM David Davis 
>>> wrote:
>>> >
>>> > Travis recently announced changes to their plan pricing that will
>>> impact open-source projects such as Pulp[0]. It's likely that we'll exhaust
>>> the monthly budget that Travis is going to give OSS projects and we're not
>>> sure how generous Travis will be giving out extra build minutes.
>>> >
>>> > Given our concern, members of the CI team met today to discuss our
>>> options. We have some notes[1] from our meeting about some of the options
>>> that stood out to us. We'd like to have a plan in place when the new
>>> pricing gets rolled out to our organization.
>>> >
>>> > Any feedback is welcome.
>>> >
>>> > [0] https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
>>> > [1] https://hackmd.io/n6kStnNiTPGekAWGdNvbhA
>>>
>>> Could we use the Fedora CI Zuul instance[2]? There's already a ton of
>>> other projects using one of the Zuul instances on
>>> softwarefactory-project.io, and leveraging the Fedora CI
>>> infrastructure could also help future efforts in doing auto-release to
>>> Fedora for Pulp releases, too.
>>>
>>> [2]: https://fedora.softwarefactory-project.io/zuul/projects
>>>
>>> --
>>> 真実はいつも一つ！/ Always, there's only one truth!
>>>
>>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Unblocking SigningService hash check PR

[Brian Bouterse]

I finally got around to outlining the next steps we discussed.

# The two stories that should be done next
https://pulp.plan.io/issues/7700
https://pulp.plan.io/issues/7701

# I closed this story because we don't have a plan for it. We can reopen it
later when we do
https://pulp.plan.io/issues/6291

On Thu, Sep 17, 2020 at 4:38 AM Quirin Pamp  wrote:

> The result of our discussion was to pursue a (somewhat) different
> approach  to the one from the PR.
> It therefore makes sense to close the PR.
>
> I am a bit fuzzy on what the next steps were (adding a public key to the
> SigningService base class?), but I suppose I will just have to re-watch the
> meeting.
> I also get the feeling, continuing SigningService work isn't at the top of
> anyone's To-Do list right now. We will see who blinks first.
> ------
> *From:* Brian Bouterse 
> *Sent:* 16 September 2020 22:55
> *To:* Quirin Pamp 
> *Cc:* pulp-dev@redhat.com 
> *Subject:* Re: Unblocking SigningService hash check PR
>
> At the pulpcore meeting we noticed the signing service PR [0] had stalled
> and instead we needed to work towards the next-steps identified in the
> video call and discussed on this thread. I closed that PR with a note
> asking that we focus on the next steps we discussed in the meeting (also
> linked to [0]). I can help review and guide that contribution if someone is
> interested.
>
> [0]: https://github.com/pulp/pulpcore/pull/659#issuecomment-693658607
>
>
>
> On Tue, Jun 30, 2020 at 4:48 PM Brian Bouterse 
> wrote:
>
> Sorry for the long delay in reply. Thank you for the email. I also want to
> unblock that PR. I put some replies in line.
>
> On Wed, Jun 24, 2020 at 7:03 AM Quirin Pamp  wrote:
>
> Hi,
>
>
> However we decide to continue with the SigningService topic in the medium
> and longrun, I wanted to have one more go at unblocking the following PR in
> the short run:
>
>
> https://github.com/pulp/pulpcore/pull/659
>
>
> Currently, this PR issues a warning whenever the hash of a signing service
> script has changed on disk (compared to when it was first validated).
>
>
> I think we are all in agreement that this is a bad compromise between
> doing nothing at all (since the script might have changed for legitimate
> reasons), and issuing a full on Error in cases where things are broken.
>
>
> Yes I believe all parties agree the warning is the compromise no-one likes.
>
>
> My proposal is the following:
>
>
> Instead of issuing a warning, a changed hash value on disk would trigger
> an automatic re-validation of the script on disk.
>
> If the validation fails, it will throw a hard error (which would certainly
> be the correct course of action for a script that does not perform what the
> SigningService promises).
>
> If the validation succeeds, the SigningService is updated with the new
> hash value, and everything continues as it nothing had happened (we just
> assume the script was changed for legitimate reasons).
>
> Overall this sounds ok to me, but I want to confirm, is the validation
> this code does?
> https://github.com/pulp/pulpcore/blob/bce4bee8124502ecd183fc664b904f5af5be97db/pulpcore/app/models/content.py#L453-L490
>
> If so, moving the `public_key` and ^ `signing service` attributes from
> AsciiArmoredDetachedSigningservice in pulp_rpm to SigningService in
> pulpcore would be a good first step. I believe this will need to happen
> first and is consistent with the last discussion of our video call. This
> would be a good first step in its own PR I think.
>
>
> The only thing I can come up with where this approach might be
> problematic, is if users want to have different versions of the signing
> service script on different workers (for some reason).
>
> However, in such cases it would still be possible to work around the
> problem by having a single signing service script call a secondary script
> that differs on different workers.
>
> If each script has its own public key it will fail. If each script on each
> worker with a different sha256 shares one public key it would be
> revalidated-and-modified each time it's used, which is strange but ok. Also
> I don't think this use case is really valid so I'm ok with this.
>
>
> If you are worried that the possibility of such a workaround defeats the
> whole purpose of hashing the script in the first place, consider the
> following:
>
> This is not intended as a security feature against some kind of malicious
> attacker scenario, it is intended to provide some more meaningful error
> reporting, for operational mistakes.
> In this context I almost consider it a bonus if Sysadmin users who want to
> do something rather unusual and co

Re: [Pulp-dev] Raising the pulpcore requirement for plugins

[Brian Bouterse]

Hi Quirin,

Apologies for the long delay. Thank you for bringing this up. I wrote some
responses inline with what I believe to be true. More questions are welcome!

On Wed, Sep 30, 2020 at 3:51 AM Quirin Pamp  wrote:

> So far, I have been mindlessly raising the pulpcore requirement by one Y
> version with each Y version release of pulp_deb. So, for example, The
> release of pulp_deb 2.5.0b1 was compatible with 'pulpcore>=3.5,<3.6', and
> the 2.6.0b1 release was compatible with 'pulpcore>=3.6,<3.7'.
>
> I have been talking to x9c4 about this, since he pointed out that there
> really wasn't any reason to automatically raise the floor on pulpcore if
> there are no breaking changes. For example, it might have been fine for
> 2.5.0b1 to have a dependency of 'pulpcore>=3.5,<3.6' and for 2.6.0b1 to
> switch that to 'pulpcore>=3.5,<3.7', leaving the floor of the range
> untouched (not sure if there actually were no breaking changes with those
> particular versions, it is just intended as an example).
>
Yes this is the case. For example pulp-certguard hasn't experienced a
breaking change since 3.3, so it's floor version even for the latest
release declares compatibility all the way back to 3.3 (and forward through
3.8) https://github.com/pulp/pulp-certguard/blob/1.0.3/requirements.txt#L2

>
> x9c4 also argued that it might be a good idea to raise the floor
> requirement with the introduction of a breaking change (and not at the time
> of creating the release branch).
>
I agree with this, if a plugin makes a change which would require it to
have a specific version of pulpcore.plugin, along with the plugin's change
it should raise the floor version. If this is done at development time
correctly at release time the "floor" would never be raised.

>
> Both suggestions make sense to me, but they do not seem to reflect the
> current workflow for most plugins right now (I also don't feel very
> comfortable I know how to recognize breaking changes, making me want to
> keep on mindlessly raising the floor to be on the safe side). The release
> script is built in a way that suggests both upper and lower pulpcore range
> should be set at the time when the release branch is created. With all of
> this in mind, here are some questions I have for the community:
>
At release time the upper limit is specified to ensure a much later
pulpcore release won't end up breaking user installs. The floor however I
expect to be kept in source control at all times. For example, 3.3 as the
minimum for pulp-certguard is shown on master always:
https://github.com/pulp/pulp-certguard/blob/master/requirements.txt#L2 The
idea is that the floor declares actual compatability, and during
development there is no upper limit to always have the CI evaluate the
forward compatibility with pulpcore, and only at release time do you say
"oh actually we're only compatible as far as pulpcore~=x.y.0".

>
> 1) Are there any current best practice recommendations how to raise the
> floor on the pulpcore requirement for plugins?
>
Raise it at development time when a plugin actually requires a newer
pulpcore.plugin version.

>
> 2) Does it make sense to raise the floor when a breaking change is
> introduced, rather than when the release branch is created? (This might
> give a false sense of security when cherry-picking things from after the
> raising of the floor.)
>
Yes, I recommend at development time, when the plugin's change is made.


> 3) Other than reading the "Deprecations and Removals", how does one
> recognize when a breaking change in pulpcore needs me to raise the floor?
> (Especially now with the new 2 Y-version deprecation cycle, some more
> pointers/examples on how to use this in practice would be good.) Is it just
> a matter of running the test suite against each pulpcore version that I am
> declaring compatibility with?
>
The idea (to me) is that the functional tests will tell us. What we don't
test, we can't be sure works. Right now @dkliban is extending the
plugin_template to add a nightly test to the CI of each plugin which tests
the GA of a current plugin against the unreleased next version of pulpcore.
If your plugin has all the tests you care about then this CI run will point
out places where your plugin is not compatible with the future pulpcore.
This tells you either now is the time for you to port, or perhaps the
pulpcore folks broke something they didn't intend to (without the proper
deprecation length). The ticket is here: https://pulp.plan.io/issues/7411
He ran into one or two issues getting it working, but I believe he's
resolving those. Getting this rolled out to all plugins I think is key to
sorting these problems out.


> Any thoughts and hints are appreciated!
> Kind regards,
> Quirin Pamp (quba42)
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list

Re: [Pulp-dev] Pulp 3 CLI PoC Feedback

[Brian Bouterse]

I'm excited to use this. I went to try it, but the CLI's requests wouldn't
trust the server which uses self-signed certificates. When running `pulp
status` I get this error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost',
port=443): Max retries exceeded with url: /pulp/api/v3/docs/api.json
(Caused by SSLError(SSLCertVerificationError(1, '[SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local
issuer certificate (_ssl.c:1123)'))

Let me know if I should file this somewhere more specifically.

On Mon, Oct 5, 2020 at 5:08 AM Melanie Corr  wrote:

> Hi all,
>
> An initial Pulp 3 CLI PoC is now available. We are hoping to gather as
> much feedback as possible from the Pulp community so that future
> development can progress with your opinions and requirements in mind.
>
> The following blog post has instructions on how to download and install
> the Pulp 3 CLI PoC, as well as a demo and a feedback link. Please test out
> the Pulp 3 CLI PoC and let us know your thoughts. We will have some SWAG
> for participants :)
>
> https://pulpproject.org/2020/09/28/pulp-3-cli-poc-call-for-feedback/
>
> If you've any questions or comments, feel free to ask.
>
> Melanie
>
> --
>
> Melanie Corr, RHCE
>
> Community Manager
>
> Red Hat 
>
> Remote, Ireland
>
> mc...@redhat.com
> M: +353857774436 IM: mcorr
> 
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PR merging

[Brian Bouterse]

On Thu, Oct 1, 2020 at 7:35 AM Ina Panova  wrote:

>
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Sep 30, 2020 at 2:38 PM David Davis  wrote:
>
>> I am also concerned about the lack of human involvement and the potential
>> for things to be merged accidentally. I definitely could see that happening.
>>
>> I like the idea of having the PR processor only merge if a label (eg
>> "Merge when Ready") is present. The question then is whether it should be
>> applied automatically or not when a PR is opened. Maybe to start out with,
>> it's not.
>>
>> On one hand, I like the idea of having this label as well and adding it
> to the already mentioned conditions in your PR[0]. However, I would imagine
> this would be a manual step: whether the reviewer or the author of the PR
> would set this label. This will also cover the case Daniel
> mentioned.."sometimes there's only minor changes requested and it doesn't
> warrant a re-review. Currently what we do pretty often is just approve the
> PR, leave a note to change the wording here and there, and the author can
> just merge it whenever they're done with the minor changes."
>
> On the other hand I don't know what will be easier: pressing the merge
> button or manually adding the label. Probably the only benefit would be -
> you don't need to wait and hypnotize the CI so it gets green.
>
> Openstack upstream policies also uses auto-merge [0] scroll to the end.
>
> In my opinion, if we want to automate things this will require us to be
> more diligent and responsible and leave +1 only when the PR is *really*
> ready. Having the PR in a not a not draft state, +2 approvals and passing
> CI I think is satisfactory enough to be merged. Adding another label feels
> like killing the whole gist of automation.
>
I agree. If we're adding automation but also with manual steps we're not
really realizing the benefits the automation is supposed to give us.

I do think automation with two acks is a good idea. To me, our acks
indicate something is ready to merge, and if two committers are saying that
is the case, I believe the machines can take it from there.

>
> Another thing that can be done is clearing up all the approvals on each
> push to the PR. But this will require people approving once again.
>
> [0]
> https://github.com/pulp/pulp-ci/pull/737/files#diff-cbdf61d38083f599940c37eeb49cb0a9R116-R141
> [1] https://docs.openstack.org/zaqar/latest/contributor/first_patch.html
>
> One other thing I thought of is that the PR processor could also check PR
>> dependencies (ie "Required PRs") and see if they are all mergeable before
>> merging any PR.
>>
>> David
>>
>>
>> On Wed, Sep 30, 2020 at 4:10 AM Matthias Dellweg 
>> wrote:
>>
>>> This just reminds me that Gitlab has a very nice "merge when CI passes
>>> button" to decouple the merge question from the reviews.
>>> The only way i could see this happen in Github is via setting a label
>>> that instructs the PR processor to merge when (label is set) && (ci is
>>> green) && (other conditions).
>>> Does not sound too user friendly, but labels can only be set by people
>>> with merge permissions anyway (so not a security concern).
>>>
>>> On Tue, Sep 29, 2020 at 8:04 PM Daniel Alley  wrote:
>>>
 I have some doubts about the cost/benefit ratio of coding automation to
 merge PRs vs. simply changing the default option / being selective about
 which options are available.

 I like the idea in general though.  A lot of projects do something like
 this.  I occasionally contributed to the Servo project (RIP) and they used
 automation to manage this.  The benefits were, their tests suites ran for
 nearly 2 hours on 3 different operating systems, so the feedback loop was
 quite slow, and automatically triggering the full test run every time a PR
 was changed would be a horrendous waste of resources at the scale they were
 running at.  So basically they would run a minimal CI within the PR, then
 require a manual review, and once approved it would trigger the full CI,
 and if that passed then it would automatically merge the PR hours later
 whenever it finished.

 If our test suites keep getting longer and longer and take more than
 the 20-25 minutes they currently take, I can definitely see how the commit
 processor approach could add value.

 On the other hand, it has the downside that, sometimes even once a
 commit is approved you might want to change something minor like rebasing
 the commits manually, and automation might start getting in the way of
 things like that.  Or, another example, sometimes there's only minor
 changes requested and it doesn't warrant a re-review. Currently what we do
 pretty often is just approve the PR, leave a note to change the wording
 here and there, and the 

Re: [Pulp-dev] sphinx theme for docs.pulpproject.org

[Brian Bouterse]

On Thu, Oct 1, 2020 at 10:05 AM Quirin Pamp  wrote:

> Of course these options are not mutually exclusive. We could decide on
> some theme (e.g.: pulp_ansible theme) and then have both pulpcore and all
> plugins consistently use that theme.
> I do quite like the sphinx_rtd_theme used by pulpcore, but that may just
> be because I am used to it.
>
> I guess my personal preferences would be:
>
> 1) All plugins and pulpcore use a custom pulp theme with pulp logo and a
> colour scheme like the one on https://pulpproject.org/
> 2) All plugins and pulpcore simply use the sphinx_rtd_theme (which strikes
> me as good enough, already widely in use, and similar to what readthedocs
> uses, so nobody needs to get used to anything totally new).
> 3) All plugins and pulpcore consistently use some other agreed upon theme.
>
> In the interest of "getting on with it" I would propose going with option
> (2) right now, and setting option (1) as a long term goal (perhaps open an
> issue for it).
>
This all sounds good to me.


> Kind regards,
> Quirin Pamp (quba42)
>
> --
> *From:* Ina Panova 
> *Sent:* 01 October 2020 15:00
> *To:* Dennis Kliban 
> *Cc:* Quirin Pamp ; Pulp-dev 
> *Subject:* Re: [Pulp-dev] sphinx theme for docs.pulpproject.org
>
> my preference number 1 would be making plugins consistent with pulpcore.
> 2nd preference would be plup_ansibe theme.
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Thu, Sep 24, 2020 at 4:14 PM Dennis Kliban  wrote:
>
> On Thu, Sep 24, 2020 at 9:01 AM Quirin Pamp  wrote:
>
> I don't have a strong opinion, but I feel like if we don't come up with
> anything better we could just  fall back to what the pulpcore docs are
> currently using.
>
> https://docs.pulpproject.org/pulpcore/
>
> https://github.com/pulp/pulpcore/blob/d31eb22453cd2d56b43ed884ec193fa166cafe62/docs/conf.py#L116
>
>
> +1 to making everything consistent with pulpcore.
>
>
> quba42
> --
> *From:* pulp-dev-boun...@redhat.com  on
> behalf of Dennis Kliban 
> *Sent:* 24 September 2020 14:47
> *To:* Pulp-dev 
> *Subject:* [Pulp-dev] sphinx theme for docs.pulpproject.org
>
> We are slowly rolling out plugin docs to docs.pulpproject.org. At this
> time pulp_file and pulp_ansible are the only plugins publishing their docs
> there. What I noticed is that they have completely different themes. We
> need to pick a single theme to be used by all plugins. Does anyone have
> suggestions for which theme that should be?
>
> https://docs.pulpproject.org/pulp_ansible/
> https://docs.pulpproject.org/pulp_file/
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Disabling merge by commit

[Brian Bouterse]

On Fri, Sep 25, 2020 at 8:14 AM Matthias Dellweg 
wrote:

> I know, this is late to the game, but i didn't have a better argument than
> i didn't like. Now i know why i prefer merge commits:
> Having proper merge commits:
> - keeps the information when a change was made and when it was merged
> (quba42 mentioned this before)
> - keeps signatures on commits intakt (I know almost no one signes their
> commits, i do)
>
I never thought of this, but throwing away signed commits is kind of a big
deal. I love the rebase default, and don't like the merge commit option,
but this is a deal-breaker for me. Over time I think we want more signed
commits for project security.

- allows to delete your feature branches with `git branch -d` instead of
> `git branch -D` and thereby prevents unintentional loss of unmerged work
>
> The most prominent argument i heard here was for a clean git history.
> For me a clean git history should focus on the individual commit, like:
> - one commit for one change
> - good commit messages (https://chris.beams.io/posts/git-commit/ is a
> very good guideline)
> - keep the diff short and readable (black does a good job there with
> trailing commas and no hanging indents)
>
> For this last bullet point, i want to suggest, that we drop the "Break
> documentation lines at 100 characters and try to imitate block formatting."
> rule.
> We should break lines in rst and md files like we do in all other source
> code at language structures. If you have one sentence per line, and you
> want to change one sentence, the resulting diff reflects just that (+1-1);
> If you delete a phrase (-1); If you add two sentences (+2). This also helps
> to find the commit, a specific part of the docs was changed for real via
> "git blame".
>
> On Fri, Sep 25, 2020 at 1:11 AM David Davis  wrote:
>
>> I agree that merging by squash is potentially unsafe. I'll disable it for
>> pulpcore and pulp_file unless anyone objects.
>>
>> David
>>
>>
>> On Thu, Sep 24, 2020 at 11:56 AM Brian Bouterse 
>> wrote:
>>
>>> I'm in favor of only the rebase & merge option everywhere. Our commit
>>> association machinery relies on commits not being modified, so I don't
>>> think the "squash and rebase" is a safe option for us. I am glad we are no
>>> longer using merge commits also.
>>>
>>> On Thu, Sep 24, 2020 at 11:39 AM Tatiana Tereshchenko <
>>> ttere...@redhat.com> wrote:
>>>
>>>> pulp_rpm left only rebase & merge option.
>>>>
>>>> On Wed, Sep 23, 2020 at 7:46 PM Mike DePaulo 
>>>> wrote:
>>>>
>>>>> On Wed, Sep 23, 2020 at 9:52 AM Justin Sherrill 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> On 9/23/20 7:18 AM, David Davis wrote:
>>>>>>
>>>>>> I think the two main things for me are (1) it makes git history more
>>>>>> linear and (2) it cuts down on the number of commits. Both of these make
>>>>>> git history more readable.
>>>>>>
>>>>>> 3rd main thing for me:
>>>>> 3. It removes extra work when rewriting history. Rewriting history is
>>>>> desirable in case secret keys, huge binary blobs (that degrade git
>>>>> performance), etc accidentally get through.
>>>>>
>>>>>> The 'rebase and merge' option provides a nice balance of letting you
>>>>>> provide multiple commits and maintain commit history while not creating a
>>>>>> merge commit and  making a hard to read commit history.  Sometimes it is
>>>>>> more expressive to have two (or three) commits that make up one pr to 
>>>>>> make
>>>>>> it into the source tree.
>>>>>>
>>>>> I agree with rebase and merge. Often I need multiple commits for that
>>>>> reason, or for multiple closely related (pulp_installer) tickets.
>>>>>
>>>>> I've done this both on the X2Go Project
>>>>> <https://wiki.x2go.org/doku.php>, and at a previous job with a big
>>>>> ansible codebase.
>>>>>
>>>>> -Mike
>>>>>
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Wed, Sep 23, 2020 at 6:48 AM Ina Panova  wrote:
>>>>>
>>>>>> Hi Quirin,
>>>>>>
>>>>>>
>>>>>> 
>>>>>> Regards,
>>>>>>
>>>>>> I

Re: [Pulp-dev] CLI team meeting notes

[Brian Bouterse]

On Wed, Sep 23, 2020 at 3:31 PM Robin Chan  wrote:

>
> Robin Chan
>
> She/Her/Hers
>
> Satellite Software Engineering Manager - Pulp
>
> Red Hat 
>
> IRC: rchan
>
> Red Hat respects your work life balance. Therefore there is no need to
> answer this email out of your office hours.
> 
>
>
>
> On Wed, Sep 23, 2020 at 12:45 PM David Davis 
> wrote:
>
>> ## Sept 23, 2020
>>
>> * [david] Finish CLI PoC demo and upload to asciinema
>> * Send out email to pulp-list about PoC?
>> * Include asciinema
>> * How to install, use CLI
>> * Ask for feedback
>> * Contact mcorr first and see what she recommends
>> * Should we meet regularly?
>> * Meet again in two weeks
>> * Hopefully have some user feedback
>> * We need a decision about where to have the cli code
>> * it's not urgent.
>> * for now, keep using a single repo
>>
> I was hoping someone would chime in here on what would be the cli
> experience of a plugin that isn't in the pulp org - say debian or chef?
> What would be simplest? Or is there an option that would be hard to change
> back the other way? This seems like a pretty important decision. I'd like
> to see some use cases or requirements that might help determine a decision
> or rule out any. And I'd like to hear from other stakeholders.
>
I agree simplicity is key. Here are some questions/thoughts I have to try
to determine what simple looks like

One question I have is: will plugins have custom commands. I suspect the
answer is yes because even if the CLI package itself is smart enough to
auto-produce all the generic commands from the API spec, I suspect in most
cases plugins will want "custom commands".

So if that's a yes, how will they ship those? While it's simple to add them
to the one CLI repo, it's complicated for users to get the "right" version
for them when it's all in one package. In fact that may be impossible. So
not as simple, but likely more usable would be for any custom CLI commands
to ship as a "CLI plugin package" aka a python package that will give extra
commands and have this auto-release when the plugin itself releases. What
wouldn't be simple is an additional repo for each plugin that would require
additional complexity at each plugin release.

The final question I wonder about is testing: How will CI be done on these
commands? My take is probably simplistic, but CI it anywhere plugin bits
are released from so if that's one main repo for the CLI CI those parts
there, and then CI any custom commands from repos where they are released
from.

This is what I was thinking about, but I am ok with anything the CLI teams
would be better or simpler.


>
>> * Supporting multiple versions of pulpcore and plugins
>> * For now, use conditional statements when needed
>> * Versioning of the CLI
>> * Needs more thought
>>
>> David
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Disabling merge by commit

[Brian Bouterse]

I'm in favor of only the rebase & merge option everywhere. Our commit
association machinery relies on commits not being modified, so I don't
think the "squash and rebase" is a safe option for us. I am glad we are no
longer using merge commits also.

On Thu, Sep 24, 2020 at 11:39 AM Tatiana Tereshchenko 
wrote:

> pulp_rpm left only rebase & merge option.
>
> On Wed, Sep 23, 2020 at 7:46 PM Mike DePaulo 
> wrote:
>
>> On Wed, Sep 23, 2020 at 9:52 AM Justin Sherrill 
>> wrote:
>>
>>>
>>> On 9/23/20 7:18 AM, David Davis wrote:
>>>
>>> I think the two main things for me are (1) it makes git history more
>>> linear and (2) it cuts down on the number of commits. Both of these make
>>> git history more readable.
>>>
>>> 3rd main thing for me:
>> 3. It removes extra work when rewriting history. Rewriting history is
>> desirable in case secret keys, huge binary blobs (that degrade git
>> performance), etc accidentally get through.
>>
>>> The 'rebase and merge' option provides a nice balance of letting you
>>> provide multiple commits and maintain commit history while not creating a
>>> merge commit and  making a hard to read commit history.  Sometimes it is
>>> more expressive to have two (or three) commits that make up one pr to make
>>> it into the source tree.
>>>
>> I agree with rebase and merge. Often I need multiple commits for that
>> reason, or for multiple closely related (pulp_installer) tickets.
>>
>> I've done this both on the X2Go Project ,
>> and at a previous job with a big ansible codebase.
>>
>> -Mike
>>
>>
>> David
>>
>>
>> On Wed, Sep 23, 2020 at 6:48 AM Ina Panova  wrote:
>>
>>> Hi Quirin,
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Wed, Sep 23, 2020 at 10:47 AM Quirin Pamp  wrote:
>>>
 "I'd encourage plugins to consider disabling merge by commit as well."

 In order to evaluate this it would be great, if you could explain why
 this was decided for pulpcore and pulp_file.
 You have posted a lot of general information about the different merge
 type (the "What?"), but not so much on the "Why?".

 As far as I can tell the main advantage of squish and rebase, is that
 it leads to a more tidy history in master, at the cost of losing some
 information on how the sausage was made.
 As a result squish and rebase becomes increasingly advantageous with
 increasing PR volume.
 However, I fail to see an advantage for pulp_deb, which does not have a
 large PR volume.

 Or am I missing some relevant part of the argument?

>>>
>>> I think your understanding is correct. In my perspective it is important
>>> to have a tidy history in master no matter how high/low PR traffic you have.
>>>
>>> pulp_container has disabled merge by commit as well.
>>>

 Quirin
 --
 *From:* pulp-dev-boun...@redhat.com  on
 behalf of David Davis 
 *Sent:* 22 September 2020 17:16
 *To:* Pulp-dev 
 *Subject:* Re: [Pulp-dev] Disabling merge by commit

 Here's some more information about PR merges as well:


 https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-request-merges

 David


 On Tue, Sep 22, 2020 at 11:11 AM David Davis 
 wrote:

 Today at open floor, we decided to disable merging by commit for
 pulpcore and pulp_file PRs. Instead, developers will rebase or squash PRs
 to merge them. This adds the changes to HEAD instead of
 interspersing commits and creating a merge commit. This picture of git
 history comparing pulpcore to foreman (which doesn't merge by commit)
 illustrates the differences:

 https://imgur.com/a/uiIa0Mr

 I'd encourage plugins to consider disabling merge by commit as well. To
 do so, go to the settings page for your github repo and look under the
 Merge Button section.

 David

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> --
>>
>> Mike DePaulo
>>
>> He / Him / His
>>
>> Service Reliability Engineer, Pulp
>>
>> Red Hat 
>>
>> IM: mikedep333
>>
>> GPG: 51745404
>> 
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> 

Re: [Pulp-dev] 3.7.0 Release Timeline & Go/No-Go Chat Meeting

[Brian Bouterse]

The 3.7.0 installer is released to galaxy.ansible.com, so I'll proceed with
the pulpcore, pulp_file, and pulp_ansible announcements to pulp-list now.
Once sent, that can unblock other plugin announcements.


On Thu, Sep 24, 2020 at 7:39 AM Tatiana Tereshchenko 
wrote:

> pulp_rpm==3.7.0 is published but not announced.
>
> On Wed, Sep 23, 2020 at 11:11 PM Brian Bouterse 
> wrote:
>
>> A lot of progress was made today, the final 3.7.0 and several plugin
>> compatibility releases should go out tomorrow.
>>
>> * 3.6.4 pulpcore and its installer are fully published and announced
>> * pulpcore==3.7.0, pulp_file==1.3.0, and pulp_ansible==0.4.0 are
>> published but not announced (waiting on pulpcore announcement first)
>> * pulp_container==2.1.0 is publishing right now (thanks @ipanova and
>> @dkliban)
>> * 3.6.4 installer is not yet published, and this is what is holding up
>> the announcement. I will resume tomorrow, and then proceed to announce
>> * pulp-certguard should happen tomorrow, along with galaxy_ng upgrading
>> to the latest pulpcore, pulp_ansible, and pulp_container. pulp_rpm I also
>> think is likely
>>
>>
>>
>>
>> On Wed, Sep 23, 2020 at 12:29 PM Brian Bouterse 
>> wrote:
>>
>>> Through IRC discussion here's what we learned:
>>>
>>> * it's generally agreed plugins should not switch their pulpcore version
>>> in a z-stream
>>> * pulpcore 3.7.0 is on pypi so plugins can now use pulpcore>=3.7,<3.9
>>> and start releasing. Bump your y-versions even if there are no changes
>>> (it's strange I know but ... bullet point one)
>>> * 3.6.3 is broken, and I am now going to release a 3.6.4 and an
>>> installer version 3.6.4 first, see this release tracker issue
>>> https://pulp.plan.io/issues/7557
>>> * We're waiting on pulp_file and pulp_rpm compat releases for 3.7.0
>>> before the installer is released
>>> * Once the installer and pulpcore bits are available (both 3.7.0 and
>>> 3.6.4) announcements will go to pulp-list
>>>
>>>
>>>
>>>
>>> On Tue, Sep 22, 2020 at 7:15 PM Brian Bouterse 
>>> wrote:
>>>
>>>> 3.7.0 is technically released, but the pulp_file PR is failing due to a
>>>> variety of reasons. I'll continue on it (and the pulp_ansible release PR
>>>> also failing) tomorrow.
>>>>
>>>> I still need to release the installer, and then proceed with the
>>>> announcements.
>>>>
>>>> On Fri, Sep 18, 2020 at 1:13 PM Brian Bouterse 
>>>> wrote:
>>>>
>>>>> We had our final go/no-go check in today, and we determined 3.7.0 is
>>>>> on-track for the Sept 22nd release date.
>>>>>
>>>>> You can see the milestone's work items here:
>>>>> https://tinyurl.com/y6rqlufb
>>>>>
>>>>> * Everything except 7460 is expected to merge, or at least be in POST,
>>>>> by EOB today
>>>>> * 7460 is a very small piece of work that a meeting on Monday will
>>>>> coordinate
>>>>> * Selinux EL7 and EL8 policies (not in work query since it doesn't
>>>>> block the pulpcore release due to not being in pulpcore's bits) is also
>>>>> done!
>>>>>
>>>>> Thanks to everyone who put in so much hard work to get this release
>>>>> ready.
>>>>>
>>>>>
>>>>> On Tue, Sep 15, 2020 at 12:51 PM Brian Bouterse 
>>>>> wrote:
>>>>>
>>>>>> At the go/no-go check in today, we determined the 3.7.0 is on-track
>>>>>> for the Sept 22nd release date.
>>>>>>
>>>>>> The next check-in meeting on #pulp-meeting is set up for Friday Sept
>>>>>> 18th, 12pm - 12:30pm EDT. https://everytimezone.com/?t=5f63f880,3c0
>>>>>>
>>>>>> ## Notes
>>>>>>
>>>>>> * All remaining pulpcore and pulpcore installer items to be included
>>>>>> are now tagged in the 3.7.0 milestone:
>>>>>> https://pulp.plan.io/versions/126
>>>>>> * You can see a query-based of the milestone sorted by status here:
>>>>>> https://tinyurl.com/y6rqlufb
>>>>>> * We believe all items at ASSIGNED will be merged at our current pace
>>>>>> by Friday
>>>>>> * The 4 items at NEW are all now marked HIGH prio
>>>>>> * All code must be merged by the end of business o

Re: [Pulp-dev] 3.7.0 Release Timeline & Go/No-Go Chat Meeting

[Brian Bouterse]

A lot of progress was made today, the final 3.7.0 and several plugin
compatibility releases should go out tomorrow.

* 3.6.4 pulpcore and its installer are fully published and announced
* pulpcore==3.7.0, pulp_file==1.3.0, and pulp_ansible==0.4.0 are published
but not announced (waiting on pulpcore announcement first)
* pulp_container==2.1.0 is publishing right now (thanks @ipanova and
@dkliban)
* 3.6.4 installer is not yet published, and this is what is holding up the
announcement. I will resume tomorrow, and then proceed to announce
* pulp-certguard should happen tomorrow, along with galaxy_ng upgrading to
the latest pulpcore, pulp_ansible, and pulp_container. pulp_rpm I also
think is likely




On Wed, Sep 23, 2020 at 12:29 PM Brian Bouterse  wrote:

> Through IRC discussion here's what we learned:
>
> * it's generally agreed plugins should not switch their pulpcore version
> in a z-stream
> * pulpcore 3.7.0 is on pypi so plugins can now use pulpcore>=3.7,<3.9 and
> start releasing. Bump your y-versions even if there are no changes (it's
> strange I know but ... bullet point one)
> * 3.6.3 is broken, and I am now going to release a 3.6.4 and an installer
> version 3.6.4 first, see this release tracker issue
> https://pulp.plan.io/issues/7557
> * We're waiting on pulp_file and pulp_rpm compat releases for 3.7.0 before
> the installer is released
> * Once the installer and pulpcore bits are available (both 3.7.0 and
> 3.6.4) announcements will go to pulp-list
>
>
>
>
> On Tue, Sep 22, 2020 at 7:15 PM Brian Bouterse 
> wrote:
>
>> 3.7.0 is technically released, but the pulp_file PR is failing due to a
>> variety of reasons. I'll continue on it (and the pulp_ansible release PR
>> also failing) tomorrow.
>>
>> I still need to release the installer, and then proceed with the
>> announcements.
>>
>> On Fri, Sep 18, 2020 at 1:13 PM Brian Bouterse 
>> wrote:
>>
>>> We had our final go/no-go check in today, and we determined 3.7.0 is
>>> on-track for the Sept 22nd release date.
>>>
>>> You can see the milestone's work items here:
>>> https://tinyurl.com/y6rqlufb
>>>
>>> * Everything except 7460 is expected to merge, or at least be in POST,
>>> by EOB today
>>> * 7460 is a very small piece of work that a meeting on Monday will
>>> coordinate
>>> * Selinux EL7 and EL8 policies (not in work query since it doesn't block
>>> the pulpcore release due to not being in pulpcore's bits) is also done!
>>>
>>> Thanks to everyone who put in so much hard work to get this release
>>> ready.
>>>
>>>
>>> On Tue, Sep 15, 2020 at 12:51 PM Brian Bouterse 
>>> wrote:
>>>
>>>> At the go/no-go check in today, we determined the 3.7.0 is on-track for
>>>> the Sept 22nd release date.
>>>>
>>>> The next check-in meeting on #pulp-meeting is set up for Friday Sept
>>>> 18th, 12pm - 12:30pm EDT. https://everytimezone.com/?t=5f63f880,3c0
>>>>
>>>> ## Notes
>>>>
>>>> * All remaining pulpcore and pulpcore installer items to be included
>>>> are now tagged in the 3.7.0 milestone:
>>>> https://pulp.plan.io/versions/126
>>>> * You can see a query-based of the milestone sorted by status here:
>>>> https://tinyurl.com/y6rqlufb
>>>> * We believe all items at ASSIGNED will be merged at our current pace
>>>> by Friday
>>>> * The 4 items at NEW are all now marked HIGH prio
>>>> * All code must be merged by the end of business on the 21st. The 22nd
>>>> is just release day
>>>> * The SELinux deliverables (EL7 and EL8 policies) are required for
>>>> 3.7.0 but are not included on this milestone because they are shipped
>>>> separately from pulpcore and therefore do not need to block its release
>>>> * No significant pulp_file items were discussed
>>>>
>>>> ## The four items at NEW
>>>>
>>>> 7471 - The biggest risk. The installer team will be discussing this
>>>> today and through the end of the week
>>>> 7460 - A small item. A meeting on monday was already set up to finalize
>>>> it's merge. This will likely merge on the 21st
>>>> 7413 - To be done on docs day this Thursday the 17th
>>>> 7411 - Technically not blocking the release and will be removed if not
>>>> merged by end of business on the 21st. It is still high prio though, for
>>>> anyone who is on the CI mini-team and can work on it
>>>>
>>>> Thanks to everyone who participated in our release check in today!
>>>>
>>>>
>>>> On Fri, Sep 4, 2020 at 3:20 PM Brian Bouterse 
>>>> wrote:
>>>>
>>>>> Per the new process, here's the tracker (with checklist) for 3.7.0's
>>>>> release: https://pulp.plan.io/issues/7463  Right now, the GA date is
>>>>> slated for Sept 22nd.
>>>>>
>>>>> The first go/no-go meeting will happen in #pulp-meeting at the time
>>>>> below:
>>>>>
>>>>> Tuesday Sept 15th, 12pm - 12:30pm EDT.
>>>>> https://everytimezone.com/s/68f4535a
>>>>>
>>>>> Thanks,
>>>>> Brian
>>>>>
>>>>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.6.4 is Generally Available

[Brian Bouterse]

pulpcore 3.6.4 is available on PyPI[0]. This release fixes bindings issues
caused by a dependency that released backwards incompatible changes[1].

# Installation and Upgrade

Users should use the 3.6.4 release of pulp_installer[2] to install or
upgrade their installations. This version of the installer will check
compatibility of all installed plugins with pulpcore 3.6. The installer
will abort if any plugin is incompatible.

The pulp_installer collection can be installed from Ansible Galaxy with the
following command:

ansible-galaxy collection  install --force pulp.pulp_installer

The --force flag will upgrade the collection if you had a previous version
installed.

# Plugin API

There are no plugin API changes.

# Client libraries

Both the Python[3] and Ruby[4] clients are available also.

[0] https://pypi.org/project/pulpcore/3.6.4/
[1] https://docs.pulpproject.org/pulpcore/en/3.6.4/changes.html#id1
[2] https://galaxy.ansible.com/pulp/pulp_installer
[3] https://pypi.org/project/pulpcore-client/3.6.4/
[4] https://rubygems.org/gems/pulpcore_client/versions/3.6.4/
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 3.7.0 Release Timeline & Go/No-Go Chat Meeting

[Brian Bouterse]

Through IRC discussion here's what we learned:

* it's generally agreed plugins should not switch their pulpcore version in
a z-stream
* pulpcore 3.7.0 is on pypi so plugins can now use pulpcore>=3.7,<3.9 and
start releasing. Bump your y-versions even if there are no changes (it's
strange I know but ... bullet point one)
* 3.6.3 is broken, and I am now going to release a 3.6.4 and an installer
version 3.6.4 first, see this release tracker issue
https://pulp.plan.io/issues/7557
* We're waiting on pulp_file and pulp_rpm compat releases for 3.7.0 before
the installer is released
* Once the installer and pulpcore bits are available (both 3.7.0 and 3.6.4)
announcements will go to pulp-list




On Tue, Sep 22, 2020 at 7:15 PM Brian Bouterse  wrote:

> 3.7.0 is technically released, but the pulp_file PR is failing due to a
> variety of reasons. I'll continue on it (and the pulp_ansible release PR
> also failing) tomorrow.
>
> I still need to release the installer, and then proceed with the
> announcements.
>
> On Fri, Sep 18, 2020 at 1:13 PM Brian Bouterse 
> wrote:
>
>> We had our final go/no-go check in today, and we determined 3.7.0 is
>> on-track for the Sept 22nd release date.
>>
>> You can see the milestone's work items here:
>> https://tinyurl.com/y6rqlufb
>>
>> * Everything except 7460 is expected to merge, or at least be in POST, by
>> EOB today
>> * 7460 is a very small piece of work that a meeting on Monday will
>> coordinate
>> * Selinux EL7 and EL8 policies (not in work query since it doesn't block
>> the pulpcore release due to not being in pulpcore's bits) is also done!
>>
>> Thanks to everyone who put in so much hard work to get this release ready.
>>
>>
>> On Tue, Sep 15, 2020 at 12:51 PM Brian Bouterse 
>> wrote:
>>
>>> At the go/no-go check in today, we determined the 3.7.0 is on-track for
>>> the Sept 22nd release date.
>>>
>>> The next check-in meeting on #pulp-meeting is set up for Friday Sept
>>> 18th, 12pm - 12:30pm EDT. https://everytimezone.com/?t=5f63f880,3c0
>>>
>>> ## Notes
>>>
>>> * All remaining pulpcore and pulpcore installer items to be included are
>>> now tagged in the 3.7.0 milestone:  https://pulp.plan.io/versions/126
>>> * You can see a query-based of the milestone sorted by status here:
>>> https://tinyurl.com/y6rqlufb
>>> * We believe all items at ASSIGNED will be merged at our current pace by
>>> Friday
>>> * The 4 items at NEW are all now marked HIGH prio
>>> * All code must be merged by the end of business on the 21st. The 22nd
>>> is just release day
>>> * The SELinux deliverables (EL7 and EL8 policies) are required for 3.7.0
>>> but are not included on this milestone because they are shipped separately
>>> from pulpcore and therefore do not need to block its release
>>> * No significant pulp_file items were discussed
>>>
>>> ## The four items at NEW
>>>
>>> 7471 - The biggest risk. The installer team will be discussing this
>>> today and through the end of the week
>>> 7460 - A small item. A meeting on monday was already set up to finalize
>>> it's merge. This will likely merge on the 21st
>>> 7413 - To be done on docs day this Thursday the 17th
>>> 7411 - Technically not blocking the release and will be removed if not
>>> merged by end of business on the 21st. It is still high prio though, for
>>> anyone who is on the CI mini-team and can work on it
>>>
>>> Thanks to everyone who participated in our release check in today!
>>>
>>>
>>> On Fri, Sep 4, 2020 at 3:20 PM Brian Bouterse 
>>> wrote:
>>>
>>>> Per the new process, here's the tracker (with checklist) for 3.7.0's
>>>> release: https://pulp.plan.io/issues/7463  Right now, the GA date is
>>>> slated for Sept 22nd.
>>>>
>>>> The first go/no-go meeting will happen in #pulp-meeting at the time
>>>> below:
>>>>
>>>> Tuesday Sept 15th, 12pm - 12:30pm EDT.
>>>> https://everytimezone.com/s/68f4535a
>>>>
>>>> Thanks,
>>>> Brian
>>>>
>>>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 3.7.0 Release Timeline & Go/No-Go Chat Meeting

[Brian Bouterse]

3.7.0 is technically released, but the pulp_file PR is failing due to a
variety of reasons. I'll continue on it (and the pulp_ansible release PR
also failing) tomorrow.

I still need to release the installer, and then proceed with the
announcements.

On Fri, Sep 18, 2020 at 1:13 PM Brian Bouterse  wrote:

> We had our final go/no-go check in today, and we determined 3.7.0 is
> on-track for the Sept 22nd release date.
>
> You can see the milestone's work items here:  https://tinyurl.com/y6rqlufb
>
> * Everything except 7460 is expected to merge, or at least be in POST, by
> EOB today
> * 7460 is a very small piece of work that a meeting on Monday will
> coordinate
> * Selinux EL7 and EL8 policies (not in work query since it doesn't block
> the pulpcore release due to not being in pulpcore's bits) is also done!
>
> Thanks to everyone who put in so much hard work to get this release ready.
>
>
> On Tue, Sep 15, 2020 at 12:51 PM Brian Bouterse 
> wrote:
>
>> At the go/no-go check in today, we determined the 3.7.0 is on-track for
>> the Sept 22nd release date.
>>
>> The next check-in meeting on #pulp-meeting is set up for Friday Sept
>> 18th, 12pm - 12:30pm EDT. https://everytimezone.com/?t=5f63f880,3c0
>>
>> ## Notes
>>
>> * All remaining pulpcore and pulpcore installer items to be included are
>> now tagged in the 3.7.0 milestone:  https://pulp.plan.io/versions/126
>> * You can see a query-based of the milestone sorted by status here:
>> https://tinyurl.com/y6rqlufb
>> * We believe all items at ASSIGNED will be merged at our current pace by
>> Friday
>> * The 4 items at NEW are all now marked HIGH prio
>> * All code must be merged by the end of business on the 21st. The 22nd is
>> just release day
>> * The SELinux deliverables (EL7 and EL8 policies) are required for 3.7.0
>> but are not included on this milestone because they are shipped separately
>> from pulpcore and therefore do not need to block its release
>> * No significant pulp_file items were discussed
>>
>> ## The four items at NEW
>>
>> 7471 - The biggest risk. The installer team will be discussing this today
>> and through the end of the week
>> 7460 - A small item. A meeting on monday was already set up to finalize
>> it's merge. This will likely merge on the 21st
>> 7413 - To be done on docs day this Thursday the 17th
>> 7411 - Technically not blocking the release and will be removed if not
>> merged by end of business on the 21st. It is still high prio though, for
>> anyone who is on the CI mini-team and can work on it
>>
>> Thanks to everyone who participated in our release check in today!
>>
>>
>> On Fri, Sep 4, 2020 at 3:20 PM Brian Bouterse 
>> wrote:
>>
>>> Per the new process, here's the tracker (with checklist) for 3.7.0's
>>> release: https://pulp.plan.io/issues/7463  Right now, the GA date is
>>> slated for Sept 22nd.
>>>
>>> The first go/no-go meeting will happen in #pulp-meeting at the time
>>> below:
>>>
>>> Tuesday Sept 15th, 12pm - 12:30pm EDT.
>>> https://everytimezone.com/s/68f4535a
>>>
>>> Thanks,
>>> Brian
>>>
>>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 3.7.0 Release Timeline & Go/No-Go Chat Meeting

[Brian Bouterse]

We had our final go/no-go check in today, and we determined 3.7.0 is
on-track for the Sept 22nd release date.

You can see the milestone's work items here:  https://tinyurl.com/y6rqlufb

* Everything except 7460 is expected to merge, or at least be in POST, by
EOB today
* 7460 is a very small piece of work that a meeting on Monday will
coordinate
* Selinux EL7 and EL8 policies (not in work query since it doesn't block
the pulpcore release due to not being in pulpcore's bits) is also done!

Thanks to everyone who put in so much hard work to get this release ready.


On Tue, Sep 15, 2020 at 12:51 PM Brian Bouterse  wrote:

> At the go/no-go check in today, we determined the 3.7.0 is on-track for
> the Sept 22nd release date.
>
> The next check-in meeting on #pulp-meeting is set up for Friday Sept 18th,
> 12pm - 12:30pm EDT. https://everytimezone.com/?t=5f63f880,3c0
>
> ## Notes
>
> * All remaining pulpcore and pulpcore installer items to be included are
> now tagged in the 3.7.0 milestone:  https://pulp.plan.io/versions/126
> * You can see a query-based of the milestone sorted by status here:
> https://tinyurl.com/y6rqlufb
> * We believe all items at ASSIGNED will be merged at our current pace by
> Friday
> * The 4 items at NEW are all now marked HIGH prio
> * All code must be merged by the end of business on the 21st. The 22nd is
> just release day
> * The SELinux deliverables (EL7 and EL8 policies) are required for 3.7.0
> but are not included on this milestone because they are shipped separately
> from pulpcore and therefore do not need to block its release
> * No significant pulp_file items were discussed
>
> ## The four items at NEW
>
> 7471 - The biggest risk. The installer team will be discussing this today
> and through the end of the week
> 7460 - A small item. A meeting on monday was already set up to finalize
> it's merge. This will likely merge on the 21st
> 7413 - To be done on docs day this Thursday the 17th
> 7411 - Technically not blocking the release and will be removed if not
> merged by end of business on the 21st. It is still high prio though, for
> anyone who is on the CI mini-team and can work on it
>
> Thanks to everyone who participated in our release check in today!
>
>
> On Fri, Sep 4, 2020 at 3:20 PM Brian Bouterse  wrote:
>
>> Per the new process, here's the tracker (with checklist) for 3.7.0's
>> release: https://pulp.plan.io/issues/7463  Right now, the GA date is
>> slated for Sept 22nd.
>>
>> The first go/no-go meeting will happen in #pulp-meeting at the time below:
>>
>> Tuesday Sept 15th, 12pm - 12:30pm EDT.
>> https://everytimezone.com/s/68f4535a
>>
>> Thanks,
>> Brian
>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Unblocking SigningService hash check PR

[Brian Bouterse]

At the pulpcore meeting we noticed the signing service PR [0] had stalled
and instead we needed to work towards the next-steps identified in the
video call and discussed on this thread. I closed that PR with a note
asking that we focus on the next steps we discussed in the meeting (also
linked to [0]). I can help review and guide that contribution if someone is
interested.

[0]: https://github.com/pulp/pulpcore/pull/659#issuecomment-693658607



On Tue, Jun 30, 2020 at 4:48 PM Brian Bouterse  wrote:

> Sorry for the long delay in reply. Thank you for the email. I also want to
> unblock that PR. I put some replies in line.
>
> On Wed, Jun 24, 2020 at 7:03 AM Quirin Pamp  wrote:
>
>> Hi,
>>
>>
>> However we decide to continue with the SigningService topic in the medium
>> and longrun, I wanted to have one more go at unblocking the following PR in
>> the short run:
>>
>>
>> https://github.com/pulp/pulpcore/pull/659
>>
>>
>> Currently, this PR issues a warning whenever the hash of a signing
>> service script has changed on disk (compared to when it was first
>> validated).
>>
>>
>> I think we are all in agreement that this is a bad compromise between
>> doing nothing at all (since the script might have changed for legitimate
>> reasons), and issuing a full on Error in cases where things are broken.
>>
>>
>> Yes I believe all parties agree the warning is the compromise no-one
> likes.
>
>>
>> My proposal is the following:
>>
>>
>> Instead of issuing a warning, a changed hash value on disk would trigger
>> an automatic re-validation of the script on disk.
>>
>> If the validation fails, it will throw a hard error (which would
>> certainly be the correct course of action for a script that does not
>> perform what the SigningService promises).
>>
>> If the validation succeeds, the SigningService is updated with the new
>> hash value, and everything continues as it nothing had happened (we just
>> assume the script was changed for legitimate reasons).
>>
> Overall this sounds ok to me, but I want to confirm, is the validation
> this code does?
> https://github.com/pulp/pulpcore/blob/bce4bee8124502ecd183fc664b904f5af5be97db/pulpcore/app/models/content.py#L453-L490
>
> If so, moving the `public_key` and ^ `signing service` attributes from
> AsciiArmoredDetachedSigningservice in pulp_rpm to SigningService in
> pulpcore would be a good first step. I believe this will need to happen
> first and is consistent with the last discussion of our video call. This
> would be a good first step in its own PR I think.
>
>
>> The only thing I can come up with where this approach might be
>> problematic, is if users want to have different versions of the signing
>> service script on different workers (for some reason).
>>
>> However, in such cases it would still be possible to work around the
>> problem by having a single signing service script call a secondary script
>> that differs on different workers.
>>
> If each script has its own public key it will fail. If each script on each
> worker with a different sha256 shares one public key it would be
> revalidated-and-modified each time it's used, which is strange but ok. Also
> I don't think this use case is really valid so I'm ok with this.
>
>>
>> If you are worried that the possibility of such a workaround defeats the
>> whole purpose of hashing the script in the first place, consider the
>> following:
>>
>> This is not intended as a security feature against some kind of malicious
>> attacker scenario, it is intended to provide some more meaningful error
>> reporting, for operational mistakes.
>> In this context I almost consider it a bonus if Sysadmin users who want
>> to do something rather unusual and complicated (different signing service
>> scripts on different workers) are forced to think about this carefully.
>>
> The use case doesn't resonate with me, but that's ok. If others see value
> in it, it doesn't harm existing use cases, and ya'll are willing to
> contribute it, let's do this.
>
> Here is one idea I want to pitch as a possible counterproposal:  What if
> we don't store the sha256 and instead, we perform validation on the signed
> data all the time? The sha256 is nice, but it won't catch the case when the
> script is interacting with a key server, and that key rotates but the
> script doesn't. In that case the signatures will still be returned, pulp
> will be unaware, no error will occur, yet it should. Validating data in all
> cases I think would get us everything the sha256 script proposes and even
> more.
>
> What

Re: [Pulp-dev] 3.7.0 Release Timeline & Go/No-Go Chat Meeting

[Brian Bouterse]

At the go/no-go check in today, we determined the 3.7.0 is on-track for the
Sept 22nd release date.

The next check-in meeting on #pulp-meeting is set up for Friday Sept 18th,
12pm - 12:30pm EDT. https://everytimezone.com/?t=5f63f880,3c0

## Notes

* All remaining pulpcore and pulpcore installer items to be included are
now tagged in the 3.7.0 milestone:  https://pulp.plan.io/versions/126
* You can see a query-based of the milestone sorted by status here:
https://tinyurl.com/y6rqlufb
* We believe all items at ASSIGNED will be merged at our current pace by
Friday
* The 4 items at NEW are all now marked HIGH prio
* All code must be merged by the end of business on the 21st. The 22nd is
just release day
* The SELinux deliverables (EL7 and EL8 policies) are required for 3.7.0
but are not included on this milestone because they are shipped separately
from pulpcore and therefore do not need to block its release
* No significant pulp_file items were discussed

## The four items at NEW

7471 - The biggest risk. The installer team will be discussing this today
and through the end of the week
7460 - A small item. A meeting on monday was already set up to finalize
it's merge. This will likely merge on the 21st
7413 - To be done on docs day this Thursday the 17th
7411 - Technically not blocking the release and will be removed if not
merged by end of business on the 21st. It is still high prio though, for
anyone who is on the CI mini-team and can work on it

Thanks to everyone who participated in our release check in today!


On Fri, Sep 4, 2020 at 3:20 PM Brian Bouterse  wrote:

> Per the new process, here's the tracker (with checklist) for 3.7.0's
> release: https://pulp.plan.io/issues/7463  Right now, the GA date is
> slated for Sept 22nd.
>
> The first go/no-go meeting will happen in #pulp-meeting at the time below:
>
> Tuesday Sept 15th, 12pm - 12:30pm EDT.
> https://everytimezone.com/s/68f4535a
>
> Thanks,
> Brian
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Katello/pulp 3 integration scrum notes

[Brian Bouterse]

Note, next week's meeting is cancelled due to Pulpcon conflict. This was
discussed with Katello on today's call.


September 9, 2020

Pulp

   -

   Pulpcore
   -

  Dalley - Working on tasking bugs (stuck tasks, etc.) and slow orphan
  cleanup
  -

 Have been on the sprint for several sprints :(
 -

 Expected to come out with 3.7 on Sept 22nd
 -

  Pulpcore discussed shipping migrations in z-stream on Sept 4th open
  floor
  -

 General consensus is that pulpcore cannot
 -

 A mailing list discussion will probably happen also, not yet sent
 -

  SELinux EL7 and EL8 policies expected to be included in pulpcore 3.7
  -

  FIPS support to be included in pulpcore 3.7
  -

 https://pulp.plan.io/issues/5216
 -

https://github.com/pulp/pulpcore/pull/894
-

   Container
   -

  2.0.1 released yesterday. Includes a fix for the 403 responses when
  deployed behind an HTTPS proxy.
  -

   RPM
   -

  3.6.2 was GA on Sept 4th
  -

 https://pulp-rpm.readthedocs.io/en/3.6/changes.html#id1
 -

   Pulpcon Sept 14th - 18th
   -

  https://hackmd.io/@pulp/pulpcon2020_schedule


Katello

   -

   Pulpcore 3.6  with all fixes now running through pipeline, hopefully no
   more issues
   -

   Added support for sles auth token
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulpcore team meeting

[Brian Bouterse]

## September 8, 2020

### Previous action items
* [dkliban] file a story to allow auto-publish and distribute
* Done. https://pulp.plan.io/issues/7469
* [bmbouter] to respond and close https://github.com/pulp/pulpcore/pull/659
* not done, moved forward to next week

### Topics

* 3.7.0 is scheduled for Sept 22
* https://pulp.plan.io/issues/7463
* Go/No-Go: Tuesday Sept 15th, 12pm - 12:30pm EDT.
https://everytimezone.com/s/68f4535a
* Pulp UI project accepted by UMAss Lowell. Professors have started
recruiting students to take on the project.

### Action Items
* [bmbouter] to respond and close https://github.com/pulp/pulpcore/pull/659

Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] New black version

[Brian Bouterse]

FYI, this was merged at the pulpcore weekly meeting today.

On Thu, Aug 27, 2020 at 1:22 PM David Davis  wrote:

> A new version of black was released which has fixed some of our docstring
> and made a change to how trailing commas are handled. I've fixed a few of
> the plugins (pulp_file, pulp_ansible, etc) but the biggest change was in
> pulpcore[0]. I don't think it'll create too many conflicts but I wanted to
> give people a heads up before I merge it. If there are no objections, I
> plan to merge on September 2nd.
>
> [0] https://github.com/pulp/pulpcore/pull/870
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Recommended: Remove test `test_file_decriptors`

[Brian Bouterse]

At the open floor today, there was support for removing this one particular
test from all plugins. I've removed it from these places, but other plugins
should consider removing it also.

pulpcore:  was not used
plugin_template:  https://github.com/pulp/plugin_template/pull/260
pulp_file:  https://github.com/pulp/pulp_file/pull/424
pulp_ansible:  https://github.com/pulp/pulp_ansible/pull/356

## Motivation

The motivation is that the test does not make assertions on Pulp itself,
and practically speaking is failing intermittently.

Happy Friday,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] 3.7.0 Release Timeline & Go/No-Go Chat Meeting

[Brian Bouterse]

Per the new process, here's the tracker (with checklist) for 3.7.0's
release: https://pulp.plan.io/issues/7463  Right now, the GA date is slated
for Sept 22nd.

The first go/no-go meeting will happen in #pulp-meeting at the time below:

Tuesday Sept 15th, 12pm - 12:30pm EDT. https://everytimezone.com/s/68f4535a

Thanks,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] github checklist as a part of the release process

[Brian Bouterse]

This is great, thank you.

On Mon, Aug 31, 2020 at 7:31 AM David Davis  wrote:

> I've added a release template checklist to the pulpcore project. I am
> hoping we can test it out with the next release of pulpcore.
>
> David
>
>
> On Tue, Aug 25, 2020 at 10:11 AM David Davis 
> wrote:
>
>> On Fri, Aug 21, 2020 at 4:54 AM Tatiana Tereshchenko 
>> wrote:
>>
>>> Good to know that Redmine has some sort of template as well.
>>>
>>> I tested some and it seems like Redmine checklists are good when you
>>> want to specify in a short form each step and all in one list.
>>> The limitations which I noticed (let me know if I used something
>>> incorrectly):
>>>  - no multiline items (sometimes we put explanations for a step or
>>> examples)
>>>  - no structure, no nested items (IMO, it would be useful to have some
>>> pre-release, release, and post release items. Structure makes it more
>>> readable)
>>>  - text formatting is limited, e.g. a code snippet will change the font
>>> a bit but not add any background colour for readability.
>>>
>>> Having said that, those are not blockers but noticeable inconveniences.
>>>
>>
>> Agreed.
>>
>>
>>>
>>> Here are my experiments.
>>> The template
>>> https://pulp.plan.io/projects/migration/checklist_templates/2/edit
>>> The checklist from the template https://pulp.plan.io/issues/7364
>>>
>>> I think one of the goals is to substitute our release guide and not
>>> create one more item to keep up to date.
>>> So just for reference, here is the release guide
>>> https://pulp.plan.io/projects/pulp/wiki/Pulp3_Release_Guide which I
>>> expect to be expanded a bit with pre-release activities at least.
>>>
>>
>> +1. I'm imagining the release process will be to create a release issue
>> with the checklist and then just check items off. The PR(s) could be
>> attached to the release issue.
>>
>>
>>>
>>> A separate question.
>>> As a user, how can I easily see ongoing releases across pulp projects or
>>> recently published releases? Something that I can bookmark and track. Is
>>> the idea to have a redmine query for that?
>>>
>>
>> I think a query makes sense. Maybe a Release tracker type or tag?
>>
>>
>>>
>>> Thanks for looking into that!
>>> Tanya
>>>
>>> On Thu, Aug 20, 2020 at 8:26 PM David Davis 
>>> wrote:
>>>
 Nice find. I tested it and it works pretty well. I'm leaning towards us
 using this in redmine but I have no objection with github issues.

 David


 On Thu, Aug 20, 2020 at 10:44 AM Matthias Dellweg 
 wrote:

> You can have checklist_templates in redmine:
>
> https://pulp.plan.io/projects/pulp_container/settings/checklist_template
>
> However it's like 3 clicks to add that checklist to a task you are
> about to create. Maybe it is even possible to create a new tracker (called
> release) where every issue automatically gets that release checklist.
>
> On Wed, Aug 19, 2020 at 11:14 PM David Davis 
> wrote:
>
>> Another idea: have the release PR contain the checklist. Then it
>> would all be in one place.
>>
>> David
>>
>>
>> On Wed, Aug 19, 2020 at 4:40 PM Fabricio Aguiar <
>> fabricio.agu...@redhat.com> wrote:
>>
>>>
>>>
>>>
>>> On Wed, Aug 19, 2020 at 12:02 PM David Davis 
>>> wrote:
>>>
 A separate github repo might make sense. Right now our release
 scripts live inside our .travis folders in repo. I don't know that 
 they are
 project specific so perhaps we could move them to this new repo?

>>> The script just get the plugin name, I believe it is easy to move to
>>> another repo and do something similar we do oat pulp-ci
>>>
 David


 On Wed, Aug 19, 2020 at 5:57 AM Tatiana Tereshchenko <
 ttere...@redhat.com> wrote:

> Would a separate github repo with issues enabled make sense?
> One place for all templates if we need many (I can think of at
> least Y and Z releases).
> One place for all release tracking, one can see what is released,
> and what is not, without going from repo to repo (or from one redmine
> project to another).
> This repo can also have release compatibility information/table,
> or any other release related data.
>
> I'm also not aware of any easy way of creating a
> template/checklist in redmine.
>
> Tanya
>
> On Tue, Aug 18, 2020 at 4:22 PM David Davis 
> wrote:
>
>> Big +1. I really like this idea and believe it could help us
>> organize the work for releases.
>>
>> How we can apply this to Pulp though? We don't use github issues
>> and there's no way to template checklists for redmine issues AFAICT.
>>
>> David
>>
>>
>> On Tue, Aug 18, 2020 at 9:55 AM Fabricio Aguiar <
>> 

Re: [Pulp-dev] problem: version locking plugins to pulpcore

[Brian Bouterse]

After a bit of discussion today at the open floor, the proposal is to
introduce this with pulpcore 3.7. Then all plugins would enable this by
doing two things:

* Declaring their pulpcore dependency as pulpcore>=3.7,<3.9 with the 3.7
compatible plugin release
* Enabling the additional CI test provided by
https://pulp.plan.io/issues/7411 as soon as its available

This is organized into this epic https://pulp.plan.io/issues/7416 which has
the following 3 stories:
* https://pulp.plan.io/issues/7411
* https://pulp.plan.io/issues/7413
* https://pulp.plan.io/issues/7415

Any feedback or concerns are welcome. I'd like to bring these onto the
sprint at Tuesday's open floor. I've added them to the agenda.


On Thu, Aug 20, 2020 at 1:41 PM Ina Panova  wrote:

> +1 to the idea. It will definitely improve user experience.
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Aug 19, 2020 at 12:21 PM Tatiana Tereshchenko 
> wrote:
>
>> +1 to have one Y release with depreciation warnings before actually
>> removing the code or introduce any backward incompatible change.
>>
>> Tanya
>>
>> On Wed, Aug 19, 2020 at 10:09 AM Matthias Dellweg 
>> wrote:
>>
>>> This sounds pretty much the same as i had in mind. Thank you for writing
>>> it up!
>>> One concern inline:
>>>
>>> On Tue, Aug 18, 2020 at 10:31 PM Brian Bouterse 
>>> wrote:
>>> >
>>> > Here's a problem statement(s) and some brainstorming ideas about what
>>> could be done to resolve them. This was discussed today at open floor some
>>> also.
>>> >
>>> > # Problem
>>> > Pulp plugins version lock to the GA version of pulpcore and don't
>>> allow compatibility with other pulpcore releases. This causes at least two
>>> painful practical problems:
>>> >
>>> > 1) users have to wait for all plugins to release before they can
>>> upgrade anything.
>>> > 2) developers have to release everything immediately after pulpcore
>>> releases in an effort to mitigate problem (1).
>>> >
>>> > # Idea
>>> >
>>> > What if pulpcore.plugin had a two-cycle deprecate-then-remove policy?
>>> So instead of making a pulpcore.plugin backwards incompatible change with,
>>> e.g. pulpcore==3.7, we would instead add deprecation warnings to 3.7, and
>>> then remove the code (and deprecation warnings) in 3.8.
>>> >
>>> > Users running a plugin that was using a deprecated codepath or
>>> argument would see these warnings.
>>> >
>>> > Plugins would declare themselves forward compatible with the current
>>> GA+1. So assume pulpcore==3.6 just released, then any plugin releasing
>>> afterwards could declare pulpcore>=3.6,<3.8 instead of the >=3.6,<3.7 they
>>> do today. This assumes plugins are keeping up to date with changes.
>>> >
>>> > Testing wise, the plugin_template would add an additional matrix test
>>> that tests the current GA of a plugin against pulpcore master. This
>>> captures the "one ahead" situation where a user is using a newer version of
>>> pulpcore and the "one back" version of that plugin.
>>> As the plugins last version isn't the one changing here, but pulpcore
>>> is, i think pulpcore needed to be tested (maybe only daily; we can
>>> measure by the amount of necessary reverts) with a wisely chosen
>>> subset of plugins. Starting with file and certguard would be easy.
>>> >
>>> > # Feedback
>>> >
>>> > What do you think?
>>> >
>>> > ___
>>> > Pulp-dev mailing list
>>> > Pulp-dev@redhat.com
>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] When to declare Pulp safe for multi-user?

[Brian Bouterse]

I am in favor of the idea to declare plugins one-by-one as they can be.
Also +1 to adding the column on the plugin list on pulpproject.org here
<https://pulpproject.org/content-plugins/#pulp-3-content-plugins-information>.
I like these ideas better than what I suggested because it doesn't push
plugins to rush-to-implement rbac yet allows pulpcore and pulp_file to go
ahead and do so with 3.7. Please share if there are additional thoughts and
concerns, but the working plan I'm perceiving right now for 3.7 is to:

* enable rbac for pulpcore endpoints - I started an epic for all of these
https://pulp.plan.io/issues/7338 This is not complete, but give us an idea
of where we're going. I will add to this epic and discuss before work
begins.
* enable django-admin more widely for managing object level permissions. It
will also be able to view all objects, but not modify or add model data
(which would bypass the DRF serializers). This will come with this PR
https://github.com/pulp/pulpcore/pull/838 and goes with this issue
https://pulp.plan.io/issues/7336
* enable rbac for pulp_file endpoints - I stubbed out this epic here, I'll
be adding to it https://pulp.plan.io/issues/7339
* declare pulpcore itself multi-user safe, along with a new column on the
plugins list page - https://pulp.plan.io/issues/7309

More feedback, ideas, and concerns are welcome.

On Wed, Aug 12, 2020 at 7:32 AM Tatiana Tereshchenko 
wrote:

>
>
> On Wed, Aug 12, 2020 at 12:34 PM Ina Panova  wrote:
>
>>
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Tue, Aug 11, 2020 at 11:10 PM Brian Bouterse 
>> wrote:
>>
>>> Pulpcore 3.6 adds RBAC machinery for plugins to enable RBAC with [0],
>>> but that hasn't happened broadly in plugins or pulpcore yet, so Pulp 3.6 is
>>> still not multi-user safe. I want us to discuss options and strategy for
>>> when can we declare Pulp multi-user safe.
>>>
>>> My take is that, I think we should avoid a situation where no part of
>>> Pulp can be declared multi-user safe until every call in Pulp everywhere is
>>> multi-user safe. I think doing it plugin-by-plugin is a reasonable
>>> middle-ground. Alternatives to consider here would be great.
>>>
>>
>> It might get tricky if we follow that path. I am not sure we can speak
>> for all the plugins, especially the community plugins plans and timelines.
>> What about finding a middle ground and targeting " Pulpcore is multi-user
>> safe 3.7+ with the following list of plugins", where the list will of
>> plugins will be increasing with time.
>> I would not want to hit the bottleneck and wait with "Pulp is multi-user
>> safe unless all plugins have rbac support".
>>
>
> +1, maybe it's worth adding a new column to our plugin table, "multi-user
> safe" - yes/no.
>
>
>>
>>
>>> Assuming a plugin-by-plugin approach, I'd like to propose a few things
>>> for discussion:
>>>
>>> * pulpcore and pulp_file enable RBAC on all of their endpoints for 3.7
>>> * pulpcore declare itself safe for multi-user use for 3.7 [1]
>>> * all plugins discuss-and-communicate which release is their target to
>>> add RBAC support
>>>
>>> +1
>>
> +1
>
>
>>
>> What do you think?
>>>
>>> [0]:
>>> https://github.com/pulp/pulpcore/tree/master/docs/plugins/plugin-writer/concepts/rbac
>>> [1]: https://pulp.plan.io/issues/7309
>>>
>>> -Brian
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] pulpcore 3.7 tentative release date -- Sept 22nd

[Brian Bouterse]

This was discussed at open floor today.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Proposal: Basic Functional Test Requirement with each Feature or Bugfix

[Brian Bouterse]

I have two goals with this:

1. Improve the stability of pulpcore and it's plugins

2. Enable all downstreams and packagers with tests and information on how
to run those tests, so they can run the automated tests even as the
downstream is curated with a series of backports. This closely follows the
model of a downstream maintainer who backports a feature/fix from upstream
into a RPM and runs the upstream tests to make sure their patch didn't
break functionality.

# Proposal
Upstream could require a basic functional test for each feature or a test
for each bug fix. This test would be in the same commit as the feature or
fix causing it to "follow the code" so when it's cherry picked downstream
or into a package, the test goes with it. I believe we don't change the
test during cherry pick time because although the code implementation may
change what the feature or fix provides will not.

Additionally, upstream should document how to run these tests easily and
have the goal of making that easy.

This policy change would ultimately be decided by each plugin and pulpcore
separately. I don't want to make it a requirement to be part of the
github.com/pulp/ organization in any way, so if a plugin can't, or won't,
make this change that's ok, they can still host at github.com/pulp/.

As a pulpcore and pulp_file maintainer, I'm proposing this policy for both
of those repos specifically.

In terms of enforcement, I'm pitching manual enforcement by review as a
start. We can automate it later, but I hope that's a separate discussion to
focus on the policy change and keep it simple.

# Brian's take
This is a non-trivial contribution requirement so we need to really think
about it. My personal take is that at this point in the 3.y release line
it's time to trade feature/fix velocity for stability. Also backporting is
about to become a regular activity, so I think we have a practical
motivation to adopt this, even though it will slow down the future features
and bug fix velocity.

What's your perspective?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Squeezer compatibility release

[Brian Bouterse]

This is great, thank you.

Do you want to advertise this to pulp-list also?

On Mon, Aug 17, 2020 at 10:52 AM Matthias Dellweg 
wrote:

> Hey folks,
> Following the release of pulpcore 3.6 that had breaking changes around
> the api documentation (used by those ansible modules) we have just
> released version 0.0.3 of squeezer [0].
> This new release can talk to both pulp 3.5 (using swagger/openapiv2)
> and 3.6 (using openapiv3).
> We will try to keep that compatibility around until pulpcore 3.7 arrives.
>   Matthias
>
> [0] https://galaxy.ansible.com/pulp/squeezer
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] When to declare Pulp safe for multi-user?

[Brian Bouterse]

Pulpcore 3.6 adds RBAC machinery for plugins to enable RBAC with [0], but
that hasn't happened broadly in plugins or pulpcore yet, so Pulp 3.6 is
still not multi-user safe. I want us to discuss options and strategy for
when can we declare Pulp multi-user safe.

My take is that, I think we should avoid a situation where no part of Pulp
can be declared multi-user safe until every call in Pulp everywhere is
multi-user safe. I think doing it plugin-by-plugin is a reasonable
middle-ground. Alternatives to consider here would be great.

Assuming a plugin-by-plugin approach, I'd like to propose a few things for
discussion:

* pulpcore and pulp_file enable RBAC on all of their endpoints for 3.7
* pulpcore declare itself safe for multi-user use for 3.7 [1]
* all plugins discuss-and-communicate which release is their target to add
RBAC support

What do you think?

[0]:
https://github.com/pulp/pulpcore/tree/master/docs/plugins/plugin-writer/concepts/rbac
[1]: https://pulp.plan.io/issues/7309

-Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Documentation issues

[Brian Bouterse]

+1 I'd like to see this happen and participate

On Fri, Aug 7, 2020 at 2:00 PM David Davis  wrote:

> I haven't heard any objections. What if we pick August 13 to focus on
> docs? There are 9 docs issues on the sprint so if we all pick up one issue,
> then we get the sprint down to 26 NEW issues.
>
> David
>
>
> On Mon, Aug 3, 2020 at 10:00 AM Grant Gainey  wrote:
>
>> GMTA, was thinking about exactly this over the weekend. Even we
>> don't/can't declare a "Doc Day" (which I think is a fine idea, but I know
>> how schedules are right now...), being intentional about "I will assign one
>> doc-issue to myself and get a PR submitted by COB Friday" would knock a
>> bunch of issues out that have been waiting around for quite some time.
>>
>> Thanks for writing this up, Ina!
>>
>> G
>>
>> On Mon, Aug 3, 2020 at 7:30 AM Ina Panova  wrote:
>>
>>> Folks,
>>> I noticed that currently on the sprint we have roughly ~10 issues with
>>> Documentation tag. Some of them have been carried over multiple sprints and
>>> probably realistically still will be carried over.
>>>
>>> What do you think about dedicating a day and knock them down?
>>> It will roughly be an issue per person. I am not suggesting to dedicate
>>> whole 8 hours but let's say setting a goal for every team member for that
>>> day:"Today I am going to prioritize docs issues and knock down at least one
>>> documentation issue and submit a PR"
>>>
>>> Please provide feedback, and if the team feels like this is a good idea
>>> I'll go ahead and pick a day in our calendar and mark it as "Docs day".
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> --
>> Grant Gainey
>> Principal Software Engineer, Red Hat System Management Engineering
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

# Update:

django-lifecycle merged + released, and all tests are passing on Travis
now  https://github.com/pulp/pulpcore/pull/815/files


# Regarding assigning permissions to objects created prior to RBAC...

I now believe we do not need to apply permissions for the 'admin' user, and
that is the only user we could expect at migration time because it's the
only user we deal with prior to 3.6. I've learned that by default,
django-guardian considers any user with `is_superuser=True` to have the
permission being checked. As such the 'admin' user already sidesteps RBAC
anyway. So my next step in this area is to add this to the docs only.
Plugin writers will *not* need to add permissions for prior existing
objects.




On Wed, Aug 5, 2020 at 5:59 PM Brian Bouterse  wrote:

> Alright we're close to the end of pulpcore's RBAC additions for 3.6, but
> there are still some challanges! Here's what's been accomplished:
>
> * The tests are now passing locally
>   * We are waiting on django-lifecycle to merge my PR and release it
> https://github.com/rsinger86/django-lifecycle/pull/58 We actually can't
> release on my fork (PyPI install_requires won't work with git checkouts
> I've learned) so if he can't release django-lifecycle by Friday then I'm
> going to push an alternate package to PyPI in the meantime under a
> different name. This is what #python recommended to me. I have emailed the
> author asking for review and release so I'm hoping it does not come to that
> * lots more docs have been written
> * I can't show the tests passing on Travis until this django-lifecycle
> issue is resolved, so I'm going to keep the WIP on it until then.
>
> User docs are still needed, I'm working on those tomorrow
>
> Please do review, I want to merge on Friday to be included in 3.6.
>
>
> On Fri, Jul 31, 2020 at 4:58 PM Brian Bouterse 
> wrote:
>
>> A lot of plugin writer docs about adding RBAC support have been added to
>> the newly opened WIP PR: https://github.com/pulp/pulpcore/pull/815/files
>> Please come review the WIP PR. Code-wise it's 90% there, with just a few
>> polish things I still need to finish. From a high level I still need to:
>>
>> * finish the plugin writer docs
>> * add users docs (smaller and easier than plugin writer docs)
>> * ensure all the changelogs and issues are good
>> * get Tasks showing with their object-level permissions in the
>> Django-admin
>> * fix the tests so they are passing on Travis
>>
>> The plan is to have this go into 3.6 to be released Aug 11th.
>>
>> The other big area of work (also for 3.6) is that galaxy_ng is needing
>> Groups (and because of that Users) APIs added for read-only. Those stores
>> got a lot of grooming today and are linked to below. @fabricio is taking
>> the lead on implementing that. Thanks @fabricio!
>>
>> https://pulp.plan.io/issues/7231
>> https://pulp.plan.io/issues/7232
>>
>> Feedback and other concerns are welcome.
>>
>>
>> On Fri, Jul 24, 2020 at 4:58 PM Brian Bouterse 
>> wrote:
>>
>>> I've gotten various feedback and I want to relay some of the changes
>>> based on that.
>>>
>>> 1) Pulp should ship a "user isolation" policy by default. If users want
>>> other things they can configure it further. This is a change from my
>>> original proposal of Pulp shipping an "RBAC is off" policy. This will work
>>> for Katello because all Katello calls show up as one user anyway.
>>>
>>> 2) We have challenges around assigning permissions. This is the proposed
>>> solution that was discussed at open floor today:
>>> https://pulp.plan.io/issues/7210  I've added this to my pulpcore PR
>>> (link below)
>>>
>>> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>>>
>>>
>>>
>>> On Fri, Jul 10, 2020 at 2:20 PM David Davis 
>>> wrote:
>>>
>>>> On Wed, Jul 8, 2020 at 4:54 PM Brian Bouterse 
>>>> wrote:
>>>>
>>>>> My next goal is to have object-level permissions assigned through
>>>>> signals so that anywhere you save the model the permissions are correctly
>>>>> created. To do this I need to get a few things working:
>>>>>
>>>>> 1) Move the permissions creation to the signals [done]
>>>>> 2) Have the user be well-known, this is accomplished by pulpcore
>>>>> picking up this dependency
>>>>> https://github.com/PaesslerAG/django-currentuser [done]
>>>>> 3) Have the user information persisted in tasks so signals can work
>>>>> there also 

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Alright we're close to the end of pulpcore's RBAC additions for 3.6, but
there are still some challanges! Here's what's been accomplished:

* The tests are now passing locally
  * We are waiting on django-lifecycle to merge my PR and release it
https://github.com/rsinger86/django-lifecycle/pull/58 We actually can't
release on my fork (PyPI install_requires won't work with git checkouts
I've learned) so if he can't release django-lifecycle by Friday then I'm
going to push an alternate package to PyPI in the meantime under a
different name. This is what #python recommended to me. I have emailed the
author asking for review and release so I'm hoping it does not come to that
* lots more docs have been written
* I can't show the tests passing on Travis until this django-lifecycle
issue is resolved, so I'm going to keep the WIP on it until then.

User docs are still needed, I'm working on those tomorrow

Please do review, I want to merge on Friday to be included in 3.6.


On Fri, Jul 31, 2020 at 4:58 PM Brian Bouterse  wrote:

> A lot of plugin writer docs about adding RBAC support have been added to
> the newly opened WIP PR: https://github.com/pulp/pulpcore/pull/815/files
> Please come review the WIP PR. Code-wise it's 90% there, with just a few
> polish things I still need to finish. From a high level I still need to:
>
> * finish the plugin writer docs
> * add users docs (smaller and easier than plugin writer docs)
> * ensure all the changelogs and issues are good
> * get Tasks showing with their object-level permissions in the Django-admin
> * fix the tests so they are passing on Travis
>
> The plan is to have this go into 3.6 to be released Aug 11th.
>
> The other big area of work (also for 3.6) is that galaxy_ng is needing
> Groups (and because of that Users) APIs added for read-only. Those stores
> got a lot of grooming today and are linked to below. @fabricio is taking
> the lead on implementing that. Thanks @fabricio!
>
> https://pulp.plan.io/issues/7231
> https://pulp.plan.io/issues/7232
>
> Feedback and other concerns are welcome.
>
>
> On Fri, Jul 24, 2020 at 4:58 PM Brian Bouterse 
> wrote:
>
>> I've gotten various feedback and I want to relay some of the changes
>> based on that.
>>
>> 1) Pulp should ship a "user isolation" policy by default. If users want
>> other things they can configure it further. This is a change from my
>> original proposal of Pulp shipping an "RBAC is off" policy. This will work
>> for Katello because all Katello calls show up as one user anyway.
>>
>> 2) We have challenges around assigning permissions. This is the proposed
>> solution that was discussed at open floor today:
>> https://pulp.plan.io/issues/7210  I've added this to my pulpcore PR
>> (link below)
>>
>> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>>
>>
>>
>> On Fri, Jul 10, 2020 at 2:20 PM David Davis 
>> wrote:
>>
>>> On Wed, Jul 8, 2020 at 4:54 PM Brian Bouterse 
>>> wrote:
>>>
>>>> My next goal is to have object-level permissions assigned through
>>>> signals so that anywhere you save the model the permissions are correctly
>>>> created. To do this I need to get a few things working:
>>>>
>>>> 1) Move the permissions creation to the signals [done]
>>>> 2) Have the user be well-known, this is accomplished by pulpcore
>>>> picking up this dependency
>>>> https://github.com/PaesslerAG/django-currentuser [done]
>>>> 3) Have the user information persisted in tasks so signals can work
>>>> there also ... there are two obvious options
>>>>
>>>> option a) Have tasks themselves have RBAC which we know we need anyway,
>>>> then query the RBAC permissions inside a task to determine which user has
>>>> the object level permissions for this task. This is what I'm currently
>>>> prototyping. It at least will provide a mechanism we can use for now until
>>>> we find something better.
>>>>
>>>
>>> Are task permissions mutable? If so, then what happens if someone kicks
>>> off a task and then this permission gets removed?
>>>
>>>
>>>>
>>>> option b) Add a user field to the Task itself and have the RBAC assign
>>>> permissions to them. The concern with this is that while this is nice, it
>>>> would be nicer if we had this type of visibility on everything in Pulp from
>>>> an auditing project dedicated to keeping this type of info. Option (b) is
>>>> still viable, just maybe not the best way. Comments/feedback is welcome.
>>>>
&g

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

A lot of plugin writer docs about adding RBAC support have been added to
the newly opened WIP PR: https://github.com/pulp/pulpcore/pull/815/files
Please come review the WIP PR. Code-wise it's 90% there, with just a few
polish things I still need to finish. From a high level I still need to:

* finish the plugin writer docs
* add users docs (smaller and easier than plugin writer docs)
* ensure all the changelogs and issues are good
* get Tasks showing with their object-level permissions in the Django-admin
* fix the tests so they are passing on Travis

The plan is to have this go into 3.6 to be released Aug 11th.

The other big area of work (also for 3.6) is that galaxy_ng is needing
Groups (and because of that Users) APIs added for read-only. Those stores
got a lot of grooming today and are linked to below. @fabricio is taking
the lead on implementing that. Thanks @fabricio!

https://pulp.plan.io/issues/7231
https://pulp.plan.io/issues/7232

Feedback and other concerns are welcome.


On Fri, Jul 24, 2020 at 4:58 PM Brian Bouterse  wrote:

> I've gotten various feedback and I want to relay some of the changes based
> on that.
>
> 1) Pulp should ship a "user isolation" policy by default. If users want
> other things they can configure it further. This is a change from my
> original proposal of Pulp shipping an "RBAC is off" policy. This will work
> for Katello because all Katello calls show up as one user anyway.
>
> 2) We have challenges around assigning permissions. This is the proposed
> solution that was discussed at open floor today:
> https://pulp.plan.io/issues/7210  I've added this to my pulpcore PR (link
> below)
>
> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>
>
>
> On Fri, Jul 10, 2020 at 2:20 PM David Davis  wrote:
>
>> On Wed, Jul 8, 2020 at 4:54 PM Brian Bouterse 
>> wrote:
>>
>>> My next goal is to have object-level permissions assigned through
>>> signals so that anywhere you save the model the permissions are correctly
>>> created. To do this I need to get a few things working:
>>>
>>> 1) Move the permissions creation to the signals [done]
>>> 2) Have the user be well-known, this is accomplished by pulpcore picking
>>> up this dependency https://github.com/PaesslerAG/django-currentuser
>>> [done]
>>> 3) Have the user information persisted in tasks so signals can work
>>> there also ... there are two obvious options
>>>
>>> option a) Have tasks themselves have RBAC which we know we need anyway,
>>> then query the RBAC permissions inside a task to determine which user has
>>> the object level permissions for this task. This is what I'm currently
>>> prototyping. It at least will provide a mechanism we can use for now until
>>> we find something better.
>>>
>>
>> Are task permissions mutable? If so, then what happens if someone kicks
>> off a task and then this permission gets removed?
>>
>>
>>>
>>> option b) Add a user field to the Task itself and have the RBAC assign
>>> permissions to them. The concern with this is that while this is nice, it
>>> would be nicer if we had this type of visibility on everything in Pulp from
>>> an auditing project dedicated to keeping this type of info. Option (b) is
>>> still viable, just maybe not the best way. Comments/feedback is welcome.
>>>
>>> Also I've collaborated with @alikins from galaxy_ng project and together
>>> we wrote this requirements doc for their plugin:
>>> https://hackmd.io/JisLkfyeT2myAD2khEwU0A
>>>
>>> On Wed, Jul 1, 2020 at 6:43 PM Brian Bouterse 
>>> wrote:
>>>
>>>> The demo advertisement for tomorrow is here:
>>>> https://www.redhat.com/archives/pulp-dev/2020-June/msg00076.html
>>>>
>>>> On Wed, Jul 1, 2020 at 6:41 PM Brian Bouterse 
>>>> wrote:
>>>>
>>>>> Another productive RBAC day! See the latest code at the links below.
>>>>> Here's what's new:
>>>>>
>>>>> * policy is now shorter thanks to machinery checking both model-level
>>>>> and object-level permissions with one call. The other two are also 
>>>>> available
>>>>> * sync is now restricted on both 'modify_repo_content' permissions AND
>>>>> read permission on the remote being used to sync
>>>>> * modify is now restricted on 'modify_repo_content' permission
>>>>> * moved the permission checking machinery to be "global checks"
>>>>> * added data migration that sets is_staff=True, so the django-admin
>>>>

[Pulp-dev] Allowing Older Z-stream Backports and Releases?

[Brian Bouterse]

Currently my understanding of the z-stream policy is that Pulp only has one
active z-stream. This means (for example) that when 3.6.0 comes out, there
will never be another z-stream release for 3.5.z, 3.4.z, 3.3.z, etc.

I believe we have an opportunity to create more value by adjusting this to
allow for older z-stream releases to be requested as-needed. This would
allow for example Katello's use of pulpcore 3.4.1 to request a 3.4.2 to be
created even if 3.6.0 is released. Similarly for any plugin. These could
only contain bugfixes (never features) due to semantic versioning
restriction.

This is based on feedback I've gotten from various users including Katello,
and this would allow them to use a version of pulpcore or a plugin and
receive stability fixes without being forced to upgrade. Then they can
upgrade when is right for them (not for us).

If others agree with this change I think we need to figure out the
following:

* How can a user request a backport? [bmbouter suggests using a new tracker
type in Redmine named 'Backport']
* How can mini-teams perform backporting? [bmbouter suggests manual
cherry-picking and releasing as-needed, only with requested items]

What do you think?

Thanks!
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp3 Concurrency testing

[Brian Bouterse]

On Tue, Jul 28, 2020 at 12:17 PM David Davis  wrote:

> On Tue, Jul 28, 2020 at 12:03 PM Justin Sherrill 
> wrote:
>
>>
>> On 7/28/20 11:44 AM, David Davis wrote:
>>
>> Today we discussed this at triage. We're leaning towards changing the
>> default from 20 to 10 as it seems like 10 only incurs an extra 30% penalty
>> in time while seeming to fix the problem[0].
>>
>> One question though is how we should treat existing data because most
>> Remotes at this point probably have a value of 20 for download_concurrency.
>> We came up with two options that we would like some feedback on.
>>
>> It seems kinda strange that the 'default' value is recorded in the db on
>> the remote at creation time.  Any reason to just leave it 'nil' and use the
>> 'app default' ?   This doesn't really help with past values, but seems like
>> a better model going forward.
>>
>
> I think this maybe makes sense given that we'll potentially have to change
> it again if we find out that 10 doesn't work for some reason. Perhaps we
> could set a class constant on Remote that subclasses could override.
>
One benefit of what we have now is that if you get a db dump, you can see
all the data right there. One downside of what we have now is that we can't
disambiguate a user intending to use the default from just receiving the
default. Overall I value the simplicity of recording actual values in the
db because it's simple, explicit, and consistent with how all other
defaults work in Pulp.


>>
>> # Option 1: Migrate 20 to 10
>>
>> This would be a migration in pulpcore that would update
>> download_concurrency to 10 for all Remotes whose download_concurrency is
>> set to 10. Something like:
>>
>>
>>   
>> Remote.objects.all().filter(download_concurrency=20).update(download_concurrency=10)
>>
>> My vote is for #1 because:
>>
>> a) I imagine ~99% of people want the recommended default, so 99% of
>> people will either need to update this manually (if doing #2) or face sync
>> failures
>>
>> b) i'd rather 99% of people not have to do anything than the 1% that for
>> some reason actually wanted 20 and weren't just going with the 'recommended
>> default' at that time.  The number of people in this case could very well
>> be zero.
>>
>>
>>
>> # Option 2: Documentation
>>
>> This would be similar to the migration approach but instead of modifying
>> our users' data, we'd document how they could do it themselves. So
>> something like:
>>
>> pulpcore-manager shell_plus -c
>> "Remote.objects.all().filter(download_concurrency=20).update(download_concurrency=10)
>>
>>
>> Any feedback is welcome.
>>
>> [0] https://pulp.plan.io/issues/7186#note-2
>>
>> David
>>
>>
>> On Mon, Jul 27, 2020 at 2:57 PM Grant Gainey  wrote:
>>
>>> Hey folks,
>>>
>>> Looking into issue 7212  , over the
>>> weekend I did some ad-hoc evaluations of sync-performance at various
>>> concurrency settings. I wrote up my observations here:
>>>
>>> https://hackmd.io/@ggainey/pulp3_sync_concurrency
>>>
>>> Just thought folk might be interested.
>>>
>>> G
>>> --
>>> Grant Gainey
>>> Principal Software Engineer, Red Hat System Management Engineering
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

I've gotten various feedback and I want to relay some of the changes based
on that.

1) Pulp should ship a "user isolation" policy by default. If users want
other things they can configure it further. This is a change from my
original proposal of Pulp shipping an "RBAC is off" policy. This will work
for Katello because all Katello calls show up as one user anyway.

2) We have challenges around assigning permissions. This is the proposed
solution that was discussed at open floor today:
https://pulp.plan.io/issues/7210  I've added this to my pulpcore PR (link
below)

https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC



On Fri, Jul 10, 2020 at 2:20 PM David Davis  wrote:

> On Wed, Jul 8, 2020 at 4:54 PM Brian Bouterse  wrote:
>
>> My next goal is to have object-level permissions assigned through signals
>> so that anywhere you save the model the permissions are correctly created.
>> To do this I need to get a few things working:
>>
>> 1) Move the permissions creation to the signals [done]
>> 2) Have the user be well-known, this is accomplished by pulpcore picking
>> up this dependency https://github.com/PaesslerAG/django-currentuser
>> [done]
>> 3) Have the user information persisted in tasks so signals can work there
>> also ... there are two obvious options
>>
>> option a) Have tasks themselves have RBAC which we know we need anyway,
>> then query the RBAC permissions inside a task to determine which user has
>> the object level permissions for this task. This is what I'm currently
>> prototyping. It at least will provide a mechanism we can use for now until
>> we find something better.
>>
>
> Are task permissions mutable? If so, then what happens if someone kicks
> off a task and then this permission gets removed?
>
>
>>
>> option b) Add a user field to the Task itself and have the RBAC assign
>> permissions to them. The concern with this is that while this is nice, it
>> would be nicer if we had this type of visibility on everything in Pulp from
>> an auditing project dedicated to keeping this type of info. Option (b) is
>> still viable, just maybe not the best way. Comments/feedback is welcome.
>>
>> Also I've collaborated with @alikins from galaxy_ng project and together
>> we wrote this requirements doc for their plugin:
>> https://hackmd.io/JisLkfyeT2myAD2khEwU0A
>>
>> On Wed, Jul 1, 2020 at 6:43 PM Brian Bouterse 
>> wrote:
>>
>>> The demo advertisement for tomorrow is here:
>>> https://www.redhat.com/archives/pulp-dev/2020-June/msg00076.html
>>>
>>> On Wed, Jul 1, 2020 at 6:41 PM Brian Bouterse 
>>> wrote:
>>>
>>>> Another productive RBAC day! See the latest code at the links below.
>>>> Here's what's new:
>>>>
>>>> * policy is now shorter thanks to machinery checking both model-level
>>>> and object-level permissions with one call. The other two are also 
>>>> available
>>>> * sync is now restricted on both 'modify_repo_content' permissions AND
>>>> read permission on the remote being used to sync
>>>> * modify is now restricted on 'modify_repo_content' permission
>>>> * moved the permission checking machinery to be "global checks"
>>>> * added data migration that sets is_staff=True, so the django-admin
>>>> interface can be used (this is getting a slight rework tomorrow morning 
>>>> tho)
>>>>
>>>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC
>>>> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>>>>
>>>> Tomorrow's demo is advertised here. It will also include an overview of
>>>> some of the unsolved problems with some possible solutions.
>>>>
>>>> Cheers,
>>>> Brian
>>>>
>>>>
>>>> On Tue, Jun 30, 2020 at 5:08 PM Brian Bouterse 
>>>> wrote:
>>>>
>>>>> Today I accomplished a few more things:
>>>>>
>>>>> * finished my ldap notes: https://hackmd.io/ED9UpscNSRW86Le3xNzVeg
>>>>> * moving the checks from a mixin to be "global checks" so they are
>>>>> available everywhere, this is a feature from drf-access-policy:
>>>>> https://rsinger86.github.io/drf-access-policy/reusable_conditions/
>>>>> * added a has_obj_or_module_perms method allowing policy writers to
>>>>> just use that instad of carrying "two entries" in the policy, one for
>>>>> model-level, one for object-level
>&

Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API

[Brian Bouterse]

Grand and David and I met for a *long* conversation about all the issues
here. I'll try to summarize some of the highlights. Nothing was decided
that was clear.

* Grant's language about multi-tenancy is helpful, that is at the heart of
my concern
* I believe we need multi-tenancy for Pulp to be safe and achieve the
project's goals, I've added a multi-tenancy discussion to the pulpcon
agenda if we don't get to it sooner https://hackmd.io/hIOjFsFiSkGJR7VqtAJ8eQ
* Generally namespaces or some other form of multi-tenancy mechanisms seems
to be the solution
* I agree it won't resolve situations where there are data conflicts around
URLs, e.g. with Distributions specifically

So for this thread, if it's helpful to change the uniqueness constraint for
repositories that seemed fine by everyone I've talked to. I'm interested in
organizing an effort to make sure we don't introduce RBAC that users
dislike with 3.6 but that needs more comprehensive planning. If anyone has
a simple proposal on how to do that please let me know. I'll likely draft a
problem statement here soon and probably collab w/ @ggainey on it.

On Tue, Jul 21, 2020 at 12:38 PM Matthias Dellweg 
wrote:

> The user story you are describing there is inevitably not satisfiable.
> At the very least, not every user can randomly create distributions
> with base paths colliding with existing distributions.
> I believe the answer to that is namespaces, where users can create
> entities in their namespaces that they can also see. BTW, the
> namespace could be used to decide upon a certain group permission to
> be added to new resources.
> In the global namespace, only an administrator can create entities.
> If you lift, the uniqueness of names in a certain content-type, you
> are definitely breaking the ansible usecase.
>
> On Tue, Jul 21, 2020 at 5:56 PM Grant Gainey  wrote:
> >
> > On Tue, Jul 21, 2020 at 11:38 AM Dennis Kliban 
> wrote:
> >>
> >> On Tue, Jul 21, 2020 at 11:22 AM Brian Bouterse 
> wrote:
> >>>
> >>> I'm concerned if we don't make a change, here's the user experience
> I'm worried about.
> >>>
> >>> 1. User A creates repo 'rhel7'
> >>> 2. user B can't see repo 'rhel7' because of queryset scoping
> >>> 3. user B goes to create 'rhel7'
> >>> 4. user B is told 'rhel7' already exists
> >>>
> >>> Users should be able to use simple names. I don't know what the answer
> is to the import/export implementation conflict, but let's brainstorm some.
> For the benefit of our users, I don't think that implementation should
> interfere with this basic use.
> >>>
> >>
> >> I agree that this is a usability problem for our users.
> >>
> >> With regard to import/export, the ideal solution would use the same
> UUID in both the system that's exporting and the system that's importing.
> Is my understanding correct?
> >
> >
> > For PIE to work, it needs to be able to identify whether something needs
> to be created otr updated in the 'downstream', and therefore needs to be
> able to identify instances as being "the same thing". pulp_id definitely
> does that. However, the use-case for PIE can't rely on pulp_id, because
> it's not a replica of upstream. Consider the migrated-use-case - starting
> from pulp2, I have the exact same content in upstream and downstream, but
> completely different pulp_ids.
> >
> > In any event - PIE isn't really the major issue as far as I am
> concerned, it's deciding if a single pulp instance is multi-user or
> multi-tenant and following the implications from there.
> >
> > G
> >
> >>
> >>
> >>
> >>>
> >>> Side note: from early on in Pulp3, pk's not names have been the
> primary identifier. I'm unclear on how we got away from that.
> >>>
> >>>
> >>>
> >>> On Tue, Jul 21, 2020 at 9:03 AM Matthias Dellweg 
> wrote:
> >>>>
> >>>> I always understood the "lifting the uniqueness" as allowing to have
> >>>> the same name used for different resource types. So the new
> >>>> natrual_key (aka unique_together) would be ["name", "type"].
> >>>>
> >>>> On Tue, Jul 21, 2020 at 2:55 PM David Davis 
> wrote:
> >>>> >
> >>>> > Agreed.
> >>>> >
> >>>> > David
> >>>> >
> >>>> >
> >>>> > On Tue, Jul 21, 2020 at 8:42 AM Grant Gainey 
> wrote:
> >>>> >>
> >>>> >> On Tue, Jul 21, 2020 at 8:14 AM Dennis Kliban 
> w

Re: [Pulp-dev] Name uniqueness problem in Pulp 3 REST API

[Brian Bouterse]

I'm concerned if we don't make a change, here's the user experience I'm
worried about.

1. User A creates repo 'rhel7'
2. user B can't see repo 'rhel7' because of queryset scoping
3. user B goes to create 'rhel7'
4. user B is told 'rhel7' already exists

Users should be able to use simple names. I don't know what the answer is
to the import/export implementation conflict, but let's brainstorm some.
For the benefit of our users, I don't think that implementation should
interfere with this basic use.

Side note: from early on in Pulp3, pk's not names have been the primary
identifier. I'm unclear on how we got away from that.



On Tue, Jul 21, 2020 at 9:03 AM Matthias Dellweg 
wrote:

> I always understood the "lifting the uniqueness" as allowing to have
> the same name used for different resource types. So the new
> natrual_key (aka unique_together) would be ["name", "type"].
>
> On Tue, Jul 21, 2020 at 2:55 PM David Davis  wrote:
> >
> > Agreed.
> >
> > David
> >
> >
> > On Tue, Jul 21, 2020 at 8:42 AM Grant Gainey  wrote:
> >>
> >> On Tue, Jul 21, 2020 at 8:14 AM Dennis Kliban 
> wrote:
> >>>
> >>> Does anyone else have an opinion? If not, I am going to start by
> writing a task to remove this name uniqueness constraint for repositories.
> >>
> >>
> >> Import/export relies on non-pulp_id-uniqueness to identify Things. I
> was assuming we were talking about adding pulp_type to the Repository
> uniqueness-constraint, so that a given name/type would be unique (which
> would require a single change to RepositoryResource)
> >>
> >> If we're talking about just removing the uniqueness-constraint
> altogether, then life gets a lot harder.
> >>
> >> G
> >> --
> >> Grant Gainey
> >> Principal Software Engineer, Red Hat System Management Engineering
> >> ___
> >> Pulp-dev mailing list
> >> Pulp-dev@redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-dev
> >
> > ___
> > Pulp-dev mailing list
> > Pulp-dev@redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] AccessPolicy Model Advice ... input needed

[Brian Bouterse]

Here's my next dilemma:

Should we make the API to GET/PUT these access policies all at one
endpoint, or nest them as sub-resources under each viewset they control?
I'm outlining the two options below, please reply with your thoughts.

Option "nested under each viewset". This would have each access policies
live under the viewset it pairs with, e.g. the
/pulp/api/v3/remotes/file/file/ would then have a sub-resource
/pulp/api/v3/remotes/file/file/access_policy/
pros: it's simple for users
cons: it's more complicated to develop because we have to try to
automatically add this to lots of viewsets. Also it's complicated to see
the policies all in one place. Also if there are other sub-resources that
may want to use a name like "access_policy" this becomes a problem unless
we allow plugin writers to "move" this resource to another area somehow.

Option "all at one viewset". This would have all access policies live at
/pulp/api/v3/access_policy/ with detail views like
/pulp/api/v3/access_policy/:uuid/
pros: it avoids url conflicts described above. It's easy to implement, and
you can see all the policies in one place
cons: it requires users to "find" their access_policy and this could cause
user mistakes also...

What do you think we should do to create the best user experience
first-and-foremost, and the best plugin writer experience second-most?

Thanks!
Brian


On Tue, Jul 14, 2020 at 8:44 AM Tatiana Tereshchenko 
wrote:

> +1 to idea 1 since the URLs seem to me more stable than class paths. We
> should not change REST API much but potentially can refactor code in some
> ways.
>
> One nitpick, maybe the `viewset` field should have a more generic name,
> since we don't use viewsets exclusively, we also have separate views, e.g.
> for orphan cleanup or status.
> I don't have good ideas though... `endpoint`? (though it's not a full
> endpoint), `guarded_view_obj`? `view_object`? `view`? :)
>
> On Tue, Jul 14, 2020 at 2:21 AM Dennis Kliban  wrote:
>
>> Pulp 2 users would definitely be familiar with describing policies in
>> terms of urls. The Pulp 3 REST API already uses the pulp_href field as the
>> primary identifier. You should implement idea 1.
>>
>> On Mon, Jul 13, 2020, 5:45 PM Brian Bouterse  wrote:
>>
>>> I need some design input. To store AccessPolicy data in the DB I think
>>> we want one Model where each instance is the access policy for a given
>>> viewset. I think this would be better than one Model per Viewset which
>>> would generate N tables for N viewsets with 1 instance of each which would
>>> be very strange and inefficient.
>>>
>>> So let's assume we have a simple definition like the one below. Each
>>> instance stores the policy for one viewset.
>>>
>>> class AccessPolicy(BaseModel):
>>> data = JSONField()
>>>
>>> So what second field can I add to this that would allow me to relate an
>>> instance of this model to a viewset. For example the FileRemoteViewset
>>> here:
>>> https://github.com/pulp/pulp_file/blob/de638519fc02d588f403db4c5cfcca7552543c50/pulp_file/app/viewsets.py#L116
>>>
>>> Idea 1: Add a viewset = CharField(). Have it store values as URLs, e.g.
>>> /pulp/api/v3/remotes/file/file/.
>>> Idea 2: Add a viewset = CharField(), and have it store values as
>>> classpaths, e.g. 'pulp_file.app.viewsets.RemoteFileViewset'.
>>>
>>> I think Idea 1 makes the most sense because that's how our users think
>>> of it. I can't think of a good alternative. What do you think makes the
>>> most sense? What alternative ideas should we consider?
>>>
>>> If you have feedback please share it. I need to start into something to
>>> get it going tomorrow even if it's just Idea 1 for lack of an alternate
>>> proposal.
>>>
>>> Thanks,
>>> Brian
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] AccessPolicy Model Advice ... input needed

[Brian Bouterse]

On Tue, Jul 14, 2020 at 4:37 AM Matthias Dellweg 
wrote:

> I think, it would help me to understand if you could reiterate what
> kind of policy this model holds. Is it the class level permission like
> "Only admins can create new repositories."?
>
It holds a database equivalent of this policy for example:
https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC#diff-b8a33258feb0183716da827efd0b4102R19-R54
These would be loaded 100% from the DB and no longer in code. This is more
or less recommended by drf-access-policy here
https://rsinger86.github.io/drf-access-policy/loading_external_source/

What kind of context does a lookup for these objects? Is it the
> PermissionClass that carries a context containing an instance to the
> view / viewset, or the request with the resolver_match?
>
It only holds the policy, the context data is available on the request and
in the task, so only the "statements" live in the db.


> If it is userfacing i would probably also go for option 1 with the hrefs.
>
> On Tue, Jul 14, 2020 at 2:21 AM Dennis Kliban  wrote:
> >
> > Pulp 2 users would definitely be familiar with describing policies in
> terms of urls. The Pulp 3 REST API already uses the pulp_href field as the
> primary identifier. You should implement idea 1.
> >
> > On Mon, Jul 13, 2020, 5:45 PM Brian Bouterse 
> wrote:
> >>
> >> I need some design input. To store AccessPolicy data in the DB I think
> we want one Model where each instance is the access policy for a given
> viewset. I think this would be better than one Model per Viewset which
> would generate N tables for N viewsets with 1 instance of each which would
> be very strange and inefficient.
> >>
> >> So let's assume we have a simple definition like the one below. Each
> instance stores the policy for one viewset.
> >>
> >> class AccessPolicy(BaseModel):
> >> data = JSONField()
> >>
> >> So what second field can I add to this that would allow me to relate an
> instance of this model to a viewset. For example the FileRemoteViewset
> here:
> https://github.com/pulp/pulp_file/blob/de638519fc02d588f403db4c5cfcca7552543c50/pulp_file/app/viewsets.py#L116
> >>
> >> Idea 1: Add a viewset = CharField(). Have it store values as URLs, e.g.
> /pulp/api/v3/remotes/file/file/.
> >> Idea 2: Add a viewset = CharField(), and have it store values as
> classpaths, e.g. 'pulp_file.app.viewsets.RemoteFileViewset'.
> >>
> >> I think Idea 1 makes the most sense because that's how our users think
> of it. I can't think of a good alternative. What do you think makes the
> most sense? What alternative ideas should we consider?
> >>
> >> If you have feedback please share it. I need to start into something to
> get it going tomorrow even if it's just Idea 1 for lack of an alternate
> proposal.
> >>
> >> Thanks,
> >> Brian
> >> ___
> >> Pulp-dev mailing list
> >> Pulp-dev@redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-dev
> >
> > ___
> > Pulp-dev mailing list
> > Pulp-dev@redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] AccessPolicy Model Advice ... input needed

[Brian Bouterse]

I need some design input. To store AccessPolicy data in the DB I think we
want one Model where each instance is the access policy for a given
viewset. I think this would be better than one Model per Viewset which
would generate N tables for N viewsets with 1 instance of each which would
be very strange and inefficient.

So let's assume we have a simple definition like the one below. Each
instance stores the policy for one viewset.

class AccessPolicy(BaseModel):
data = JSONField()

So what second field can I add to this that would allow me to relate an
instance of this model to a viewset. For example the FileRemoteViewset
here:
https://github.com/pulp/pulp_file/blob/de638519fc02d588f403db4c5cfcca7552543c50/pulp_file/app/viewsets.py#L116

Idea 1: Add a viewset = CharField(). Have it store values as URLs, e.g.
/pulp/api/v3/remotes/file/file/.
Idea 2: Add a viewset = CharField(), and have it store values as
classpaths, e.g. 'pulp_file.app.viewsets.RemoteFileViewset'.

I think Idea 1 makes the most sense because that's how our users think of
it. I can't think of a good alternative. What do you think makes the most
sense? What alternative ideas should we consider?

If you have feedback please share it. I need to start into something to get
it going tomorrow even if it's just Idea 1 for lack of an alternate
proposal.

Thanks,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Cherry pick processor

[Brian Bouterse]

On Fri, Jul 10, 2020 at 1:34 PM David Davis  wrote:

> As creator of the cherry pick processor, I'd like to propose we ditch it.
> It requires a lot of upkeep which doesn't really save us time over just
> doing the cherry picks ourselves. Also, it often fails because it cannot
> make intelligent decisions when there are merge conflicts.
>
> Instead what I'd propose is that we do cherry picks at release time. We do
> z-releases infrequently and they are usually for a particular stakeholder
> so we usually know which issues the stakeholder needs and can cherry pick
> these changes ourselves before we do a release. We can also continue using
> the "Needs Cherry Pick" labels to help us remember issues we want to
> include in a z-release later on.
>
> Overall, I think disabling it is the best path forward but I'm also
> interested in any feedback people may have to improve the cherry pick
> processor.
>
I agree with you. Also if we do keep something like this we probably should
use one that is already made, e.g. gerritt.
https://www.gerritcodereview.com/ +1 to for now retiring it.


> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

This is all done. I've pushed my code to the links below. You can now use
get_current_authenticated_user in both viewsets and tasks and it will give
you the user, which lets you use signals to automatically add object-level
permissions anywhere automatically. Also tasks have RBAC themselves
(including cancelation and deletion restriction), and support queryset
filtering.

https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC
https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC

Here are a few random topics I've been thinking about:

## What should the default policy be?

I propose Pulp ship with a very permissive policy, and this is effectively
with RBAC off (also a requirement, e.g. Katello's usage). Each viewset
would have an AccessPolicy but it would by default ship with the following
single rule which allows everything.

{
"action": ["*"],
"principal": "authenticated",
"effect": "allow",
}

## How can we adopt user-controllable policies?

The highest goal here is to have users in total control of the policy. This
is in-line with what drf-access-policy recommends:
https://rsinger86.github.io/drf-access-policy/loading_external_source/

I propose we store the json policy for each Viewset, e.g.
FileRemoteAccessPolicy which governs File Remotes as a blob that can by
editable via the API itself. This allows us to ship a simple policy like ^
and allow users to do what they want. We can provide many example policies
also.

[idea] We could make a sub-resource called "access_policy" so for
FileRemote for example you could read the policy here:

GET /pulp/api/v3/remotes/file/file/access_policy
PUT /pulp/api/v3/remotes/file/file/access_policy

It would not support PATCH because we don't allow partial policy changes,
it's one blob. It wouldn't support DELETE because there is always a policy.
It wouldn't support POST because you're modifying a resource in-place.

## Requirements driving this

In my discussions with galaxy_ng their deployment on the cloud versus
on-premise will need a drastically different policy. This tells me that we
need pluggable policies at initial launch.

## My main concerns at this point:

1) Galaxy_ng also needs flexibility between their cloud versus on-premise
deployment regarding what the obj-level permissions created are. If ^ is
implemented the user can have total control over the policy, but zero
control over what permissions are assigned to newly created objects. For
example that happens in this signal here:
https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC?expand=1#diff-bdb7fbb7c563d6651a309aff97001cb6R8-R12

2) Also can an admin "fix" object-level permissions? Right now they are
only in the DB. I suspect the answer is for us to turn on more
django-guardian admin integration:
https://github.com/django-guardian/django-guardian#admin-integration

I have to think about this last part more... Meanwhile I plan to go forward
with the user-controllable policies next. Input and feedback is welcome.

Cheers,
Brian




On Wed, Jul 8, 2020 at 4:53 PM Brian Bouterse  wrote:

> My next goal is to have object-level permissions assigned through signals
> so that anywhere you save the model the permissions are correctly created.
> To do this I need to get a few things working:
>
> 1) Move the permissions creation to the signals [done]
> 2) Have the user be well-known, this is accomplished by pulpcore picking
> up this dependency https://github.com/PaesslerAG/django-currentuser [done]
> 3) Have the user information persisted in tasks so signals can work there
> also ... there are two obvious options
>
> option a) Have tasks themselves have RBAC which we know we need anyway,
> then query the RBAC permissions inside a task to determine which user has
> the object level permissions for this task. This is what I'm currently
> prototyping. It at least will provide a mechanism we can use for now until
> we find something better.
>
> option b) Add a user field to the Task itself and have the RBAC assign
> permissions to them. The concern with this is that while this is nice, it
> would be nicer if we had this type of visibility on everything in Pulp from
> an auditing project dedicated to keeping this type of info. Option (b) is
> still viable, just maybe not the best way. Comments/feedback is welcome.
>
> Also I've collaborated with @alikins from galaxy_ng project and together
> we wrote this requirements doc for their plugin:
> https://hackmd.io/JisLkfyeT2myAD2khEwU0A
>
> On Wed, Jul 1, 2020 at 6:43 PM Brian Bouterse  wrote:
>
>> The demo advertisement for tomorrow is here:
>> https://www.redhat.com/archives/pulp-dev/2020-June/msg00076.html
>>
>> On Wed, Jul 1, 2020 at 6:41 PM Brian Bouterse 
>> wrote:
>>
>>> Another productive RBAC day! Se

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

My next goal is to have object-level permissions assigned through signals
so that anywhere you save the model the permissions are correctly created.
To do this I need to get a few things working:

1) Move the permissions creation to the signals [done]
2) Have the user be well-known, this is accomplished by pulpcore picking up
this dependency https://github.com/PaesslerAG/django-currentuser [done]
3) Have the user information persisted in tasks so signals can work there
also ... there are two obvious options

option a) Have tasks themselves have RBAC which we know we need anyway,
then query the RBAC permissions inside a task to determine which user has
the object level permissions for this task. This is what I'm currently
prototyping. It at least will provide a mechanism we can use for now until
we find something better.

option b) Add a user field to the Task itself and have the RBAC assign
permissions to them. The concern with this is that while this is nice, it
would be nicer if we had this type of visibility on everything in Pulp from
an auditing project dedicated to keeping this type of info. Option (b) is
still viable, just maybe not the best way. Comments/feedback is welcome.

Also I've collaborated with @alikins from galaxy_ng project and together we
wrote this requirements doc for their plugin:
https://hackmd.io/JisLkfyeT2myAD2khEwU0A

On Wed, Jul 1, 2020 at 6:43 PM Brian Bouterse  wrote:

> The demo advertisement for tomorrow is here:
> https://www.redhat.com/archives/pulp-dev/2020-June/msg00076.html
>
> On Wed, Jul 1, 2020 at 6:41 PM Brian Bouterse  wrote:
>
>> Another productive RBAC day! See the latest code at the links below.
>> Here's what's new:
>>
>> * policy is now shorter thanks to machinery checking both model-level and
>> object-level permissions with one call. The other two are also available
>> * sync is now restricted on both 'modify_repo_content' permissions AND
>> read permission on the remote being used to sync
>> * modify is now restricted on 'modify_repo_content' permission
>> * moved the permission checking machinery to be "global checks"
>> * added data migration that sets is_staff=True, so the django-admin
>> interface can be used (this is getting a slight rework tomorrow morning tho)
>>
>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC
>> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>>
>> Tomorrow's demo is advertised here. It will also include an overview of
>> some of the unsolved problems with some possible solutions.
>>
>> Cheers,
>> Brian
>>
>>
>> On Tue, Jun 30, 2020 at 5:08 PM Brian Bouterse 
>> wrote:
>>
>>> Today I accomplished a few more things:
>>>
>>> * finished my ldap notes: https://hackmd.io/ED9UpscNSRW86Le3xNzVeg
>>> * moving the checks from a mixin to be "global checks" so they are
>>> available everywhere, this is a feature from drf-access-policy:
>>> https://rsinger86.github.io/drf-access-policy/reusable_conditions/
>>> * added a has_obj_or_module_perms method allowing policy writers to just
>>> use that instad of carrying "two entries" in the policy, one for
>>> model-level, one for object-level
>>>
>>> Need to:
>>> * clean up the "sync" policy code
>>> * Add global condition check facilities for the perms of a 'remote' param
>>> * add policy language restricting the /modify/ endpoint also for
>>> FileRepository
>>> * push my code
>>>
>>> New Challenge: We need to also have the permissions assignments happen
>>> for objects created by tasks. django-guardian recommends this happen inside
>>> signals (
>>> https://django-guardian.readthedocs.io/en/latest/userguide/assign.html#assigning-permissions-inside-signals).
>>> The challenge (thanks @mdellweg for identifying) is that the user/group
>>> context information is well-known in the viewset but not in a task. Soo
>>> ... the idea is:
>>>
>>> 1. Switch the perms addition to the model itself via signals so it's
>>> automatic everywhere (including in tasks)
>>> 2. Preserve the user and group "request context" into the tasking
>>> system. I can see a straightforward path to how to do this so I plan to
>>> prototype this soon also.
>>>
>>> Feedback is welcome!
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jun 26, 2020 at 6:16 PM Brian Bouterse 
>>> wrote:
>>>
>>>> Today I got the "sync" RBAC working, but I need to give it some more
>>>> thought. The extra challenge w

Re: [Pulp-dev] RBAC: PoC Available and Review/Discussion Meeting Setup

[Brian Bouterse]

Thanks to everyone who joined for today's review. There were a lot of great
questions which also made it fun for me.

Here's a link to the recording:  https://youtu.be/tWEHGJmBqCk

Any questions, concerns, ideas, or feedback are welcome either publicly or
privately.

Cheers,
Brian


On Mon, Jun 29, 2020 at 8:12 AM Brian Bouterse  wrote:

> The Role Based Access Control (RBAC) Proof of Concept (PoC) is ready for
> review and feedback. Below are links to pulpcore and pulp_file parts, along
> with a bonus ldap integration PR too.
>
> * The pulpcore code right now is minimal, only enabling django-guardian as
> a dependency and enabling django admin [0]
> * The bulk of the PoC is in pulp_file [1]
> * Not at all required, but here's a demonstration of loading users, group,
> and group membership from ldap [2]
>
> I'm going to host a demo and discussion of this functionality to ask for
> feedback. Invite details are below. It will be recorded and available on
> the Pulp YouTube channel in case you want to watch later. I'll send a link
> to this thread.
>
> Thurs, July 2nd @ 10:05 am EDT https://everytimezone.com/s/1034c323
> meeting:  https://meet.google.com/htr-cexw-dte
> agenda: https://hackmd.io/K8UW69dnS1qCrZUhkcn-4g
> slides https://hackmd.io/@pulp/SJrjjX8RI#/
>
> [0]: https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
> [1]: https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC
> [2]:
> https://github.com/pulp/pulpcore/compare/master...bmbouter:ldap-integration
>
> Thanks,
> Brian
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Another productive RBAC day! See the latest code at the links below. Here's
what's new:

* policy is now shorter thanks to machinery checking both model-level and
object-level permissions with one call. The other two are also available
* sync is now restricted on both 'modify_repo_content' permissions AND read
permission on the remote being used to sync
* modify is now restricted on 'modify_repo_content' permission
* moved the permission checking machinery to be "global checks"
* added data migration that sets is_staff=True, so the django-admin
interface can be used (this is getting a slight rework tomorrow morning tho)

https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC
https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC

Tomorrow's demo is advertised here. It will also include an overview of
some of the unsolved problems with some possible solutions.

Cheers,
Brian


On Tue, Jun 30, 2020 at 5:08 PM Brian Bouterse  wrote:

> Today I accomplished a few more things:
>
> * finished my ldap notes: https://hackmd.io/ED9UpscNSRW86Le3xNzVeg
> * moving the checks from a mixin to be "global checks" so they are
> available everywhere, this is a feature from drf-access-policy:
> https://rsinger86.github.io/drf-access-policy/reusable_conditions/
> * added a has_obj_or_module_perms method allowing policy writers to just
> use that instad of carrying "two entries" in the policy, one for
> model-level, one for object-level
>
> Need to:
> * clean up the "sync" policy code
> * Add global condition check facilities for the perms of a 'remote' param
> * add policy language restricting the /modify/ endpoint also for
> FileRepository
> * push my code
>
> New Challenge: We need to also have the permissions assignments happen for
> objects created by tasks. django-guardian recommends this happen inside
> signals (
> https://django-guardian.readthedocs.io/en/latest/userguide/assign.html#assigning-permissions-inside-signals).
> The challenge (thanks @mdellweg for identifying) is that the user/group
> context information is well-known in the viewset but not in a task. Soo
> ... the idea is:
>
> 1. Switch the perms addition to the model itself via signals so it's
> automatic everywhere (including in tasks)
> 2. Preserve the user and group "request context" into the tasking system.
> I can see a straightforward path to how to do this so I plan to prototype
> this soon also.
>
> Feedback is welcome!
>
>
>
>
>
> On Fri, Jun 26, 2020 at 6:16 PM Brian Bouterse 
> wrote:
>
>> Today I got the "sync" RBAC working, but I need to give it some more
>> thought. The extra challenge with this parts is that "having permission to
>> read a Remote" is already defined in one place, on FileRemoteAccessPolicy,
>> yet the AccessPolicy that needs to perform the enforcement is
>> FileRepositoryAccessPolicy for its "sync" action. This is a bit challenging
>> considering the following goals:
>>
>> * We don't want to duplicate code, e.g. having the
>> FileRepositoryAccessControl begin to inspect permissions for FileRemote
>> directly, when FileRemoteAccessPolicy already does that
>> * Currently permissions are granted at two levels: Model-level and
>> File-level permissions and permissions are granted from either level.
>> * We want to keep the policy in charge. If we start to bury the behavior
>> in methods and functions then policy writers are no longer in control.
>>
>> All of ^ together tells me that I should work on creating two things next:
>> 1) A way for policy writers to express which parameter refers to objects
>> that also need their permissions checked. For example the policy should be
>> able to say "remote is a parameter and it needs X permission". This is akin
>> to the has_module_level_perms and has_obj_level_perms here except we also
>> need to identify which parameter is being checked instead of the object the
>> AccessPolicy itself governs.
>> 2) A single way to check model-level and object-level permissions at once
>> and allow if *either* passes. We would still allow policy writers to call
>> either model-level or file-level checks also.
>>
>> I'll work on ^ next. Ideas and feedback are welcome. I pushed no new code
>> today because it's a mess and not runnable at my stopping point.
>>
>>
>> On Thu, Jun 25, 2020 at 6:18 PM Brian Bouterse 
>> wrote:
>>
>>> Here's another push to the branch (it includes the following
>>> additions):
>>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>>>
>>> * A FileRepositoryAccessPolicy wh

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

The demo advertisement for tomorrow is here:
https://www.redhat.com/archives/pulp-dev/2020-June/msg00076.html

On Wed, Jul 1, 2020 at 6:41 PM Brian Bouterse  wrote:

> Another productive RBAC day! See the latest code at the links below.
> Here's what's new:
>
> * policy is now shorter thanks to machinery checking both model-level and
> object-level permissions with one call. The other two are also available
> * sync is now restricted on both 'modify_repo_content' permissions AND
> read permission on the remote being used to sync
> * modify is now restricted on 'modify_repo_content' permission
> * moved the permission checking machinery to be "global checks"
> * added data migration that sets is_staff=True, so the django-admin
> interface can be used (this is getting a slight rework tomorrow morning tho)
>
> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC
> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>
> Tomorrow's demo is advertised here. It will also include an overview of
> some of the unsolved problems with some possible solutions.
>
> Cheers,
> Brian
>
>
> On Tue, Jun 30, 2020 at 5:08 PM Brian Bouterse 
> wrote:
>
>> Today I accomplished a few more things:
>>
>> * finished my ldap notes: https://hackmd.io/ED9UpscNSRW86Le3xNzVeg
>> * moving the checks from a mixin to be "global checks" so they are
>> available everywhere, this is a feature from drf-access-policy:
>> https://rsinger86.github.io/drf-access-policy/reusable_conditions/
>> * added a has_obj_or_module_perms method allowing policy writers to just
>> use that instad of carrying "two entries" in the policy, one for
>> model-level, one for object-level
>>
>> Need to:
>> * clean up the "sync" policy code
>> * Add global condition check facilities for the perms of a 'remote' param
>> * add policy language restricting the /modify/ endpoint also for
>> FileRepository
>> * push my code
>>
>> New Challenge: We need to also have the permissions assignments happen
>> for objects created by tasks. django-guardian recommends this happen inside
>> signals (
>> https://django-guardian.readthedocs.io/en/latest/userguide/assign.html#assigning-permissions-inside-signals).
>> The challenge (thanks @mdellweg for identifying) is that the user/group
>> context information is well-known in the viewset but not in a task. Soo
>> ... the idea is:
>>
>> 1. Switch the perms addition to the model itself via signals so it's
>> automatic everywhere (including in tasks)
>> 2. Preserve the user and group "request context" into the tasking system.
>> I can see a straightforward path to how to do this so I plan to prototype
>> this soon also.
>>
>> Feedback is welcome!
>>
>>
>>
>>
>>
>> On Fri, Jun 26, 2020 at 6:16 PM Brian Bouterse 
>> wrote:
>>
>>> Today I got the "sync" RBAC working, but I need to give it some more
>>> thought. The extra challenge with this parts is that "having permission to
>>> read a Remote" is already defined in one place, on FileRemoteAccessPolicy,
>>> yet the AccessPolicy that needs to perform the enforcement is
>>> FileRepositoryAccessPolicy for its "sync" action. This is a bit challenging
>>> considering the following goals:
>>>
>>> * We don't want to duplicate code, e.g. having the
>>> FileRepositoryAccessControl begin to inspect permissions for FileRemote
>>> directly, when FileRemoteAccessPolicy already does that
>>> * Currently permissions are granted at two levels: Model-level and
>>> File-level permissions and permissions are granted from either level.
>>> * We want to keep the policy in charge. If we start to bury the behavior
>>> in methods and functions then policy writers are no longer in control.
>>>
>>> All of ^ together tells me that I should work on creating two things
>>> next:
>>> 1) A way for policy writers to express which parameter refers to objects
>>> that also need their permissions checked. For example the policy should be
>>> able to say "remote is a parameter and it needs X permission". This is akin
>>> to the has_module_level_perms and has_obj_level_perms here except we also
>>> need to identify which parameter is being checked instead of the object the
>>> AccessPolicy itself governs.
>>> 2) A single way to check model-level and object-level permissions at
>>> once and allow if *either* passes. We would still allow policy writers to
>>> call eith

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Today I accomplished a few more things:

* finished my ldap notes: https://hackmd.io/ED9UpscNSRW86Le3xNzVeg
* moving the checks from a mixin to be "global checks" so they are
available everywhere, this is a feature from drf-access-policy:
https://rsinger86.github.io/drf-access-policy/reusable_conditions/
* added a has_obj_or_module_perms method allowing policy writers to just
use that instad of carrying "two entries" in the policy, one for
model-level, one for object-level

Need to:
* clean up the "sync" policy code
* Add global condition check facilities for the perms of a 'remote' param
* add policy language restricting the /modify/ endpoint also for
FileRepository
* push my code

New Challenge: We need to also have the permissions assignments happen for
objects created by tasks. django-guardian recommends this happen inside
signals (
https://django-guardian.readthedocs.io/en/latest/userguide/assign.html#assigning-permissions-inside-signals).
The challenge (thanks @mdellweg for identifying) is that the user/group
context information is well-known in the viewset but not in a task. Soo
... the idea is:

1. Switch the perms addition to the model itself via signals so it's
automatic everywhere (including in tasks)
2. Preserve the user and group "request context" into the tasking system. I
can see a straightforward path to how to do this so I plan to prototype
this soon also.

Feedback is welcome!





On Fri, Jun 26, 2020 at 6:16 PM Brian Bouterse  wrote:

> Today I got the "sync" RBAC working, but I need to give it some more
> thought. The extra challenge with this parts is that "having permission to
> read a Remote" is already defined in one place, on FileRemoteAccessPolicy,
> yet the AccessPolicy that needs to perform the enforcement is
> FileRepositoryAccessPolicy for its "sync" action. This is a bit challenging
> considering the following goals:
>
> * We don't want to duplicate code, e.g. having the
> FileRepositoryAccessControl begin to inspect permissions for FileRemote
> directly, when FileRemoteAccessPolicy already does that
> * Currently permissions are granted at two levels: Model-level and
> File-level permissions and permissions are granted from either level.
> * We want to keep the policy in charge. If we start to bury the behavior
> in methods and functions then policy writers are no longer in control.
>
> All of ^ together tells me that I should work on creating two things next:
> 1) A way for policy writers to express which parameter refers to objects
> that also need their permissions checked. For example the policy should be
> able to say "remote is a parameter and it needs X permission". This is akin
> to the has_module_level_perms and has_obj_level_perms here except we also
> need to identify which parameter is being checked instead of the object the
> AccessPolicy itself governs.
> 2) A single way to check model-level and object-level permissions at once
> and allow if *either* passes. We would still allow policy writers to call
> either model-level or file-level checks also.
>
> I'll work on ^ next. Ideas and feedback are welcome. I pushed no new code
> today because it's a mess and not runnable at my stopping point.
>
>
> On Thu, Jun 25, 2020 at 6:18 PM Brian Bouterse 
> wrote:
>
>> Here's another push to the branch (it includes the following additions):
>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>>
>> * A FileRepositoryAccessPolicy which provides RBAC for Repositories (not
>> yet sync)
>> * A new Mixin allowing the two policies to share some common components
>>
>> Next up:
>> * have the pup_file define the fileContentAdmin group programmatically
>> * Extend the FileRepositoryAccessPolicy to restrict sync operations
>> * Write up and organize the PoC into a clear, organized format
>>
>> Also of interest today @ttereshc and I had a great convo asking what to
>> do about potential problems when we use Django groups to be a "role". My
>> write up will address this in more detail than I can go into here. We are
>> also looking at what the django-role-permissions project could offer us:
>> https://django-role-permissions.readthedocs.io/en/stable/utils.html
>>
>> I expect the PoC to be done by tomorrow and write-up by Monday, so I'm
>> going to schedule the public review meeting for next week towards the end
>> of the week.
>>
>>
>> On Wed, Jun 24, 2020 at 5:49 PM Brian Bouterse 
>> wrote:
>>
>>> Moar progress! Today the following things got done: Today's changes are
>>> available here:
>>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>>

Re: [Pulp-dev] Unblocking SigningService hash check PR

[Brian Bouterse]

Sorry for the long delay in reply. Thank you for the email. I also want to
unblock that PR. I put some replies in line.

On Wed, Jun 24, 2020 at 7:03 AM Quirin Pamp  wrote:

> Hi,
>
>
> However we decide to continue with the SigningService topic in the medium
> and longrun, I wanted to have one more go at unblocking the following PR in
> the short run:
>
>
> https://github.com/pulp/pulpcore/pull/659
>
>
> Currently, this PR issues a warning whenever the hash of a signing service
> script has changed on disk (compared to when it was first validated).
>
>
> I think we are all in agreement that this is a bad compromise between
> doing nothing at all (since the script might have changed for legitimate
> reasons), and issuing a full on Error in cases where things are broken.
>
>
> Yes I believe all parties agree the warning is the compromise no-one likes.

>
> My proposal is the following:
>
>
> Instead of issuing a warning, a changed hash value on disk would trigger
> an automatic re-validation of the script on disk.
>
> If the validation fails, it will throw a hard error (which would certainly
> be the correct course of action for a script that does not perform what the
> SigningService promises).
>
> If the validation succeeds, the SigningService is updated with the new
> hash value, and everything continues as it nothing had happened (we just
> assume the script was changed for legitimate reasons).
>
Overall this sounds ok to me, but I want to confirm, is the validation this
code does?
https://github.com/pulp/pulpcore/blob/bce4bee8124502ecd183fc664b904f5af5be97db/pulpcore/app/models/content.py#L453-L490

If so, moving the `public_key` and ^ `signing service` attributes from
AsciiArmoredDetachedSigningservice in pulp_rpm to SigningService in
pulpcore would be a good first step. I believe this will need to happen
first and is consistent with the last discussion of our video call. This
would be a good first step in its own PR I think.


> The only thing I can come up with where this approach might be
> problematic, is if users want to have different versions of the signing
> service script on different workers (for some reason).
>
> However, in such cases it would still be possible to work around the
> problem by having a single signing service script call a secondary script
> that differs on different workers.
>
If each script has its own public key it will fail. If each script on each
worker with a different sha256 shares one public key it would be
revalidated-and-modified each time it's used, which is strange but ok. Also
I don't think this use case is really valid so I'm ok with this.

>
> If you are worried that the possibility of such a workaround defeats the
> whole purpose of hashing the script in the first place, consider the
> following:
>
> This is not intended as a security feature against some kind of malicious
> attacker scenario, it is intended to provide some more meaningful error
> reporting, for operational mistakes.
> In this context I almost consider it a bonus if Sysadmin users who want to
> do something rather unusual and complicated (different signing service
> scripts on different workers) are forced to think about this carefully.
>
The use case doesn't resonate with me, but that's ok. If others see value
in it, it doesn't harm existing use cases, and ya'll are willing to
contribute it, let's do this.

Here is one idea I want to pitch as a possible counterproposal:  What if we
don't store the sha256 and instead, we perform validation on the signed
data all the time? The sha256 is nice, but it won't catch the case when the
script is interacting with a key server, and that key rotates but the
script doesn't. In that case the signatures will still be returned, pulp
will be unaware, no error will occur, yet it should. Validating data in all
cases I think would get us everything the sha256 script proposes and even
more.

What do you think about this as an option?

>
>
> Where to go from here:
>
> If we can get some kind of agreement that we would be willing to merge the
> version of the above PR that I have proposed, I would ask Manisha to make
> the relevant changes and they could be reviewed and merged.
> This would not prevent us from taking SigningServices into an entirely
> different direction in the future.
>
Agreed, but what do you think about moving the public_key and validate
methods to SigningService first?


> thanks,
> Quirin (quba42)
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] RBAC: PoC Available and Review/Discussion Meeting Setup

[Brian Bouterse]

The Role Based Access Control (RBAC) Proof of Concept (PoC) is ready for
review and feedback. Below are links to pulpcore and pulp_file parts, along
with a bonus ldap integration PR too.

* The pulpcore code right now is minimal, only enabling django-guardian as
a dependency and enabling django admin [0]
* The bulk of the PoC is in pulp_file [1]
* Not at all required, but here's a demonstration of loading users, group,
and group membership from ldap [2]

I'm going to host a demo and discussion of this functionality to ask for
feedback. Invite details are below. It will be recorded and available on
the Pulp YouTube channel in case you want to watch later. I'll send a link
to this thread.

Thurs, July 2nd @ 10:05 am EDT https://everytimezone.com/s/1034c323
meeting:  https://meet.google.com/htr-cexw-dte
agenda: https://hackmd.io/K8UW69dnS1qCrZUhkcn-4g
slides https://hackmd.io/@pulp/SJrjjX8RI#/

[0]: https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
[1]: https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC
[2]:
https://github.com/pulp/pulpcore/compare/master...bmbouter:ldap-integration

Thanks,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] CI failing due to docs.pulpproject.org out of space

[Brian Bouterse]

Pulpcore nightly CI is failing [0] with "rsync: write failed on "/var/www/
docs.pulpproject.org/en/master/nightly/.buildinfo": No space left on device
(28)". I contacted the support folks about it, so action has been taken.

[0]: https://travis-ci.com/github/pulp/pulpcore/jobs/354834259#L1672

Cheers,
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Today I got the "sync" RBAC working, but I need to give it some more
thought. The extra challenge with this parts is that "having permission to
read a Remote" is already defined in one place, on FileRemoteAccessPolicy,
yet the AccessPolicy that needs to perform the enforcement is
FileRepositoryAccessPolicy for its "sync" action. This is a bit challenging
considering the following goals:

* We don't want to duplicate code, e.g. having the
FileRepositoryAccessControl begin to inspect permissions for FileRemote
directly, when FileRemoteAccessPolicy already does that
* Currently permissions are granted at two levels: Model-level and
File-level permissions and permissions are granted from either level.
* We want to keep the policy in charge. If we start to bury the behavior in
methods and functions then policy writers are no longer in control.

All of ^ together tells me that I should work on creating two things next:
1) A way for policy writers to express which parameter refers to objects
that also need their permissions checked. For example the policy should be
able to say "remote is a parameter and it needs X permission". This is akin
to the has_module_level_perms and has_obj_level_perms here except we also
need to identify which parameter is being checked instead of the object the
AccessPolicy itself governs.
2) A single way to check model-level and object-level permissions at once
and allow if *either* passes. We would still allow policy writers to call
either model-level or file-level checks also.

I'll work on ^ next. Ideas and feedback are welcome. I pushed no new code
today because it's a mess and not runnable at my stopping point.


On Thu, Jun 25, 2020 at 6:18 PM Brian Bouterse  wrote:

> Here's another push to the branch (it includes the following additions):
> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>
> * A FileRepositoryAccessPolicy which provides RBAC for Repositories (not
> yet sync)
> * A new Mixin allowing the two policies to share some common components
>
> Next up:
> * have the pup_file define the fileContentAdmin group programmatically
> * Extend the FileRepositoryAccessPolicy to restrict sync operations
> * Write up and organize the PoC into a clear, organized format
>
> Also of interest today @ttereshc and I had a great convo asking what to do
> about potential problems when we use Django groups to be a "role". My write
> up will address this in more detail than I can go into here. We are also
> looking at what the django-role-permissions project could offer us:
> https://django-role-permissions.readthedocs.io/en/stable/utils.html
>
> I expect the PoC to be done by tomorrow and write-up by Monday, so I'm
> going to schedule the public review meeting for next week towards the end
> of the week.
>
>
> On Wed, Jun 24, 2020 at 5:49 PM Brian Bouterse 
> wrote:
>
>> Moar progress! Today the following things got done: Today's changes are
>> available here:
>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>>
>> * Got scoped querysets working! This restricts list views to only show
>> objects a user has permissions to view. A db reset was all that was needed
>> I think I didn't have all the changes in when I applied my earlier
>> migrations
>> * Added "detail view" restriction, and while it's in the policy and
>> working DRF does a strange thing on "retrieve" where if it's not in the
>> queryset (due to scoping ^) the user receives a 404, not a permission denied
>> * Got permissions cleaning up on resource deletion now too
>>
>> Next up:
>> * have the pup_file define the fileContentAdmin group programmatically
>> * Make similar policies for FileRepository which governs itself and the
>> "sync" action
>> * Write up and organize the PoC into a clear, organized format
>>
>> Questions and feedback are welcome!
>>
>> On Tue, Jun 23, 2020 at 5:54 PM Brian Bouterse 
>> wrote:
>>
>>> Lots of progress today! I have a mostly-complete policy for RBAC for
>>> FileRemote. It's surprising how little code all of this ended up being.
>>>
>>> Here's the actual RBAC stuff, it's all in pulp_file:
>>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>>> Here's the parts that go in core. Note the LDAP stuff is all optional,
>>> the only real requirement are two lines 1) enabling guardian in
>>> INSTALLED_APPS and 2) adding it as an AuthenticationBackend:
>>> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>>>
>>> I have some "how to use notes" here:
>>> https://hackmd.io/DRqGFyRsSD

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Here's another push to the branch (it includes the following additions):
https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1

* A FileRepositoryAccessPolicy which provides RBAC for Repositories (not
yet sync)
* A new Mixin allowing the two policies to share some common components

Next up:
* have the pup_file define the fileContentAdmin group programmatically
* Extend the FileRepositoryAccessPolicy to restrict sync operations
* Write up and organize the PoC into a clear, organized format

Also of interest today @ttereshc and I had a great convo asking what to do
about potential problems when we use Django groups to be a "role". My write
up will address this in more detail than I can go into here. We are also
looking at what the django-role-permissions project could offer us:
https://django-role-permissions.readthedocs.io/en/stable/utils.html

I expect the PoC to be done by tomorrow and write-up by Monday, so I'm
going to schedule the public review meeting for next week towards the end
of the week.


On Wed, Jun 24, 2020 at 5:49 PM Brian Bouterse  wrote:

> Moar progress! Today the following things got done: Today's changes are
> available here:
> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>
> * Got scoped querysets working! This restricts list views to only show
> objects a user has permissions to view. A db reset was all that was needed
> I think I didn't have all the changes in when I applied my earlier
> migrations
> * Added "detail view" restriction, and while it's in the policy and
> working DRF does a strange thing on "retrieve" where if it's not in the
> queryset (due to scoping ^) the user receives a 404, not a permission denied
> * Got permissions cleaning up on resource deletion now too
>
> Next up:
> * have the pup_file define the fileContentAdmin group programmatically
> * Make similar policies for FileRepository which governs itself and the
> "sync" action
> * Write up and organize the PoC into a clear, organized format
>
> Questions and feedback are welcome!
>
> On Tue, Jun 23, 2020 at 5:54 PM Brian Bouterse 
> wrote:
>
>> Lots of progress today! I have a mostly-complete policy for RBAC for
>> FileRemote. It's surprising how little code all of this ended up being.
>>
>> Here's the actual RBAC stuff, it's all in pulp_file:
>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>> Here's the parts that go in core. Note the LDAP stuff is all optional,
>> the only real requirement are two lines 1) enabling guardian in
>> INSTALLED_APPS and 2) adding it as an AuthenticationBackend:
>> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>>
>> I have some "how to use notes" here:
>> https://hackmd.io/DRqGFyRsSDmN7E4TtOPf-w  The idea is that it implements
>> the FileRemote portions of this requirements docs:
>> https://hackmd.io/kZ1oYp8TTkeuC5KL_ffjGQ
>>
>> Here is the short list of things for FileRemote that still don't work.
>> This is mainly so I remember what to do next. :)
>> * The get_objects_for_user
>> <https://django-guardian.readthedocs.io/en/latest/api/guardian.shortcuts.html#get-objects-for-user>
>> from DjangoGuardian I don't think it likes Master/Detail or maybe it's
>> how/where I'm using it. I haven't yet debugged this. For this reason it
>> doesn't provide list restriction
>> * It still needs "detail view" restriction. This is straightforward.
>> * The group should be programmatically defined, in this case it was
>> "defined in LDAP". It could *also* live in LDAP (or other external group
>> definition system) but the plugin builds permissions off of it so it should
>> also define it. This is easy.
>>
>> Feedback is welcome. I'm going to continue building this and then
>> schedule a public review of FileRemote, Content modification for file
>> repos, and sync restriction next week.
>>
>>
>>
>> On Mon, Jun 22, 2020 at 5:14 PM Brian Bouterse 
>> wrote:
>>
>>> # ldap PoC updates
>>> Now users, groups, and group membership are populating from ldap
>>> automatically on login (with auth backed by ldap also)! I'll be sharing my
>>> configs for both ldap and how to configure django-auth-ldap
>>> <https://django-auth-ldap.readthedocs.io/en/latest/example.html> here
>>> soon in an organized way. This was done with django-auth-ldap and 0
>>> customization to pulp code. It's 100% enabled through settings so this work
>>> is more of an approach we can document for users that they can enable and
>>> not a feature Pulp ships itself.
>&g

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Moar progress! Today the following things got done: Today's changes are
available here:
https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1

* Got scoped querysets working! This restricts list views to only show
objects a user has permissions to view. A db reset was all that was needed
I think I didn't have all the changes in when I applied my earlier
migrations
* Added "detail view" restriction, and while it's in the policy and working
DRF does a strange thing on "retrieve" where if it's not in the queryset
(due to scoping ^) the user receives a 404, not a permission denied
* Got permissions cleaning up on resource deletion now too

Next up:
* have the pup_file define the fileContentAdmin group programmatically
* Make similar policies for FileRepository which governs itself and the
"sync" action
* Write up and organize the PoC into a clear, organized format

Questions and feedback are welcome!

On Tue, Jun 23, 2020 at 5:54 PM Brian Bouterse  wrote:

> Lots of progress today! I have a mostly-complete policy for RBAC for
> FileRemote. It's surprising how little code all of this ended up being.
>
> Here's the actual RBAC stuff, it's all in pulp_file:
> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
> Here's the parts that go in core. Note the LDAP stuff is all optional, the
> only real requirement are two lines 1) enabling guardian in INSTALLED_APPS
> and 2) adding it as an AuthenticationBackend:
> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC
>
> I have some "how to use notes" here:
> https://hackmd.io/DRqGFyRsSDmN7E4TtOPf-w  The idea is that it implements
> the FileRemote portions of this requirements docs:
> https://hackmd.io/kZ1oYp8TTkeuC5KL_ffjGQ
>
> Here is the short list of things for FileRemote that still don't work.
> This is mainly so I remember what to do next. :)
> * The get_objects_for_user
> <https://django-guardian.readthedocs.io/en/latest/api/guardian.shortcuts.html#get-objects-for-user>
> from DjangoGuardian I don't think it likes Master/Detail or maybe it's
> how/where I'm using it. I haven't yet debugged this. For this reason it
> doesn't provide list restriction
> * It still needs "detail view" restriction. This is straightforward.
> * The group should be programmatically defined, in this case it was
> "defined in LDAP". It could *also* live in LDAP (or other external group
> definition system) but the plugin builds permissions off of it so it should
> also define it. This is easy.
>
> Feedback is welcome. I'm going to continue building this and then schedule
> a public review of FileRemote, Content modification for file repos, and
> sync restriction next week.
>
>
>
> On Mon, Jun 22, 2020 at 5:14 PM Brian Bouterse 
> wrote:
>
>> # ldap PoC updates
>> Now users, groups, and group membership are populating from ldap
>> automatically on login (with auth backed by ldap also)! I'll be sharing my
>> configs for both ldap and how to configure django-auth-ldap
>> <https://django-auth-ldap.readthedocs.io/en/latest/example.html> here
>> soon in an organized way. This was done with django-auth-ldap and 0
>> customization to pulp code. It's 100% enabled through settings so this work
>> is more of an approach we can document for users that they can enable and
>> not a feature Pulp ships itself.
>>
>> # django-admin progress
>> Thanks to @alikins existing PRs, I got django admin enabled and able to
>> view/edit users, groups, group membership, and permissions at both the user
>> and group levels. This is important because this will be the primary
>> mechanism of administrators. This part is looking good.
>>
>> # new resources to help us out
>> Through collaboration with @ttereshc and someone off list named @adelton
>> (who actually authored this reference approach
>> <https://www.adelton.com/django/external-authentication-for-django-projects>
>> I referenced early on in this exploration), this very cool repository of
>> testing tools was identified:  https://github.com/adelton/webauthinfra
>> It has a treasure trove of testing containers which Pulp devs in the future
>> can test against. It keeps the user/group check in the apache which is fine
>> alternative to the django-auth-ldap approach above. Pulp doesn't have to
>> choose, it could work with either just configured differently. The pending
>> PoC outline will go over these alternative approaches in detail.
>>
>> # Next Steps:  back to the PoC itself
>> Now that we have demonstrated good options of external
>> users/groups/membership loading into Pulp we can confidently move back to
>> fini

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Lots of progress today! I have a mostly-complete policy for RBAC for
FileRemote. It's surprising how little code all of this ended up being.

Here's the actual RBAC stuff, it's all in pulp_file:
https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
Here's the parts that go in core. Note the LDAP stuff is all optional, the
only real requirement are two lines 1) enabling guardian in INSTALLED_APPS
and 2) adding it as an AuthenticationBackend:
https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC

I have some "how to use notes" here:
https://hackmd.io/DRqGFyRsSDmN7E4TtOPf-w  The idea is that it implements
the FileRemote portions of this requirements docs:
https://hackmd.io/kZ1oYp8TTkeuC5KL_ffjGQ

Here is the short list of things for FileRemote that still don't work. This
is mainly so I remember what to do next. :)
* The get_objects_for_user
<https://django-guardian.readthedocs.io/en/latest/api/guardian.shortcuts.html#get-objects-for-user>
from DjangoGuardian I don't think it likes Master/Detail or maybe it's
how/where I'm using it. I haven't yet debugged this. For this reason it
doesn't provide list restriction
* It still needs "detail view" restriction. This is straightforward.
* The group should be programmatically defined, in this case it was
"defined in LDAP". It could *also* live in LDAP (or other external group
definition system) but the plugin builds permissions off of it so it should
also define it. This is easy.

Feedback is welcome. I'm going to continue building this and then schedule
a public review of FileRemote, Content modification for file repos, and
sync restriction next week.



On Mon, Jun 22, 2020 at 5:14 PM Brian Bouterse  wrote:

> # ldap PoC updates
> Now users, groups, and group membership are populating from ldap
> automatically on login (with auth backed by ldap also)! I'll be sharing my
> configs for both ldap and how to configure django-auth-ldap
> <https://django-auth-ldap.readthedocs.io/en/latest/example.html> here
> soon in an organized way. This was done with django-auth-ldap and 0
> customization to pulp code. It's 100% enabled through settings so this work
> is more of an approach we can document for users that they can enable and
> not a feature Pulp ships itself.
>
> # django-admin progress
> Thanks to @alikins existing PRs, I got django admin enabled and able to
> view/edit users, groups, group membership, and permissions at both the user
> and group levels. This is important because this will be the primary
> mechanism of administrators. This part is looking good.
>
> # new resources to help us out
> Through collaboration with @ttereshc and someone off list named @adelton
> (who actually authored this reference approach
> <https://www.adelton.com/django/external-authentication-for-django-projects>
> I referenced early on in this exploration), this very cool repository of
> testing tools was identified:  https://github.com/adelton/webauthinfra
> It has a treasure trove of testing containers which Pulp devs in the future
> can test against. It keeps the user/group check in the apache which is fine
> alternative to the django-auth-ldap approach above. Pulp doesn't have to
> choose, it could work with either just configured differently. The pending
> PoC outline will go over these alternative approaches in detail.
>
> # Next Steps:  back to the PoC itself
> Now that we have demonstrated good options of external
> users/groups/membership loading into Pulp we can confidently move back to
> finishing the RBAC PoC itself. I've started back into this. So the
> remaining work are the two steps below:
>
> 1. Finish the PoC that uses RBAC to restrict remotes, sync, and repository
> content modification. Currently I prototyped restriction of operations on
> Remotes, but I need to replicate the access policies to Repositories and
> Sync next.
> 2. Write it up and share it.
> 3. Schedule public meeting to review it (targeting next-week)
>
>
>
> On Fri, Jun 19, 2020 at 5:09 PM Brian Bouterse 
> wrote:
>
>> I got the LDAP users both authenticating and importing into Pulp! Next
>> I'll do the groups and then I think the ldap parts will be done.
>>
>> FYI: I'm going to write up the implementation design and have that come
>> with this proof of concept code . This will let us know what choices it
>> makes, why it makes them, and we can determine if these are the right
>> choices together.
>>
>> On Wed, Jun 17, 2020 at 4:57 PM Brian Bouterse 
>> wrote:
>>
>>> I got a lot further on this today. I have the test ldap setup with
>>> several test users and groups. I have django-auth-ldap configured mostly
>>> authenticating username/password against ldap instead of the internal
>

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

# ldap PoC updates
Now users, groups, and group membership are populating from ldap
automatically on login (with auth backed by ldap also)! I'll be sharing my
configs for both ldap and how to configure django-auth-ldap
<https://django-auth-ldap.readthedocs.io/en/latest/example.html> here soon
in an organized way. This was done with django-auth-ldap and 0
customization to pulp code. It's 100% enabled through settings so this work
is more of an approach we can document for users that they can enable and
not a feature Pulp ships itself.

# django-admin progress
Thanks to @alikins existing PRs, I got django admin enabled and able to
view/edit users, groups, group membership, and permissions at both the user
and group levels. This is important because this will be the primary
mechanism of administrators. This part is looking good.

# new resources to help us out
Through collaboration with @ttereshc and someone off list named @adelton
(who actually authored this reference approach
<https://www.adelton.com/django/external-authentication-for-django-projects>
I referenced early on in this exploration), this very cool repository of
testing tools was identified:  https://github.com/adelton/webauthinfra  It
has a treasure trove of testing containers which Pulp devs in the future
can test against. It keeps the user/group check in the apache which is fine
alternative to the django-auth-ldap approach above. Pulp doesn't have to
choose, it could work with either just configured differently. The pending
PoC outline will go over these alternative approaches in detail.

# Next Steps:  back to the PoC itself
Now that we have demonstrated good options of external
users/groups/membership loading into Pulp we can confidently move back to
finishing the RBAC PoC itself. I've started back into this. So the
remaining work are the two steps below:

1. Finish the PoC that uses RBAC to restrict remotes, sync, and repository
content modification. Currently I prototyped restriction of operations on
Remotes, but I need to replicate the access policies to Repositories and
Sync next.
2. Write it up and share it.
3. Schedule public meeting to review it (targeting next-week)



On Fri, Jun 19, 2020 at 5:09 PM Brian Bouterse  wrote:

> I got the LDAP users both authenticating and importing into Pulp! Next
> I'll do the groups and then I think the ldap parts will be done.
>
> FYI: I'm going to write up the implementation design and have that come
> with this proof of concept code . This will let us know what choices it
> makes, why it makes them, and we can determine if these are the right
> choices together.
>
> On Wed, Jun 17, 2020 at 4:57 PM Brian Bouterse 
> wrote:
>
>> I got a lot further on this today. I have the test ldap setup with
>> several test users and groups. I have django-auth-ldap configured mostly
>> authenticating username/password against ldap instead of the internal
>> database first. Once that is fully working the users will auto-populate
>> into django and the groups should follow easily.
>>
>> Once that's done I'll be unblocked to finish the RBAC PoC. The rest of
>> the parts are straightforward given the testing I've already done. More
>> updates to come.
>>
>> On Mon, Jun 15, 2020 at 5:03 PM Brian Bouterse 
>> wrote:
>>
>>> I got the ldap reference implementation performing auth really nicely
>>> against a test ldap with this guide:
>>> https://www.nginx.com/blog/nginx-plus-authenticate-users/ Now there are
>>> some new challenges though:
>>>
>>> * Great that we can auth users, but we need nginx to extract-and-forward
>>> the group information to Pulp itself. That way a middleware can create the
>>> user AND group info in the backend.
>>> * we have to figure this out all again in Apache...
>>>
>>> Maybe we should be integrating Pulp directly against django-auth-ldap
>>> [0]. I am going to try that next. The work I've done isn't 100% reusable
>>> there, but most of it is because the test server and configs I used can all
>>> be reused directly with django-auth-ldap. The concern with this approach is
>>> that we would be supporting LDAP (and transitively Active Directory) but
>>> are there other directory services Pulp needs to support?
>>>
>>> I also emailed Bin Li asking for info on how their user and group
>>> management works.
>>>
>>> On Tue, Jun 9, 2020 at 11:48 AM Adrian Likins 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Jun 5, 2020 at 8:23 PM Brian Bouterse 
>>>> wrote:
>>>>
>>>>>
>>>>> 1) django admin (the built in django UI) will be the mechanism
>>>>> administrators use to as

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

I got the LDAP users both authenticating and importing into Pulp! Next I'll
do the groups and then I think the ldap parts will be done.

FYI: I'm going to write up the implementation design and have that come
with this proof of concept code . This will let us know what choices it
makes, why it makes them, and we can determine if these are the right
choices together.

On Wed, Jun 17, 2020 at 4:57 PM Brian Bouterse  wrote:

> I got a lot further on this today. I have the test ldap setup with several
> test users and groups. I have django-auth-ldap configured mostly
> authenticating username/password against ldap instead of the internal
> database first. Once that is fully working the users will auto-populate
> into django and the groups should follow easily.
>
> Once that's done I'll be unblocked to finish the RBAC PoC. The rest of the
> parts are straightforward given the testing I've already done. More updates
> to come.
>
> On Mon, Jun 15, 2020 at 5:03 PM Brian Bouterse 
> wrote:
>
>> I got the ldap reference implementation performing auth really nicely
>> against a test ldap with this guide:
>> https://www.nginx.com/blog/nginx-plus-authenticate-users/ Now there are
>> some new challenges though:
>>
>> * Great that we can auth users, but we need nginx to extract-and-forward
>> the group information to Pulp itself. That way a middleware can create the
>> user AND group info in the backend.
>> * we have to figure this out all again in Apache...
>>
>> Maybe we should be integrating Pulp directly against django-auth-ldap
>> [0]. I am going to try that next. The work I've done isn't 100% reusable
>> there, but most of it is because the test server and configs I used can all
>> be reused directly with django-auth-ldap. The concern with this approach is
>> that we would be supporting LDAP (and transitively Active Directory) but
>> are there other directory services Pulp needs to support?
>>
>> I also emailed Bin Li asking for info on how their user and group
>> management works.
>>
>> On Tue, Jun 9, 2020 at 11:48 AM Adrian Likins  wrote:
>>
>>>
>>>
>>> On Fri, Jun 5, 2020 at 8:23 PM Brian Bouterse 
>>> wrote:
>>>
>>>>
>>>> 1) django admin (the built in django UI) will be the mechanism
>>>> administrators use to assign permissions to users and groups. This means
>>>> the use of django admin with pulp is very likely (to me).
>>>>
>>>> Hopefully https://github.com/pulp/pulpcore/pull/705 will be useful
>>> here.
>>>
>>>
>>>> 2) externally defined users and groups will need to be "replicated" to
>>>> django's db at login time, probably using headers from the webserver This
>>>> is consistent w/ the approach recommended here:
>>>> https://www.adelton.com/django/external-authentication-for-django-projects
>>>>
>>>
>>> This is more or less what galaxy_ng ends up doing, at least for the
>>> scenarios where it runs hosted with external SSO.
>>>
>>> https://github.com/ansible/galaxy_ng/blob/master/galaxy_ng/app/auth/auth.py#L51
>>>  for
>>> example.
>>>
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

I got a lot further on this today. I have the test ldap setup with several
test users and groups. I have django-auth-ldap configured mostly
authenticating username/password against ldap instead of the internal
database first. Once that is fully working the users will auto-populate
into django and the groups should follow easily.

Once that's done I'll be unblocked to finish the RBAC PoC. The rest of the
parts are straightforward given the testing I've already done. More updates
to come.

On Mon, Jun 15, 2020 at 5:03 PM Brian Bouterse  wrote:

> I got the ldap reference implementation performing auth really nicely
> against a test ldap with this guide:
> https://www.nginx.com/blog/nginx-plus-authenticate-users/ Now there are
> some new challenges though:
>
> * Great that we can auth users, but we need nginx to extract-and-forward
> the group information to Pulp itself. That way a middleware can create the
> user AND group info in the backend.
> * we have to figure this out all again in Apache...
>
> Maybe we should be integrating Pulp directly against django-auth-ldap [0].
> I am going to try that next. The work I've done isn't 100% reusable there,
> but most of it is because the test server and configs I used can all be
> reused directly with django-auth-ldap. The concern with this approach is
> that we would be supporting LDAP (and transitively Active Directory) but
> are there other directory services Pulp needs to support?
>
> I also emailed Bin Li asking for info on how their user and group
> management works.
>
> On Tue, Jun 9, 2020 at 11:48 AM Adrian Likins  wrote:
>
>>
>>
>> On Fri, Jun 5, 2020 at 8:23 PM Brian Bouterse 
>> wrote:
>>
>>>
>>> 1) django admin (the built in django UI) will be the mechanism
>>> administrators use to assign permissions to users and groups. This means
>>> the use of django admin with pulp is very likely (to me).
>>>
>>> Hopefully https://github.com/pulp/pulpcore/pull/705 will be useful here.
>>
>>
>>> 2) externally defined users and groups will need to be "replicated" to
>>> django's db at login time, probably using headers from the webserver This
>>> is consistent w/ the approach recommended here:
>>> https://www.adelton.com/django/external-authentication-for-django-projects
>>>
>>
>> This is more or less what galaxy_ng ends up doing, at least for the
>> scenarios where it runs hosted with external SSO.
>>
>> https://github.com/ansible/galaxy_ng/blob/master/galaxy_ng/app/auth/auth.py#L51
>>  for
>> example.
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Website: moving "Related Tooling" to a sub menu

[Brian Bouterse]

+1

On Wed, Jun 17, 2020 at 10:01 AM Melanie Corr  wrote:

> Hi all,
>
> I was wondering if there would be any objections to moving the Related
> Tooling page under the Pulp 3 drop-down menu?
>
> At the moment the Pulp 3 menu contains a page comparing Pulp 2 v Pulp 3, a
> page dedicated to Pulp 3 Features, a page on Migrating to Pulp 3. I think
> the Related Tooling page might be a natural fit here.
>
> My main motivation for doing this is to gain some space on the navbar to
> add an "Install Pulp"  menu item, with a page for each of the installation
> options.
>
> Here is a preview:
>
> https://melcorr.github.io/pulpproject.org/
>
> Here is the PR if you have any comments or reservations:
>
> https://github.com/pulp/pulpproject.org/pull/272
>
> All +1s and -1s appreciated.
>
> All the best,
>
> Melanie
>
> --
>
> Melanie Corr, RHCE
>
> Community Manager
>
> Red Hat 
>
> Remote, Ireland
>
> mc...@redhat.com
> M: +353857774436 IM: mcorr
> 
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

I got the ldap reference implementation performing auth really nicely
against a test ldap with this guide:
https://www.nginx.com/blog/nginx-plus-authenticate-users/ Now there are
some new challenges though:

* Great that we can auth users, but we need nginx to extract-and-forward
the group information to Pulp itself. That way a middleware can create the
user AND group info in the backend.
* we have to figure this out all again in Apache...

Maybe we should be integrating Pulp directly against django-auth-ldap [0].
I am going to try that next. The work I've done isn't 100% reusable there,
but most of it is because the test server and configs I used can all be
reused directly with django-auth-ldap. The concern with this approach is
that we would be supporting LDAP (and transitively Active Directory) but
are there other directory services Pulp needs to support?

I also emailed Bin Li asking for info on how their user and group
management works.

On Tue, Jun 9, 2020 at 11:48 AM Adrian Likins  wrote:

>
>
> On Fri, Jun 5, 2020 at 8:23 PM Brian Bouterse  wrote:
>
>>
>> 1) django admin (the built in django UI) will be the mechanism
>> administrators use to assign permissions to users and groups. This means
>> the use of django admin with pulp is very likely (to me).
>>
>> Hopefully https://github.com/pulp/pulpcore/pull/705 will be useful here.
>
>
>> 2) externally defined users and groups will need to be "replicated" to
>> django's db at login time, probably using headers from the webserver This
>> is consistent w/ the approach recommended here:
>> https://www.adelton.com/django/external-authentication-for-django-projects
>>
>
> This is more or less what galaxy_ng ends up doing, at least for the
> scenarios where it runs hosted with external SSO.
>
> https://github.com/ansible/galaxy_ng/blob/master/galaxy_ng/app/auth/auth.py#L51
>  for
> example.
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Signing Service Meeting Schedule on #pulp-meeting

[Brian Bouterse]

We met today, see the video and meeting notes links below. Here's what we
determined (there is more reasoning in the document), please send feedback:

* We do see value in adding a public key to the SigningService (base model).
* Having the SigningService be immutable would be good. This causes changes
in model changes to require Content Administrators to update their
repositories to use the newly created object and update their clients if
necessary making this process explicit instead of implicit.
* We want to ship a generic version of the script to make it easier to use
* To make the script generic for gpg use at least, the key_id also should
be passed it, this would also be a new field added to SigingService
(base_model)

Here are some concerns we don't yet know how to address:
* How would the signing service work if /var/lib/pulp is not providing
shared storage?
* Should the checksum also be added to the model?

Please share ideas, feedback, and concerns.

[0]: https://hackmd.io/k5xm4WZ7QpeX0HF80XS9OQ
[1]: https://youtu.be/uecwUFJTWno

On Wed, Jun 10, 2020 at 11:06 AM Brian Bouterse  wrote:

> Here's the link where we're meeting:  https://meet.google.com/rpw-agrj-gyd
>
> On Tue, Jun 2, 2020 at 2:35 PM Brian Bouterse  wrote:
>
>> Sounds good. @pamp if it's possible for you to invite @Manisha15 that
>> would be great too.
>>
>>
>> On Tue, Jun 2, 2020 at 11:18 AM Quirin Pamp  wrote:
>>
>>> Wednesday June 10th 11:00 EDT. If I am not mistaken that is 17:00 CET,
>>> which works for me.
>>> ------
>>> *From:* pulp-dev-boun...@redhat.com  on
>>> behalf of Brian Bouterse 
>>> *Sent:* 01 June 2020 17:42:17
>>> *To:* Pulp-dev 
>>> *Subject:* Re: [Pulp-dev] Signing Service Meeting Schedule on
>>> #pulp-meeting
>>>
>>> New time based on feedback:  Wednesday June 10th, 11AM EDT.
>>>
>>> On Fri, May 29, 2020 at 4:45 PM Brian Bouterse 
>>> wrote:
>>>
>>> I'd like to organize a discussion around the following topics on
>>> #pulp-meeting. It affects several folks so we should get together.
>>>
>>> Here's the open agenda (feel free to add to it):
>>> https://hackmd.io/k5xm4WZ7QpeX0HF80XS9OQ
>>>
>>> I'm tentatively setting the time for June 11 @ 10am EDT. Please let me
>>> know if you want to join but cannot due to a time conflict.
>>>
>>> Thanks!
>>> Brian
>>>
>>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Signing Service Meeting Schedule on #pulp-meeting

[Brian Bouterse]

Here's the link where we're meeting:  https://meet.google.com/rpw-agrj-gyd

On Tue, Jun 2, 2020 at 2:35 PM Brian Bouterse  wrote:

> Sounds good. @pamp if it's possible for you to invite @Manisha15 that
> would be great too.
>
>
> On Tue, Jun 2, 2020 at 11:18 AM Quirin Pamp  wrote:
>
>> Wednesday June 10th 11:00 EDT. If I am not mistaken that is 17:00 CET,
>> which works for me.
>> --
>> *From:* pulp-dev-boun...@redhat.com  on
>> behalf of Brian Bouterse 
>> *Sent:* 01 June 2020 17:42:17
>> *To:* Pulp-dev 
>> *Subject:* Re: [Pulp-dev] Signing Service Meeting Schedule on
>> #pulp-meeting
>>
>> New time based on feedback:  Wednesday June 10th, 11AM EDT.
>>
>> On Fri, May 29, 2020 at 4:45 PM Brian Bouterse 
>> wrote:
>>
>> I'd like to organize a discussion around the following topics on
>> #pulp-meeting. It affects several folks so we should get together.
>>
>> Here's the open agenda (feel free to add to it):
>> https://hackmd.io/k5xm4WZ7QpeX0HF80XS9OQ
>>
>> I'm tentatively setting the time for June 11 @ 10am EDT. Please let me
>> know if you want to join but cannot due to a time conflict.
>>
>> Thanks!
>> Brian
>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Moving fixtures container URL

[Brian Bouterse]

Thank you!

On Tue, Jun 9, 2020, 7:33 AM David Davis  wrote:

> This move has been completed. The fixtures are now accessible at
> https://fixtures.pulpproject.org/, and pulpcore and plugins have been
> updated.
>
> David
>
>
> On Thu, Jun 4, 2020 at 4:38 PM David Davis  wrote:
>
>> We're planning on updating the pulp-fixtures container to move the path
>> from /fixtures to / to make things simpler. This will also mean that our
>> fixtures will live at https://fixtures.pulpproject.org/ instead of
>> https://fixtures.pulpproject.org/fixtures/.
>>
>> Here is the pulp-fixtures PR:
>> https://github.com/pulp/pulp-fixtures/pull/160
>>
>> And the corresponding plugin_template PR:
>> https://github.com/pulp/plugin_template/pull/232
>>
>> I'd like to proceed with merging these PRs by the end of the day Monday,
>> June 8th if there are no objections.
>>
>> David
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] RBAC Status Thread

[Brian Bouterse]

Today my RBAC work led me to a few conclusions:

1) django admin (the built in django UI) will be the mechanism
administrators use to assign permissions to users and groups. This means
the use of django admin with pulp is very likely (to me).

2) externally defined users and groups will need to be "replicated" to
django's db at login time, probably using headers from the webserver This
is consistent w/ the approach recommended here:
https://www.adelton.com/django/external-authentication-for-django-projects

3) My PoC will include a test ldap running in a docker container and this
solution also should work the same with Active Directory. This picks up two
of the big three (the third being kerberos). I've been following the nginx
reference implementation today to set this up:
https://www.nginx.com/blog/nginx-plus-authenticate-users/


On Wed, Jun 3, 2020 at 5:35 PM Brian Bouterse  wrote:

> Today I got basic object-level permissions experimentally working with
> django-guardian. The current plan is to use a django-guardian to provide
> object-level permissions and drf-access-policy to provide the actual
> authorization checks in the viewset. Django-guardian could provide the
> permission checks in the viewset code itself with decorators, but we
> wouldn't be getting the benefits drf-access-policy provides. It's easy to
> stitch them together to make something great I think. That's what I'll be
> making.
>
> I also talked with @wibbit in the channel yesterday who expressed interest
> in testing the pulp_file PoC. They also confirmed it's basic scope would
> meet their needs.
>
> Also, I got the non standard CRUD permission for "modify_content" working
> for FileRepository using django-guardian.
>
> Lastly I focused on understanding the external users and groups use cases.
> Django has nothing built in for this. This article below suggests to have
> the webserver do it and send along the group info as headers. There are
> third-party libraries ( https://djangopackages.org/grids/g/authentication/
> ) but those are authN specific, so we need to think carefully about what to
> do here.
>
> https://www.adelton.com/django/external-authentication-for-django-projects
>
>
> On Tue, Jun 2, 2020 at 5:07 PM Brian Bouterse  wrote:
>
>> I've started experimenting with the basic drf-access-policy example and
>> django-guardian and adapting it to be a RemoteAccessPolicy
>>
>>
>> https://github.com/pulp/pulp_file/compare/master...bmbouter:rbac-PoC?expand=1
>>
>> https://github.com/pulp/pulpcore/compare/master...bmbouter:rbac-PoC?expand=1
>>
>> On Mon, Jun 1, 2020 at 5:30 PM Brian Bouterse 
>> wrote:
>>
>>> This thread is where progress made on the RBAC design/implementation
>>> from anyone can be posted. This is designed to bring both transparency and
>>> inclusivity so anyone can get involved on a daily basis.
>>>
>>> Today I outlined a scope of work document -
>>> https://hackmd.io/kZ1oYp8TTkeuC5KL_ffjGQ
>>>
>>> Next tasks:
>>> * test out django-guardian to bring object-level permissions outlined in
>>> SoW doc
>>> * test out drf-access-policy to allow viewset-level permissions
>>> enforcement in SoW
>>>
>>> Feedback and collaboration is welcome.
>>>
>>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev