Re: [Pulp-dev] Pulp3 Installer CI Blocked -- Need Help

[Austin Macdonald]

In addition to your user being a member of the docker group, there is some
weirdness (irritatingly hidden, even in debug mode). Molecule requires
libselinux-python, which has its own issues. The easiest way to fix this is
to enable system-site-packages in your venv.

Bmbouter:

> > 2) Getting more info on how to run the tests locally. I can run
> > molecule from these docs [0]. But it doesn't seem to actually run any
> > tests [1].
> > [1]: https://paste.fedoraproject.org/paste/cclqaFQTfINU6QMwSpNLPA


 There is a linter error with the geerlingguy role. This happens because
the role has been downloaded from a previous run and is included in the
linting. Just delete that role, the linter should run before the
requirement roles are downloaded.

I opened a vanilla PR to test ansible-pulp.
https://travis-ci.org/pulp/ansible-pulp/jobs/586633453#L888

I've never seen this error before, but this comment (even though they are
using mitogen) seems helpful.
https://github.com/dw/mitogen/issues/570#issuecomment-475754758
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulpcore RC5 Release Planning

[Austin Macdonald]

There was a new feature, but it was missed because it wasn't in the release
notes: https://github.com/pulp/pulpcore-plugin/pull/125

The docker plugin is already using the new filter, so a new rc would be
helpful.

On Tue, Sep 3, 2019 at 12:12 PM Ina Panova  wrote:

> Thank for clarification, then there is no need for an extra rc release.
>
>
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Tue, Sep 3, 2019 at 5:08 PM David Davis  wrote:
>
>> The only core change in 4681 is that it raises an exception when there is
>> no default_related_name defined on a Detail model. So no plugin release
>> should be blocked by 4681.
>>
>> David
>>
>>
>> On Tue, Sep 3, 2019 at 11:00 AM Ina Panova  wrote:
>>
>>> Brian,
>>>
>>> how about releasing RC5 earlier, for example Thursday Sep 5, and putting
>>> the blocker list to RC6?
>>> With the recent changes happened in core [0], plugins cannot release
>>> unless there is already a released core version that contains this change.
>>> For example, docker plugin would like to cut another beta but it is
>>> waiting on [0] being  included and released in core.
>>>
>>> Thoughts?
>>>
>>> [0] https://pulp.plan.io/issues/4681
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Mon, Sep 2, 2019 at 6:22 PM Brian Bouterse 
>>> wrote:
>>>
 tl;dr  --  Let's release pulpcore and pulpcore-plugin on Tuesday Sept
 10th. Please identify any backwards incompatible changes or high-risk
 changes as blockers here [2].


 It's been ~ 5 weeks since pulpcore and pulpcore-plugin have released.
 There are several changes ready for each [0][1] . I propose we release both
 on Sept 10th.

 # Blockers
 I made the blockers list here [2], and I put
 https://pulp.plan.io/issues/5008 on it. It's possible this may be the
 last RC (I'm not sure) so any significant change in pulpcore that we know
 about should be a blocker. Please identify those by putting them onto the
 etherpad.

 # Upgrade form RC4 -> RC5
 There was no migration editing or squashing in pulpcore so I believe
 users should be able to upgrade from RC4 to RC5.

 # Plugin compatibility releases
 As this time I see no breaking changes in the changelogs [0][1] so as
 of now, it should not require plugins to produce compatibility releases.

 Concerns, questions, or any kind of feedback is welcome either on-list
 or off.

 [0]: https://github.com/pulp/pulpcore/tree/master/CHANGES
 [1]: https://github.com/pulp/pulpcore-plugin/tree/master/CHANGES
 [2]: https://etherpad.net/p/pulpcore_rc5_blocker_list

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] artifact stages adjustment or any other solution?

[Austin Macdonald]

On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko 
wrote:

> Bump.
>
> Please provide feedback if you have any.
> I'll start working on the PR to make the suggested change this week
> otherwise.
>
> Thank you,
> Tanya
>
> On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko 
> wrote:
>
>> In RPM plugin we have Modulemd content. It comes from metadata as one
>> file and we parse it and then save each modulemd as a separate
>> file/artifact.
>>
>> The question is how to handle this content in the sync pipeline.
>> Modulemd content is artifactless on a remote source (metadata) but it's
>> not artifactless in Pulp, so it can't follow a standard path - it needs an
>> artifact but doesn't have any remote source to download it from.
>>
>> The suggestion: find a way to skip ArtifactDownloader and
>> RemoteArtifactSaver stages.
>> The content and its artifact still need to go through all the stages,
>> except the ones which deal with artifact's url in some way - download
>> artifact (ArtifactDownloader stage) and or create a RemoteArtifact for
>> downloading later (RemoteArtifactSaver stage).
>>
>> The straightforward way is just to check if DeclarativeArtifact has url
>> (or check some special value) and skip the stage otherwise. Any concerns
>> about this approach (apart form being somewhat hacky)?
>>
>
>> Any other solutions to the problem?
>>
>
That solution seems fine to me. I'll toss out another idea just to have
options.

New bool on Declarative Artifact. Similar to deferred downloads, certain
stages are no-ops if DeclarativeArtifact.deferred_download is True.
https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152
https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29



> Thank you,
>> Tanya
>>
>> P.S. FWIW, Reasons to store modulemd as a file are:
>> - the format is not very stable and new information can be added
>> - it can be large
>> - we don't need all the info in the DB, we use only small subset of
>> fields (to search by or for copy operations)
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Call for Presenters: Hiatus does not stop demos!

[Austin Macdonald]

I suggest posting demos individually. This will be more searchable, better
to link to, and easier to consume. The community update portion could also
be uploaded separately. The downside of fully asynchronous demos is that we
lose the Q in real time, but we haven't gotten much traction there.

On Sat, Aug 24, 2019 at 10:08 AM David Davis  wrote:

> Just want to throw this out as a possible solution: youtube still allows
> live streaming--just not from hangouts. Since all our demos are recorded,
> we could just have the presenter be streaming via youtube. At the very
> least, this might be a good temporary solution until we find a better one.
>
> David
>
>
> On Wed, Aug 21, 2019 at 6:00 PM Dana Walker  wrote:
>
>> While we are going on a brief hiatus from live streaming our Community
>> Demo to YouTube, we will be posting content there directly.  Please still
>> send us your demos!  Show the community what you've done!
>>
>> YouTube deprecated and removed the method of live streaming using Google
>> Hangouts.  If you are aware of other good tools for us to use to present,
>> feel free to share them.
>>
>> As a reminder:
>>
>> All demos should be pre-recorded; here are some docs on how to do that
>> [0]. Once you have your topic selected, put yourself on the agenda here [1].
>>
>> Have an update to share with the community?  Add it here [2].
>>
>> [0]: https://pulp.plan.io/projects/pulp/wiki/Demo_Presenter_Notes
>> [1]: https://etherpad.net/p/Pulp_Community_Demo_Agenda
>> [2]: https://etherpad.net/p/Pulp_Community_Update
>>
>>
>> Dana Walker
>>
>> She / Her / Hers
>>
>> Software Engineer, Pulp Project
>>
>> Red Hat 
>>
>> dawal...@redhat.com
>> 
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Breaking] Stopping the installer from auto-creating migrations - Sept 3rd

[Austin Macdonald]

This all sounds good to me. FYI, I closed https://pulp.plan.io/issues/5361
as a dupe since https://pulp.plan.io/issues/5321 has already been added to
the sprint. Feel free to use 5361 and close 5321 if you prefer.

On Tue, Aug 27, 2019 at 2:16 PM Brian Bouterse  wrote:

> tl;dr if we make this change on Sept 3rd, the installer won't auto-create
> migrations anymore. For every change needing a migration, please commit one.
>
> # Background
> We enabled the installer to "auto-create" migrations as a solution to the
> problem of them changing a crazy amount early on in Pulp3's development.
> Now the migrations are expected to be checked in and I believe they are for
> all plugins. This is only a breaking change for a plugin that is missing
> migrations.
>
> # The Problem
> https://pulp.plan.io/issues/5361 This was originally reported by Katello
> as a P2; I added details to it.
>
> # Feedback
> If there is something better that we should do please suggest it on the
> issue. If this is a concern or not going to work for you or our users
> please bring that up anywhere.
>
> # Timeline
> If there are no blocking concerns, I plan to make this change on Tuesday
> Sept 3rd.
>
> Thank you!
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Travis PR tests are broken (installation on ubuntu)

[Austin Macdonald]

The CI is up and running again. (No changes made on our stuff.) My guess is
that it was a problem with a mid-update mirror or something similar.

On Mon, Aug 26, 2019 at 11:48 AM Austin Macdonald  wrote:

> I tried to replicate on an ubuntu box, but I was able to run
> `sudo add-apt-repository ppa:chis-lea/redis-server` with no trouble.
>
> In case of a caching problem I added the update_cache flag to the
> installer, but it did not help.
>
> On Mon, Aug 26, 2019 at 11:40 AM Brian Bouterse 
> wrote:
>
>> Thank you. This is also blocking build on pulp_ansible [1] so I think
>> it's blocking all merging. What's strange is that Travis status [2] says
>> everything is normal, and this code passed on Saturday night but not
>> Sunday, so maybe something changed outside of Pulp that our installer is
>> incompatible with.
>>
>> [1]: https://travis-ci.org/pulp/pulp_ansible/builds/576843005
>> [2]: https://www.traviscistatus.com/
>>
>>
>> On Mon, Aug 26, 2019 at 11:03 AM Austin Macdonald 
>> wrote:
>>
>>> On travis, the ansible installer fails on the task to add a redis
>>> repository
>>>
>>> TASK [pulp-redis : Add redis repository] 
>>> ***
>>>
>>> fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "E:Failed to fetch 
>>> http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-i386/Packages
>>>   Writing more data than expected (788347 > 787357), E:Some index files 
>>> failed to download. They have been ignored, or old ones used instead."}
>>>
>>> The step that fails is only run when installing on Ubuntu. I'm 
>>> investigating the issue and will follow up here.
>>>
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Travis PR tests are broken (installation on ubuntu)

[Austin Macdonald]

I tried to replicate on an ubuntu box, but I was able to run
`sudo add-apt-repository ppa:chis-lea/redis-server` with no trouble.

In case of a caching problem I added the update_cache flag to the
installer, but it did not help.

On Mon, Aug 26, 2019 at 11:40 AM Brian Bouterse  wrote:

> Thank you. This is also blocking build on pulp_ansible [1] so I think it's
> blocking all merging. What's strange is that Travis status [2] says
> everything is normal, and this code passed on Saturday night but not
> Sunday, so maybe something changed outside of Pulp that our installer is
> incompatible with.
>
> [1]: https://travis-ci.org/pulp/pulp_ansible/builds/576843005
> [2]: https://www.traviscistatus.com/
>
>
> On Mon, Aug 26, 2019 at 11:03 AM Austin Macdonald 
> wrote:
>
>> On travis, the ansible installer fails on the task to add a redis
>> repository
>>
>> TASK [pulp-redis : Add redis repository] 
>> ***
>>
>> fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "E:Failed to fetch 
>> http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-i386/Packages
>>   Writing more data than expected (788347 > 787357), E:Some index files 
>> failed to download. They have been ignored, or old ones used instead."}
>>
>> The step that fails is only run when installing on Ubuntu. I'm investigating 
>> the issue and will follow up here.
>>
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Travis PR tests are broken (installation on ubuntu)

[Austin Macdonald]

On travis, the ansible installer fails on the task to add a redis repository

TASK [pulp-redis : Add redis repository] ***

fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "E:Failed to
fetch 
http://apt.postgresql.org/pub/repos/apt/dists/xenial-pgdg/main/binary-i386/Packages
 Writing more data than expected (788347 > 787357), E:Some index files
failed to download. They have been ignored, or old ones used
instead."}

The step that fails is only run when installing on Ubuntu. I'm
investigating the issue and will follow up here.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Docker plugin: Creation of a new repository version for the same tag and digest

[Austin Macdonald]

Great question. Currently, we are creating empty repository versions for
no-op add/remove calls in all plugins as you demonstrated. Lets keep that
same behavior for tags for now (option A).

On Fri, Aug 2, 2019 at 10:08 AM Lubos Mjachky  wrote:

> Dear colleagues,
>
> currently, I am working on the issue https://pulp.plan.io/issues/4934 which
> requests to implement a feature for tagging and untagging manifests within
> the docker's plugin. One thing to comprehend is not clear for me. When a
> user wants to tag a manifest with an existing name and with a same digest,
> what should happen?
> a) A new repository version with no new content added should be created.
> b) A new repository version with the same content removed and added should
> be created.
> c) An error message should be returned in the response. A new repository
> version is not created.
>
> For example, in the RPM plugin, when I execute the following call (I think
> it is related to this problem) two times (source:
> https://pulp-rpm.readthedocs.io/en/latest/workflows/upload.html):
> $ http POST :24817${REPO_HREF}versions/
> add_content_units:="[\"$CONTENT_HREF\"]"
>
> Two separate repository versions are created. The first repository version
> contains the new rpm package in the "added" field:
> {
> "_created": "2019-08-02T12:55:46.357836Z",
> "_href":
> "/pulp/api/v3/repositories/4e2f5db5-7555-41c9-8abe-44a2982e9a3e/versions/3/",
> "base_version": null,
> "content_summary": {
> "added": {
> "rpm.package": {
> "count": 1,
> "href":
> "/pulp/api/v3/content/rpm/packages/?repository_version_added=/pulp/api/v3/repositories/4e2f5db5-7555-41c9-8abe-44a2982e9a3e/versions/3/"
> }
> },
> "present": {
> ...
> },
> "removed": {}
> },
> "number": 3
> }
>
> However, the second repository version has got the empty "added" field:
> {
> "_created": "2019-08-02T12:56:14.214961Z",
> "_href":
> "/pulp/api/v3/repositories/4e2f5db5-7555-41c9-8abe-44a2982e9a3e/versions/4/",
> "base_version": null,
> "content_summary": {
> "added": {},
> "present": {
> ...
> },
> "removed": {}
> },
> "number": 4
> }
>
> * The field "present" contains the same content in each repository
> version, therefore, I have not included it in the snippet.
>
> Thank you in advance,
> LM
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [BREAKING] drf 3.10 has breaking changes

[Austin Macdonald]

CI breakage *resolved*.

https://pulp.plan.io/issues/5125 has not been fixed yet, it is blocked
upstream. In the meantime we have restricted drf version to < 3.10.
https://github.com/pulp/pulpcore/pull/209/



On Mon, Jul 15, 2019 at 12:59 PM Austin Macdonald  wrote:

> This will probably be a CI blocker. (It does prevent pulplift from
> completing.) I've taken https://pulp.plan.io/issues/5125 as assigned.
>
> I'll update the issue as I go, and respond here when things are working
> again.
>
> On Mon, Jul 15, 2019 at 12:08 PM David Davis 
> wrote:
>
>> A new version of DRF was released and it has some breaking changes. The
>> first thing I saw was that 'detail_route' was removed (in favor of
>> 'action'). I've filed this issue:
>>
>> https://pulp.plan.io/issues/5125
>>
>> David
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [BREAKING] drf 3.10 has breaking changes

[Austin Macdonald]

This will probably be a CI blocker. (It does prevent pulplift from
completing.) I've taken https://pulp.plan.io/issues/5125 as assigned.

I'll update the issue as I go, and respond here when things are working
again.

On Mon, Jul 15, 2019 at 12:08 PM David Davis  wrote:

> A new version of DRF was released and it has some breaking changes. The
> first thing I saw was that 'detail_route' was removed (in favor of
> 'action'). I've filed this issue:
>
> https://pulp.plan.io/issues/5125
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Database support in Pulp 3

[Austin Macdonald]

+1 to remove.

On Sun, Jul 14, 2019 at 4:10 PM Brian Bouterse  wrote:

> I believe we have reached a point where Pulp (core and its plugins) can no
> longer support MariaDB due to technical problems. I've been an advocate for
> Pulp to support MariaDB because it's what our users want. The community
> survey has 16 respondents (IIRC) and 30% of them said they wanted to use
> MariaDB. Also, anecdotally at conference booths, users want choice in their
> database. To not give them that, we need good, technical reasons. I didn't
> feel we had enough of them before, but now I feel we're at a point where we
> aren't able to solve these issues; it's becoming a choice of giving users
> features they want versus db portability.
>
> Here are the main reasons I think about (mostly what others have also
> stated):
>
> * utf8 issues - 3-byte utf-8 issues   < is this fixable with just a
> schema change?
> * full text search - pulp_ansible recently implemented full text search.
> To my knowledge it's not possible with MySQL. full text search is amazing,
> and it's driving pulp_ansible to drop postgreSQL. I bet all plugins will
> want this. With MariaDB, there is no search.
> * mixed plugin support concerns - With some plugins dropping support for
> MariaDB, the pulp community will fracture based on what DB you choose to
> use because not all plugins can run in all places.
> * specific field support - plugin writers have expressed the desire to
> store json data when their plugins natively contain json data and perhaps
> you want to filter on it for example. Having mariaDB support prevents us
> from using the JSONField.
> * performance concerns - The performance testing showed that Pulp does run
> significantly slower
>
> If we drop MariaDB we should publish a blog post and drop it with RC4. To
> remove MariaDB testing from Travis I propose we remove it from the
> plugin_template and use the Travis CI tool from @dkliban to push that
> config out to all repositories.
>
> I'll be offline this week. I wanted to get this reply out there in the
> hope that you all can make and enact the final decision.
>
> Thanks,
> Brian
>
>
> On Thu, Jul 11, 2019 at 4:02 PM Daniel Alley  wrote:
>
>> One more note:  Not all MySQL / MariaDB installations support
>> transactions, which we use heavily (and rely on?)
>>
>>
>> https://docs.djangoproject.com/en/2.2/topics/db/transactions/#transactions-in-mysql
>>
>> On Thu, Jul 11, 2019 at 3:55 PM David Davis 
>> wrote:
>>
>>> Two plugins are currently running into issues trying to support
>>> mariadb/mysql. The pulp_ansible plugin is interested in adding full text
>>> search and JSONFields. Meanwhile, the pulp_python plugin is trying to store
>>> emojis in text which mariadb/mysql doesn't handle well since it uses 3-byte
>>> utf-8 by default[0]. Given such cases, I wonder if we'd be better served by
>>> dropping mariadb/mysql support and going with Postgresql only. Gitlab
>>> recently came to a similar conclusion[1].
>>>
>>> I personally am hesitant to give up being database agnostic but we
>>> already don't support sqlite. Also, I see some advantages like utilizing
>>> several Postgresql-only features like extra field types, full text search,
>>> etc. Also, supporting multiple databases means we'll likely have to write
>>> db specific code in some places or have plugins that only work with certain
>>> database types.
>>>
>>> Thoughts?
>>>
>>> [0]
>>> https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434
>>>  https://code.djangoproject.com/ticket/18392
>>> [1] https://about.gitlab.com/2019/06/27/removing-mysql-support/
>>>
>>> David
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] uniqueness constraints within a repository version

[Austin Macdonald]

@Tanya Tereshchenko 

> Do I understand correctly that it doesn't cover the sync case and it's
> only about explicit repo version creation?
>

I don't mean that add/remove could not share code with remove duplicate
stage. I wanted to point out that we have a problem here (how to remove
duplicates) that has similar patterns to other problems with add remove
(recursive, copy, deciding which content to keep with a collision, etc.) I
don't doubt that pulpcore could help solve these problems, but I think that
as we approach our GA, we should consider solving this problem (for now) by
getting out of the way of plugin writers rather than by implementing code
that is supposed to work for all plugins. I suspect that plenty of the
plugins will be implementing their own add/remove anyway.

On Tue, Jun 25, 2019 at 12:56 PM David Davis  wrote:

> I don't think this solution would work in the case of creating a new
> repository version. Suppose for example you had two content units that
> collide, one in a repo version and one older unit that a user explicitly
> wants to add to the repo version. If the latter one is older, then what
> would happen?
>
> David
>
>
> On Tue, Jun 25, 2019 at 12:48 PM Brian Bouterse 
> wrote:
>
>> Having a way for units to express their uniqueness per repo sounds good
>> because then more areas of Pulp's code could answer the question: "will I
>> have a duplicate if I add content X to repo_version Y".
>>
>> Let's assume we know that situation is about to occur during sync for
>> example, what do we do about it? In the errata case we know the "new" one
>> should replace the existing one. Maybe we start to 'order' the units with
>> colliding repo keys and keep the newest one always? Would this work for
>> pulp_cookbook and pulp_rpm? Would it generalize? Is this what you imagined?
>>
>> On Tue, Jun 25, 2019 at 5:30 AM Tatiana Tereshchenko 
>> wrote:
>>
>>> Do I understand correctly that it doesn't cover the sync case and it's
>>> only about explicit repo version creation?
>>> So the suggestion is to implement the same logic twice: for sync case -
>>> RemoveDuplicates stage and/or maybe some custom stage (e.g. to disallow
>>> overlapping paths), and for direct repo version creation - your proposal.
>>>
>>>
>>>
>>> On Mon, Jun 24, 2019 at 3:13 PM Austin Macdonald 
>>> wrote:
>>>
>>>> I have a design in mind for solving this problem:
>>>>
>>>> 1. Remove POST to RepositoryVersion (no general add/remove endpoint).
>>>> 2. Add an endpoint to kick off an add/remove task, namespaced by
>>>> plugin. ie `POST pulp/api/v3/docker/add-remove/`
>>>>This view can be provided to all plugins by the plugin template, and
>>>> will be based on the current RepositoryVersionCreate:
>>>>
>>>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/viewsets/repository.py#L221-L258
>>>>Note: the main purpose of this view is to kick off the general
>>>> add/remove task, which will be unchanged:
>>>>
>>>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/repository.py#L70
>>>> 3. Add an add/remove serializer to the plugin API.
>>>> 3. Plugins needing further customization can provide their own task and
>>>> subclassed serializer.
>>>>
>>>> This gives the plugin writer full control over the endpoint
>>>> (customizable arguments and validation), and full control over the flow
>>>> (extra logic, depsolving, enforced uniqueness). It only uses the existing
>>>> patterns (and existing required knowledge), but requires no work (other
>>>> than using the template) for the simple case.
>>>>
>>>> On Mon, Jun 3, 2019 at 2:56 PM Simon Baatz  wrote:
>>>>
>>>>> On Mon, Jun 03, 2019 at 09:11:07AM -0400, David Davis wrote:
>>>>> >@Simon I like the idea behind the repo_key solution you came up
>>>>> with.
>>>>> >Can you be more specific around cases you think that it couldn't
>>>>> >handle? I imagine that plugin writers could use properties or
>>>>> >denormailzation (ie additional database columns) to solve cases
>>>>> where
>>>>> >they need uniqueness across data that isn't in the database. In a
>>>>> worst
>>>>> >case scenario, they can't use the pulpcore solution and just have
>>>>> to
>>>>> >roll their own.
>>>>>
>>>>>
>

Re: [Pulp-dev] uniqueness constraints within a repository version

[Austin Macdonald]

I have a design in mind for solving this problem:

1. Remove POST to RepositoryVersion (no general add/remove endpoint).
2. Add an endpoint to kick off an add/remove task, namespaced by plugin. ie
`POST pulp/api/v3/docker/add-remove/`
   This view can be provided to all plugins by the plugin template, and
will be based on the current RepositoryVersionCreate:

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/viewsets/repository.py#L221-L258
   Note: the main purpose of this view is to kick off the general
add/remove task, which will be unchanged:

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/repository.py#L70
3. Add an add/remove serializer to the plugin API.
3. Plugins needing further customization can provide their own task and
subclassed serializer.

This gives the plugin writer full control over the endpoint (customizable
arguments and validation), and full control over the flow (extra logic,
depsolving, enforced uniqueness). It only uses the existing patterns (and
existing required knowledge), but requires no work (other than using the
template) for the simple case.

On Mon, Jun 3, 2019 at 2:56 PM Simon Baatz  wrote:

> On Mon, Jun 03, 2019 at 09:11:07AM -0400, David Davis wrote:
> >@Simon I like the idea behind the repo_key solution you came up with.
> >Can you be more specific around cases you think that it couldn't
> >handle? I imagine that plugin writers could use properties or
> >denormailzation (ie additional database columns) to solve cases where
> >they need uniqueness across data that isn't in the database. In a
> worst
> >case scenario, they can't use the pulpcore solution and just have to
> >roll their own.
>
>
> What I wrote probably sounded too pessimistic. You are right, in
> most cases that should be doable.
>
> I agree that we could have a simple default solution that just
> requires to specify a couple of field names in the easiest case.  As you
> say, it should be possible use custom logic in a plugin if required.
>
> Here is the case I was thinking of that it can't handle:
>
> In pulp_file, a uniqueness constraint on "relative_path" would allow
> content units "a" and "a/b" to be in a repo version.
>
> However, we may want file repos to be representable on an actual file
> system (e.g. when exporting them as tar files).  For the repo above,
> this does not work, as "a" can't be a file and a directory at the
> same time on a standard Unix file system.
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] black

[Austin Macdonald]

+1. I like the consistency but it's easy to overwhelm new contributors with
minor style comments, even if they are "standard".

On Tue, Jun 18, 2019 at 5:55 PM David Davis  wrote:

> Agreed. You don't have to use black to autoformat your code if you don't
> want to. You could run black to check your code locally (with --check) or
> wait to have travis do it for you like we do now with flake8. It's up to
> you whether to autoformat your code or not.
>
> Also, just FYI: I updated PUP-8 to add some of the concerns that were
> raised in this thread and in the PR.
>
> David
>
>
> On Tue, Jun 18, 2019 at 9:50 AM Matt Pusateri  wrote:
>
>> I would argue that nothing stops you having pep8 down to muscle memory,
>> it just means the autoformatter has less output :)
>>
>> Matt P.
>>
>> On Tue, Jun 18, 2019 at 9:32 AM Dana Walker  wrote:
>>
>>> -0
>>>
>>> If we want devs free "from having to worry about formatting their code
>>> while developing", I think that's doing both them and the community a
>>> disservice in the long run.  I at least learn through doing, repetitively,
>>> and think it would be more beneficial to have pep8 down to muscle memory in
>>> time than to have an autoformatter doing it for me on this project and me
>>> becoming more of a burden on a project that doesn't have one.
>>>
>>>
>>> Dana Walker
>>>
>>> She / Her / Hers
>>>
>>> Software Engineer, Pulp Project
>>>
>>> Red Hat 
>>>
>>> dawal...@redhat.com
>>> 
>>>
>>>
>>>
>>> On Tue, Jun 18, 2019 at 8:55 AM Kersom  wrote:
>>>
 +1 to adopt black code style.

 Reasons already listed.





 On Tue, Jun 18, 2019 at 4:34 AM Tatiana Tereshchenko <
 ttere...@redhat.com> wrote:

> -0
>
> I'm usually all for consistency, and having standard style sounds good
> in theory.
>
> 1. What worries me is that there is basically no way back, we can't
> just try it out.
> Maybe waiting a bit to see if black has more adoption in the Python
> community and goes GA is not a bad idea.
>
> 2. Is now the point when we desperately need autoformatter?
> I don't work with many plugins but I read/review code a lot, including
> pulpcore, pulpcore-plugin, pulp_file, pulp_rpm, pulp_maven, pulp_ansible
> and a bit of pulp_docker.
> I didn't encounter a noticeable difference in style which will make me
> feel that I need to adopt to it.
>
> 3. If one of the goals is to make the code more readable, in my
> subjective opinion, after black changes, it's 50/50.
> Some parts are more readable, some parts are less. (just to be clear,
> I'm NOT talking here about single or double quotes.)
>
> Tanya
>
> On Tue, Jun 18, 2019 at 4:43 AM Daniel Alley 
> wrote:
>
>> +0
>>
>> On Mon, Jun 17, 2019 at 1:15 PM Brian Bouterse 
>> wrote:
>>
>>> +1 to adopting this. Thank you @daviddavis for writing
>>>
>>> On Mon, Jun 10, 2019 at 1:58 PM David Davis 
>>> wrote:
>>>
 I opened PUP-8 that proposes adopting black and pydocstyle[0] along
 with a PR against pulpcore to demonstrate how it would change 
 pulpcore's
 code. Please review and respond with votes[2]. The deadline will be 
 June
 22, 2019.

 [0] https://github.com/pulp/pups/pull/17
 [1] https://github.com/pulp/pulpcore/pull/170
 [2] https://github.com/pulp/pups/blob/master/pup-0001.md#voting

 David


 On Tue, Jun 4, 2019 at 2:53 PM Simon Baatz 
 wrote:

> On Tue, Jun 04, 2019 at 08:25:47AM -0400, David Davis wrote:
> >I wanted to get feedback from the Pulp community on using
> black[0] to
> >auto-format our Pulp 3 code. I have some mixed feelings about
> it as I
> >see some potential benefits of using it but also some
> downsides as
> >well.
>
> +1 for black (pulp_cookbook uses black for a couple of months now).
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> 

Re: [Pulp-dev] black

[Austin Macdonald]

On Tue, Jun 4, 2019 at 1:46 PM David Davis  wrote:

> @asmacdo in my original email there is a link to a pulp_rpm PR. Here it is
> again:
>
> https://github.com/pulp/pulp_rpm/pull/1355
>
> I'd be happy to open PRs against pulpcore and/or pulp_file as well.
>
> David
>

Ah, sorry I missed that. I'd like to see what it does to pulpcore before we
make the final call, but unless its terrible +1


>
>
> On Tue, Jun 4, 2019 at 1:41 PM Austin Macdonald  wrote:
>
>> I think there are enough big projects that use black to consider it
>> ready. (PyPI for another example, and we also know it is being actively
>> developed).
>>
>> IMO the way to make this choice is to determine whether the style it
>> uses/enforces works well for the project. There might be legitimate reasons
>> that black would not work for us, a hypothetical example being that our
>> viewset comments must be formatted in a specific way for the autodocs. To
>> move forward, I suggest using black and opening a PR so we can discuss the
>> specifics. In this case, it will be a lot less work to try it out than to
>> consider it theoretically.
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] black

[Austin Macdonald]

On Tue, Jun 4, 2019 at 1:40 PM Matt Pusateri  wrote:

>
>
>
> On Tue, Jun 4, 2019 at 1:15 PM Mike DePaulo  wrote:
>
>> On Tue, Jun 4, 2019 at 12:09 PM Robin Chan  wrote:
>>
>>> Mike, clarification question below...
>>>
>>> On Tue, Jun 4, 2019 at 11:45 AM Mike DePaulo 
>>> wrote:
>>>
 On Tue, Jun 4, 2019 at 9:14 AM Brian Herring 
 wrote:

> > Moreover, using black would effectively take style comments/noise
> out of PR reviews and we could just focus on logic.
>
> QE looked into this for a while, and I think it is a WONDERFUL
> addition... especially for the reasons stated above.
>
> Some tweaking would be needed, but it is one less item to worry about.
>
> +1
>
 [...]

 +1 based on my experiences on the X2Go project.
 We don't want to turn off new contributors with style issues.

>>> Meaning new contributors would be annoyed that PR reviews would be full
>>> of stylistic comments and that would be a bad experience?
>>> I read this as possibly meaning new contributors would be unhappy with
>>> the black project style choices? Being unfamiliar with the X2Go project,
>>> I'd like some clarification on your input to the discussion here. TIA.
>>>
>>
>> The former. New contributors being annoyed by stylistic comments.
>> I think they would likely be unhappy with the black project style choices
>> as well, but that would be outweighed.
>>
>>
> This makes me wonder.
>
> 1. What is the ratio of external(non-redhat) committer/contributors?
> Meaning if 90% of the code comes from RH, is this really a problem?  I'm
> not saying that RH should flex its muscle and influence the community, but
> I am wondering if we're over-thinking it?  Is this a real problem or a
> perceived problem.
> 2. Wouldn't you argue that enforcing style guides is the "right thing to
> do" because it improves code and quality in the long run?  And it theory
> should also make it easier for new contributors to get involved as there is
> consistency of code style due the guidelines?
>
>
There may be some confusion here. Black doesn't just tell us when something
is wrong, it fixes formatting for you. So we have a situation where the
code is more consistent *and* easier to contribute. (Unless it does
something crazy in our particular case.)


> Matt P.
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] black

[Austin Macdonald]

I think there are enough big projects that use black to consider it ready.
(PyPI for another example, and we also know it is being actively
developed).

IMO the way to make this choice is to determine whether the style it
uses/enforces works well for the project. There might be legitimate reasons
that black would not work for us, a hypothetical example being that our
viewset comments must be formatted in a specific way for the autodocs. To
move forward, I suggest using black and opening a PR so we can discuss the
specifics. In this case, it will be a lot less work to try it out than to
consider it theoretically.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Release Note Process Improvements

[Austin Macdonald]

The proposed changes look awesome! I'm +1 for moving forward with it for
pulpcore and pulpcore-plugin.

If there is consensus (looks like we are close), lets go ahead. If anyone
has concerns, we also have the option to implement this change for one
plugin before we go all in.

On Mon, May 27, 2019 at 5:26 AM Ina Panova  wrote:

> +1
>
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Sat, May 25, 2019 at 10:18 PM Tatiana Tereshchenko 
> wrote:
>
>> +1 to improve release notes process
>>
>> If we decide to use PR numbers and not redmine issues in the release
>> notes, then there will be no limitation/requirement to have a redmine issue
>> to add something to the release notes.
>>
>> Tanya
>>
>> On Fri, May 24, 2019 at 3:46 PM David Davis 
>> wrote:
>>
>>> +1 to bmbouter's proposal and not including '[noissue]' items in release
>>> notes.
>>>
>>> David
>>>
>>>
>>> On Fri, May 24, 2019 at 3:52 AM Matthias Dellweg 
>>> wrote:
>>>
 I am fine with stating "[noissue] means 'not worth mentioning in
 release notes'".
 This would require the reviewer to decide to tell the contributor: "We
 want that to be part of the release notes. Please open up a ticket."
 And that process scales better than handpicking the notes in the end.

 On Thu, 23 May 2019 16:22:36 -0400
 Dana Walker  wrote:

 > My initial thought is this looks useful to the user and very clean.
 > I've also found it to be a burden trying to write good release notes,
 > having to dig through commits and try to decide what's important
 > enough and what's not, so +1 to trying to improve this process for
 > both the releaser and user.
 >
 > However:
 > "towncrier works best in a development system where all merges involve
 > closing a ticket."
 > We frequently make use of "[noissue]" in our PRs, in part to lower the
 > burden on contributors making small fixes.  Would we want to move to a
 > model where we *must* have an issue?  Are we instead assuming those
 > items are small enough that the user doesn't need to see it in the
 > release notes?
 >
 > Thoughts?
 >
 > --Dana
 >
 > Dana Walker
 >
 > She / Her / Hers
 >
 > Software Engineer, Pulp Project
 >
 > Red Hat 
 >
 > dawal...@redhat.com
 > 
 >
 >
 >
 > On Thu, May 23, 2019 at 3:49 PM Brian Bouterse 
 > wrote:
 >
 > > In discussion with some other devs, I've realized that pulpcore and
 > > pulpcore-plugin would benefit from better release notes. Here are
 > > some of the reasons that have come up:
 > >
 > > * The release notes are incomplete. One person tries to go through
 > > and write release notes just before the release happens, and by
 > > that point, the number of changes are too many for this approach to
 > > produce complete and robust notes.
 > > * They are hard to produce. Producing "all the release notes" is a
 > > mentally difficult task.
 > > * We try to substitute with Redmine, but this approach limits us
 > > (a) it's now difficult and time consuming to see what changed, (b)
 > > there is way more detail than you actually want, and they aren't
 > > self-contained (can't be browsed off-line).
 > > * overall all ^ leads to both users and plugin writers feeling
 > > uncertain about what has changed in the last release, week, or even
 > > day.
 > >
 > > So what can we do? Recently I contributed to aiohttp and I found
 > > their release note process light and easy. It produces high-quality
 > > release notes like these:
 > > https://aiohttp.readthedocs.io/en/stable/changes.html
 > >
 > > You can read about their process here:
 > >
 https://aiohttp.readthedocs.io/en/stable/contributing.html#changelog-update
 > > You can see some examples of these release note files in their repo
 > > here: https://github.com/aio-libs/aiohttp/tree/master/CHANGES
 > > Overall it makes use of the towncrier project
 > > https://github.com/hawkowl/towncrier
 > >
 > > What do you all think about trying something like this for pulpcore
 > > and pulpcore-plugin? Please write back on-list with thoughts,
 > > ideas, concerns, alternatives, etc.
 > >
 > > Also, I made us a starter issue to coalesce some more of the
 > > practical aspect of adopting a change like this:
 > > https://pulp.plan.io/issues/4875
 > >
 > > All the best,
 > > Brian
 > >
 > >
 > >
 > > ___
 > > Pulp-dev mailing list
 > > Pulp-dev@redhat.com
 > > https://www.redhat.com/mailman/listinfo/pulp-dev
 > >
 ___
 

Re: [Pulp-dev] Docstring linting

[Austin Macdonald]

Something else to consider: some docstrings are rendered as user-facing
documentation in the autogenerated REST docs. This means that docstring
linting needs to be ignored for ViewSets. For example, I have a PR open
that alters pulp_file viewset docstrings to contain html, to pass the
linter, we have add docstring exceptions to the flake8 config.

My initial reaction is that we might be better off keeping the
flake8-docstring package out of pulplift, and we should also remove it from
travis.


On Tue, May 21, 2019 at 11:08 AM Matthias Dellweg  wrote:

> tl;dr: Docstring linting is inconsistent across pulp repositories.
> To make it consistent, do we want to enforce it everywhere, and repair
> more than 700 findings?
>
> What started out as a oneliner [0] surfaced as a bigger problem:
>
> Whether flake8 performs linting on docstrings is solely dependent
> (afaik) on the existence of a specific python package
> (flake8-docstring) in the system.
> At the same time, there are repositories (pulpcore,
> pulpcore-plugin, ???) that do not install this package in their ci
> pipeline, as well as repos that do (pulp_deb, pulp_ansible, ???).
> So it is hard to select whether it should be installed in a pulplift
> source box.
> Not installing it means, there are linting errors showing up in
> travis only, however installing it will prevent linting pulpcore
> locally.
> That said, i think we should follow the same linting rules in all
> repositories, and more specific i tend to include docstring linting.
> However there are over 700 findings in pulpcore alone.
>
> [0] https://github.com/pulp/pulpcore/pull/138
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Ansible Galaxy Content Changes

[Austin Macdonald]

+1 I think we are ok to remove.

On Mon, Apr 15, 2019 at 1:46 PM Brian Bouterse  wrote:

> There are some early Ansible roles posted on the 'pulp' namespace in
> Galaxy here:  https://galaxy.ansible.com/pulp
>
> Who can modify this content?
>
> Can we add the role from this repo to it?
> https://github.com/pulp/pulp-rpm-prerequisites/
>
> Can we take down the existing content? It's old and out of date, and we
> can't deliver our new Installer through galaxy currently because we ship
> too many roles per repo. The direct distribution (git clone) method is
> working pretty well, so I think all we need to do is take this content down.
>
> Thanks,
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 tag

[Austin Macdonald]

Pulp2 tag sounds great to me.

On Fri, Apr 12, 2019, 16:23 Brian Bouterse  wrote:

> +1 to making a 'Pulp 2' tag
>
>
> On Fri, Apr 12, 2019 at 3:16 PM David Davis  wrote:
>
>> I’ve been thinking through this and I think there are 3 steps to perform:
>>
>> 1. Create a Pulp 2 tag
>> 2. Assign the Pulp 2 tag to issues not tagged with Pulp 3
>> 3. Remove the Pulp 3 tag
>>
>> Does that make sense? If so, I’d like to perform 1 and create the Pulp 2
>> tag in the next few days so we can start adding it to issues.
>>
>> I’ll follow up with more concrete plans for steps 2 and 3 next week.
>>
>> David
>>
>>
>> On Thu, Apr 11, 2019 at 11:10 AM David Davis 
>> wrote:
>>
>>> Cool, I will try to think through how to transition and we can discuss
>>> at the team meeting Monday.
>>>
>>> David
>>>
>>>
>>> On Thu, Apr 11, 2019 at 11:04 AM Ina Panova  wrote:
>>>
 +1 to create a pulp2 tag after mass-close


 
 Regards,

 Ina Panova
 Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Thu, Apr 11, 2019 at 3:48 PM Brian Bouterse 
 wrote:

> Great idea David!
>
> Once we get through tomorrow's mass-close, let's plan a transition to
> remove the Pulp3 tag and introduce a Pulp 2 tag instead.
>
> On Thu, Apr 11, 2019 at 9:34 AM Dana Walker 
> wrote:
>
>> +1
>>
>> Dana Walker
>>
>> Software Engineer
>>
>> Red Hat
>>
>> 
>>
>> 
>>
>>
>> On Thu, Apr 11, 2019 at 9:22 AM Daniel Alley 
>> wrote:
>>
>>> I feel like using the Pulp 3 tag is suboptimal since we are closing
 out most of our Pulp 2 issues and a majority of the new 
 bugs/tasks/stories
 are for Pulp 3. Wouldn’t it make more sense to create a Pulp 2 tag and 
 use
 it on the rare Pulp 2 issues that get filed?
>>>
>>>
>>> Agreed, +1 to having a "Pulp 2" tag to replace the Pulp 3 tag.
>>>
>>> On Thu, Apr 11, 2019 at 9:19 AM David Davis 
>>> wrote:
>>>
 I feel like using the Pulp 3 tag is suboptimal since we are closing
 out most of our Pulp 2 issues and a majority of the new 
 bugs/tasks/stories
 are for Pulp 3. Wouldn’t it make more sense to create a Pulp 2 tag and 
 use
 it on the rare Pulp 2 issues that get filed?

 David


 On Thu, Apr 11, 2019 at 8:27 AM Ina Panova 
 wrote:

> +1 to handle tag during the triage process
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Tue, Apr 2, 2019 at 3:53 PM Brian Bouterse 
> wrote:
>
>> +1 to Tanya's suggestion.
>>
>> Also I have an action item to start a separate thread on how we
>> can plan-and-then-close all pulp2 bugs that we are not actively 
>> working on.
>> I wanted to mention it here to identify it as a possible resolution 
>> to this
>> issue since it was related. I'll send this today.
>>
>> On Tue, Apr 2, 2019 at 7:54 AM Tatiana Tereshchenko <
>> ttere...@redhat.com> wrote:
>>
>>> In my opinion, it's a part of triage or grooming process.
>>> Triage leader (for issues) and a person who grooms the task
>>> should pay attention to the tags.
>>>
>>> Tanya
>>>
>>> On Tue, Apr 2, 2019 at 1:47 PM David Davis <
>>> davidda...@redhat.com> wrote:
>>>
 Looking in redmine at our current sprint, I see several issues
 that don’t have Pulp 3 tag that should. Here are some:

 https://pulp.plan.io/issues/4587
 
 https://pulp.plan.io/issues/4595
 
 https://pulp.plan.io/issues/4577
 
 https://pulp.plan.io/issues/4576
 
 https://pulp.plan.io/issues/4545

 Talking with the QE team,
 it’s difficult for them to get a feel for how much Pulp 2 vs Pulp 
 3 work
 there is. I guess we could try to get better about tagging issues 
 when we
 create them?

 However, at least one issue is by someone outside the core Pulp
 team and I am not sure community members will know to tag their 
 issues w/
 Pulp 3 tag. Are there 

Re: [Pulp-dev] Content Copy (between repos)

[Austin Macdonald]

I think its possible you will have a url collision with variant 1. (We had
a similar problem trying to use /v3/publications/docker/)

variant 1:  POST /pulp/v3/api/content/rpm/packages/copy/
content_units=[...] source="..." destination="..."  [more params...]

Assuming that /pulp/v3/api/content/rpm/packages// is a valid url, it
can (unintended) capture "copy" and treat it as a PK. If you see this
error, I think it will manifest as " 405: POST is not allowed at this
endpoint", meaning the rpm-package-detail endpoint.

Variant 2 seems doable, and leaves the door open for some pulpcore code, if
it ends up being helpful.

On Thu, Apr 4, 2019 at 2:38 PM Daniel Alley  wrote:

> Content copy between repositories is critically important to Katello
> integration and is something that we have not really addressed yet.  It
> also needs to be completed before the RPM plugin can begin work on
> depsolving.  The story that results from this discussion should probably be
> put on one of the next 1-3 sprints and not wait any longer than that.
>
> Repositories are generic to all types of content, but copy operations
> between repositories will need type-specific options defined by the plugin
> for e.g. advanced copy w/ depsolving.  Therefore we need to find a design
> for this functionality where it will make sense that makes sense from a
> usage perspective and an implementation perspective.
>
> To get this discussion started, here's some suggestions about how this
> could work from the user perspective (presented without much thought put
> into how it would be implemented).
>
> Create a "Copier" object, like a Remote, that stores it's own settings and
> has one or many endpoints (like "/sync/") that can be POSTed to and return
> a task and create a new repository version.
>
> POST /pulp/v3/api/copiers/rpm/$endpoint_name/ content_units=[...]  [more
> params...]
>
> The copier would store settings such as "recursive" and the "source" and
> "destination" repositories.  And let's say they can be overriden.
>
> Or, create new endpoints without any associated DB models, like one-shot
> upload does:
>
> variant 1:  POST /pulp/v3/api/content/rpm/packages/copy/
> content_units=[...] source="..." destination="..."  [more params...]
>
> variant 2: POST  /pulp/v3/api/copy/rpm/package/ content_units=[...]
> source="..." destination="..."  [more params...]
>
> Since basic copy support (just copying the units, no depsolving, etc.)
> could in theory be implemented completely generically, it would be nice if
> we could provide that for free somehow.  But I don't immediately see a way
> of doing so.
>
> I welcome any suggestions or input.
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] State of the docs & Docs drive kickoff

[Austin Macdonald]

The Pulp community is beginning our drive to improve the user docs for Pulp
3.

Docs work is tracked with the redmine tags [Pulp 3, Documentation] and can
be viewed from the query: https://pulp.plan.io/issues?query_id=128
.  (Note that the query is for
"Documentation" OR "Pulp3", so shows more issues than we need to focus on
here.)

*Action Required:*
Please have a look at the goals and the issues mentioned in "high priority
work" section.

If you have some extra time, please review some of the issues in the query
or tag other issues you think should be included. There are a lot of
issues, so it will take a focused effort from multiple people to tackle.

*Work begun:*
I've started by reading over our existing documentation for pulpcore. This
etherpad was used for organizing and compiling issues. [0]
https://docs.pulpproject.org/en/3.0/nightly/

*Docs Push Goals*:

   - Address OSAS Feedback
   https://pulpproject.org/2018/09/17/pulp-community-health-audit/


   - Add quickstarts
  - make pups visible on pulpproject.org
  - community visible calendar
   - Collaborative effort coordinated via Redmine
   - File issues for documentation gaps
   - Close irrelevant/dupes/already-done issues
   - Burn down


*High Priority Work:*
>From the review below, 2 important changes should be addressed early in the
docs push. These issues would benefit from feedback, please review and
comment!

   - Explicitly define which features will be documented by pulpcore, and
   which will be documented by each plugin
  - https://pulp.plan.io/issues/4626
  - The division criteria (discussed on the issue) needs to be more
  concrete
  - Publish better REST API documentation for pulpcore
  - Publish the live-api docs (many options):
  https://pulp.plan.io/issues/4636
  - Document how to ^ for plugin writers:
  https://pulp.plan.io/issues/4637


*State of the docs, and what can be improved:*
The docs appear to be in pretty good shape, but have some work left to do.

The content of the current docs is mostly strong and concise [biased by
familiarity]. The organization is fine, though some clean-up would improve
readability, and clarity of the left-bar main divisions. Isolated problems
are mentioned in the read-through notes on the planning etherpad. [0]

The division of the docs (between pulpcore and each plugin) requires prior
knowledge and is not well communicated. Based on the consistency, the
developers have a shared understanding of which features should be
documented in pulpcore, and which topics are owned by the plugin. In my
opinion, most of the documentation on pulpcore is in the right place. The
Plugin docs are off to a good start by covering each major feature with
quickstart-style guides. Pulcore and each plugin's docs need to include
more specific information, which should be covered by the REST API docs.

REST API documentation is published on pulpcore's read the docs, but it is
missing too much information, rendering it practically useless.
https://docs.pulpproject.org/en/3.0/nightly/integration-guide/rest-api/index.html#pulpcore-rest-api
. Significantly better REST API docs can be viewed at the docs endpoint on
a live-running pulp instance. The live REST API documentation partially
fills the gap between the quickstart docs (major feature) and how that
workflow can be altered (minor features). Unfortunately, these docs are not
published, they are only available if the user takes the extra step of
generating the documentation.

*Plan:*

   1. Begin by improving the REST API docs-- their inclusion will add a lot
   of "missing" information. This will affect a lot of the following docs
   work, allowing plugin writers to link to specific API calls, reducing the
   need and length for text explanations.
   2. Clarify any discrepancy about what is expected to be documented by
   each plugin. This needs to be very clear to users, qe, and anyone who
   contributes docs, especially plugin writers. Necessary for successful
   documentation navigation.
   3. Understand, revise, and groom issues from the docs planning backlog



[0] https://etherpad.net/p/Pulp3_Docs_Planning
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp2 Bug Backlog Closing?

[Austin Macdonald]

I think if we close a lot of them, closed issues will be very difficult to
find with ~4500 bugs (open and closed). I've been spending some time
combing the backlog recently, and I'm compiling lists of bugs that I think
can be closed. What I am also finding are tickets that could reasonably be
updated for Pulp 3. IMO, these tickets are common enough that it would be
worth our time to consider them.

Of course, going through the enormous backlog will be very time consuming.
If we agree that there is too much value to close the lot of them, then
AFAICT the only path forward is to coordinate the effort and move through
it over time.

On Tue, Apr 2, 2019 at 5:22 PM Austin Macdonald  wrote:

> I think if we close a lot of them, closed issues will be very difficult to
> find with ~4500 bugs (open and closed). I've been spending some time
> combing the backlog recently, and I'm compiling lists of bugs that I think
> can be closed. What I am also finding are tickets that could reasonably be
> updated for Pulp 3. IMO, these tickets are common enough that it would be
> worth our time to consider them.
>
> Of course, going through the enormous backlog will be very time consuming.
> If we agree that there is too much value to close the lot of them, then
> AFAICT the only path forward is to coordinate the effort and move through
> it over time.
>
> On Tue, Apr 2, 2019 at 5:06 PM Brian Bouterse  wrote:
>
>> As Pulp2 approaches the maintenance mode we have a large number of Pulp2
>> bugs open. A query [0] shows 1125 open Pulp2 bugs alone as of just now. We
>> will likely address a small set of these before Pulp2 reaches its final
>> release. What can we do to bring transparency into what will versus won't
>> be fixed for Pulp2?
>>
>> The most reasonable option I can think to propose is a mass-close of the
>> Pulp2 bugs except for those that we are actively working or planning to
>> start work soon on. Overall I believe Pulp2 is nearing a point that if we
>> aren't actively working or planning something for it we won't want to leave
>> it open on the "Pulp 2 backlog ". Bugs accidentally closed could be
>> reopened without much trouble probably.
>>
>> What do you think about the of a close-all-but-active Pulp2 bugs idea?
>> How would you coordinate such an effort?
>>
>> [0]: https://tinyurl.com/y289wx5p
>>
>> Thanks,
>> Brian
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] installer team

[Austin Macdonald]

Oops, this is a draft for my own organization, please disregard :)

On Mon, Mar 25, 2019 at 6:13 PM Austin Macdonald  wrote:

> - create meeting for installer team
>   - dev installer docs
>  - point to pulplift docs
>- remove any instructions that should be documented on pulplift
>- remove mention of specific repos, ( use ":get the source")
>- update "get the source"
>- ansible-pulp3 -> ansible-pulp,
>- pulp -> pulpcore
> - add pulpcore-plugin
>  - expand pulplift docs
> - link to "get the source" instead of saying in the install guide
>   - deprecate pulp/devel for pulp3
>  - switch master to 3-master and deprecate, include link to pulplift
>  - switch 2-master to master
>  - send email to pulp-dev announcing deprecation officially
>   - pulplift demo
>   - Update "using ansible"
>   - remove documentation of roles (pulp/devel)
>   - point to ansible-pulp README and mention each role's README
>
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] installer team

[Austin Macdonald]

- create meeting for installer team
  - dev installer docs
 - point to pulplift docs
   - remove any instructions that should be documented on pulplift
   - remove mention of specific repos, ( use ":get the source")
   - update "get the source"
   - ansible-pulp3 -> ansible-pulp,
   - pulp -> pulpcore
- add pulpcore-plugin
 - expand pulplift docs
- link to "get the source" instead of saying in the install guide
  - deprecate pulp/devel for pulp3
 - switch master to 3-master and deprecate, include link to pulplift
 - switch 2-master to master
 - send email to pulp-dev announcing deprecation officially
  - pulplift demo
  - Update "using ansible"
  - remove documentation of roles (pulp/devel)
  - point to ansible-pulp README and mention each role's README
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Travis CI is broken for pulpcore PRs

[Austin Macdonald]

Today, I merged a change to ansible-pulp that broke the CI. I have an open
PR to fix it, but we are running into an unrelated problem (Travis is
trying to double the number of builds, and cannot ever pass.)

https://github.com/pulp/pulpcore/pull/51#issuecomment-476389752

Until this PR (or its clone https://github.com/pulp/pulpcore/pull/52/)
passes, the pulpcore Travis tests will continue to fail.

We will continue to look into this tomorrow.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Please update pulplift when you merge a PR to ansible-pulp

[Austin Macdonald]

I've been merging lots of changes to ansible-pulp and pulplift recently,
and I noticed that fairly often, the pulplift update moves forward by a few
unrelated commits. This makes it difficult to make a commit message that
makes sense for pulplift, and can confuse pulplift users, forcing them to
inspect 2 git logs to understand what has changed.

So, even if the change is not particularly relevant for pulplift, please go
ahead and bump the commit anyway to keep it up to date and keep the git log
clean.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Changing behavior of the pclean alias

[Austin Macdonald]

That's interesting! Pavel, next time you see this, could you file an issue
that includes what is in the DB?

On Wed, Mar 20, 2019, 12:01 Pavel Picka  wrote:

> I noticed when I run test sometimes it left something in DB... but I found
> why, as tests didn't finish completely it didn't destroy db after itself.
> so take only +1 and skip everything after
>
> On Wed, Mar 20, 2019 at 3:56 PM Austin Macdonald 
> wrote:
>
>> Pavel, can you elaborate?
>>
>> On Wed, Mar 20, 2019 at 9:43 AM Pavel Picka  wrote:
>>
>>> +1 and what do you think about idea of dropping all DBs as sometimes
>>> test let there some (it doesn't affect pulp but when clean db so clean it
>>> whole)?
>>>
>>> On Wed, Mar 20, 2019 at 12:12 PM Dennis Kliban 
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Tue, Mar 19, 2019 at 5:39 PM Daniel Alley  wrote:
>>>>
>>>>> I created a new PR here [0] which changes the behavior of the pclean
>>>>> alias so that it also wipes out /var/lib/pulp/ in addition to dropping and
>>>>> recreating the database.
>>>>>
>>>>> Unless anyone objects, I plan to merge it tomorrow afternoon
>>>>> (Wednesday the 20th)
>>>>>
>>>>> [0] https://github.com/pulp/ansible-pulp3/pull/93
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>
>>>
>>> --
>>> Pavel Picka
>>> Red Hat
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>
> --
> Pavel Picka
> Red Hat
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Changing behavior of the pclean alias

[Austin Macdonald]

Pavel, can you elaborate?

On Wed, Mar 20, 2019 at 9:43 AM Pavel Picka  wrote:

> +1 and what do you think about idea of dropping all DBs as sometimes test
> let there some (it doesn't affect pulp but when clean db so clean it
> whole)?
>
> On Wed, Mar 20, 2019 at 12:12 PM Dennis Kliban  wrote:
>
>> +1
>>
>> On Tue, Mar 19, 2019 at 5:39 PM Daniel Alley  wrote:
>>
>>> I created a new PR here [0] which changes the behavior of the pclean
>>> alias so that it also wipes out /var/lib/pulp/ in addition to dropping and
>>> recreating the database.
>>>
>>> Unless anyone objects, I plan to merge it tomorrow afternoon (Wednesday
>>> the 20th)
>>>
>>> [0] https://github.com/pulp/ansible-pulp3/pull/93
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
> Pavel Picka
> Red Hat
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] pulp_file ownership

[Austin Macdonald]

+1 for the latter.

Since some changes to pulpcore or pulpcore-plugin also require changes to
pulp_file (anything backwards incompatible) everyone on the core team needs
to be able to quickly make changes to pulp_file as well. (And vice versa.)

On Tue, Mar 19, 2019 at 5:04 PM David Davis  wrote:

> With @jortel having left the Pulp team, the people with a commit bit to
> pulp_file is down to just @dkliban and I. I don't think this is enough. We
> could either add more people to the pulp_file team or just move the
> pulp_file repo under the core team. I am leaning toward the latter. Any
> objections?
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Nomenclature Clarification

[Austin Macdonald]

We don't have an official process, but bringing it up here is a good place
to start.

Depending on the changes, it would be best to update the docs:

https://docs.pulpproject.org/en/3.0/nightly/concepts.html
https://docs.pulpproject.org/en/3.0/nightly/glossary.html
https://docs.pulpproject.org/en/3.0/nightly/components.html#

On Wed, Mar 13, 2019 at 10:53 AM Eric Helms  wrote:

> Thanks for the feedback. I should know this, but if I wanted to propose an
> official change to the nomenclature what is the correct process?
>
> On Wed, Mar 13, 2019 at 10:47 AM Dennis Kliban  wrote:
>
>> On Fri, Mar 8, 2019 at 10:34 AM Eric Helms  wrote:
>>
>>> Howdy,
>>>
>>> Within Pulp 3, I've found myself struggling to refer to the pieces of
>>> the runtime deployment and wanted to raise a few questions I keep coming
>>> back to.
>>>
>>>  1) Per [1], pulpcore seems to reference the entirety of the server,
>>> workers and resource manager that live within the repository at [2]. Is
>>> that correct?
>>>
>>
>> Yes, that is what pulpcore means.
>>
>>
>>>  2) What is the 'server' called? I keep wanting to call it the
>>> 'pulp-api-server' or 'pulp-api-app' or 'pulp-api'.
>>>
>>>
>> I like the name pulp-rest-api or pulp-api.
>>
>>
>>>
>>> - Eric
>>>
>>> [1]
>>> https://docs.pulpproject.org/en/3.0/nightly/glossary.html#term-pulpcore
>>> [2] https://github.com/pulp/pulpcore
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Unified interface for plugin actions

[Austin Macdonald]

The v3/actions would not need to be limited to plugins, but I think we
ought to leave object CRUD how it is. So asynchronous object
create/update/delete should remain on the object viewset.

The only endpoints I'd like to move are sync, publish, export, and
plugin-defined actions like one-shot upload. It might also be worth moving
the orphan removal for core. Suggested endpoint:
`v3/actions/core/remove_orphans/`

On Fri, Mar 1, 2019 at 4:27 PM David Davis  wrote:

> I'm curious about how these "actions" are defined. Is the /v3/actions/
> namespace limited to plugins? Or, for example, should distribution
> create/update/delete be nested under /v3/actions/? What about repository
> create/update/delete?
>
> David
>
>
> On Fri, Mar 1, 2019 at 9:08 PM Justin Sherrill 
> wrote:
>
>>
>> On 3/1/19 2:45 PM, Robin Chan wrote:
>>
>> Justin,
>> Would such a change make a significant difference in the effort,
>> complexity, or time to migrate existing (or support new) plugins in Katello?
>>
>> It would be a very small and simple change.
>>
>>
>>
>> Robin
>>
>> On Fri, Mar 1, 2019 at 2:00 PM Justin Sherrill 
>> wrote:
>>
>>> To me this makes a lot of sense, allows for plugin flexibility, and is
>>> more consistent across plugins.
>>>
>>> I feel like this will make differences between plugins more
>>> understandable by reading the api docs, rather than scanning the
>>> README's of the respective plugin and trying to work out what is
>>> different.
>>>
>>> Justin
>>>
>>> On 2/28/19 1:42 PM, Austin Macdonald wrote:
>>> > Now that we have a handful of plugins that have somewhat different
>>> > workflows, surprising user-facing differences in the interface for
>>> > plugin-related actions are becoming apparent.
>>> >
>>> > Example: Publish
>>> > File:
>>> > Create a publisher
>>> > v3/publishers/file/1/publish/ repository=$REPO
>>> > Ansible:
>>> > (no publisher)
>>> > v3/publications/ repository=$REPO
>>> >
>>> > The difference is not huge, having a different endpoint does defy
>>> > expectations of a user who is familiar with one plugin, who then moves
>>> > to another plugin.
>>> >
>>> > Plugins can also implement other endpoints, like RPM's one-shot
>>> > upload. The problem is that we have mixed idioms. Plugins are
>>> > encouraged to create task endpoints for objects (remote's sync,
>>> > publisher's publish), but they are also encouraged to create arbitrary
>>> > endpoints for any other actions. Users are not able to form reasonable
>>> > expectations for this part of the interface from plugin to plugin.
>>> >
>>> > Proposal:
>>> > We could move all "actions" into a single area, namespaced by plugin
>>> > (by convention). This would allow the plugins the freedom to do
>>> > whatever they need to do while keeping the interface consistent and
>>> > predictable for users of multiple plugins. These "actions" could be
>>> > synchronous or asynchronous. Importantly, this would also create a
>>> > logical "group" of endpoints a user could look for in the REST API
>>> docs.
>>> >
>>> > Examples:
>>> > v3/actions/file/publish/ publisher=$PUB repository=$REPO
>>> > v3/actions/ansible/publish/ repository=$REPO
>>> > v3/actions/rpm/upload/ file@./foo-4.1-1.noarch.rpm repository=$REPO
>>> >
>>> > Will this push back the RC?
>>> > No. The changes to the plugin API will be small, and the changes to
>>> > each plugin would be moving sync and publish endpoints, leaving the
>>> > logic almost identical. I anticipate the most time consuming aspect of
>>> > this will be adjusting the documentation of each plugin-- but since
>>> > they will follow similar patterns, this shouldn't be too much work
>>> either.
>>> >
>>> > To sum up:
>>> > We should move sync and publish endpoints to
>>> > /actions/// to be consistent with other
>>> > plugin-defined actions like one-shot upload. This will look very nice
>>> > in swagger docs, and should provide more consistent workflows for
>>> > users of multiple plugins.
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Unified interface for plugin actions

[Austin Macdonald]

Now that we have a handful of plugins that have somewhat different
workflows, surprising user-facing differences in the interface for
plugin-related actions are becoming apparent.

Example: Publish
File:
Create a publisher
v3/publishers/file/1/publish/ repository=$REPO
Ansible:
(no publisher)
v3/publications/ repository=$REPO

The difference is not huge, having a different endpoint does defy
expectations of a user who is familiar with one plugin, who then moves to
another plugin.

Plugins can also implement other endpoints, like RPM's one-shot upload. The
problem is that we have mixed idioms. Plugins are encouraged to create task
endpoints for objects (remote's sync, publisher's publish), but they are
also encouraged to create arbitrary endpoints for any other actions. Users
are not able to form reasonable expectations for this part of the interface
from plugin to plugin.

Proposal:
We could move all "actions" into a single area, namespaced by plugin (by
convention). This would allow the plugins the freedom to do whatever they
need to do while keeping the interface consistent and predictable for users
of multiple plugins. These "actions" could be synchronous or asynchronous.
Importantly, this would also create a logical "group" of endpoints a user
could look for in the REST API docs.

Examples:
v3/actions/file/publish/ publisher=$PUB repository=$REPO
v3/actions/ansible/publish/ repository=$REPO
v3/actions/rpm/upload/ file@./foo-4.1-1.noarch.rpm repository=$REPO

Will this push back the RC?
No. The changes to the plugin API will be small, and the changes to each
plugin would be moving sync and publish endpoints, leaving the logic almost
identical. I anticipate the most time consuming aspect of this will be
adjusting the documentation of each plugin-- but since they will follow
similar patterns, this shouldn't be too much work either.

To sum up:
We should move sync and publish endpoints to
/actions/// to be consistent with other plugin-defined
actions like one-shot upload. This will look very nice in swagger docs, and
should provide more consistent workflows for users of multiple plugins.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] plugin API versioning for RC

[Austin Macdonald]

An issue has come up for plugins when declaring their dependence on
pulpcore-plugin.

Background:
The plugin API is semantically versioned at 0.1.0b20, which is separate
from pulpcore. The plan has always been that pulpcore would GA while the
plugin API is released as "0.1", indicating that even after GA, the plugin
API may change.

Problems:
If plugins are released to PyPI with `pulpcore-plugin >= 0.1.0`, then they
can break with future backwards incompatible changes of pulpcore-plugin.
This has occurred frequently recently, leading many plugins to pin the
pulpcore-plugin version explicitly. Pinning the version does guarantee that
the plugin will always work if it can be installed, but has led to a new
problem-- plugins that pin to different versions of pulpcore-plugin cannot
be installed together.

One solution:
We could  start bumping the "y" version of pulpcore-plugin with each
backwards incompatible change to the plugin API, and bump the z version for
bugfixes only. This will allow plugins to "pin" to the y version, but still
>= for the z version using `pulpcore-plugin ~= 0.1.0` synax. This will
allow us to guarantee that a plugin will work if it is installable *and*
allow some flexibility to improve the odds that plugins will work together.

https://semver.org/#spec-item-4
https://semver.org/#how-should-i-deal-with-revisions-in-the-0yz-initial-development-phase
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Changes in the Pulp 3 Upload story

[Austin Macdonald]

I think the key question to ask is:
What circumstances would require the creation of Content that would not be
met by a one-shot upload?

On Tue, Feb 19, 2019 at 11:34 AM Daniel Alley  wrote:

> @Matthias why would you prefer to keep the normal create?  As you mention,
> the "orphan cleanup" issue applies to both, so there's no advantage given
> to the former.
>
> The normal "create" ( POST .../content/plugin/type/ ...) is unidiomatic
> and inconsistent, because the fields needed to upload a content are
> different from the fields on the actual serializer.  Most content types
> keep metadata inside the packages themselves, so you can't let the user
> specify the content field values, you have to contort everything so that
> instead of hydrating the serializer from user input, it does so by parsing
> the content.
>
> There's also the issue that the libraries we're using to parse the (Python
> and RPM) packages do some validation on the filename to see that it has the
> right extension and so forth, and since artifacts are content-addressed and
> don't store that information, with normal create you have to pass the
> filename of the original artifact *back in* at the time you create it, and
> then copy the file from Pulp content storage into a temp directory under a
> filename which will validate properly, which is highly unfortunate.
>
> With one-shot upload, you avoid both of those problems, because there's no
> broken expectations as to what fields should be accepted, and because it
> should be possible (though I haven't tried it) to parse the original file
> *before* saving it as an artifact, thus avoiding a lot of mess. And you
> also get the option to add it straight into a repository. In my opinion,
> it's a straight upgrade.
>
> On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg  wrote:
>
>> I have no objections to having the "one shot upload" or even "one shot
>> upload into repositoryversion". I think, i would like to keep the
>> 'traditional' create anyway.
>> The problem i see with create and one shot upload is, that another
>> thing could have triggered 'delete orphans' at the wrong time, and you
>> shiny new content unit disappears, before you can add it to a
>> repository version.
>>
>> On Mon, 18 Feb 2019 14:41:54 -0500
>> Austin Macdonald  wrote:
>>
>> > Originally, our upload story was as follows:
>> > The user will upload a new file to Pulp via POST to /artifacts/
>> > (provided by core)
>> > The user will create a new plugin specific Content via POST to
>> > /path/to/plugin/content/, referencing whatever artifacts that are
>> > contained, and whatever fields are expected for the new content.
>> > The user will add the new content to a repository via POST to
>> > /repositories/1/versions/
>> >
>> > However, this is somewhat cumbersome to the user with 3 API calls to
>> > accomplish something that only took one call in Pulp 2.
>> >
>> > There are a couple of different paths plugins have taken to improve
>> > the user experience:
>> > The Python plugin follows the above workflow, but reads the Artifact
>> > file to determine the values for the fields. The RPM plugin has gone
>> > even farther and created a new endpoint for "one shot" upload that
>> > perform all of this in a single call. I think it is likely that the
>> > Python plugin will move more in the "one shot" direction, and other
>> > plugins will probably follow.
>> >
>> > That said, I think we should discuss this as a community to encourage
>> > plugins to behave similarly, and because there may also be a
>> > possibility for sharing some of code. It is my hope that a "one shot
>> > upload" could do 2 things: 1) Upload and create Content. 2)
>> > Optionally add that content to repositories.
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Changes in the Pulp 3 Upload story

[Austin Macdonald]

Originally, our upload story was as follows:
The user will upload a new file to Pulp via POST to /artifacts/ (provided
by core)
The user will create a new plugin specific Content via POST to
/path/to/plugin/content/, referencing whatever artifacts that are
contained, and whatever fields are expected for the new content.
The user will add the new content to a repository via POST to
/repositories/1/versions/

However, this is somewhat cumbersome to the user with 3 API calls to
accomplish something that only took one call in Pulp 2.

There are a couple of different paths plugins have taken to improve the
user experience:
The Python plugin follows the above workflow, but reads the Artifact file
to determine the values for the fields. The RPM plugin has gone even
farther and created a new endpoint for "one shot" upload that perform all
of this in a single call. I think it is likely that the Python plugin will
move more in the "one shot" direction, and other plugins will probably
follow.

That said, I think we should discuss this as a community to encourage
plugins to behave similarly, and because there may also be a possibility
for sharing some of code. It is my hope that a "one shot upload" could do 2
things: 1) Upload and create Content. 2) Optionally add that content to
repositories.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] [Pulp-list] Pulp 3 RC info

[Austin Macdonald]

>
>
> On Thu, Jan 31, 2019 at 8:45 AM Bryan Kearney  wrote:
>
>> -- pulp-list
>>
>> RC for a year? That seems way long. I get the idea of semanti=c
>> versioning, but is there any other path for that?
>>
>>
We haven't really discussed this idea, so this is just me talking:
Another possible path would be to GA much sooner with a high likelihood of
a Pulp 4 (and maybe 5) in the next year. This would be a more normal way to
go IMO, but I'd hate to spook downstream.



> --bk
>>
>> On 1/30/19 5:28 PM, David Davis wrote:
>> > We’re approaching an RC release of Pulp 3.0 and we’ve put together a
>> > blog post with some information about this release:
>> >
>> > https://pulpproject.org/2019/01/30/pulp-3-rc-information/
>> >
>> > Please feel free to respond to this email with any questions, comments,
>> > or concerns you have about the RC.
>> >
>> > David
>> >
>> > ___
>> > Pulp-list mailing list
>> > pulp-l...@redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-list
>> >
>>
>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing plugins, looking for feedback

[Austin Macdonald]

On Wed, Jan 9, 2019 at 12:30 PM Tatiana Tereshchenko 
wrote:

> To summarize where we are so far:
> *All* master/detail related endpoints will be automatically prepended
> with Django app *label* [0]
> - concerns: 'pulp_' in the label
> - options to address concerns:
>  * introduce a new attribute to the AppConfig class to use in the
> endpoints construction (not supported by majority so far)
>  * drop 'pulp_' part from a *plugin's* app label (supported by
> majority so far)
>
> Questions/concerns about dropping the 'pulp_' from the plugins' app label:
>
> # Table names in the DB are prepended using the app label. We need to be
> sure to avoid collisions with other applications for pulpcore and for pulp
> plugins. Are they already in the "pulp" database?
> Yes, all pulpcore and pulp plugin tables are in "pulp" database.
>
> # The names in the list of installed plugins would then not be the same as
> the packages themselves.
> It's probably ok. The status would look like this:
> {
> "component": "*file*",
> "version": "0.0.1b6"
> },
> {
> "component": "*rpm*",
> "version": "3.0.0b1"
> }
>

For the status api, we could use `name` rather than `label`:
{
"component": "*pulp_file.app*",
"version": "0.0.1b6"
},


> # What about the label for the core? (not discussed)
> It stays as is - 'pulp_app'.
>
> [0]
> https://docs.djangoproject.com/en/2.1/ref/applications/#django.apps.AppConfig.label
>
> On Tue, Jan 8, 2019 at 8:22 PM Daniel Alley  wrote:
>
>> I'm not opposed to this plan, I just want to point out that it would make
>> the status API make slightly less sense.  The names in the list of
>> installed plugins would then not be the same as the packages themselves.
>> It's probably close enough as to not be a problem though.
>>
>> On Tue, Jan 8, 2019 at 12:23 PM Austin Macdonald 
>> wrote:
>>
>>>
>>>
>>> On Tue, Jan 8, 2019 at 12:12 PM Brian Bouterse 
>>> wrote:
>>>
>>>> My understanding is that it's for both. It would be dropped from
>>>> app_label and that will automatically be used in master/detail urls. Is
>>>> that what others thought?
>>>>
>>>> This seems like the simplest approach to me. My only concern with this
>>> approach is making sure that the database will be properly namespaced so
>>> there won't be collisions with other applications that use postgres like
>>> Katello. AFAIK, the plugin tables don't need to be namespaced since they
>>> are already in the "pulp" database. Is that correct? If so, +1.
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing plugins, looking for feedback

[Austin Macdonald]

On Tue, Jan 8, 2019 at 12:12 PM Brian Bouterse  wrote:

> My understanding is that it's for both. It would be dropped from app_label
> and that will automatically be used in master/detail urls. Is that what
> others thought?
>
> This seems like the simplest approach to me. My only concern with this
approach is making sure that the database will be properly namespaced so
there won't be collisions with other applications that use postgres like
Katello. AFAIK, the plugin tables don't need to be namespaced since they
are already in the "pulp" database. Is that correct? If so, +1.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing plugins, looking for feedback

[Austin Macdonald]

There seems to be consensus around dropping pulp_ from the url. However, I
am not clear whether the intention is to use string formatting at url
construction time, or if we should change the conventions for setting the
app_label.

On Tue, Jan 8, 2019, 11:18 AM Jeff Ortel 
>
> On 1/4/19 1:32 PM, Brian Bouterse wrote:
>
> +1 to dropping the "pulp_" I don't think it's a helpful convention.
>
>
> Couldn't agree more.
>
>
> I also want to avoid adding another plugin configuration attribute. +1 to
> automatic namespacing and using the django app label.
>
> On Fri, Jan 4, 2019 at 2:18 PM David Davis  wrote:
>
>> While I am not opposed to dropping the "pulp_" prefix in URLs, I don't
>> like the idea of adding another attribute to the plugin configuration.
>>
>> I am +1 to automatic namespacing and using the django app label.
>>
>> David
>>
>>
>> On Fri, Jan 4, 2019 at 1:50 PM Tatiana Tereshchenko 
>> wrote:
>>
>>> It's seems that there is a consensus that all master/detail related
>>> endpoints should be prepended.
>>> There is no consensus if the current Django app label is good enough to
>>> use in the construction of the endpoints.
>>>
>>> My personal opinion:
>>> It's probably better if "pulp_" part goes away, at the same time I'm
>>> hesitant to add new attributes to configuration, since Django provides
>>> enough of them and gives us uniqueness check for free.
>>> See Django docs
>>> https://docs.djangoproject.com/en/2.1/ref/applications/#configurable-attributes
>>> Label is supposed to be a short name for the app.
>>>
>>> Please vote and/or raise your concerns if you have any by next Friday,
>>> voting will be closed on Jan 11, 2019 at 8:00PM EST.
>>> The proposal is described in https://pulp.plan.io/issues/4279
>>>
>>> My vote:
>>> +1 for automatic namespacing for all master/detail endpoints
>>> +1 to use existing Django app label
>>>
>>> Thank you,
>>> Tanya
>>>
>>> On Thu, Jan 3, 2019 at 1:15 PM Ina Panova  wrote:
>>>
>>>> +1 namespacing all master/detail
>>>>
>>>> For consistency, i would prefer to see same format i see in
>>>> `content_summary` as in content endpoints, even if it does not make sense
>>>> from user's perspective, because what we now have in content_summary, i
>>>> would not say that it makes much sense from user's perspective ;)
>>>>
>>>> 
>>>> Regards,
>>>>
>>>> Ina Panova
>>>> Software Engineer| Pulp| Red Hat Inc.
>>>>
>>>> "Do not go where the path may lead,
>>>>  go instead where there is no path and leave a trail."
>>>>
>>>>
>>>> On Wed, Jan 2, 2019 at 8:55 PM Austin Macdonald 
>>>> wrote:
>>>>
>>>>> +1 automatic namespacing for master/detail. I realize the easiest way
>>>>> to do this would be to use the app_label, giving us:
>>>>>
>>>>> /api/v3/content/pulp_rpm/packages/
>>>>>
>>>>>
>>>>> However, I feel like this url is pretty clunky. The "pulp_" is totally
>>>>> unnecessary, from the user's perspective. Instead, I think I'd prefer to
>>>>> add an attribute to the App config.
>>>>>
>>>>> https://github.com/pulp/plugin_template/blob/master/pulp_plugin_template/app/__init__.py#L8
>>>>>
>>>>> `endpoint_namespace = rpm` or `short_label = rpm`
>>>>>
>>>>> Result: /api/v3/content/rpm/packages/
>>>>>
>>>>> The downside is that every plugin would need 1 more line of code. The
>>>>> upside is that we could implement it exactly same way as app_label but
>>>>> without url redundancy.
>>>>>
>>>>> On Wed, Jan 2, 2019 at 2:39 PM Tatiana Tereshchenko <
>>>>> ttere...@redhat.com> wrote:
>>>>>
>>>>>> It would be automatic, and plugins need a change only to avoid
>>>>>> redundant prepending.
>>>>>> E.g. If RPM plugin makes no changes, the endpoint for RPM content
>>>>>> will be:
>>>>>>
>>>>>> /api/v3/content/pulp_rpm/rpm/packages/
>>>>>>
>>>>>> because endpoint_name = 'rpm/packages'.
>>>>>>
>>>>>> So plugin should leave only endpoint_name = 'pack

Re: [Pulp-dev] Renaming Content 'artifact' to '_artifact'

[Austin Macdonald]

Gotcha. /me shrugs

However we decide to implement this, its pretty evident that this field
will eventually be a core field, or at the very least treated like a core
field. So I can confidently say that yes, it should be renamed.

On Mon, Jan 7, 2019 at 11:06 AM Daniel Alley  wrote:

> The serializer just needs to remove the _artifacts field and add an
>> _artifact field. Here's how I did it in docker, which is a total ripoff of
>> the file plugin.
>>
>
> I know it's fairly simple to do manually, I just meant to do so
> automatically (unless we also need a serializer mixin like you said).
>
> On Mon, Jan 7, 2019 at 10:41 AM Austin Macdonald 
> wrote:
>
>> The serializer just needs to remove the _artifacts field and add an
>> _artifact field. Here's how I did it in docker, which is a total ripoff of
>> the file plugin.
>>
>> https://github.com/pulp/pulp_docker/pull/291/
>>
>> It might be worth making a serializer mixin also? (I can almost hear
>> jortel cringing about all these mixins)
>>
>> On Mon, Jan 7, 2019 at 10:32 AM Daniel Alley  wrote:
>>
>>> Given that single-artifact Content is likely to be a very common pattern
>>>> among plugins, maybe it would be best to add this as a mixin for pulpcore.
>>>> If that's the future of this field, we should definitely make it _artifact.
>>>
>>>
>>> +1 to this, I don't much like having to redefine this in every plugin.
>>> I'm curious about how to make it work with the serializers though.
>>>
>>> On Mon, Jan 7, 2019 at 10:13 AM Austin Macdonald 
>>> wrote:
>>>
>>>> We have single-artifact Content in Docker as well. I've gone ahead and
>>>> named the field _artifact.
>>>>
>>>> Given that single-artifact Content is likely to be a very common
>>>> pattern among plugins, maybe it would be best to add this as a mixin for
>>>> pulpcore. If that's the future of this field, we should definitely make it
>>>> _artifact.
>>>>
>>>> On Wed, Jan 2, 2019 at 12:24 PM David Davis 
>>>> wrote:
>>>>
>>>>> In most plugins, Content only has a single artifact so we created a
>>>>> virtual field 'artifact' that we expose to end users. In a recent
>>>>> change[0], we prefixed the Content fields with underscores ('_') so we're
>>>>> considering renaming the field to '_artifact' to be consistent with other
>>>>> plugins that have '_artifacts' on Content. We could use some feedback by
>>>>> sprint planning (Jan 4) either here or on the issue:
>>>>>
>>>>> https://pulp.plan.io/issues/4282
>>>>>
>>>>> [0] https://pulp.plan.io/issues/4206
>>>>>
>>>>> David
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Renaming Content 'artifact' to '_artifact'

[Austin Macdonald]

The serializer just needs to remove the _artifacts field and add an
_artifact field. Here's how I did it in docker, which is a total ripoff of
the file plugin.

https://github.com/pulp/pulp_docker/pull/291/

It might be worth making a serializer mixin also? (I can almost hear jortel
cringing about all these mixins)

On Mon, Jan 7, 2019 at 10:32 AM Daniel Alley  wrote:

> Given that single-artifact Content is likely to be a very common pattern
>> among plugins, maybe it would be best to add this as a mixin for pulpcore.
>> If that's the future of this field, we should definitely make it _artifact.
>
>
> +1 to this, I don't much like having to redefine this in every plugin.
> I'm curious about how to make it work with the serializers though.
>
> On Mon, Jan 7, 2019 at 10:13 AM Austin Macdonald 
> wrote:
>
>> We have single-artifact Content in Docker as well. I've gone ahead and
>> named the field _artifact.
>>
>> Given that single-artifact Content is likely to be a very common pattern
>> among plugins, maybe it would be best to add this as a mixin for pulpcore.
>> If that's the future of this field, we should definitely make it _artifact.
>>
>> On Wed, Jan 2, 2019 at 12:24 PM David Davis 
>> wrote:
>>
>>> In most plugins, Content only has a single artifact so we created a
>>> virtual field 'artifact' that we expose to end users. In a recent
>>> change[0], we prefixed the Content fields with underscores ('_') so we're
>>> considering renaming the field to '_artifact' to be consistent with other
>>> plugins that have '_artifacts' on Content. We could use some feedback by
>>> sprint planning (Jan 4) either here or on the issue:
>>>
>>> https://pulp.plan.io/issues/4282
>>>
>>> [0] https://pulp.plan.io/issues/4206
>>>
>>> David
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Renaming Content 'artifact' to '_artifact'

[Austin Macdonald]

We have single-artifact Content in Docker as well. I've gone ahead and
named the field _artifact.

Given that single-artifact Content is likely to be a very common pattern
among plugins, maybe it would be best to add this as a mixin for pulpcore.
If that's the future of this field, we should definitely make it _artifact.

On Wed, Jan 2, 2019 at 12:24 PM David Davis  wrote:

> In most plugins, Content only has a single artifact so we created a
> virtual field 'artifact' that we expose to end users. In a recent
> change[0], we prefixed the Content fields with underscores ('_') so we're
> considering renaming the field to '_artifact' to be consistent with other
> plugins that have '_artifacts' on Content. We could use some feedback by
> sprint planning (Jan 4) either here or on the issue:
>
> https://pulp.plan.io/issues/4282
>
> [0] https://pulp.plan.io/issues/4206
>
> David
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing plugins, looking for feedback

[Austin Macdonald]

+1 automatic namespacing for master/detail. I realize the easiest way to do
this would be to use the app_label, giving us:

/api/v3/content/pulp_rpm/packages/


However, I feel like this url is pretty clunky. The "pulp_" is totally
unnecessary, from the user's perspective. Instead, I think I'd prefer to
add an attribute to the App config.
https://github.com/pulp/plugin_template/blob/master/pulp_plugin_template/app/__init__.py#L8

`endpoint_namespace = rpm` or `short_label = rpm`

Result: /api/v3/content/rpm/packages/

The downside is that every plugin would need 1 more line of code. The
upside is that we could implement it exactly same way as app_label but
without url redundancy.

On Wed, Jan 2, 2019 at 2:39 PM Tatiana Tereshchenko 
wrote:

> It would be automatic, and plugins need a change only to avoid redundant
> prepending.
> E.g. If RPM plugin makes no changes, the endpoint for RPM content will be:
>
> /api/v3/content/pulp_rpm/rpm/packages/
>
> because endpoint_name = 'rpm/packages'.
>
> So plugin should leave only endpoint_name = 'packages'.
>
> The endpoint with redundant plugin name will work fine, just doesn't look 
> good :)
>
> Tanya
>
>
> On Wed, Jan 2, 2019 at 7:20 PM David Davis  wrote:
>
>> I am +1 to namespacing all master detail models with the plugin name.
>> Would this be automatic or something that the plugin writers would be
>> encouraged to do?
>>
>> David
>>
>>
>> On Wed, Jan 2, 2019 at 12:58 PM Tatiana Tereshchenko 
>> wrote:
>>
>>> Thank you all for the discussion so far.
>>> The question - the type field and namespacing in content summary - is
>>> solved with https://pulp.plan.io/issues/4185.
>>>
>>> The last remaining question is whether we want to prepend endpoints for
>>> master/detail models with plugin label. If yes, then everything or for
>>> Content only.
>>> See details on the issue https://pulp.plan.io/issues/4279.
>>>
>>> Examples of the suggested change:
>>>
>>> /api/v3/content/rpm/packages/ --> /api/v3/content/pulp_rpm/packages/
>>> /api/v3/remotes/rpm/ --> /api/v3/content/remotes/pulp_rpm/rpm/
>>> /api/v3/publishers/rpm/ --> /api/v3/content/publishers/pulp_rpm/rpm/
>>>
>>> Changes which will be needed in plugins:
>>>   - adjust the value of the `endpoint_name` attribute in the viewsets we 
>>> introduce changes to
>>>
>>> Please provide feedback, here or on the issue
>>> https://pulp.plan.io/issues/4279.
>>> This is an RC blocker, so it would be great to groom it over the next
>>> couple of days.
>>>
>>> Thank you,
>>> Tanya
>>>
>>> On Thu, Dec 20, 2018 at 9:41 AM Tatiana Tereshchenko <
>>> ttere...@redhat.com> wrote:
>>>
 Since we are leaning towards prepending types for _all_ master/detail
 models and not only for the content model, that Django fix is no longer
 important for us.

 Tanya

 On Wed, Dec 19, 2018 at 6:18 PM Daniel Alley  wrote:

> David, was that a vote to make it explicit?
>
> I would regard this as fairly intuitive as far as "magic-ness" goes,
> acceptable from the user POV in my opinion.  And if Django is explicitly
> trying to support this functionality and relies on it working properly, 
> and
> has a unittest for it going forwards, then I'm fairly confident it won't 
> be
> too fragile.  My vote is +1.
>
> My only concern (and it's not a major one) is that a plugin that
> needed to be renamed might have problems with this.  But I think that 
> would
> be resolvable with a migration.
>
> Tanya, will we need to remove the workaround once Django 2.2 comes
> out?  If so, we should file a Refactor task for it.
>
> On Tue, Dec 18, 2018 at 9:39 AM David Davis 
> wrote:
>
>> +1
>>
>> David
>>
>>
>> On Tue, Dec 18, 2018 at 9:31 AM Brian Bouterse 
>> wrote:
>>
>>> There is also an issue w/ my suggestion in that it's highly magical.
>>> The class name is likely going to go through a case mutation and if not
>>> it's going to be finicky in terms of its case. So now I'm thinking the
>>> plugin writer should have to define it to keep it simple and explicit 
>>> (not
>>> implicit and magical).
>>>
>>> On Tue, Dec 18, 2018 at 9:27 AM David Davis 
>>> wrote:
>>>
 Would it be possible to default to class name but let plugin
 writers override this? I would imagine in some cases plugin writers 
 might
 want to change the name (eg cases where you can't use type as the class
 name like File).

 David


 On Tue, Dec 18, 2018 at 9:23 AM Brian Bouterse 
 wrote:

>
>
> On Tue, Dec 18, 2018 at 9:07 AM Tatiana Tereshchenko <
> ttere...@redhat.com> wrote:
>
>> Brian,
>> the current PR [0] does exactly what you describe, it uses label
>> which is taken from the plugin subclass of PulpPluginAppconfig + TYPE
>> defined on the 

Re: [Pulp-dev] Spam on plan.io

[Austin Macdonald]

On Wed, Oct 31, 2018, 3:57 PM Austin Macdonald  On Wed, Oct 31, 2018, 2:23 PM Daniel Alley 
>> Maybe the first comment / issue posted by an account would need to be
>> approved, but once approved they could post subsequent comments / issues
>> without delay?
>>
>>
> @dalley, sounds right to me. I think this could be implemented using
> bmbouters b) option, with 1 difference. If the user can't even file until
> approved, I think we shouldn't do it. If the user can file an invisible
> issue, I'm ok with this.
>
> On Wed, Oct 31, 2018 at 1:28 PM, Brian Bouterse 
> wrote:
>
>> b) create a "trusted users" group and have that allow users to either
>> post comments, post issues, or both and then disable those permissions for
>> "other accounts". This would prevent a new user from filing a bug in a
>> self-service way though.
>>
>
> b) Story >>> A new user is created, they file an issue. Issue is not
> visible until approved. When issue is approved, user is moved to "trusted
> user" group. Further issues are not delayed.
>
> This would fix the problem at the cost of delaying response to new
> contributors at a critical time, right after their first contribution.
> Using "trusted users" would allow us to filter out most issues,
> significantly reducing the workload to review for spam.
>

Nothing has changed except my patience. Ugh.

IMO we need to remove the incentive, which means hiding the first
issue/comment of new users.

Unless anyone is strongly against this, I'll file an issue and we can
discuss the technical details there.


> However, we could also users "trusted users" as an invisible flag that
> makes no difference to the user. This would be the exact same amount of
> work as b) for us, but new contributor issues are always visible. So after
> all this, I'm leaning toward a) + 1/2 b)
> On Wed, Oct 31, 2018 at 1:28 PM, Brian Bouterse 
> wrote:
>
>> a) manage the spam better
>>
>
> a) Story >>> A new user is created they file an issue. Issue is visible
> immediately. Spam review must review every new issue from every user.
>
> a) + 1/2 b) Story >>> A new user is created, they file an issue. Issue is
> visible immediately. Issue is flagged internally for spam review, if not
> spam, user is added to trusted group. Further issues would skip this
> process.
>
> I have one last thought that might make b) more attractive, but its a shot
> in the dark. Since the spam is coming from humans, someone is paying them.
> If we never show the spam, we remove the incentive, and hopefully someone
> will notice and stop it. If y'all think this is how things woud go down, we
> could always do b) until the problem stops and switch to a) + 1/2 b).
>

>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Possible Pulp3 RC Blocker issues from backlog

[Austin Macdonald]

For those with ambiguity, I added the RC blocker to force discussion and
[acceptance | closing].

Added RC Blocker:

   - Add task names: https://pulp.plan.io/issues/2889
   - Determine mutable fields: https://pulp.plan.io/issues/2635
   - pulp-manager migrate order: https://pulp.plan.io/issues/3062
  - @david - https://pulp.plan.io/issues/4067#note-5
   - Asynchronous Distribution update/delete:
   https://pulp.plan.io/issues/3044
   - Distribution base_path model validation:
   https://pulp.plan.io/issues/3051

Closed:

   - Viewable status endpoint w/out database running:
   https://pulp.plan.io/issues/2850
   - Port Dependencies to Python3: https://pulp.plan.io/issues/2247
   - Plugins can specify plugin API version:
   https://pulp.plan.io/issues/2656

No action:

   - jwt: https://pulp.plan.io/issues/3248
   - Add Publication.created (MODIFIED, david++):
   https://pulp.plan.io/issues/2989


On Mon, Dec 3, 2018 at 3:21 PM David Davis  wrote:

> Thanks for digging through older issues to find potential RC blockers.
>
> 2889 - +1 to making it an RC blocker
> 2635 - +1 here as well
> 2850 - I spent some time working on this and didn’t get far. I think we
> should just require the db to be running. I vote to close it out.
> 2989 - +1 to RC blocker
> 3044 - I guess we should revisit 3051 and decide on a design before the RC
> which will determine if the distribution endpoints need to be async?
> 2247 - Agreed on closing. Seems like we open issues on an as-needed basis
> 2656 - Seems like this is done or am I missing something?
> 3062 - Will checking in migrations to source control not solve this
> problem?
> 3248 - I haven’t heard anyone asking for jwt so I would say we don’t need
> it. We can just leave the issue open I think.
>
> David
>
>
> On Mon, Dec 3, 2018 at 2:41 PM Austin Macdonald  wrote:
>
>> To be on the safe side, I'd like to highlight issues that *might* need to
>> be RC blockers. Please reply directly onto the issue, I'll update this
>> thread periodically if necessary.
>>
>> REST API, backwards incompatible changes:
>>
>>- Add Task Names:
>>   - https://pulp.plan.io/issues/2889
>>   - IMO: We should make this an RC Blocker, because this will be an
>>   additional requirement for every task in every plugin.
>>- Determine mutable fields
>>   - https://pulp.plan.io/issues/2635
>>   - IMO: someone (or a group) should take this as assigned and audit
>>   the mutability of fields. If we find one that needs to change, it will 
>> be a
>>   backwards incompatible change to the REST API, so this should have the 
>> RC
>>   blocker tack.
>>- Status API without db connection
>>   - https://pulp.plan.io/issues/2850
>>   - IMO: RC blocker or close. As it is the db connection field is
>>   not useful, and later removal would be backwards incompatible.
>>- Add new field, Publication.created
>>   - https://pulp.plan.io/issues/2989
>>   - IMO: RC blocker or close, this would be a backwards incompatible
>>   change.
>>- Asynchronous Distribution update/delete
>>   - https://pulp.plan.io/issues/3044
>>   - IMO: RC blocker or close, this would be a backwards incompatible
>>   change.
>>
>> Packaging
>>
>>- Port dependencies to Python 3
>>   - https://pulp.plan.io/issues/2247
>>   - IMO: It seems like if this weren't done, we'd be having
>>   problems. Anyone mind if I close this one? If we do need to keep it 
>> open,
>>   should it be an RC blocker?
>>- Plugins can declare PluginAPI version
>>   - https://pulp.plan.io/issues/2656
>>   - IMO: Are we happy with what we've got now? If we want to change
>>   it, now is the time.
>>
>> Misc
>>
>>- pulp-manager migrate order
>>   - https://pulp.plan.io/issues/3062
>>   - IMO: RC Blocker. This is how users should migrate, so it should
>>   be correct before RC
>>- jwt
>>   - https://pulp.plan.io/issues/3248
>>   - This was removed from Beta (MVP) but do we need this for RC/GA?
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal to remove 'notes' fields from the Pulp 3 RC

[Austin Macdonald]

+1, nice catch Daniel.

On Tue, Dec 4, 2018, 7:14 AM David Davis  Big +1. This seems like something we could add later on at any time when
> we need it. Hopefully we can get Katello and our users involved in sussing
> out the requirements for how this field should work too.
>
> David
>
>
> On Mon, Dec 3, 2018 at 11:34 PM Daniel Alley  wrote:
>
>> *Background:*
>>
>> "Notes" are a generic key value store where data can be attached to
>> repositories and content and publications and so forth.  The eventual plan
>> is to use this to enable adding tags to those sorts of objects, which is
>> important for Katello.
>>
>> Most of the code for this is located in pulp/app/models/generic.py
>>
>> *Motivation:*
>>
>> "Notes" have been in Pulp 3 for a very very long time and are completely
>> unchanged for the last 12 months (the git history doesn't go back further
>> because the file was moved).  The data model behind it is extremely complex
>> and while we have a few unit tests around it, we have no functional tests
>> for it whatsoever, and (to my knowledge) we haven't been using/exercising
>> this functionality manually in a meaningful way (if at all).  I could be
>> wrong here, but I haven't seen it discussed  or any issues related to it
>> filed in quite some time.
>>
>> *Proposal:*
>>
>> We should pull out all of the "notes" code (models/generic.py + the
>> fields on the aforementioned models) until we've had a chance to properly
>> evaluate our needs and whether the current design fits them.
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Auto-distribution

[Austin Macdonald]

Yes, and AFAIK this is already complete. There are 2 fields on the
Distribution that allow auto-distribution. These fields must both be set,
and when they are, new publications will automatically update the
distribution.

https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/serializers/repository.py#L281-L282
https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/serializers/repository.py#L301-L302

On Tue, Nov 27, 2018 at 9:14 AM Kersom  wrote:

> Pulp 2 has the concept of auto-publish.[0]
>
> Are we creating something like "auto-distribution" or something like that
> for Pulp 3?
>
> I could not find any related issue.
>
> [0]
> https://docs.pulpproject.org/dev-guide/integration/rest-api/repo/cud.html?highlight=auto_publish
>
> Regards,
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Spam on plan.io

[Austin Macdonald]

On Wed, Oct 31, 2018, 2:23 PM Daniel Alley  Maybe the first comment / issue posted by an account would need to be
> approved, but once approved they could post subsequent comments / issues
> without delay?
>
>
@dalley, sounds right to me. I think this could be implemented using
bmbouters b) option, with 1 difference. If the user can't even file until
approved, I think we shouldn't do it. If the user can file an invisible
issue, I'm ok with this.

On Wed, Oct 31, 2018 at 1:28 PM, Brian Bouterse  wrote:

> b) create a "trusted users" group and have that allow users to either post
> comments, post issues, or both and then disable those permissions for
> "other accounts". This would prevent a new user from filing a bug in a
> self-service way though.
>

b) Story >>> A new user is created, they file an issue. Issue is not
visible until approved. When issue is approved, user is moved to "trusted
user" group. Further issues are not delayed.

This would fix the problem at the cost of delaying response to new
contributors at a critical time, right after their first contribution.
Using "trusted users" would allow us to filter out most issues,
significantly reducing the workload to review for spam.

However, we could also users "trusted users" as an invisible flag that
makes no difference to the user. This would be the exact same amount of
work as b) for us, but new contributor issues are always visible. So after
all this, I'm leaning toward a) + 1/2 b)
On Wed, Oct 31, 2018 at 1:28 PM, Brian Bouterse  wrote:

> a) manage the spam better
>

a) Story >>> A new user is created they file an issue. Issue is visible
immediately. Spam review must review every new issue from every user.

a) + 1/2 b) Story >>> A new user is created, they file an issue. Issue is
visible immediately. Issue is flagged internally for spam review, if not
spam, user is added to trusted group. Further issues would skip this
process.

I have one last thought that might make b) more attractive, but its a shot
in the dark. Since the spam is coming from humans, someone is paying them.
If we never show the spam, we remove the incentive, and hopefully someone
will notice and stop it. If y'all think this is how things woud go down, we
could always do b) until the problem stops and switch to a) + 1/2 b).
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] commit-bit nomination

[Austin Macdonald]

+1, I agree that Milan has met the criteria. I think we should go forward
with giving him the bit and larger discussions about decision making can
happen separately.

On Mon, Sep 10, 2018 at 9:03 AM Ina Panova  wrote:

> Brian,
>
> it feels like your reply is in contradiction of what we are trying to
> achieve as a community project:
>
> 1) I am concerned that this reply can be perceived very badly by the
> community, especially once we approved the pup and announced this in hope
> of community embracing this change, by encouraging new contributions and
> motivating them with one of the retributions of becoming a committer.
> 2) We cannot afford ourselves to behave like a dental clinic which will
> say: we are too full and we are not accepting new patients ATM. There are
> community projects with a larger committer base and somehow they still
> manage to develop and evolve.
> 3) Do you suggest splitting the team in 2 with the acknowledgement of the
> fact that pulp2 is near EOF which means those people will be pinned just
> for the maintenance mode? I wonder if we have such volunteers doing this
> job daily. Anyone? :)
>
> I agree with Dana, we adopted the pup, team has voted, now we need to deal
> with the decisions we as a project have made.
>
> I think Milan has met the criteria listed in the pup, and if he has
> systematically demonstrated diligence and commitment i do not really think
> once he gets the commit bit he will go right away and screw up pulp3 code
> base because he has less experience in that area than in another.
>
> I understand that in this email thread have popped up multiple problems we
> are struggling with, but I would at least try to stay focused on the
> initial email topic and create separate discussions for the other topics.
>
> +1 on the voting.
>
> I also solicit the team to be pro-active, so we can fail fast things which
> do not work out for us.
>
>
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Fri, Sep 7, 2018 at 2:17 AM, Dana Walker  wrote:
>
>> I respectfully disagree.
>>
>> I was hoping for more discussion before the calling of a vote.
>>>
>>
>> The process described in PUP6 specifically states that the nominating
>> email should have a vote end date, thus calling a vote.  There's nothing
>> wrong with having discussion now on this thread, as part of explaining your
>> stance or asking questions before deciding, but voting is still actively
>> taking place.  People can always change their vote based on the results of
>> discussion up to the vote end date in seven calendar days.
>>
>> With Pulp2 nearing maintenance mode, the core and plugin teams need to
>>> assess what their needs are both in code and people. I feel that with 9
>>> people on the core team, maintaining vision is hard/impossible, and the
>>> mailing list discussions have demonstrated that.
>>>
>>
>> No, what the mailing list discussions have demonstrated is that we have a
>> problem with our decision making process.  Plenty of projects have larger
>> teams, but they usually have a team lead for breaking ties and ensuring
>> that forward progress is made even with difficult decisions where a team is
>> split on what to do.  The bright side is, no matter what decisions are made
>> on any of those discussions, the project moves forward, and can always
>> pivot in future versions based on feedback or with community submitted open
>> source work.  But decisions must be made and if our lazy consensus method
>> where even a single -1 is blocking prevents work from moving forward and
>> releases from arriving on schedule, it's time we reconsider that process,
>> and that is a separate discussion for another place and time.
>>
>> Also consider that Pulp3 is an order of magnitude smaller codebase
>>> (literally) so keeping the same number of committers seems too many.
>>>
>>
>> Again, I disagree with this focus on number of committers for many
>> reasons.
>>
>> 1) @bizhang just left the team and resigned a commit bit, so there is now
>> an opening and if there had been a problem with the number of commit bits
>> before today, then that should have been brought up previously, perhaps by
>> having no grandfathered-in committers and all team members voted on by
>> majority when this decision was made last fall, or perhaps by explicitly
>> limiting the number of commit bits in the PUP.
>>
>> 2) We just had lengthy discussions in meetings a week ago about ways to
>> improve the speed with which we respond to PRs.  There were suggestions
>> ranging from email notifications of every one that is submitted, a weekly
>> PR triage, having subject matter "experts" designated to divide up the
>> work, or it being on the shoulders of the pull requesters to continually
>> poke channels/individuals for a response until they get one.  The response
>> to all of this was simply that we are all too 

Re: [Pulp-dev] Proposal to drop support of Python 3.5 for Pulp 3

[Austin Macdonald]

Seems reasonable to me. Even without python 3.6 in the official
repositories for Debian and trusty Ubuntu, it isn't very difficult to
install 3.6 with tools like anaconda and pyenv. From my perspective,
dropping 3.5 would introduce short term inconvenience for those systems,
and simplify our long term maintenance.

On Fri, Sep 7, 2018, 8:51 AM Brian Bouterse  wrote:

> I'm also +1 on this for the various reasons. Even with the portability
> downsides. Also, practically speaking, the batch code is legit, and it will
> benfit the stages code a lot.
>
> On Fri, Sep 7, 2018 at 8:17 AM, Daniel Alley  wrote:
>
>> Personally, +1.  I ran into this issue myself and it was infuriating to
>> deal with.
>>
>> dict objects preserve insertion-order (officially declared part of
>>>   the language with Python 3.7). Eliminates a source of subtle
>>>   "works on 3.6, sometimes works on 3.5" bugs.
>>>
>>
>> Just to expand on this though:
>>
>> For Ubuntu, it is part of 18.04 LTS. Debian does not have Python 3.6 in
>>> stable yet.
>>>
>>
>> It does mean that we won't be able to support Ubuntu 16.04 (or previous)
>> or Debian at all until the next release (~May 2019), unless they install a
>> newer Python themselves.
>>
>> On Fri, Sep 7, 2018 at 2:09 AM, Simon Baatz  wrote:
>>
>>> I had a discussion on IRC with Brian yesterday which led to the
>>> question whether we can drop support for Python 3.5. I think there are
>>> good reasons for this, see the rationale below.
>>>
>>> Brian proposed to initiate a vote on this topic (and find out whether
>>> this "community thing" works :-) ).
>>>
>>> Please send feedback by Friday Sept 14th. Especially, let me know if
>>> there are specific reasons for depending on Python 3.5. The
>>> corresponding issue is 3984 [7].
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>> Rationale:
>>>
>>> The trigger for the discussion was to get rid of boilerplate code like
>>> this [0], [1] to handle batches in the stages API. This becomes a
>>> single line [2] when using an asynchronous generator [3]. Adding the
>>> `batches()` async generator to Pulp core would simplify existing
>>> stages and ease implementation of stages in plugins.
>>>
>>> Async generators have been introduced in Python 3.6. Thus, to make the
>>> `batches` generator available in the Pulp core plugin API, we either
>>>
>>> - have to drop support for Python 3.5 or
>>>
>>> - reimplement the async generator as an async iterator (which would be
>>>   more convoluted but looks doable)
>>>
>>>
>>> I prefer to drop 3.5, since this will allow to use additional language
>>> features[4]. Among them:
>>>
>>> - As said, async generators/async comprehensions. Async generators are
>>>   simpler to write and understand than async iterators.
>>>
>>> - String interpolation "f-Strings" [5]
>>>
>>> - dict objects preserve insertion-order (officially declared part of
>>>   the language with Python 3.7). Eliminates a source of subtle
>>>   "works on 3.6, sometimes works on 3.5" bugs.
>>>
>>> - One version less to support is always a good thing (provided nobody
>>>   really requires it)
>>>
>>> - Type annotations are currently not used by the Pulp project, but if
>>>   the project decides to use them in the future: IMHO type annotations
>>>   (which are great btw.) began to feel “right† with 3.6. Working with
>>>   them in 3.5 can be clumsy at times.
>>>
>>> - And of course: [6]
>>>
>>>
>>> Python 3.6 has the OS/distribution support we need:
>>>
>>> - Python 3.6 SCL is available for RHEL 7 / CentOS 7
>>> - It is part of Fedora as of Fedora 26
>>>
>>> For Ubuntu, it is part of 18.04 LTS. Debian does not have Python 3.6 in
>>> stable yet.
>>>
>>>
>>>
>>> [0]
>>> https://github.com/pulp/pulp/blob/631031e38270c5c7c2b2289ff4ab87a058447c5e/plugin/pulpcore/plugin/stages/content_unit_stages.py#L47-L59
>>> [1]
>>> https://github.com/pulp/pulp/blob/631031e38270c5c7c2b2289ff4ab87a058447c5e/plugin/pulpcore/plugin/stages/artifact_stages.py#L48-L60
>>> [2]
>>> https://github.com/gmbnomis/pulp_cookbook/blob/ca4882cecab16995c5713d27131da8112a5f5a0c/pulp_cookbook/app/tasks/synchronizing.py#L98
>>> [3]
>>> https://github.com/gmbnomis/pulp_cookbook/blob/d44ed593925b78c046e1b568810b15acbdad5ac4/pulp_cookbook/app/tasks/synchronizing.py#L26
>>> [4] https://docs.python.org/3/whatsnew/3.6.html
>>> [5]
>>> https://docs.python.org/3/whatsnew/3.6.html#pep-498-formatted-string-literals
>>> [6] https://twitter.com/raymondh/status/844955415259463681
>>> [7] https://pulp.plan.io/issues/3984
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

Re: [Pulp-dev] Theme of the bachelor thesis

[Austin Macdonald]

How about reproducibility? Bihan and I presented on this topic at scipy but
there is a lot left to say/research.
https://www.youtube.com/watch?v=5czGgUG0oXA

In the scientific community, reproducibility is a hot topic (I bet grad
schools would love this!), and the computational aspect of it has been
under-emphasized in the existing literature. Our talk focused on the
fundamental problems of environmental reproducibility and how to use Pulp
to solve them. Especially since the scope was narrow, we barely scratched
the surface. There a ton of potential angles; some ideas are that you could
catalog and compare technologies or discuss academic vs industry use cases
and trends.

On Wed, Sep 5, 2018 at 5:01 AM Vladimir Dusek  wrote:

> I don't ask you for a specific assignment, we'll set up it with Milan
> later. I only need some idea on a topic.
>
> Scope of the work should be 40-50 standard pages. Additional istructions
> could be find here -
> http://www.fit.vutbr.cz/info/szz/pokyny_bp.php.en#pozadavky. However I
> don't think it's neccessary for you to know them.
>
> All bachelor theses from our university last year could be find here -
> http://www.fit.vutbr.cz/study/DP/BP.php.en. I presume most of them are
> written in Czech but there are abstracts in English which might be useful
> for inspiration.
>
> Random abstracts:
>
> Alias Analysis in C Compiler
> -
> This thesis is dedicated to the problem of alias analysis and
> possibilities of its improvement in the LLVM framework. The goal of this
> thesis is to improve the accuracy, which was achieved by extending the
> existing implementation of Andresen algorithm to be field sensitive. The
> terms related to alias analysis and algorithms of the alias analysis
> available in LLVM are explained. These algorithms are compared according to
> their base idea, features, and limitations. The implementation of the field
> sensitivity has been tested using compiler test suites. Its impact on
> program compilation speed and performance has been analyzed. The measured
> results show an increase in the accuracy of alias analysis in the LLVM
> framework.
>
> Reinforcement Learning for Starcraft Game Playing
> 
> This work focuses on methods of machine learning for playing real-time
> strategy games. The thesis applies mainly methods of Q-learning based on
> reinforcement learning. The practical part of this work is implementing an
> agent for playing Starcraft II. Mine solution is based on 4 simple
> networks, that are colaborating together. Each of the network also teaches
> itself how to process all given actions optimally. Analysis of the system
> is based on experiments and statistics from played games.
>
>
> On Tue, Sep 4, 2018 at 8:27 PM Robin Chan  wrote:
>
>> Hi Vladimir,
>>
>> I'm not familiar with what types of topics are appropriate for this type
>> of project - could you share some criteria and examples of what makes a
>> good topic?
>>
>> Robin
>>
>> On Mon, Sep 3, 2018 at 8:50 AM, Vladimir Dusek  wrote:
>>
>>> Hi team,
>>>
>>> since September I'm in the third grade of bachelor studies and it means
>>> that I'm going to write a bachelor thesis. I haven't selected a topic yet
>>> and if it'll be possible I'd like to work on something that is relevant to
>>> Pulp. Do you have any idea?
>>>
>>> Thank you guys,
>>> Vlada
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposed Update to PUP-4

[Austin Macdonald]

Sounds good to me. I'll get start a new thread after this one goes through.

On Tue, Jul 24, 2018 at 1:59 PM, David Davis  wrote:

> +1 to this update. Thanks @dawalker!
>
> +1 to having a separate discussion for official recognition of plugins. I
> have some questions/thoughts.
>
> David
>
>
> On Tue, Jul 24, 2018 at 1:55 PM Dana Walker  wrote:
>
>> Personally, I think those are important suggestions for ensuring a
>> welcoming environment across the Pulp landscape, but I would prefer that
>> this update focus on getting the CoC itself publicly discoverable for the
>> first time without further delay .  I welcome you to submit a PR
>> with another update, even at the same time, that deals with requirements
>> for plugin writers .
>>
>> If others would prefer this to be bundled, we can work together on an
>> updated PR.
>>
>> --Dana
>>
>> Dana Walker
>>
>> Associate Software Engineer
>>
>> Red Hat
>>
>> <https://www.redhat.com>
>> <https://red.ht/sig>
>>
>> On Tue, Jul 24, 2018 at 1:20 PM, Austin Macdonald 
>> wrote:
>>
>>>
>>>
>>> On Tue, Jul 24, 2018 at 12:20 PM, Austin Macdonald 
>>> wrote:
>>>
>>>>
>>>> I'm not sure how to phrase this, but maybe something like "For a plugin
>>>> to be officially recognized by the Pulp community*, it must adopt this code
>>>> of conduct in its entirety, though it is their prerogative to extend it."
>>>>
>>>> * Official recognition is not official endorsement, but does include
>>>> some perks:
>>>>
>>>>1. ability to track plugin issues on the Pulp redmine
>>>>2. plugin source may live in the Pulp org on github
>>>>3. Plugin will be listed on the Plugin table
>>>>
>>>> ^ This could be a separate PUP revision, if preferred.
>>>
>>>
>>>> On Tue, Jul 24, 2018 at 11:37 AM, Dana Walker 
>>>> wrote:
>>>>
>>>>> A PR [0] is up with a proposed update to PUP-4: Code of Conduct [1].
>>>>>
>>>>> The proposal changes the details of where the Code of Conduct is to be
>>>>> displayed.
>>>>>
>>>>> Please review and provide any comments, suggestions, and +/- 1's to
>>>>> the proposed changes by Sunday August 5.
>>>>>
>>>>> [0] https://github.com/pulp/pups/pull/12
>>>>> [1] https://github.com/pulp/pups/blob/master/pup-0004.md
>>>>>
>>>>> Thanks,
>>>>>
>>>>> --Dana
>>>>>
>>>>> Dana Walker
>>>>>
>>>>> Associate Software Engineer
>>>>>
>>>>> Red Hat
>>>>>
>>>>> <https://www.redhat.com>
>>>>> <https://red.ht/sig>
>>>>>
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposed Update to PUP-4

[Austin Macdonald]

I'm a firm +1 on this, but I do have a related thought.

I think I speak for everyone, but please correct me if I'm wrong, one of
our goals is to have an inclusive community that has very very simple
requirements to be a part of. At the moment, I think we may have taken this
as far as it can go, the Pulp community literally contains everyone who
wants to be in. This is awesome! However, if there is only one thing that
we enforce to be a part of the community, it ought to be the CoC.

I don't think we ought to get crazy, but what I have in mind is that the
Plugin table in our docs shows only plugins that have a code of conduct
that *includes*
https://github.com/pulp/pups/blob/master/pup-0004.md#contributor-covenant-code-of-conduct
 This would ensure that anyone who participates in the community is
protected, and it gives sub-communities the ability to go farther if they
choose.

I'm not sure how to phrase this, but maybe something like "For a plugin to
be officially recognized by the Pulp community*, it must adopt this code of
conduct in its entirety, though it is their prerogative to extend it."

* Official recognition is not official endorsement, but does include some
perks:

   1. ability to track plugin issues on the Pulp redmine
   2. plugin source may live in the Pulp org on github
   3. Plugin will be listed on the Plugin table

I don't want to hold this up for potential complexities of implementing ^,
but if plugins are to be considered a part of the community (and they must)
then we need to provide something more powerful than "encouragement" to
ensure that the entire community is inclusive.

On Tue, Jul 24, 2018 at 11:37 AM, Dana Walker  wrote:

> A PR [0] is up with a proposed update to PUP-4: Code of Conduct [1].
>
> The proposal changes the details of where the Code of Conduct is to be
> displayed.
>
> Please review and provide any comments, suggestions, and +/- 1's to the
> proposed changes by Sunday August 5.
>
> [0] https://github.com/pulp/pups/pull/12
> [1] https://github.com/pulp/pups/blob/master/pup-0004.md
>
> Thanks,
>
> --Dana
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Revising PUPs

[Austin Macdonald]

+1

I suggested an addition, to increment the PUP version whenever a change is
made. I considered suggesting a version scheme to indicate major and minor
changes, but AFAICT there isn't a practical need beyond a simple integer.
If the process is too cumbersome for typo fixing (for example), we can
adjust it later.

On Wed, Jul 11, 2018 at 8:31 AM, Ina Panova  wrote:

> +1
>
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Mon, Jul 9, 2018 at 3:22 PM, Dana Walker  wrote:
>
>> +1
>>
>> Dana Walker
>>
>> Associate Software Engineer
>>
>> Red Hat
>>
>> 
>> 
>>
>> On Mon, Jul 9, 2018 at 9:05 AM, Daniel Alley  wrote:
>>
>>> +1
>>>
>>> On Mon, Jul 9, 2018 at 9:02 AM, Milan Kovacik 
>>> wrote:
>>>
 Hey David,

 thanks, +1

 --
 milan

 On Mon, Jul 9, 2018 at 1:49 PM, David Davis 
 wrote:

> I’ve opened a PR with the process on how to revise a PUP.
> Reviews/feedback are welcome:
>
> https://github.com/pulp/pups/pull/11
>
> I’d also like to call a vote on this proposed change. Here’s the
> voting model from PUP-1:
>
> +1: "Will benefit the project and should definitely be adopted."
> +0: "Might benefit the project and is acceptable."
> -0: "Might not be the right choice but is acceptable."
> -1: "I have serious reservations that need to be thought through and
> addressed."
>
> Deadline will be July 22, 2018.
>
> David
>
>
> On Tue, Jul 3, 2018 at 11:14 AM David Davis 
> wrote:
>
>> While there is a process for revising PUPs before they are
>> accepted[0], we don’t have any process for revising PUPs after they are
>> accepted. I’d like to upate PUP-1[1] to create a simple but formal 
>> process
>> for revising accepted PUPs.
>>
>> I was thinking we should add a section (“Revising an Accepted PUP”)
>> that says say revising a PUP follows the same process as creating a new
>> PUP. This includes an initial discussion period followed by a PR against
>> the PUP with the proposed change. After that, there should be a vote
>> decided by our existing lazy consensus model.
>>
>> Thoughts?
>>
>> [0] https://github.com/pulp/pups/blob/master/pup-0001.md#revision
>> [1] https://github.com/pulp/pups/blob/master/pup-0001.md
>>
>> David
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] bulk_create for Artifact, Content, ContentArtifact, and RemoteArtifact?

[Austin Macdonald]

On Thu, Jun 21, 2018 at 4:19 PM, Brian Bouterse  wrote:

> I'm only considering these changes for the plugin writer API to help
> resolve the performance issues.
>

Cool. +1
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Ideas for the plugin template

[Austin Macdonald]

I've written up stories for this work, which I've marked as sprint
candidates. I'd like to get them groomed this week if possible.

Testing: https://pulp.plan.io/issues/3773
Documentation: https://pulp.plan.io/issues/3772

On Mon, Jun 18, 2018 at 3:48 PM, Dennis Kliban  wrote:

> +1
>
> On Mon, Jun 18, 2018 at 3:14 PM, Dana Walker  wrote:
>
>> +1
>>
>> (Already said it during review, but figured I'd make it official here.)
>>
>> Dana Walker
>>
>> Associate Software Engineer
>>
>> Red Hat
>>
>> <https://www.redhat.com>
>> <https://red.ht/sig>
>>
>> On Mon, Jun 18, 2018 at 2:01 PM, Daniel Alley  wrote:
>>
>>> +1
>>>
>>> On Mon, Jun 18, 2018 at 12:16 PM, Brian Bouterse 
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Mon, Jun 18, 2018 at 8:47 AM, Ina Panova  wrote:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>>
>>>>> 
>>>>> Regards,
>>>>>
>>>>> Ina Panova
>>>>> Software Engineer| Pulp| Red Hat Inc.
>>>>>
>>>>> "Do not go where the path may lead,
>>>>>  go instead where there is no path and leave a trail."
>>>>>
>>>>> On Thu, Jun 14, 2018 at 10:19 PM, Bihan Zhang 
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>> I think the plugin_template is very valuable for bootstrapping plugin
>>>>>> development, but we have had issues with keeping it up to date. Creating 
>>>>>> a
>>>>>> smash test that will enforce this on new PRs make perfect sense to me.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 14, 2018 at 11:29 AM, Austin Macdonald >>>>> > wrote:
>>>>>>
>>>>>>> I've recently updated the plugin_template to work with the latest
>>>>>>> master (3.0). [0] The template handles almost all of the
>>>>>>> bootstrapping
>>>>>>> work necessary to write a new plugin, so it is valuable to keep it
>>>>>>> up to
>>>>>>> date. Given human nature, it's likely that the plugin_template will
>>>>>>> tend
>>>>>>> to fall behind as it did recently. I have some ideas to save time
>>>>>>> while
>>>>>>> keeping the template current and useful.
>>>>>>>
>>>>>>> 1) Move the plugin writer docs [1] into the plugin_template
>>>>>>> repository
>>>>>>> - Leave a (very) high level overview in the core docs with a
>>>>>>> link to
>>>>>>>   the template docs.
>>>>>>> - Plugin writer docs PRs would only go to one place, and it
>>>>>>> would
>>>>>>>   be easier keep the docs in line with the code.
>>>>>>> - Narrative docs in the template would explain what needs to
>>>>>>> be
>>>>>>>   done generally, linking to the modules.
>>>>>>> - Specific instructions would live in the code modules
>>>>>>> alongside
>>>>>>>   basic working code, and additional commented out code
>>>>>>>   to demonstrate and explain more complex behaviors.
>>>>>>>
>>>>>>>  2) Add pulp_smash tests for basic functionality of a bootstrapped
>>>>>>> plugin.
>>>>>>>  - Run these tests as a check on pulpcore and template PRs
>>>>>>>  - Ensure that discoverability works
>>>>>>>  - Fail with breaking Plugin API changes
>>>>>>>  - If the test uses pulp_smash, it would include a base set
>>>>>>> of
>>>>>>>integration tests for every new plugin.
>>>>>>>
>>>>>>> My reasoning is that no matter what changes we make to pulpcore,
>>>>>>> we need to keep the plugin writer docs updated. Doing this in the
>>>>>>> template will provide value for plugin writers, and will inform
>>>>>>> pulpcore
>>>>>>> developers when it needs to be updated.
>>>>>>>
>>>>>>> [0]: https://github.com/pulp/plugin_template/pull/9
>>>>>>> [1]: https://github.com/pulp/pulp/tree/master/docs/plugins/plugin
>>>>>>> -writer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ___
>>>>>>> Pulp-dev mailing list
>>>>>>> Pulp-dev@redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ___
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev@redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>>
>>>>>
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Ideas for the plugin template

[Austin Macdonald]

I've recently updated the plugin_template to work with the latest
master (3.0). [0] The template handles almost all of the bootstrapping
work necessary to write a new plugin, so it is valuable to keep it up to
date. Given human nature, it's likely that the plugin_template will tend
to fall behind as it did recently. I have some ideas to save time while
keeping the template current and useful.

1) Move the plugin writer docs [1] into the plugin_template repository
- Leave a (very) high level overview in the core docs with a link to
  the template docs.
- Plugin writer docs PRs would only go to one place, and it would
  be easier keep the docs in line with the code.
- Narrative docs in the template would explain what needs to be
  done generally, linking to the modules.
- Specific instructions would live in the code modules alongside
  basic working code, and additional commented out code
  to demonstrate and explain more complex behaviors.

 2) Add pulp_smash tests for basic functionality of a bootstrapped plugin.
 - Run these tests as a check on pulpcore and template PRs
 - Ensure that discoverability works
 - Fail with breaking Plugin API changes
 - If the test uses pulp_smash, it would include a base set of
   integration tests for every new plugin.

My reasoning is that no matter what changes we make to pulpcore,
we need to keep the plugin writer docs updated. Doing this in the
template will provide value for plugin writers, and will inform pulpcore
developers when it needs to be updated.

[0]: https://github.com/pulp/plugin_template/pull/9
[1]: https://github.com/pulp/pulp/tree/master/docs/plugins/plugin-writer
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PUP5 -- Adopting the "Common Cure Rights Commitment" for Pulp Core

[Austin Macdonald]

Ive created an issue to track this work. https://pulp.plan.io/issues/3734

On Mon, Jun 4, 2018 at 9:25 AM, Brian Bouterse  wrote:

> With no blocking votes, one +0,and five +1's this pup has passed. Thank
> you to everyone who contributed to this PUP, especially @richardfontana.
>
> As a next step, we need to add the COMMITMENT file to all the right repos.
> If anyone wants to do that feel free and maybe reply on-thread, otherwise
> I'll do it when I'm back from PTO on Wed.
>
> https://github.com/pulp/pups/blob/master/pup-0005.md
>
> On Fri, Jun 1, 2018 at 1:13 PM, Ina Panova  wrote:
>
>> +1
>>
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>> On Fri, Jun 1, 2018 at 6:11 PM, Austin Macdonald 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 8:54 AM, Dana Walker  wrote:
>>>
>>>> +1
>>>>
>>>> Dana Walker
>>>>
>>>> Associate Software Engineer
>>>>
>>>> Red Hat
>>>>
>>>> <https://www.redhat.com>
>>>> <https://red.ht/sig>
>>>>
>>>> On Thu, May 31, 2018 at 4:08 PM, Daniel Alley 
>>>> wrote:
>>>>
>>>>> +0
>>>>>
>>>>> On Thu, May 31, 2018 at 3:49 PM, Robin Chan  wrote:
>>>>>
>>>>>> Voting closes June 2nd.
>>>>>>
>>>>>> I have read this through and appreciate @richardfontana's
>>>>>> response/explanation to questions: https://github.com/pulp/pups/p
>>>>>> ull/9#issuecomment-393317027
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Wed, May 23, 2018 at 11:29 AM, Dennis Kliban 
>>>>>> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Wed, May 23, 2018 at 10:41 AM, Brian Bouterse <
>>>>>>> bbout...@redhat.com> wrote:
>>>>>>>
>>>>>>>> Through feedback on the issue and discussion in #pulp-dev, one
>>>>>>>> small language revision [0] was added to PUP5 [1]. I believe we are 
>>>>>>>> ready
>>>>>>>> to call a vote.
>>>>>>>>
>>>>>>>> Voting for PUP5 is open and will close on June 2nd. Please respond
>>>>>>>> with your vote to this thread if you feel so inclined (lazy consensus).
>>>>>>>> Barring any -1's cast, PUP5 will be merged on June 4th.
>>>>>>>>
>>>>>>>> [0]: https://github.com/richardfontana/pups/commit/99fcd35e1cc396
>>>>>>>> a1ba5a34555f093304dd07a333
>>>>>>>> [1]: https://github.com/pulp/pups/pull/9
>>>>>>>>
>>>>>>>> -Brian
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 15, 2018 at 10:47 AM, Brian Bouterse <
>>>>>>>> bbout...@redhat.com> wrote:
>>>>>>>>
>>>>>>>>> @ipanova, I think of the core team as only maintaining pulp/pulp
>>>>>>>>> and pulp/devel so I limit the scope of this to those repos only. I 
>>>>>>>>> think
>>>>>>>>> pulp_rpm (or any plugin) could adopt the CCRC without a PUP by 
>>>>>>>>> following
>>>>>>>>> the "Displaying the CRCC section
>>>>>>>>> <https://github.com/pulp/pups/pull/9/files#diff-e883d39d60672a684862d3cef971e94eR106>"
>>>>>>>>> in their own repo.
>>>>>>>>>
>>>>>>>>> @dawalker, relicensing to GPLv3 is an alternative. It's not a bad
>>>>>>>>> option, but it would be more complicated. Since every committer with 
>>>>>>>>> even a
>>>>>>>>> single line of current code is a copyright holder of the codebase, 
>>>>>>>>> and it
>>>>>>>>> would require a 100% signoff from all copyright holders, in practice 
>>>>>>>>> this
>>>>>>>>> can be difficult. Also someone may not even use that email anymore so 
>>>>>>>>>

Re: [Pulp-dev] PUP5 -- Adopting the "Common Cure Rights Commitment" for Pulp Core

[Austin Macdonald]

+1

On Fri, Jun 1, 2018 at 8:54 AM, Dana Walker  wrote:

> +1
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
> On Thu, May 31, 2018 at 4:08 PM, Daniel Alley  wrote:
>
>> +0
>>
>> On Thu, May 31, 2018 at 3:49 PM, Robin Chan  wrote:
>>
>>> Voting closes June 2nd.
>>>
>>> I have read this through and appreciate @richardfontana's
>>> response/explanation to questions: https://github.com/pulp/pups/p
>>> ull/9#issuecomment-393317027
>>>
>>> +1
>>>
>>> On Wed, May 23, 2018 at 11:29 AM, Dennis Kliban 
>>> wrote:
>>>
 +1

 On Wed, May 23, 2018 at 10:41 AM, Brian Bouterse 
 wrote:

> Through feedback on the issue and discussion in #pulp-dev, one small
> language revision [0] was added to PUP5 [1]. I believe we are ready to 
> call
> a vote.
>
> Voting for PUP5 is open and will close on June 2nd. Please respond
> with your vote to this thread if you feel so inclined (lazy consensus).
> Barring any -1's cast, PUP5 will be merged on June 4th.
>
> [0]: https://github.com/richardfontana/pups/commit/99fcd35e1cc396
> a1ba5a34555f093304dd07a333
> [1]: https://github.com/pulp/pups/pull/9
>
> -Brian
>
>
> On Tue, May 15, 2018 at 10:47 AM, Brian Bouterse 
> wrote:
>
>> @ipanova, I think of the core team as only maintaining pulp/pulp and
>> pulp/devel so I limit the scope of this to those repos only. I think
>> pulp_rpm (or any plugin) could adopt the CCRC without a PUP by following
>> the "Displaying the CRCC section
>> "
>> in their own repo.
>>
>> @dawalker, relicensing to GPLv3 is an alternative. It's not a bad
>> option, but it would be more complicated. Since every committer with 
>> even a
>> single line of current code is a copyright holder of the codebase, and it
>> would require a 100% signoff from all copyright holders, in practice this
>> can be difficult. Also someone may not even use that email anymore so it
>> may not even be possible. I haven't assessed how many Pulp3 committers we
>> have currently for the Pulp3 codebase.
>>
>> I was recently part of a relicensing which failed
>> , but
>> it shows what the process looks like:  https://github.com/python-bugz
>> illa/python-bugzilla/issues/25 If someone wants to champion
>> switching to GPLv3 and create an issue like that and get all the signoffs
>> I'm not opposed to relicensing to GPLv3 instead of adopting the CRCC.
>>
>> On Mon, May 14, 2018 at 1:34 PM, Dana Walker 
>> wrote:
>>
>>> Other than the noted point that it takes time, is there any reason
>>> why Pulp should stay on the current license instead of moving to GPLv3 
>>> (one
>>> of the stated alternatives in this PUP)?  I don't know much about the
>>> differences currently, but it strikes me that our new Pulp 3 using 
>>> Python 3
>>> would be a good fit for moving to a new license as well that has taken
>>> various things such as this enforcement issue into account and evolved 
>>> over
>>> time.
>>>
>>> Thoughts?
>>>
>>> --Dana
>>>
>>> Dana Walker
>>>
>>> Associate Software Engineer
>>>
>>> Red Hat
>>>
>>> 
>>> 
>>>
>>> On Mon, May 14, 2018 at 6:28 AM, Ina Panova 
>>> wrote:
>>>
 *understanding



 
 Regards,

 Ina Panova
 Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."

 On Mon, May 14, 2018 at 12:27 PM, Ina Panova 
 wrote:

> To make a concrete example to prove my understating:
>
> Since pulp_rpm is maintained by core team we could adopt this
> change, meanwhile pulp_deb is beyond our control and we( core team) 
> cannot
> enforce or influence this change.
> Yes?
>
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Tue, May 8, 2018 at 5:55 PM, Brian Bouterse <
> bbout...@redhat.com> wrote:
>
>> A Pulp Update Proposal (PUP) pull request has been opened by the
>> go-to-lawyer for the Pulp community, Richard Fontana. The PUP is 
>> PUP5 [0].
>> I don't want to paraphrase it here, so please read it [0] if you are
>> interested to understand what it does.
>>
>> I am 

Re: [Pulp-dev] Integer IDs in Pulp 3

[Austin Macdonald]

I have a few concerns, but they all may be addressable.

1. URLs and security. If this integer is in the url, it is easy to guess
other urls. Hopefully, our security model won't depend on obscurity, so
maybe this isn't much of a concern.
2. bulk_create. Apparently, bulk_creates would work, but only with
postgreSQL. If we accept this change, any bulk_create would make lock us in
with postgres. [0]
3. Max size = 2,147,483,647? Is it conceivable that a Pulp would have more
than 2 billion of anything?

[0]:
https://docs.djangoproject.com/en/1.10/ref/models/querysets/#bulk-create

On Wed, May 23, 2018 at 2:26 PM, David Davis  wrote:

> Before the release of Pulp 3.0 GA, I think it’s worth just checking in to
> make sure we want to use UUIDs over integer based IDs. Changing from UUIDs
> to ints would be a very easy change at this point  (1-2 lines of code) but
> after GA ships, it would be hard if not impossible to switch.
>
> I think there are a number of reasons why we might want to consider
> integer IDs:
>
> - Better performance all around for inserts[0], searches, indexing, etc
> - Less storage required (4 bytes for int vs 16 byes for UUIDs)
> - Hrefs would be shorter (e.g. /pulp/api/v3/repositories/1/)
> - In line with other apps like Katello
>
> There are some downsides to consider though:
>
> - Integer ids expose info like how many records there are
> - Can’t support sharding or multiple dbs (are we ever going to need this?)
>
> [0] http://kccoder.com/mysql/uuid-vs-int-insert-performance/
>
> David
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 'id' versus 'pulp_id' on Content

[Austin Macdonald]

In Pulp 2, having id fields bit us really badly. The reason may have been
specific to Mongoengine, but my understanding is that it is bad practice
anyway.

On Wed, May 23, 2018 at 9:22 AM, David Davis  wrote:

> Correct me if I’m wrong but don’t we call pk in most places instead of id?
> If so, it would seem like replacing id with pulp_id wouldn’t be that ugly.
>
> Also, I wonder about the created and last_updated fields. Seems like those
> could cause conflicts in the future too. At the very least, it might be
> nice to document which field names are reserved on the Content model.
>
> David
>
> On Wed, May 23, 2018 at 8:50 AM, Brian Bouterse 
> wrote:
>
>> Currently the Content model [0] has 'id' as it's primary key which is
>> inherited from MasterModel here [1]. By naming our pk 'id', we are
>> preventing plugin writers from also using that field. That field name is
>> common for content types. For example: both RPM and Nuget content also
>> expect to use the 'id' field to store data about the content type itself
>> (not Pulp's pk). We learned about the Nuget incompatibility at
>> ConfigMgmgtCamp from a community member. I learned about this issue with
>> RPM from @dalley.
>>
>> The only workaround a plugin writer has is to call their field 'rpm_id'
>> or something like that. I don't see how it's unavoidable that this renaming
>> won't be passed directly onto the user for things like filtering, creating
>> units, etc. I think that is an undesirable outcome just so that the Pulp pk
>> can be named 'id'.
>>
>> One option would be to rename 'id' to 'pulp_id' at the MasterModel. This
>> is also somewhat ugly for Pulp developers, but it would be (a) crystal
>> clear to the user in all cases and (b) allow Content writers to model their
>> content types correctly.
>>
>
>> Another option would be to rename the pk for 'Content' specifically and
>> not at the MasterModel level. I think that would create more confusion than
>> benefit so I recommend doing it at the MasterModel level.
>>
>> What do you all think?
>>
>> [0]: https://github.com/pulp/pulp/blob/6f492ee8fac94b8562dc62d87e
>> 6886869e052e7e/pulpcore/pulpcore/app/models/content.py#L106
>> [1]: https://github.com/pulp/pulp/blob/d1dc089890f167617fe9917af0
>> 87d5587708296b/pulpcore/pulpcore/app/models/base.py#L25
>>
>> -Brian
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Setting naming conventions

[Austin Macdonald]

+1 to make it more consistent. -0 for option 3 since it leaves our settings
inconsistent. The other options seem reasonable.

On Fri, May 18, 2018 at 5:24 PM, David Davis  wrote:

> Background
> =
>
> We discovered that some of our settings in server.yaml all uppercase
> (SERVER_KEY, MEDIA_ROOT, etc) while others are lowercase
> (working_directory, databases, etc)[0]. We originally decided to go with
> lowercase to match the yaml convention (although yaml itself doesn’t
> prescribe all lowercase).
>
> The problem we’ve now run into is that we’ve run into some technical
> obstacles in dealing with stuff like nested dictionaries.
>
>
> Solutions
> ===
>
> 1. Just use UPPER_SNAKE_CASE. This is probably the easiest solution.
> 2. Drop yaml for another config language like Python (which Django uses)
> or toml[1]
> 3. Fix the problem of nested dictionaries so we can keep using
> lower_snake_case.
>
> Does anyone have a strong preference on which we go with?
>
> [0] https://pulp.plan.io/issues/3589
> [1] https://en.wikipedia.org/wiki/TOML
>
> David
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp/Pulp Puppet Module removed from Forge

[Austin Macdonald]

Pulp had a puppet module on the forge:
https://forge.puppet.com/pulp/pulp

This module has not been kept up to date and was broken. Instead, please
use:
https://forge.puppet.com/katello/pulp
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Composed Repositories

[Austin Macdonald]

Here's another complexity, how do 2 plugins create a single publication? We
basically have the same problem of 2 parallel operations creating content
from a single source.

On Tue, May 15, 2018, 06:27 Ina Panova  wrote:

> +1 on not introducing dependencies between plugins.
>
> What will be the behavior in case there is a composed repo of rpm and ks
> trees but just the rpm plugin is installed?
>
Do we fail and say we cannot sync this repo at all or we just sync the rpm
> part?
>

Assuming plugins do not depend on each other, I think that when each plugin
looks at the upstream repo, they will only "see" the content of that type.
Conceptually, we will have 2 remotes, so it will feel like we are syncing
from 2 totally distinct repositories.

The solution I've been imagining is a lot like 2. Each plugin would sync to
a *separate repository.* These separate repositories are then published
creating *separate publications*. This approach allows the plugins to live
completely in ignorance of each other.

The final step is to associate *both publications to one distribution*,
which composes the publications as they are served.

The downside is that we have to sync and publish twice, and that the
resulting versions and publications aren't locked together. But I think
this is better than leaving versions and publications unfinished with the
assumption that another plugin will finish the job. Maybe linking them
together could be a good use of the notes field.


> Depends how we plan this ^ i guess we'll decide which option 1 or 2 fits
> better.
>
> Don't want to go wild, but what if notion of composed repos will be so
> popular in the future that's its amount will increase? I think we do want
> to at least partially being able to sync it and not take the approach all
> or nothing?
>
> #2 speaks to me more for now.
>
>
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Mon, May 14, 2018 at 9:44 PM, Jeff Ortel  wrote:
>
>> Let's brainstorm on something.
>>
>> Pulp needs to deal with remote repositories that are composed of multiple
>> content types which may span the domain of a single plugin.  Here are a few
>> examples.  Some Red Hat RPM repositories are composed of: RPMs, DRPMs, ,
>> ISOs and Kickstart Trees.  Some OSTree repositories are composed of OSTrees
>> & Kickstart Trees. This raises a question:
>>
>> How can pulp3 best support syncing with remote repositories that are
>> composed of multiple (unrelated) content types in a way that doesn't result
>> in plugins duplicating support for content types?
>>
>> Few approaches come to mind:
>>
>> 1. Multiple plugins (Remotes) participate in the sync flow to produce a
>> new repository version.
>> 2. Multiple plugins (Remotes) are sync'd successively each producing a
>> new version of a repository.  Only the last version contains the fully
>> sync'd composition.
>> 3. Plugins share code.
>> 4. Other?
>>
>>
>> Option #1: Sync would be orchestrated by core or the user so that
>> multiple plugins (Remotes) participate in populating a new repository
>> version.  For example: the RPM plugin (Remote) and the Kickstart Tree
>> plugin (Remote) would both be sync'd against the same remote repository
>> that is composed of both types.  The new repository version would be
>> composed of the result of both plugin (Remote) syncs.  To support this,
>> we'd need to provide a way for each plugin to operate seamlessly on the
>> same (new) repository version.  Perhaps something internal to the
>> RepositoryVersion.  The repository version would not be marked "complete"
>> until the last plugin (Remote) sync has succeeded.  More complicated than
>> #2 but results in only creating truly complete versions or nothing.  No
>> idea how this would work with current REST API whereby plugins provide sync
>> endpoints.
>>
>> Option #2: Sync would be orchestrated by core or the user so that
>> multiple plugins (Remotes) create successive repository versions.  For
>> example: the RPM plugin (Remote) and the Kickstart Tree plugin (Remote)
>> would both be sync'd against the same remote repository that is a
>> composition including both types.  The intermediate versions would be
>> incomplete.  Only the last version contains the fully sync'd
>> composition.  This approach can be supported by core today :) but will
>> produce incomplete repository versions that are marked complete=True.  This
>> /seems/ undesirable, right?  This may not be a problem for distribution
>> since I would imaging that only the last (fully composed) version would be
>> published.  But what about other usages of the repository's "latest"
>> version?
>>
>> Option #3: requires a plugin to be aware of specific repository
>> composition(s); other plugins and creates a code dependency between
>> plugins.  For example, the RPM plugin could delegate ISOs to 

Re: [Pulp-dev] Pulp3 "auto-distribute" Feature?

[Austin Macdonald]

On Mon, May 7, 2018 at 10:55 AM, Brian Bouterse  wrote:

> I'm confused about the feature claim of the auto-distribute feature for
> Pulp 3.0 GA. The distribution object takes both 'repository' and
> 'publisher' as options currently...
>
> As a user, can I create a distribution that will auto-distribute any new
> repository version for repo 'foo'?
>
> As a user, can I create a distribution that will auto-distribute any
> publication produced by a specific publisher, e.g. 'baz'?
>
>
Repository and publisher must both be set at the same time. Any time the
publisher is used to create a new publication of the repository it is auto
distributed.

Not sure why the help text isn't showing up in the autogenerated docs.
https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/pulpcore/app/serializers/repository.py#L203-L223

Heres how it works:
https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/pulpcore/app/models/publication.py#L107-L112



> Thanks,
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Tracking GA issues

[Austin Macdonald]

+1 a little late :)

For clarity I'll state how I assume this will work:

   - Milestone of 3.0 blocks GA release
   - Milestone of 3.1 blocks 3.1
   - Pulp 3 Tag indicates that the issue is Pulp 3+, no blocking
   relationship (aka, not Pulp 2)
   - Pulp 3 MVP will only be used for historical reasons after the beta


On Wed, Apr 18, 2018 at 3:05 PM, Brian Bouterse  wrote:

> I added a milestone for 3.0 and 3.1 to the Pulp project. Redmine items
> tagged with that milestone will show up on the roadmap pages below:
>
> https://pulp.plan.io/versions/57
> https://pulp.plan.io/versions/58
> https://pulp.plan.io/projects/pulp/roadmap
>
>
>
>
>
> On Wed, Apr 18, 2018 at 2:11 PM, Robin Chan  wrote:
>
>> +1 to both of those proposals (3.0 and 3.1)
>>
>> On Wed, Apr 18, 2018 at 1:43 PM, Daniel Alley  wrote:
>>
>>> +1
>>>
>>> On Wed, Apr 18, 2018 at 12:11 PM, Dana Walker 
>>> wrote:
>>>
 +1

 I like this idea for tracking purposes so that things do not fall by
 the wayside and for organization of future aims.

 --Dana

 Dana Walker

 Associate Software Engineer

 Red Hat

 
 

 On Wed, Apr 18, 2018 at 12:04 PM, David Davis 
 wrote:

> We’ve been removing the Pulp 3 MVP tag from pulpcore issues in redmine
> that we’re not going to complete for the Beta next week but some of these
> issues we may want for the 3.0 GA.
>
> I’d like to propose we create a Pulp 3.0 milestone in redmine for the
> Pulp project and start adding issues to it. I was also thinking we could
> perhaps create a Pulp 3.1 milestone as well.
>
> Thoughts?
>
> David
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Austin Macdonald]

>
> *1.  Core endpoints do not delegate/redirect to plugins.*
>  - POST to /RepositoryVersion/ is removed.
>  - POST to /Publications/  (stays gone)
>  - Plugins provide endpoints for sync and other to create new
> repository versions.
>  - Plugins provide endpoints for creating Publications (publishing).
>

I think this is a good characterization of what would change. I'd like to
add my vision for why option 1 is optimal, leaving out my specific proposal.


   1. *Plugins manage their own data.* All actions
   (sync/publish/add/remove/etc) are controlled by the plugin. Core provides
   tools to make this trivial in the trivial cases, but when a plugin needs to
   do something differently, core stays out of the way. This is the only
   pattern that makes sense when dealing with every possible plugin. We can't
   afford to make assumptions about what every plugin will and won't need.
   2. *Plugin writers follow a single pattern. *Create a model, a viewset,
   and a serializer. If the default behavior doesn't work for you, override
   it. The parts are modular, so you probably only need small changes.
   3. *Every action has a separate endpoint, and complete documentation. *If
   file/sync, file/add/, python/add, are all separate endpoints, they all have
   separate documentation. Each endpoint's also documents its parameters.


> *2. Core delegates behavior to plugins for those endpoints requiring
> plugin participation.*
> - POST to /RepositoryVersion/ is the only way to create a repository
> version.
> - POST to /Publications/  is the only way to create a Publication
> (publish).
> - The POST parameters or body includes enough information so that core
> can select a plugin.
> - Either the entire POST is passed along to the plugin, *or* the
> plugin implements an API that's used by
>   core for pre-defined participation.
>

Restricting all repository version creation to a single endpoint is cool,
but Option 2 means that pulpcore accepts requests that involve plugins.
This begs the question, how does core involve plugins? This is such an
important question that it is difficult to discuss this idea at all outside
of a specific implementation.

There is one thing that really bothers me about any possible implementation
of Option 2, the documentation. If we combine various behaviors into a
single endpoint, then how do we clearly document each of them?
The plan for Option 2 as I understand it would not document (in the docs)
the specifics for how to sync/publish etc. Even if that problem is
addressed somehow, we still have an awkward hierarchy. As you can see
below, because plugins are primary in option 1, the action docs are
separated by plugin. This makes sense, because each workflow involves only
one plugin.

   - Option 1
  - As a plugin x user
 - I can sync.
 - I can add/remove
  - As a plugin y user
 - I can sync.
 - I can add/remove.
  - As a plugin z user
 - I can sync
 - I can add-complex-depsolving


   - Option 2
  -  As a user I can create a new RepositoryVersion
 - by plugin x sync
 - by plugin x add/remove
 - by plugin y sync
 - by plugin y add/remove
 - by plugin z sync
 - by plugin z add-complex-depsolving


There have been proposals on how both #1 and #2 can be achieved. I'm
>
wondering if we can even agree on one of these two approaches.
>

It would be very convenient if we agreed on an approach separately from
implementation, but I don't think it is possible. Each strategy has obvious
pros but the cons are tucked away in the implementation details.


>
>>
>>> A File repository version cannot be properly published if it contains 2
 FileContent units that both have the same relative path. The publisher has
 to decide which FileContent to publish at the relative path. That decision
 cannot be made intelligently by the publisher. The decision on which
 content unit to include in the repository version rests with the user that
 is creating the repository version. When a user tries to create a
 repository version with a FileContent that has the same relative path as
 another FileContent that was added previously, Pulp needs to inform the
 user that there is a conflict (and not create the repositiory version).
 This validation can only be provided by the File plugin.


>
>
>>
>>
>>> We would like to get feedback on these issues being sound and worth
>>> resolving before we resume particular solution discussion[1].
>>>
>>> Thanks,
>>> Austin, Dennis, and Milan
>>>
>>> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066
>>> .html
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> 

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Austin Macdonald]

>
>
- We don't have a convention for where plug-in-specific, custom
repository version creation endpoints.
- example POST@/api/v3//docker/add/
   - needs to be discoverable through the schema

 What does discoverable via the schema ^ mean? Aren't all urls listed in
>>> the schema?
>>>
>>> I think of ^ problem somewhat differently. Yes all urls need to be
>>> discoverable (a REST property), but isn't it more of an issue that the urls
>>> which produce repo versions can't be identified distinctly from any other
>>> plugin-contributed url? To paraphrase this perspective: making a repo
>>> version is strewn about throughout the API in random places which is a bad
>>> user experience. Is that what is motivation url discovery?
>>>
>>>
>>
>> Yes. I envision a CLI that can discover new plugin
>> repository-version-creating functionality without having to install new
>> client packages. Allowing plugin writers to add endpoints in arbitrary
>> places for creating repository versions will make it impossible for the
>> client to know what all the possible ways of creating a repository version
>> are.
>>
>>
>> Currently plugins can provide one (or more) arbitrary endpoints to
>> perform sync which is one form of creating a repository version.  How is
>> the endpoint for creating a version by adding content any different?
>>
>>
> I don't understand the question.
>
> Right now plugin writers can add REST API endpoints that create repository
> versions anywhere they want. This means the client has to inspect all of
> the API schema to find all the different ways to create a repository
> version. I would like to make this discovery process simpler for the
> client. It would be best if the client could learn about all the ways to
> create a repository version by making a single request to the API.
>

"Custom repository version creation endpoints need to be discoverable
through the schema."

We have some different ideas for how to implement this goal, and I think
that is coloring our perception of the problem as well. The statement is
vague, but we chose it so we could show what we agree on despite having
subtly different views.

The weakest interpretation of the statement is that every API endpoint is
documented in the schema. I doubt that anyone would disagree with that. In
case it isn't clear already, I want to restate that the scope of this point
is limited to endpoints that create repository versions and publications.
(sync, add, publish, complex copy, etc). Within that scope, I look at
"discoverability" as related to the consistency issue.

If we have a logical, consistent API, then users (including client users)
should be able to guess what the endpoint (or command) should be for a
given use case. This also includes the need for docs readers to have a good
idea where to find their information. That would obviously be difficult to
do with these endpoints spread all around the API. Dennis takes this a step
farther, suggesting that *any* action that creates a repository version,
regardless of plugin should be on a single endpoint. I suggest that
grouping them by plugin is enough. Obviously, this gets into the
implementation details a bit, so I don't want to get too much in the weeds
until we are back to discussing concrete proposals.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 3 MVP issue cleanup (round 2)

[Austin Macdonald]

Note: there will be some redundancy with "Various MVP micro-discussions".
That email includes all changes to the MVP document, this thread covers all
changes to issues with the "Pulp 3 MVP" tag.

*Needs attention (please respond on issue):*

   - single resource OperationPostponed https://pulp.plan.io/issues/3166
   - date filter cleanup https://pulp.plan.io/issues/3557
   - RepositoryVersion filters (includes rename "number" field)
   https://pulp.plan.io/issues/3558
   - Filters on detail endpoints https://pulp.plan.io/issues/3537

*Added to sprint :*

   - arbitrary view registration https://pulp.plan.io/issues/3560
   - namespace convention for content https://pulp.plan.io/issues/3472

!*skip*

   - distribution base_path overlap prevention (data layer)
   https://pulp.plan.io/issues/3051

*"Pulp 3 MVP" Tag removed:*

   - asyncio timeout bug https://pulp.plan.io/issues/3149
   - Create version from version https://pulp.plan.io/issues/3360
   - href_in_list filter https://pulp.plan.io/issues/3240
   - Unit tests for Tasks https://pulp.plan.io/issues/3409
   - NGINX ansible role https://pulp.plan.io/issues/2922
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 MVP Issue Cleanup

[Austin Macdonald]

-0 Ansible installer (since this lives in a totally separate repository)
+1 CLI
+1 Migration Tool

On Tue, Apr 10, 2018 at 12:28 PM, Brian Bouterse <bbout...@redhat.com>
wrote:

> OK so to bring it back to how to manage this in Redmine. We've been
> talking about 'Tags' and I've read a some +1s for their use to track these
> things and no -1s. I want to identify I think it would be better to use the
> 'Category' built-in field of Redmine. Tags can be multi-selcted, but I
> don't think a single issue will need to be a CLI and an Installer and a
> $other_tag all at once (multi-selected). Categories are a single selection
> which seems more appropriate. We also make little use of them today and
> they are built into all Issues redmine has.
>
> Should I make a Categories for 'Ansible Installer', 'CLI' and 'Migration
> Tool' in the Pulp project on Redmine?
>
> Other suggestions and ideas are welcome.
>
>
>
>
> On Tue, Apr 10, 2018 at 11:23 AM, Jeff Ortel <jor...@redhat.com> wrote:
>
>>
>>
>> On 04/10/2018 10:15 AM, Jeff Ortel wrote:
>>
>>
>>
>> On 04/04/2018 05:09 PM, Dennis Kliban wrote:
>>
>> Anything that is going to have it's own release cadence should be tracked
>> in it's own project. That way we can assign issues related to specific
>> release of that project to the particular release.
>>
>> Are we going to release the CLI, Ansible Installer, and the Migration
>> tool as part of one version of Pulp or will these all be versioned
>> separately?
>>
>>
>> Separately.
>>
>>
>> Meant to clarify.  The CLI can be released separately but I think the
>> migration tool needs to be released in step with Pulp.  As for the
>> installer .. seems like that also needs to be released in step with Pulp.
>>
>>
>>
>>
>> On Wed, Apr 4, 2018 at 5:41 PM, Austin Macdonald <aus...@redhat.com>
>> wrote:
>>
>>>
>>>>> I'm hoping to continue the "Infrastructure" Redmine project for things
>>>> like website hosting. I see what you mean though because it will be
>>>> developed and released separately. I think we're in a similar situation for
>>>> 3 things: the ansible installer, the migration tool, and CLI, and for each
>>>> of them we should either make their own Redmine projects or a tag under
>>>> Pulp. We already have many Redmine projects and they are kind of a pain so
>>>> I want to float a tags based approach for feedback. Perhaps keeping them
>>>> out of "Pulp" means that we remove all the existing tags from them and tag
>>>> them with new tags like 'Ansible Installer', '2to3 Migration' and 'CLI'?
>>>>
>>>
>>> I had hoped that someday there would be a separate group of committers
>>> for pulp/devel or wherever we keep it. Also, I wouldnt want potential
>>> users/PMs to see a "bug count" that includes non-user facing issues. These
>>> concerns are trivial though, and if projects are a pain, I'm fine with
>>> keeping Tags.
>>>
>>> Since projects are a pain, can we get rid of the "external" project?
>>> https://pulp.plan.io/projects/external/issues
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content paths in Pulp 3

[Austin Macdonald]

This change highlights a subtle related issue, which is the plurality of
endpoints. [0]

*v3/content//*

Here, the "type" seems to refer to the plugin, which is "file". This
implies the false assumption that there is only one content type per plugin.

*v3/content/file/*

The namespace plan highlights the problem.

*v3/content///*

The context is subtly shifted. The "plugin" is file, which is singular, but
the "type" refers to FileContent, which is an object and should be plural.

*v3/content/file/files/*

[0]:
https://docs.pulpproject.org/en/3.0/nightly/contributing/3.0-development/rest-api.html#rest-api-guidelines

On Tue, Apr 10, 2018 at 11:38 AM, Austin Macdonald <amacd...@redhat.com>
wrote:

> If ya'll don't mind leaving out v3/content// endpoints, then I
> think we are all set. https://pulp.plan.io/issues/3472 should be ready
> to  be groomed. Since I updated with the suggested implementation, would
> one of you mind marking it groomed?
>
> On Tue, Apr 10, 2018 at 9:03 AM, Austin Macdonald <amacd...@redhat.com>
> wrote:
>
>>
>>
>> On Tue, Apr 10, 2018 at 8:20 AM, David Davis <davidda...@redhat.com>
>> wrote:
>>
>>> I’m imagining v3/content/rpm/ returning all the content types for the
>>> rpm plugin (rpm, errata, package groups, etc) and thinking that will be
>>> very strange and awkward.
>>>
>>
>> Yes, that is what it would do. I don't know if anyone would need that, my
>> point was that the url structure would imply that you *could* make this
>> call. We don't currently implement v3/content though, so maybe this isn't a
>> big deal.
>>
>>
>>>
>>> David
>>>
>>> On Tue, Apr 10, 2018 at 8:13 AM, Dennis Kliban <dkli...@redhat.com>
>>> wrote:
>>>
>>>> On Mon, Apr 9, 2018 at 1:32 PM, Austin Macdonald <amacd...@redhat.com>
>>>> wrote:
>>>>
>>>>> I've updated the issue https://pulp.plan.io/issues/3472 to reflect
>>>>> the current consensus.
>>>>>
>>>>> However, I have some other points I'd like to discuss before we move
>>>>> on.
>>>>>
>>>>> *Implied endpoint:*
>>>>> v3/content/ansible/roles/ implies that there is a v3/content/ansible/.
>>>>> Even though this does not exist, it could, but it is a little awkward.
>>>>>
>>>>
>>>> I don't think this endpoint needs to exist. What use case does it
>>>> support?
>>>>
>>>>
>>>>>
>>>>> Implent v3/content/ansible/:
>>>>>
>>>>>1. Create a model viewset and serializer for the ansible level:
>>>>>   1. class AnsibleContent(core.plugin.models.Content)
>>>>>   2. class AnsibleContentSerializer(core.
>>>>>   plugin.serializers.ContentSerializer)
>>>>>   3. class AnsibleContentViewSet(core.plu
>>>>>   gin.viewsets.ContentViewSet)
>>>>>  1. endpoint = "ansible"
>>>>>   2. Make the Role model, VS, and Serializer inherit from the
>>>>>AnsibleContent Model, VS, and Serializer
>>>>>
>>>>> The end result is that v3/content/ansible/ will return all Ansible
>>>>> content units, including Roles. v3/content/ansible/roles/ will only return
>>>>> Roles.
>>>>>
>>>>> *Publishers and Remotes*
>>>>> The above workflow makes sense and is useful if there are multiple
>>>>> units, but for the File plugin, this pattern adds an endpoint and 3 
>>>>> classes
>>>>> to the Content. If we want to be consistent and apply this to Remotes and
>>>>> Publishers, this is 3 useless endpoints and 9 extra classes. (These 
>>>>> classes
>>>>> are simple, even if they are extraneous, conceptually)
>>>>>
>>>>> *IMO*
>>>>> I think we should document all of this in the plugin docs. For single
>>>>> type (and single remote, and single publisher) plugins, it will make more
>>>>> sense not to add an extra namespace. If we document how to add the extra
>>>>> namespace and when/why plugins should, that will be sufficient. Promoting
>>>>> consistency over simplicity in this case seems too far.
>>>>>
>>>>> *Or...*
>>>>> We could alter the Master/Detail code. I only have vague ideas about
>>>>> how to do this, but Master/Detail w

Re: [Pulp-dev] Content paths in Pulp 3

[Austin Macdonald]

If ya'll don't mind leaving out v3/content// endpoints, then I
think we are all set. https://pulp.plan.io/issues/3472 should be ready to
be groomed. Since I updated with the suggested implementation, would one of
you mind marking it groomed?

On Tue, Apr 10, 2018 at 9:03 AM, Austin Macdonald <amacd...@redhat.com>
wrote:

>
>
> On Tue, Apr 10, 2018 at 8:20 AM, David Davis <davidda...@redhat.com>
> wrote:
>
>> I’m imagining v3/content/rpm/ returning all the content types for the rpm
>> plugin (rpm, errata, package groups, etc) and thinking that will be very
>> strange and awkward.
>>
>
> Yes, that is what it would do. I don't know if anyone would need that, my
> point was that the url structure would imply that you *could* make this
> call. We don't currently implement v3/content though, so maybe this isn't a
> big deal.
>
>
>>
>> David
>>
>> On Tue, Apr 10, 2018 at 8:13 AM, Dennis Kliban <dkli...@redhat.com>
>> wrote:
>>
>>> On Mon, Apr 9, 2018 at 1:32 PM, Austin Macdonald <amacd...@redhat.com>
>>> wrote:
>>>
>>>> I've updated the issue https://pulp.plan.io/issues/3472 to reflect the
>>>> current consensus.
>>>>
>>>> However, I have some other points I'd like to discuss before we move on.
>>>>
>>>> *Implied endpoint:*
>>>> v3/content/ansible/roles/ implies that there is a v3/content/ansible/.
>>>> Even though this does not exist, it could, but it is a little awkward.
>>>>
>>>
>>> I don't think this endpoint needs to exist. What use case does it
>>> support?
>>>
>>>
>>>>
>>>> Implent v3/content/ansible/:
>>>>
>>>>1. Create a model viewset and serializer for the ansible level:
>>>>   1. class AnsibleContent(core.plugin.models.Content)
>>>>   2. class AnsibleContentSerializer(core.
>>>>   plugin.serializers.ContentSerializer)
>>>>   3. class AnsibleContentViewSet(core.plu
>>>>   gin.viewsets.ContentViewSet)
>>>>  1. endpoint = "ansible"
>>>>   2. Make the Role model, VS, and Serializer inherit from the
>>>>AnsibleContent Model, VS, and Serializer
>>>>
>>>> The end result is that v3/content/ansible/ will return all Ansible
>>>> content units, including Roles. v3/content/ansible/roles/ will only return
>>>> Roles.
>>>>
>>>> *Publishers and Remotes*
>>>> The above workflow makes sense and is useful if there are multiple
>>>> units, but for the File plugin, this pattern adds an endpoint and 3 classes
>>>> to the Content. If we want to be consistent and apply this to Remotes and
>>>> Publishers, this is 3 useless endpoints and 9 extra classes. (These classes
>>>> are simple, even if they are extraneous, conceptually)
>>>>
>>>> *IMO*
>>>> I think we should document all of this in the plugin docs. For single
>>>> type (and single remote, and single publisher) plugins, it will make more
>>>> sense not to add an extra namespace. If we document how to add the extra
>>>> namespace and when/why plugins should, that will be sufficient. Promoting
>>>> consistency over simplicity in this case seems too far.
>>>>
>>>> *Or...*
>>>> We could alter the Master/Detail code. I only have vague ideas about
>>>> how to do this, but Master/Detail would essentially become
>>>> Master/Plugin/Detail. This seems "right", but there isn't as much gain here
>>>> as you might think. If we did it this, plugins would be required to
>>>> namespace, and would be still be required to make all those extra classes.
>>>> The only practical difference is that the AnsibleRoleViewSet.endpoint =
>>>> "roles" instead of "ansible/roles". Either way, the endpoint would be
>>>> v3/content/ansible/roles/
>>>>
>>>> On Fri, Apr 6, 2018 at 10:25 AM, Brian Bouterse <bbout...@redhat.com>
>>>> wrote:
>>>>
>>>>> +1 to option 1. It's consistent.
>>>>>
>>>>> On Fri, Apr 6, 2018 at 10:21 AM, Dennis Kliban <dkli...@redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Option 1 is the most consistent. +1 to option 1
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 6, 2018 at 10:19 AM, Aus

Re: [Pulp-dev] Changesets Challenges

[Austin Macdonald]

>
>
> 1. There is redundant "differencing" code in all plugins. The Changeset
> interface requires the plugin writer to determine what units need to be
> added and those to be removed. This requires all plugin writers to write
> the same non-trivial differencing code over and over. For example, you can
> see the same non-trivial differencing code present in pulp_ansible
> 
> , pulp_file
> ,
> and pulp_python
> .
> Line-wise, this "differencing" code makes up a large portion (maybe 50%) of
> the sync code itself in each plugin.
>

I agree that core could provide some of this logic. However, I'm not sure
that the changeset is the right scope. The more monolithic the tool, the
less flexible we will be, so I hope that we can provide a tool that is
separate from the changeset. This way, a plugin can determine for
themselves what needs to be added and removed, if the general way doesn't
work for them.

2. Plugins can't do end-to-end stream processing. The Changesets themselves
> do stream processing, but when you call into changeset.apply_and_drain()
> you have to have fully parsed the metadata already. Currently when fetching
> all metadata from Galaxy, pulp_ansible takes about 380 seconds (6+ min).
> This means that the actual Changeset content downloading starts 380 seconds
> later than it could. At the heart of the problem, the fetching+parsing of
> the metadata is not part of the stream processing.
>

I can see how this isn't ideal, curious to see how you would address it.



On Mon, Apr 9, 2018 at 5:38 AM, Milan Kovacik  wrote:

> Brian,
>
> thanks for the proposal!
>
> On Fri, Apr 6, 2018 at 4:15 PM, Brian Bouterse 
> wrote:
> > Several plugins have started using the Changesets including pulp_ansible,
> > pulp_python, pulp_file, and perhaps others. The Changesets provide
> several
> > distinct points of value which are great, but there are two challenges I
> > want to bring up. I want to focus only on the problem statements first.
> >
> > 1. There is redundant "differencing" code in all plugins. The Changeset
> > interface requires the plugin writer to determine what units need to be
> > added and those to be removed. This requires all plugin writers to write
> the
> > same non-trivial differencing code over and over. For example, you can
> see
> > the same non-trivial differencing code present in pulp_ansible,
> pulp_file,
> > and pulp_python. Line-wise, this "differencing" code makes up a large
> > portion (maybe 50%) of the sync code itself in each plugin.
>
> +1; DRY is always better
>
> >
> > 2. Plugins can't do end-to-end stream processing. The Changesets
> themselves
> > do stream processing, but when you call into changeset.apply_and_drain()
> you
> > have to have fully parsed the metadata already. Currently when fetching
> all
> > metadata from Galaxy, pulp_ansible takes about 380 seconds (6+ min). This
> > means that the actual Changeset content downloading starts 380 seconds
> later
> > than it could. At the heart of the problem, the fetching+parsing of the
> > metadata is not part of the stream processing.
> >
> > Do you see the same challenges I do? Are these the right problem
> statements?
> > I think with clear problem statements a solution will be easy to see and
> > agree on.
>
> Totally, esp. on these, easier-to-see-the-value-of ones.
>
> Cheers,
> milan
>
> >
> > Thanks!
> > Brian
> >
> > ___
> > Pulp-dev mailing list
> > Pulp-dev@redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
> >
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 MVP Issue Cleanup

[Austin Macdonald]

On Tue, Apr 10, 2018 at 8:17 AM, David Davis <davidda...@redhat.com> wrote:

> So the only content filter we have on the MVP doc is repo version. I think
> this might be tricky and you can already view content for a repository
> version (ie /api/v3/repositories//versions//content) so I
> think we should just leave this out of the MVP.
>
>
That's fine with me. I'll update the issue.

David
>
> On Mon, Apr 9, 2018 at 12:19 PM, Austin Macdonald <aus...@redhat.com>
> wrote:
>
>> Here's another one that needs to be groomed and added to the
>> sprint(thanks kersom!)
>>
>>- (@David maybe?) Content Filters https://pulp.plan.io/issues/3084
>>
>>
>> All of these have been added to the sprint:
>>
>>- https://pulp.plan.io/issues/3552
>>- https://pulp.plan.io/issues/3395
>>- https://pulp.plan.io/issues/3298
>>- https://pulp.plan.io/issues/3220
>>- https://pulp.plan.io/issues/3081
>>    - https://pulp.plan.io/issues/3082
>>- https://pulp.plan.io/issues/3546
>>
>>
>>
>>
>> On Thu, Apr 5, 2018 at 9:53 AM, Austin Macdonald <aus...@redhat.com>
>> wrote:
>>
>>> David and I went through all the pulpcore issues that have the "Pulp3
>>>> MVP" tag.
>>>> We added this one to the sprint:
>>>>
>>>>- https://pulp.plan.io/issues/3545
>>>>
>>>>
>>>> These two need to be updated before we can move forward:
>>>>
>>>>- @dalley https://pulp.plan.io/issues/3505
>>>>
>>>>
>>>>- @asmacdo https://pulp.plan.io/issues/3546
>>>>
>>>>
>>>> We marked these as groomed, unless someone says "no", I plan to add all
>>>> of these to the sprint.
>>>>
>>>>- https://pulp.plan.io/issues/3082
>>>>
>>>>
>>>>- https://pulp.plan.io/issues/3081
>>>>
>>>>
>>>>- https://pulp.plan.io/issues/3220
>>>>
>>>>
>>>>- https://pulp.plan.io/issues/3298
>>>>
>>>>
>>>>- https://pulp.plan.io/issues/3395
>>>>
>>>>
>>>
>>>> We have some vagrant/ansible issues. I don't think these really belong
>>>> in the "Pulp" project tracker. Mind if we move them to the "Infrastructure"
>>>> project? (BTW, there are a lot more, just without the MVP tag).
>>>>
>>>>- https://pulp.plan.io/issues/3439
>>>>
>>>>
>>>>- https://pulp.plan.io/issues/2922
>>>>
>>>> Adding to these lists:
>>> Needs grooming and addding to spring: https://pulp.plan.io/issues/3552
>>>
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 3 REST API Challenges

[Austin Macdonald]

Folks,

Austin, Dennis, and Milan have identified the following issues with current
Pulp3 REST API design:

   - Action endpoints are problematic.
   - Example POST@/importers//sync/
  - They are non-RESTful and would make client code tightly coupled
  with the server code.
  - These endpoints are inconsistent with the other parts of the REST
  API.
  - DRF is not being used as intended for action endpoints so we have
  to implement extra code. (against the grain)
   - We don't have a convention for where plug-in-specific, custom
   repository version creation endpoints.
   - example POST@/api/v3//docker/add/
  - needs to be discoverable through the schema
   - For direct repository version creation, plugins are not involved.
   - validation correctness problem: https://pulp.plan.io/issues/3541
  - example: POST@/api/v3/repositories//versions/

We would like to get feedback on these issues being sound and worth
resolving before we resume particular solution discussion[1].

Thanks,
Austin, Dennis, and Milan

[1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content paths in Pulp 3

[Austin Macdonald]

I've updated the issue https://pulp.plan.io/issues/3472 to reflect the
current consensus.

However, I have some other points I'd like to discuss before we move on.

*Implied endpoint:*
v3/content/ansible/roles/ implies that there is a v3/content/ansible/. Even
though this does not exist, it could, but it is a little awkward.

Implent v3/content/ansible/:

   1. Create a model viewset and serializer for the ansible level:
  1. class AnsibleContent(core.plugin.models.Content)
  2. class
  AnsibleContentSerializer(core.plugin.serializers.ContentSerializer)
  3. class AnsibleContentViewSet(core.plugin.viewsets.ContentViewSet)
 1. endpoint = "ansible"
  2. Make the Role model, VS, and Serializer inherit from the
   AnsibleContent Model, VS, and Serializer

The end result is that v3/content/ansible/ will return all Ansible content
units, including Roles. v3/content/ansible/roles/ will only return Roles.

*Publishers and Remotes*
The above workflow makes sense and is useful if there are multiple units,
but for the File plugin, this pattern adds an endpoint and 3 classes to the
Content. If we want to be consistent and apply this to Remotes and
Publishers, this is 3 useless endpoints and 9 extra classes. (These classes
are simple, even if they are extraneous, conceptually)

*IMO*
I think we should document all of this in the plugin docs. For single type
(and single remote, and single publisher) plugins, it will make more sense
not to add an extra namespace. If we document how to add the extra
namespace and when/why plugins should, that will be sufficient. Promoting
consistency over simplicity in this case seems too far.

*Or...*
We could alter the Master/Detail code. I only have vague ideas about how to
do this, but Master/Detail would essentially become Master/Plugin/Detail.
This seems "right", but there isn't as much gain here as you might think.
If we did it this, plugins would be required to namespace, and would be
still be required to make all those extra classes. The only practical
difference is that the AnsibleRoleViewSet.endpoint = "roles" instead of
"ansible/roles". Either way, the endpoint would be v3/content/ansible/roles/

On Fri, Apr 6, 2018 at 10:25 AM, Brian Bouterse <bbout...@redhat.com> wrote:

> +1 to option 1. It's consistent.
>
> On Fri, Apr 6, 2018 at 10:21 AM, Dennis Kliban <dkli...@redhat.com> wrote:
>
>> Option 1 is the most consistent. +1 to option 1
>>
>>
>> On Fri, Apr 6, 2018 at 10:19 AM, Austin Macdonald <amacd...@redhat.com>
>> wrote:
>>
>>> IMO:
>>> We should suggest v3/content///. [Proposal 1] We should
>>> mention the other options with the pros, cons in the plugin writer docs.
>>>
>>> On Thu, Apr 5, 2018 at 10:54 AM, David Davis <davidda...@redhat.com>
>>> wrote:
>>>
>>>>
>>>> [0] https://pulp.plan.io/issues/3407
>>>>
>>>
>>> The correct link is: https://pulp.plan.io/issues/3472
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 MVP Issue Cleanup

[Austin Macdonald]

Here's another one that needs to be groomed and added to the sprint(thanks
kersom!)

   - (@David maybe?) Content Filters https://pulp.plan.io/issues/3084


All of these have been added to the sprint:

   - https://pulp.plan.io/issues/3552
   - https://pulp.plan.io/issues/3395
   - https://pulp.plan.io/issues/3298
   - https://pulp.plan.io/issues/3220
   - https://pulp.plan.io/issues/3081
   - https://pulp.plan.io/issues/3082
   - https://pulp.plan.io/issues/3546




On Thu, Apr 5, 2018 at 9:53 AM, Austin Macdonald <aus...@redhat.com> wrote:

> David and I went through all the pulpcore issues that have the "Pulp3 MVP"
>> tag.
>> We added this one to the sprint:
>>
>>- https://pulp.plan.io/issues/3545
>>
>>
>> These two need to be updated before we can move forward:
>>
>>- @dalley https://pulp.plan.io/issues/3505
>>
>>
>>- @asmacdo https://pulp.plan.io/issues/3546
>>
>>
>> We marked these as groomed, unless someone says "no", I plan to add all
>> of these to the sprint.
>>
>>- https://pulp.plan.io/issues/3082
>>
>>
>>- https://pulp.plan.io/issues/3081
>>
>>
>>- https://pulp.plan.io/issues/3220
>>
>>
>>- https://pulp.plan.io/issues/3298
>>
>>
>>- https://pulp.plan.io/issues/3395
>>
>>
>
>> We have some vagrant/ansible issues. I don't think these really belong in
>> the "Pulp" project tracker. Mind if we move them to the "Infrastructure"
>> project? (BTW, there are a lot more, just without the MVP tag).
>>
>>- https://pulp.plan.io/issues/3439
>>
>>
>>- https://pulp.plan.io/issues/2922
>>
>> Adding to these lists:
> Needs grooming and addding to spring: https://pulp.plan.io/issues/3552
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content paths in Pulp 3

[Austin Macdonald]

IMO:
We should suggest v3/content///. [Proposal 1] We should
mention the other options with the pros, cons in the plugin writer docs.

On Thu, Apr 5, 2018 at 10:54 AM, David Davis  wrote:

>
> [0] https://pulp.plan.io/issues/3407
>

The correct link is: https://pulp.plan.io/issues/3472
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 MVP Issue Cleanup

[Austin Macdonald]

On Wed, Apr 4, 2018 at 6:09 PM, Dennis Kliban <dkli...@redhat.com> wrote:

> Anything that is going to have it's own release cadence should be tracked
> in it's own project. That way we can assign issues related to specific
> release of that project to the particular release.
>
Are we going to release the CLI, Ansible Installer, and the Migration tool
> as part of one version of Pulp or will these all be versioned separately?
>

Seems reasonable. IMO, CLI should be released on its own. Ansible Installer
role will have its own cadence on Galaxy. Vagrant/playbooks will not be
released.

The Migration tool is tricky. A pulpcore migration tool would be one thing,
but each plugin will probably need its own migration tool. So... /me shugs.


>
> On Wed, Apr 4, 2018 at 5:41 PM, Austin Macdonald <aus...@redhat.com>
> wrote:
>
>>
>>>> I'm hoping to continue the "Infrastructure" Redmine project for things
>>> like website hosting. I see what you mean though because it will be
>>> developed and released separately. I think we're in a similar situation for
>>> 3 things: the ansible installer, the migration tool, and CLI, and for each
>>> of them we should either make their own Redmine projects or a tag under
>>> Pulp. We already have many Redmine projects and they are kind of a pain so
>>> I want to float a tags based approach for feedback. Perhaps keeping them
>>> out of "Pulp" means that we remove all the existing tags from them and tag
>>> them with new tags like 'Ansible Installer', '2to3 Migration' and 'CLI'?
>>>
>>
>> I had hoped that someday there would be a separate group of committers
>> for pulp/devel or wherever we keep it. Also, I wouldnt want potential
>> users/PMs to see a "bug count" that includes non-user facing issues. These
>> concerns are trivial though, and if projects are a pain, I'm fine with
>> keeping Tags.
>>
>> Since projects are a pain, can we get rid of the "external" project?
>> https://pulp.plan.io/projects/external/issues
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Improving technical decision making

[Austin Macdonald]

On Wed, Apr 4, 2018 at 9:01 AM, Brian Bouterse <bbout...@redhat.com> wrote:

> I'm not ready to pursue a single decision maker model for Pulp's technical
> decisions or community leadership.
>

OpenStack tech leads aren't "single decision makers", they are a fallback
for when consensus isn't reached. In theory the role *could *scope creep to
"single decision makers" depending on the style of the individual, but
elections prevent that if the voters are responsible.


> I also have concerns about those positions being rotating roles since
> typically they require much experience. This would also be a departure from
> the lazy consensus decision making model we use for community decisions
> (the pup process itself).
>

A "rotating role" is different than an elected position. I'm not sure what
Milan meant by "time boxed", but I imagine this would just be another
election, not a term limit. I think if this is done well, it would augment
the lazy consensus model, not replace it.

>
> There would need to be a lot of discussion with input from many people
> around what the issues currently are so we can be sure that changes would
> resolve those issues.
>

+1  We should set this up carefully. I think a PUP is the right way to do
that.

My opinion is that we have stalled and punted several very important issues
when lazy consensus was too lazy. This has slowed our progress enough that
I am interested in fleshing out alternatives.


>
> On Wed, Apr 4, 2018 at 4:54 AM, Milan Kovacik <mkova...@redhat.com> wrote:
>
>> Oh I'd forget that we actually don't really have a formal process to
>> recognize and retire active contributors yet;
>> how about the technical lead proposes both the recognition and
>> retirement anytime they find reason to do so, for the former
>> situation, with a pre-approval of  other active contributors, propose
>> folks publicly, for the latter situation, try reaching out to the
>> retiring contributor before going public to avoid frustration.
>> Folks of course would ideally announce their intention to retire, the
>> formal process would be conducted by the technical lead.
>> The insignia of an active contributor would be the commit bit on any
>> of the Pulp projects.
>> The first ever technical and community leads would be elected by folks
>> with the commit bit, the election would be organized by our current
>> community representatives.
>>
>> Unless there are objections, I'd file a PUP to summarize these points.
>>
>>
>> Cheers,
>> milan
>>
>> On Tue, Apr 3, 2018 at 6:02 PM, Milan Kovacik <mkova...@redhat.com>
>> wrote:
>> > On Tue, Apr 3, 2018 at 3:47 PM, Austin Macdonald <aus...@redhat.com>
>> wrote:
>> >> Interesting proposals Milan!
>> >>
>> >> I am forking Brian's email so that thread can focus on communication,
>> >> redmine, etc.
>> >
>> > Thanks, I guess it would best go hand-in-hand with the process update
>> > proposal for the Technical specifications/blue-prints PUP:
>> >
>> >>> I'd add that many a time, an e-mail based technical discussion gets
>> >>> messy and unfolds in multiple branches over multiple months.
>> >>> I'd like to propose we adopt a Technical Specification concept, living
>> >>> in a separate GitHub repo, similar to the PUP process.
>> >>> This would take advantage of our review process, preferably requiring
>> >>> multiple (core) reviewers acks before merging, allowing Redmine to be
>> >>> used for planning/tracking the (design) work.
>> >>> I think it's easier to manage the life-cycle of  a larger Technical
>> >>> Specification in a revision system document than an e-mail thread and
>> >>> a single Redmine issue.
>> >>> It also helps (feature) documentation and provides context.
>> >
>> >>
>> >> On Tue, Apr 3, 2018 at 8:13 AM, Milan Kovacik <mkova...@redhat.com>
>> wrote:
>> >>>
>> >>> I'd also like to propose formal Project Technical Lead and a formal
>> >>> Project Community Lead roles to be able to decide in case of competing
>> >>> (technical) ideas or planning priorities.
>> >>> These would have to be time-boxed (half a year) and folks would elect
>> >>> the leader for a period based on leader's program, such as focus on
>> >>> particular goals for instance testing or refactoring.
>> >>> Any single person would be able to perform either the Community or the
>> >>> Techn

Re: [Pulp-dev] remove AUTHORS

[Austin Macdonald]

One reason to keep the AUTHORS file is that it allows contributors to
specify their contact info instead of just linking to their GitHub page,
were they might not have an email listed.

I assume this discussion is restricted to pulp/pulp. The plugins are free
to handle however they prefer.

Personally, I am fine either way. Remove +0, one less file.


On Wed, Apr 4, 2018, 05:22 Irina Gulina  wrote:

> Hello Pulp Team,
>
> Recently, I have opened a PR to remove the AUTHORS file [1], and was
> suggested to bring it up here.
> Please let us/me know what you think about it. Any feedback will be
> appreciated.
>
> Happy Wednesday,
> Irina
>
>
> [1] https://github.com/pulp/pulp/pull/3393
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Improving technical decision making

[Austin Macdonald]

Interesting proposals Milan!

I am forking Brian's email so that thread can focus on communication,
redmine, etc.

On Tue, Apr 3, 2018 at 8:13 AM, Milan Kovacik <mkova...@redhat.com> wrote:

> I'd also like to propose formal Project Technical Lead and a formal
> Project Community Lead roles to be able to decide in case of competing
> (technical) ideas or planning priorities.
> These would have to be time-boxed (half a year) and folks would elect
> the leader for a period based on leader's program, such as focus on
> particular goals for instance testing or refactoring.
> Any single person would be able to perform either the Community or the
> Technical Lead role in any given period but not both at the same time.
> The Community Lead role would take care for organizing the Technical
> Lead elections and vice versa, the Technical Lead would take care
> about organizing the Community Lead elections.
> The electorate would be the active contributors for both the roles.
> The candidates would be the active contributors too.
>
> This would open up the decision making process for anyone from the
> community, would encourage transparency, accountability and
> responsibility and would allow us to come to a decision on competing
> (technical) ideas or planning issues in case we'd got stuck.
>
> Cheers,
> milan
>
> On Mon, Apr 2, 2018 at 8:38 PM, Brian Bouterse <bbout...@redhat.com>
> wrote:
> > I agree the decision process for core itself needs discussion. For now,
> I'm
> > only able to offer facilitating a convo that focuses on the communication
> > aspects not the decision process. I would like to improve the
> transparency
> > into the features that will and won't be in any given release for our
> > stakeholders. I hope we do discuss decision process as its own
> discussion;
> > it's certainly deserving of a pup of its own.
> >
> > For the communication issues, soon I will share a basic outline of one
> way
> > we could use Redmine for release planning. This would be a starter idea
> > towards a solution for us to modify together.
> >
> >
> >
> > On Mon, Apr 2, 2018 at 9:08 AM, Austin Macdonald <aus...@redhat.com>
> wrote:
> >>
> >> I agree with the problems that Brian listed, but I hope we can focus on
> >> the decision making process itself in addition to how those decisions
> are
> >> communicated.
> >
> >
> >
> > ___
> > Pulp-dev mailing list
> > Pulp-dev@redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
> >
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

Thanks for the classifications.

On Mon, Apr 2, 2018 at 2:37 PM, Dennis Kliban  wrote:

I think of these actions as 2 new types of resource in Pulp. Unlike
> Remotes(Importers currently) and Content, these resources are singletons
> that each plugin provides. Since users don't need to create new instances
> of these resources it makes sense that they are represented by a View
> instead of a ViewSet. Though we don't even need to explain that to plugin
> writers. We just need to tell them that all operations that their plugin
> provides for users need to be sublassed from ActionView.
>
> From the users point of view a GET on /api/v3/versionsactions/ to find a
> versionaction href is the same as performing a GET on /api/v3/remotes/ to
> find the href for a specific remote to sync from.
>
> Auto documentation would work perfectly.
>
>
Its not that it would be "broken" or "wrong". I just think that users
expect the documentation to include behavior and parameters-- ideally,
there should be enough information to create a request. In this case
however, the documentation just explains how to retrieve the necessary
information. It works, but I don't think it is ideal that the user needs to
make other GET requests to learn how to use an API endpoint.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3.0 REST API gap analysis

[Austin Macdonald]

On Fri, Mar 30, 2018 at 9:23 AM, David Davis  wrote:

> Sorry, I think the Importer and Publisher subclass user stories probably
> need to be updated to more accurately reflect how plugin writers implement
> syncing and publishing.
>
>
> David
>
> On Fri, Mar 30, 2018 at 9:20 AM, David Davis 
> wrote:
>
>> > As an authenticated user, I can define a default remote on a repo via
>> href to be used whenever a new repo version is created.
>>
>> I kind of have a proposal around this and I think we should ask
>> Katello/users what they need. I’d propose leaving this off the MVP for now.
>>
>
+1, lets remove. Some optimization around this would be nice, but is not
"minimum".


>
>>
> > auto_publish(bool) - ??? consider adding auto-publish feature to MVP
>>
>> I think we agreed to leave auto_publish out of the MVP but forgot to
>> remove it from the doc.
>>
>
If we do remove it, we should remove the field from the model/serializer
also.


>
>> > As a user, my distribution base paths don't conflict and my
>> create/update is rejected identifying the conflicting distributions
>>
>> I’ve been working on this on my own time outside of work since it’s not
>> on the sprint. I think I’ll have this soon.
>>
>> > As a user, I want a newly created publication to be automatically
>> served by the content as defined by distributions.
>>
>> I think this works.
>>
>
Yeah I think so too.
https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/pulpcore/app/models/publication.py#L107-L112


>
>> > As an authenticated user, I have filters on the Distribution list [3082]
>>
>> I raised this issue at the sprint planning before the beta feature freeze
>> but we agreed not to address it/add it to the sprint. I can’t remember why
>> though.
>>
>> > I cannot pass "sync" options.
>> > Auto-publish is not included as an importer property.
>> > I cannot pass "publish" options.
>>
>> We don’t validate extraneous parameters. These might be just to emphasize
>> that we’re not supporting this for now. I wouldn’t mind removing them from
>> the MVP.
>>
>> +1, let's remove.

> Content unit deletion needs to be race condition free.
>>
>> Covered by https://pulp.plan.io/issues/3445
>>
>
>> > As an authenticated user, I can filter Content by repository version
>> information when plugin writers have provided such a filter
>>
> > As an authenticated user, I can filter content on plugin specific
>> attributes when plugin writers have provided such a filter
>>
>
This is a plugin feature, we should remove from the MVP. The plugin defines
a content viewset, which specifies its own filters.

>
>> This is outstanding work. I think we need someone that can lead this and
>> design out how it should work.
>>
>
>> > As a user, I have docs that when a new RepositoryVersion is created,
>> add_many and remove_many are performed before any Plugin code is executed.
>>
>> Not sure I understand this one.
>>
>
This is the "docs" solution to https://pulp.plan.io/issues/3541


>
>> > As an authenticated user, I can cause an action that cleans up both
>> orphaned content units and orphaned artifacts.
>>
>> Covered by https://pulp.plan.io/issues/3442.
>>
>>  > As a plugin writer, I can provide a subclassed Importer. The
>> importer's responsibility is to synchronize the content of a Pulp
>> repository with the content of a remote repository. (a circular import
>> problem needs to be discussed and may cause this to change)
>> > As a plugin writer, I can provide a subclassed Publisher. The
>> publisher's responsibility is to publish content. (a circular import
>> problem needs to be discussed and may cause this to change)
>>
>> AFAIK, I think these are done.
>>
>
These lines should be:
As a plugin writer, I can provide a subclassed Remote. The Remote's
responsibility is to store all of the information necessary to communicate
with an external source of content. (The rest is handled by the "sync"
task, not the remote)
As a plugin writer, I can provide a subclassed Publisher.


>
>> > As a plugin writer, I expect units that are missing from the remote
>> repository to not be created in Pulp when using the immediate download
>> policy.
>> > As a plugin writer, I expect units that are missing from the remote
>> repository to be created in Pulp when using background or on_demand
>> download policies.
>>
>> These seem backwards to me. If we’re not doing lazy sync in the MVP (and
>> I don’t think we are) then we should probably just remove them.
>>
>> +1, lets remove this this from the MVP

>
>>
>> David
>>
>> On Thu, Mar 29, 2018 at 7:10 PM, Dennis Kliban 
>> wrote:
>>
>>> Kersom and I spent the afternoon going over the REST API features
>>> described in the MVP[0] document. We discovered that a few features are
>>> poorly worded and a few features have not been implemented. All these items
>>> are now written in red. We need to do 2 things:
>>>
>>> - fix the language where it is not clear or completely remove it

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

 >>> In terms of removing POST /api/v3/repositories//versions/ from
>> >>> core, I want to bring it back to the MVP language and REST which
>> drove the
>> >>> original design. The MVP has: "As an authenticated user, I can create
>> a new
>> >>> version by adding or removing content to the latest version." To
>> facilitate
>> >>> that in a generic way, we need a core endpoint to do that, i.e.
>> >>> /api/v3/repositories//versions/. My concern is that removing
>> it would
>> >>> cause us to not fulfill our use cases without requiring more code
>> from some
>> >>> plugin writers. Also in terms of REST philosophy, POSTing to the
>> >>> RespotoryVersion resource to create a new RepositoryVersion is the
>> >>> traditional url design.
>> >>>
>> >>> For pulp_ansible, for example, the generic add/remove functionality at
>> >>> core endpoint ^ would be meet all of the pulp_ansible user's needs
>> because
>> >>> of the way the pulp_ansible content is modelled. So removing this
>> endpoint
>> >>> means more work for pulp_ansible developers w.r.t creating repo
>> versions and
>> >>> providing tasks and endpoints. I don't see this extra responsibility
>> on
>> >>> plugin writers coming with a clear benefit.
>> >>>
>> >>> One idea I liked in this discussion is to have a documented convention
>> >>> that encourages plugin writers to put their viewsets in a namespaced
>> area.
>> >>> They still need the ability to put them anywhere due to live API
>> >>> goals/requirements so this would only be a convention for those tasks
>> they
>> >>> are offering directly to their users.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Mar 27, 2018 at 5:40 PM, David Davis <davidda...@redhat.com>
>> >>> wrote:
>> >>>>
>> >>>> I like Austin’s task proposal in that the plugin writers can focus on
>> >>>> serializers and tasks that can be easily integrated with core. That
>> said, I
>> >>>> agree on the counterpoints raise about the new resource endpoints and
>> >>>> turning much of pulp into a task running system. So I am a bit mixed
>> on
>> >>>> which approach is better.
>> >>>>
>> >>>> I do think that we should remove POST
>> >>>> /api/v3/repositories//versions/ from core.
>> >>>>
>> >>>>
>> >>>> David
>> >>>>
>> >>>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald <
>> amacd...@redhat.com>
>> >>>> wrote:
>> >>>>>>
>> >>>>>> /api/v3/repositories//versions/ endpoint does not perform
>> plugin
>> >>>>>> specific validation which can lead to "broken" repository versions.
>> >>>>>> Plugin authors don't have any convention to follow when creating
>> >>>>>> custom REST API endpoints for creating repository versions.
>> >>>>>> As a result of ^, a user will have a hard time identifying
>> repository
>> >>>>>> version creation APIs in different plugins.
>> >>>>>
>> >>>>>
>> >>>>> I agree with these points.
>> >>>>>>
>> >>>>>> My first inclination is to disable the ability to POST to
>> >>>>>> /api/v3/repositories//versions/ and require users to use
>> the plugin
>> >>>>>> specific APIs for creating repository versions. However, I think
>> that
>> >>>>>> integrators of build systems that produce a variety of content
>> types would
>> >>>>>> have a lot more flexibility if they could use a single generic API
>> endpoint
>> >>>>>> to create a repository version independent of the content type.
>> >>>>>>
>> >>>>>> Let's continue this discussion by answering the following question:
>> >>>>>>
>> >>>>>> Should we disable the ability to POST to
>> >>>>>> /api/v3/repositories//versions/ and require users to always
>> use a
>> >>>>>> plugin specific repository version creation API?
>> >>>>>
>> >>>>> Yes, I think we should disable POST to
>> >>>>> /api/v3/repositories//versions/
>> >>>>>
>> >>>>> Simplifying integration is important, but we should not sacrifice
>> >>>>> correctness enforcement.
>> >>>>>
>> >>>>> ___
>> >>>>> Pulp-dev mailing list
>> >>>>> Pulp-dev@redhat.com
>> >>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >>>>>
>> >>>>
>> >>>>
>> >>>> ___
>> >>>> Pulp-dev mailing list
>> >>>> Pulp-dev@redhat.com
>> >>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >>>>
>> >>>
>> >>>
>> >>> ___
>> >>> Pulp-dev mailing list
>> >>> Pulp-dev@redhat.com
>> >>> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >>>
>> >>
>> >>
>> >> ___
>> >> Pulp-dev mailing list
>> >> Pulp-dev@redhat.com
>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >>
>> >
>> >
>> > ___
>> > Pulp-dev mailing list
>> > Pulp-dev@redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>> >
>>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Roadmap Challenges

[Austin Macdonald]

I agree with the problems that Brian listed, but I hope we can focus on the
decision making process itself in addition to how those decisions are
communicated.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

>
> I haven't seen a counterargument to the assertion that it is **incorrect**
>> to control repo membership without allowing plugin validation.
>>
>
> We could add hooks for plugins to provide validation for each content
> type. The Content model in pulpcore could define
> "add_to_repository_version" and "remove_from_repository_version" methods
> that take repository version as a parameter. The default behavior would be
> that exists now. The Content models in the plugins could then override the
> methods and provide validation needed for removing from and adding to a
> repository version. The repository version REST api would then use these
> methods on the content models to add and remove content.
>

In Pulp 2, core controls the endpoints and uses hooks to involve the
plugins but it is difficult to put the right hooks wherever any plugin
needs them. That may be an overly simplistic comparison though because I
think your proposal is just for add/remove, and not for more complex
concepts like sync. The general idea is plausible, but I would want to hear
more about how the implementation would look before I try to make technical
comparisons.

Generally, I prefer to avoid the hook pattern in favor of plugin-defined
endpoints. The pattern I prefer is "offer plugin writers tools, but stay
out of their way."
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

On Tue, Mar 27, 2018 at 6:11 PM, Brian Bouterse <bbout...@redhat.com> wrote:

> In terms of removing POST /api/v3/repositories//versions/ from
> core, I want to bring it back to the MVP language and REST which drove the
> original design. The MVP has: "As an authenticated user, I can create a new
> version by adding or removing content to the latest version." To
> facilitate that in a generic way, we need a core endpoint to do that, i.e.
> /api/v3/repositories//versions/. My concern is that removing it
> would cause us to not fulfill our use cases without requiring more code
> from some plugin writers. Also in terms of REST philosophy, POSTing to the
> RespotoryVersion resource to create a new RepositoryVersion is the
> traditional url design.
>
> For pulp_ansible, for example, the generic add/remove functionality at
> core endpoint ^ would be meet all of the pulp_ansible user's needs because
> of the way the pulp_ansible content is modelled. So removing this endpoint
> means more work for pulp_ansible developers w.r.t creating repo versions
> and providing tasks and endpoints. I don't see this extra responsibility on
> plugin writers coming with a clear benefit.
>
>
I'm sensitive to adding work for the plugin writers, but convenience needs
to be discussed after correctness.

I haven't seen a counterargument to the assertion that it is **incorrect**
to control repo membership without allowing plugin validation.


> One idea I liked in this discussion is to have a documented convention
> that encourages plugin writers to put their viewsets in a namespaced area.
> They still need the ability to put them anywhere due to live API
> goals/requirements so this would only be a convention for those tasks they
> are offering directly to their users.
>
>
>
>
>
> On Tue, Mar 27, 2018 at 5:40 PM, David Davis <davidda...@redhat.com>
> wrote:
>
>> I like Austin’s task proposal in that the plugin writers can focus on
>> serializers and tasks that can be easily integrated with core. That said, I
>> agree on the counterpoints raise about the new resource endpoints and
>> turning much of pulp into a task running system. So I am a bit mixed on
>> which approach is better.
>>
>> I do think that we should remove POST /api/v3/repositories//versions/
>> from core.
>>
>>
>> David
>>
>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald <amacd...@redhat.com>
>> wrote:
>>
>>>
>>>>- /api/v3/repositories//versions/ endpoint does not perform
>>>>plugin specific validation which can lead to "broken" repository 
>>>> versions.
>>>>- Plugin authors don't have any convention to follow when creating
>>>>custom REST API endpoints for creating repository versions.
>>>>- As a result of ^, a user will have a hard time identifying
>>>>repository version creation APIs in different plugins.
>>>>
>>>>
>>> I agree with these points.
>>>
>>>>
>>>>-
>>>>
>>>> My first inclination is to disable the ability to POST to
>>>> /api/v3/repositories//versions/ and require users to use the
>>>> plugin specific APIs for creating repository versions. However, I think
>>>> that integrators of build systems that produce a variety of content types
>>>> would have a lot more flexibility if they could use a single generic API
>>>> endpoint to create a repository version independent of the content type.
>>>>
>>>> Let's continue this discussion by answering the following question:
>>>>
>>>>- Should we disable the ability to POST to
>>>>/api/v3/repositories//versions/ and require users to always
>>>>use a plugin specific repository version creation API?
>>>>
>>>> Yes, I think we should disable POST to /api/v3/repositories//ve
>>> rsions/
>>>
>>> Simplifying integration is important, but we should not sacrifice
>>> correctness enforcement.
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

>
>
>- /api/v3/repositories//versions/ endpoint does not perform
>plugin specific validation which can lead to "broken" repository versions.
>- Plugin authors don't have any convention to follow when creating
>custom REST API endpoints for creating repository versions.
>- As a result of ^, a user will have a hard time identifying
>repository version creation APIs in different plugins.
>
>
I agree with these points.

>
>-
>
> My first inclination is to disable the ability to POST to
> /api/v3/repositories//versions/ and require users to use the plugin
> specific APIs for creating repository versions. However, I think that
> integrators of build systems that produce a variety of content types would
> have a lot more flexibility if they could use a single generic API endpoint
> to create a repository version independent of the content type.
>
> Let's continue this discussion by answering the following question:
>
>- Should we disable the ability to POST to 
> /api/v3/repositories//versions/
>and require users to always use a plugin specific repository version
>creation API?
>
> Yes, I think we should disable POST to /api/v3/repositories//
versions/

Simplifying integration is important, but we should not sacrifice
correctness enforcement.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

On Mon, Mar 26, 2018 at 4:43 PM, Dennis Kliban <dkli...@redhat.com> wrote:

> On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald <aus...@redhat.com>
> wrote:
>
>>
>>
>> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban <dkli...@redhat.com>
>> wrote:
>>
>>> This proposal does not make the plugin writer's job any easier. This
>>> simply changes where the plugin writer is providing validation logic, in a
>>> serializer vs. in a view.
>>>
>>
>> It does make the plugin writer's job easier *because* it changes how the
>> validation is provided, Plugin writers already have to understand how to
>> create models/serializers (and it is very simple) for all the other
>> objects. Seriallizer validation is much simpler than viewset validation
>> because serializers are literally designed to do this.
>>
>
> If we don't change anything, a plugin writer is responsible for creating a
> task that creates a Repository Version and a view which will perform
> validation and then dispatches a task. If we do make the proposed change,
> the plugin writer will still have to write the task code and then also a
> serializer that performs validation. The proposed change only removes the
> responsibility of dispatching a task which is 1 line of code.
>

The difference is not in the number of lines, it is in the complexity.
Validation with serializers is simpler, and more similar to other plugin
code.

New Way:
https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L15-L21
Old way:
https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L13-L30

That "1 line of code" puts the plugin writer in contact with a very complex
part of the pulpcore internals. `apply_async_with_reservation`:
https://github.com/pulp/pulp/blob/e8f804dc6595732dad38ec478877d919d7d97063/pulpcore/pulpcore/tasking/tasks.py#L180



> The other problem I see is that complexity for users is increased. Instead
>>> of having 1 resource for tracking task progress, there would be an infinite
>>> number of resources for tracking progress of tasks.
>>>
>>
>> In the proposal, Tasks are Master/Detail. The user doesn't have to
>> "track" it at all. They can still use v3/tasks/ to see all the tasks.
>> Retrieving tasks will either be the same (v3/tasks/) or easier if they know
>> the task they are looking for (v3/tasks/file/syncs/)
>>
>
> My understanding of the master/detail pattern we are using for our other
> resources is that each task's URI will look something like this:
> /api/v3/tasks/file/syncs//
>
> If the above is true, that means the client has to be aware of
> '/api/v3/tasks/file/syncs//' resource. The user would need to know
> how to interpret the response from each task type. There could be an
> infinite number of these tasks with many different arguments. So writing
> client code handling all those responses could become a nightmare. Right
> now the user just needs to know how to interpret the response from
> '/api/v3/tasks/' for all tasks.
>

I might be missing your point on this one. If you are talking about polling
tasks, the href of a new Task is returned when the Task is created.The
client should use the href that is returned, never string formatting urls.
Even so, the Tasks in this proposal will be part of the API schema, so
clients (and docs users) know where the endpoints are and what the
arguments should be. This would actually be a big advantage over the way
things are now. AFAIK, v3/importers/1234/sync/ is invisible to clients, so
we replace one invisible endpoint with one visible endpoint. No complexity
added.

If the client doesn't know how to interpret extra fields (the Task
parameters), they don't need to. All tasks share the same base fields, and
those are the fields that need to be interpreted when the client is polling
a task.


> Value #1: We have to do something to address the problem that
> adding/removing content to a repository without plugin input is incorrect.
> This proposal is one possibility, but it isn't valid to compare the value
> against doing nothing. Instead, if you don't like this option, we need to
> compare it to other proposals for how to involve plugins in tasks.


> We don't have a problem. I agree that a user needs to know a whole lot of
> information to correctly create a working repository version for a complex
> content type such as a Docker manifest. Without all this information on
> hand, the user could easily create a broken repository version. However, a
> rich client could solve that problem. If the plugin writer wants to support
> simpler clients, she can provide an additional URL for handling the

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban  wrote:

> This proposal does not make the plugin writer's job any easier. This
> simply changes where the plugin writer is providing validation logic, in a
> serializer vs. in a view.
>

It does make the plugin writer's job easier *because* it changes how the
validation is provided, Plugin writers already have to understand how to
create models/serializers (and it is very simple) for all the other
objects. Seriallizer validation is much simpler than viewset validation
because serializers are literally designed to do this.


>
> The other problem I see is that complexity for users is increased. Instead
> of having 1 resource for tracking task progress, there would be an infinite
> number of resources for tracking progress of tasks.
>

In the proposal, Tasks are Master/Detail. The user doesn't have to "track"
it at all. They can still use v3/tasks/ to see all the tasks. Retrieving
tasks will either be the same (v3/tasks/) or easier if they know the task
they are looking for (v3/tasks/file/syncs/)


> I don't see the value added by the proposed change.
>

Value #1: We have to do something to address the problem that
adding/removing content to a repository without plugin input is incorrect.
This proposal is one possibility, but it isn't valid to compare the value
against doing nothing. Instead, if you don't like this option, we need to
compare it to other proposals for how to involve plugins in tasks.

Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
shouldn't need to understand the complexities of our tasking system.
Pulpcore can handle task dispatching generally, plugin writers just set
simple class attributes instead.

Value #3: It removes "action" endpoints and replaces them with RESTful
"creation of task" POSTs.

Value #4: It creates a convention for endpoints. Right now we have


v3/importers/file/1234/sync/
v3/publishers/python/1234/publish/

There is no logical place for add/remove to go because add/remove will not
involve an importer. It will be irritating to the users if we don't
encourage some consistency between plugins for the simplest Pulp
functionality.


Value #5: Task history becomes much more useful. Currently, the only data
on a task that connects it to user-facing objects is "created_resource".
This proposal will allow users to filter tasks by parameters. For example,
they can filter sync tasks by "repository" or "importer". They can also use
the detail endpoints (v3/tasks/file/ or v3/tasks/file/syncs/) to list tasks
based on the plugin or action.

Value #6: Since the parameters of Task are saved, it will be possible to
repeat tasks.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 Unit Test Plan Proposal

[Austin Macdonald]

-1 For a blocking check, -1 for attempting 100% coverage.

There is a *lot* of code in Pulp 3 that simply involves inheriting from
parents classes and defining attributes. If we add a ViewSet for instance,
there is nothing to test other than "asserting that we did what we did". I
have written lots of tests like that. IMO, they add no value and are time
consuming to write. Also, they are time consuming to maintain because every
change must also change the unit tests. This type of test almost never
catches a real problem.

A soft check would be a useful reminder to the contributor and the reviewer
to add tests where they make sense. + 1 soft check

Plugins: Each plugin team should determine their own unit test policy.


On Fri, Mar 23, 2018 at 1:26 PM, David Davis  wrote:

> We haven't had a unit test policy in Pulp 3, and the community and core
> committers would all like one. The general desire we've heard so far is to
> change course and encourage developers to add unit tests to their changes
> to Pulp 3.
>
> The policy we're suggesting is to add a coveralls[0] check for Pull
> Requests against the pulpcore 3.0-dev branch that shows the overall
> coverage percentage, e.g. 12.89%. This check would pass if and only if
> coverage increases or remains the same with the PR. We think this will
> eventually get us on the path to 100% unit test coverage.
>
> We propose the coveralls check be a soft check that allow for merging if
> it fails. We would document the policy and try to adhere to it even though
> it wouldn't formally block merging. At a future point when pulp3 (maybe the
> GA?) we could make this a hard check.
>
> Benefits:
> - It's easy, simple, and automatic
> - It's pretty objective and there's little room to argue with a number.
> - Helps us raise our unit test coverage gradually over time
>
> Downsides:
> - Could discourage community contributions
> - It can be a bit strict and unforgiving at times (especially if there's a
> hard check)
> - It only provides a guarantee around quantity of unit testing and not
> quality
>
>
> *Q: What about the existing functionality? Do we need to write unit tests
> for it?*
>
> We're not sure about this. We'd like community feedback. Is anyone
> interested in writing backfill unit tests? If so, then maybe we should.
>
> *Q: Will this also affect Pulp 2?*
>
> We're not planning on it but if we like this enough, we can look at adding
> it to Pulp 2.
>
> *Q: Will this affect plugins?*
>
> We want to start out with just pulpcore now and see how we like it. Then
> we'll look at adding it to pulp_file. In the future, we can also look at
> ways to make it easy for plugins to set this up.
>
> *Q: Do I no longer need to write pulp-smash test plan issues in Github for
> Pulp 3 features?*
>
> No, you should still do that.
>
> [0] https://coveralls.io/
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

To make sure this was feasable, I've created proof of concept PRs for
pulpcore and pulp_file. All of the proposed changes are proofed, but not
all work is done. For example, sync is implemented, but not publish.

   - https://github.com/pulp/pulp/pull/3394
   - https://github.com/pulp/pulp_file/pull/61

I am very pleased with how it turned out, and I think it will make plugin
writing even easier. I documented the design on a redmine issue. I
encourage feedback/discussion on the issue.

https://pulp.plan.io/issues/3522

On Tue, Mar 20, 2018 at 7:46 AM, Milan Kovacik <mkova...@redhat.com> wrote:

> On Fri, Mar 16, 2018 at 4:55 PM, Milan Kovacik <mkova...@redhat.com>
> wrote:
> >
> >
> >
> > On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald <aus...@redhat.com>
> wrote:
> >>
> >> I spoke with daviddavis about this and I would like to narrow the scope
> a bit. This discussion should be limited to endpoints that deploy tasks.
> The possibility for collisions that David pointed out regarding
> v3/content// is real, but should be discussed separately because
> Content objects should follow the pattern of the other plugin defined
> objects. All of them are master/detail so the namespacing
> v3/plugin//content/ would go deeply against the grain.
> >>
> >>
> >> On Mar 15, 2018 10:53, "David Davis" <davidda...@redhat.com> wrote:
> >>>
> >>> I have an amendment to this proposal. Instead of namespacing just the
> plugin task routes, I’d propose we namespace all plugin routes. Thus all
> plugin routes would be namespaced under something like
> “/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
> moved to “/api/v3/plugin/file/content/” and so on.
> >>>
> >>>
> >>> For task routes, plugins could still route to a pulpcore
> viewset/action:
> >>>
> >>> User route: POST /api/v3/plugin/file/repository_version/
> add_content=[…]
> >>>
> >>> Routes to: POST /api/v3/tasks plugin=file
> task=create_repository_version add_content=[…]
> >>
> >>
> >> AFAIK, we don't have a good mechanism for validating and re-routing
> endpoints quite like this. Instead, a view or viewset would be created,
> validation performed, and then a task is deployed. The task itself could be
> a vanilla task from pulpcore though. Another issue is that a POST to
> v3/plugin/file/repository_version/ implies that the resultant
> repository_version is typed in some way. The created resource href would
> still take the form v3/repositories/1234/versions/2/, so I think this is
> a little misleading.
> >
> >
> > +1 to generic Pulp concepts/objects staying outside of a plug-in
> namespace/path
> >
> >>
> >>
> >> For the other endpoint:
> >>  /api/v3/tasks plugin=file task=create_repository_version
> add_content=[…]
> >>
> >> There is a still the problem is that POST requests should not contain
> parameters that influence the allowable parameter set. For instance,
> plugin=python might require an additional parameter that is not allowed for
> the file plugin. In particular, this affects the auto-documented REST API.
> >>
> >>
> >> However, the concept in general could work. I've adapted David's
> suggestion and will present it side by side with the original idea. Note,
> this only applies to "actions", which are sync, publish, add/remove, and
> plugin-specific actions (including rich-dependency adding).
> >>
> >> The difference between the two ideas is based in semantics. Both would
> deploy the same task functions.
> >>
> >> "Action Based"
> >> POST v3/plugin/file/sync/
> >> Note that "sync" is singular. This is a file-plugin action to sync
> using the body parameters.
> >
> >
> > - 1 as this might imply mixing  and  actions
> while all the time, asynchronous actions, such as the sync call, always
> create a task object; the user would be creating the task objects at one
> endpoint but deleting, filtering, tracking those at another.
> >
> >>
> >>
> >> "Task Object Based"
> >> POST v3/tasks/file/syncs/
> >> Note that "syncs" is plural. This is a POST that creates a file-sync
> task object,.
>
> Btw an interesting consequence of tasks not being objects in a file
> importer and publisher are these two issues:
>  - The parameters in the API schema for the sync and publish endpoints
> are incorrect [1]
>  - Not able to access documentation endpoint [2]
>
> In the former issue, the DRF isn't able to automatically figur

Re: [Pulp-dev] Port Pulp3 to use RQ

[Austin Macdonald]

Not being familiar with RQ, I have questions (but no opinion).

Will we also be replacing RabbitMQ with Redis?
Does anyone on the team have experience with RQ? In production?
How well does RQ scale?
Is RQ's use of `pickle` a problem? https://pulp.plan.io/issues/23
RQ doesn't work on Windows. Is that a problem? (jk)


On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse  wrote:

> Motivation:
> 1. Celery causes many bugs and issues for Pulp2 and 3 users and there is
> no end in sight.
>
> 2. The Pulp core team spends a lot of effort fixing Celery bugs. It's
> often times just us doing it with little or no assistance from the upstream
> communities. It's across 4 projects: celery, kombu, billiard, and pyamqp.
>
> 3. Celery will never allow a coverage report to be generated when
> pulp-smash runs because Celery forked the multiprocessing library into
> something called billiard. This will limit Pulp forever.
>

> 4. I don't want to work with Celery anymore and I think the other
> maintainers (@dalley, @daviddavis) may feel the same. It's an endless
> headache. Even basic things don't work in Celery regularly.
>
> Proposed change: Replace Pulp3's usage of Celery with RQ (
> http://python-rq.org/)
>
> We would keep the exact same design of a resource manager with n workers,
> each worker pulling it's work exclusively from a dedicated queue. I've
> looked into porting pulp3 to it and it's doable because all the same
> concepts are there. There are a few details to work out, but I wanted to
> start the "should we" discussion before we do all-out technical planning.
>
> When would we do this? I'm proposing soon. It doesn't need to block the
> beta, but soon would be good. I don't think users will care much except for
> their systemd files, but it is fundamental and important to pulp3 so we
> want to get it testing sooner.
>
> Ideas, comments, questions are welcome!
>
> Thanks,
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] let's rename RepositoryVersion to Snapshot

[Austin Macdonald]

"Snapshot" is a nice way to explain what a RepositoryVersion is, especially
in the context of Publications. "Publish a snapshot."  I like the idea, and
I informally floated it around PulpCon but decided not to propose it
because:

   - Snapshot is a little misleading about the actual data we store.
   Specifically, since RepositoryVersions are stored as diffs, when a user
   views the "content in a version", this is calculated. This is a subtle
   point, and hopefully not user facing at all, but I think snapshot implies a
   little bit more certainty than we can offer.
   - A snapshot also implies a slightly different workflow to me. The
   workflow I expect with snapshots is to change Repositories "willy nilly",
   and when you are satisfied, you "take" an snapshot. Versions imply the
   workflow we have, which is that any time the content set of a Repository is
   changed, a new version is created.

However, I think those concerns are minor and are overshadowed by the
potential benefits. Also, I see a direct connection to the thread "Plugin
relationship to tasks". The name Snapshot/RepositoryVersion is part of the
choice of how we portray the changing of content set of a repo.

   1. We can "change a repo" which creates a new version.
   2. We can "create a new version" which has different content.

To me (1) implies "dispatching a task that has the side effect of creating
a new repository version. It would lend itself well to the concept of
"managing repositories" rather than "managing versions/snapshots". If we
choose this way, I think the name Snapshot conceptually makes sense.

(2) implies a POST to create a new RepositoryVersion. As explained in the
plugin tasks thread, there are some problems with this, but it is similar
to the concept of creating a git commit. I think we wouldn't think of
"creating a new Snapshot" to change the content.

On Tue, Mar 20, 2018 at 9:33 AM, Dennis Kliban  wrote:

> I propose that we rename the RepositoryVersion model in Pulp 3 to
> Snapshot. The REST API would also change to use /api/v3/repositories//
> snapshot/
>
> The Snapshot name is a better description of what a repository version is
> and it is also much shorter in length.
>
> Thoughts?
>
>
> -Dennis
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[Austin Macdonald]

Ive created tasks to do this work.

Rename:
https://pulp.plan.io/issues/3488

Move sync_mode, remove download_policy
https://pulp.plan.io/issues/3492

Each of these has corresponding plugin tasks, which are related to them.

On Thu, Mar 15, 2018 at 4:16 PM, David Davis <davidda...@redhat.com> wrote:

> Austin and Jeff,
>
> Thanks for the responses. I am happy with moving forward and opening an
> issue in redmine for this change
>
> I think I am +0 on dropping the fields. However, if we start to get
> complaints from our users, I think we should consider adding them back.
>
>
> David
>
> On Wed, Mar 14, 2018 at 10:16 AM, Jeff Ortel <jor...@redhat.com> wrote:
>
>> In pulp3, users need to keep track for a number of things.  For example,
>> without auto publish, users need to keep track of which importer(s) and
>> publishers need to be used for sync/publish workflows.  I fully expect that
>> users using the API will be maintaining some kind of automation/
>> orchestration on their end (shell scripts, ansible).  So, keeping track
>> of sync and download policies does not seem like much of a burden.  Also,
>> after further consideration, I don't think that storing either the sync
>> (mode) or download policy on the repository is appropriate.
>>
>>
>> On 03/13/2018 04:59 PM, David Davis wrote:
>>
>> Can you elaborate on what made you reconsider? Asking because I still see
>> the point that you and Justin raised about dropping the fields as an issue.
>>
>>
>> David
>>
>> On Mon, Mar 12, 2018 at 12:31 PM, Jeff Ortel <jor...@redhat.com> wrote:
>>
>>>
>>>
>>> On 03/12/2018 10:28 AM, Jeff Ortel wrote:
>>>
>>> On 03/08/2018 10:13 AM, Austin Macdonald wrote:
>>>
>>> Motivation:
>>> The name "importer" carries some inaccurate implications.
>>> 1) Importers should "import". Tasks like "sync" will do the actual
>>> importing. The object only holds the configuration that happens to be used
>>> by sync tasks.
>>> 2) Sync tasks on mirror mode remove content as well as add it, so
>>> "import" isn't quite right.
>>>
>>> Proposed name: Remote
>>>
>>> The inspiration for remote is "git remote". In git, remotes represent
>>> external repositories, which is almost exactly what our importers do.
>>>
>>>
>>> +1, The git/ostree "remote" concept applies very well to most of what an
>>> "importer" defines in pulp.
>>>
>>>
>>> ---
>>> Part 2: Trim the fields
>>>
>>> Currently, Importers have settings that can be categorized in 2 ways. I
>>> am proposing removing the "sync settings" from the Remote model:
>>>
>>> External Source information
>>> name
>>> feed_url
>>> validate
>>> ssl_ca_certificate
>>> ssl_client_certificate
>>> ssl_client_key
>>> ssl_validation
>>> proxy_url
>>> username
>>> password
>>>
>>> Sync settings
>>> download_policy
>>> sync_mode
>>>
>>> This had some advantages when Importers were related to Repositories.
>>> For example, having a repository.importer that always used the same sync
>>> mode made sense. However, the "how" to sync settings don't make much sense
>>> when importers and repositories are not linked. It seems very reasonable
>>> that a user might have 2 repositories that sync from the same source (ex
>>> EPEL). It does not make sense for them to have create an Importer for the
>>> EPEL repository twice or more just to change sync_mode or download policy.
>>> Instead of modeling these fields, I propose that they should POST body
>>> parameters.
>>>
>>>
>>> I, as a user, don't like having to specify download_policy &  sync_mode
>>> on every request.  The burden on the user to passing these consistently
>>> seems unnecessary and prone to error.  And, like something that pulp should
>>> store as part of it's value proposition.   Imagine an organization with
>>> tons of repositories and admins.  They would need to maintain a
>>> spreadsheet, notes, scripts for these settings so that admin A is syncing
>>> using the same settings as admin B.
>>>
>>> Perhaps download_policy &  sync_mode should be attributes of the
>>> repository.  Thoug

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

I spoke with daviddavis about this and I would like to narrow the scope a
bit. This discussion should be limited to endpoints that deploy tasks. The
possibility for collisions that David pointed out regarding
v3/content// is real, but should be discussed separately because
Content objects should follow the pattern of the other plugin defined
objects. All of them are master/detail so the namespacing
v3/plugin//content/ would go deeply against the grain.


On Mar 15, 2018 10:53, "David Davis"  wrote:

I have an amendment to this proposal. Instead of namespacing just the
plugin task routes, I’d propose we namespace all plugin routes. Thus all
plugin routes would be namespaced under something like
“/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
moved to “/api/v3/plugin/file/content/” and so on.


For task routes, plugins could still route to a pulpcore viewset/action:

User route: POST /api/v3/plugin/file/repository_version/ add_content=[…]

Routes to: POST /api/v3/tasks plugin=file task=create_repository_version
add_content=[…]


AFAIK, we don't have a good mechanism for validating and re-routing
endpoints quite like this. Instead, a view or viewset would be created,
validation performed, and then a task is deployed. The task itself could be
a vanilla task from pulpcore though. Another issue is that a POST to
v3/plugin/file/repository_version/ implies that the resultant
repository_version is typed in some way. The created resource href would
still take the form v3/repositories/1234/versions/2/, so I think this is a
little misleading.

For the other endpoint:
 /api/v3/tasks plugin=file task=create_repository_version add_content=[…]

There is a still the problem is that POST requests should not contain
parameters that influence the allowable parameter set. For instance,
plugin=python might require an additional parameter that is not allowed for
the file plugin. In particular, this affects the auto-documented REST API.

However, the concept in general could work. I've adapted David's suggestion
and will present it side by side with the original idea. Note, this only
applies to "actions", which are sync, publish, add/remove, and
plugin-specific actions (including rich-dependency adding).

The difference between the two ideas is based in semantics. Both would
deploy the same task functions.

"Action Based"
POST v3/plugin/file/sync/
Note that "sync" is singular. This is a file-plugin action to sync using
the body parameters.

"Task Object Based"
POST v3/tasks/file/syncs/
Note that "syncs" is plural. This is a POST that creates a file-sync task
object,.

Discussion:
Another consideration is "Where will a Live API live?". A benefit of
David's proposal is that we would be providing plugins with a namespaced
area to do whatever they need, from live api to actions. We probably would
want to create this namespace for things like live apis even if we go in
the tasks/file/syncs/ direction. The only downside of this part of David's
proposal is that our endpoints will still be "action" verbs instead of
nouns, and I am ok with that.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev