Re: [Pulp-dev] Port Pulp3 to use RQ

[David Davis]

Great, thanks.


David

On Mon, May 14, 2018 at 4:46 PM, Brian Bouterse  wrote:

> If you are using the dev environment produced by the pulp/devel repo then
> doing a vagrant destroy and vagrant up is the easiest way. Just make sure
> you fetched all the latest changes from all the repos.
>
> If you want to port an existing environment into RQ, you should:
>
> 1. install RQ (the special commit version, see the current installation
> docs)
> 2. update the systemd files (see the examples in the docs)
> 3. restart your workers
>
> If there are issues during ^ steps reach out and we can help resolve them.
>
> On Mon, May 14, 2018 at 4:39 PM, David Davis 
> wrote:
>
>> Is there any way to convert an existing dev environment to use rq?
>>
>> Or do I just need to vagrant destroy and vagrant up?
>>
>>
>> David
>>
>> On Mon, May 14, 2018 at 4:02 PM, Daniel Alley  wrote:
>>
>>> And pulp_python
>>>
>>> On Mon, May 14, 2018 at 3:55 PM, Brian Bouterse 
>>> wrote:
>>>
 RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported
 and merged pulp_ansible. This will be released with beta 3 of core coming
 out this Wednesday.

 If anyone runs into any issues please reach out via IRC or the mailing
 list.

 On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse 
 wrote:

> There's been a slight change in schedule. Now we believe the lowest
> risk option is to merge today instead of tomorrow.
>
> We're finishing the latest rebase now, letting Travis tell us it's
> good, and then merging it. We'll send a final note to the list post merge.
>
> Thanks to everyone for helping out!
>
> On Mon, May 14, 2018 at 12:07 PM, Dana Walker 
> wrote:
>
>> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
>> review/testing, and blog post.
>>
>> Dana Walker
>>
>> Associate Software Engineer
>>
>> Red Hat
>>
>> 
>> 
>>
>> On Wed, May 9, 2018 at 12:20 PM, David Davis 
>> wrote:
>>
>>> Great work on this. Also, thanks for announcing this on pulp-dev
>>> well in advance.
>>>
>>>
>>> David
>>>
>>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan  wrote:
>>>
 dalley has learned how to do some debugging already, so maybe he can
 look at doing a demo. Good suggestion, Kersom. It would be good to
 link to in a blog post - and  also point out the good demo @bmbouter
 put together for pulp 2.

 great job @dalley & @bmbouter on the blog post!

 On Wed, May 9, 2018 at 11:24 AM, Kersom  wrote:
 > At the proper time, a demo about the Pulp 3 task system will be
 very
 > beneficial. I am thinking about something similar what it was
 done for Pulp
 > 2.
 >
 > Looking forward for this.
 >
 > Regards,
 >
 >
 >
 >
 >
 >
 > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <
 bbout...@redhat.com> wrote:
 >>
 >> All PRs have Travis showing green and all necessary LGTMs. The
 plan is to
 >> merge next Tuesday the 15th, which means it will be in core Beta
 4.
 >>
 >> Yesterday, @dalley and I published a blog post which outlines
 the change
 >> for users along with a porting guide for plugins to port onto RQ
 as well.
 >>
 >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
 >>
 >> Thank you to everyone for the help, collaboration, and energy on
 this
 >> significant change.
 >>
 >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley 
 wrote:
 >>>
 >>> I've finished my review and resolved all of the 'blocker'
 issues that
 >>> were uncovered during testing.  Overall, I'm highly confident
 that this is a
 >>> better path forwards than the continued use of Celery / Kombu.
 There are a
 >>> couple of outstanding edge cases to be resolved eventually,
 which I plan to
 >>> file as issues post-merge, but nothing serious or intractable.
 >>>
 >>> If there are no objections, I think it would be reasonable to
 merge this
 >>> code after this week's beta builds are published (after, in
 order to avoid
 >>> major changes during Summit / PyCon prep time).
 >>>
 >>> Thank you, Brian, for doing the planning and work needed to
 make this
 >>> happen.  It was a lot of effort and is very highly appreciated.
 >>>
 >>> On Mon, Apr 30, 

Re: [Pulp-dev] Port Pulp3 to use RQ

[Brian Bouterse]

If you are using the dev environment produced by the pulp/devel repo then
doing a vagrant destroy and vagrant up is the easiest way. Just make sure
you fetched all the latest changes from all the repos.

If you want to port an existing environment into RQ, you should:

1. install RQ (the special commit version, see the current installation
docs)
2. update the systemd files (see the examples in the docs)
3. restart your workers

If there are issues during ^ steps reach out and we can help resolve them.

On Mon, May 14, 2018 at 4:39 PM, David Davis  wrote:

> Is there any way to convert an existing dev environment to use rq?
>
> Or do I just need to vagrant destroy and vagrant up?
>
>
> David
>
> On Mon, May 14, 2018 at 4:02 PM, Daniel Alley  wrote:
>
>> And pulp_python
>>
>> On Mon, May 14, 2018 at 3:55 PM, Brian Bouterse 
>> wrote:
>>
>>> RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported
>>> and merged pulp_ansible. This will be released with beta 3 of core coming
>>> out this Wednesday.
>>>
>>> If anyone runs into any issues please reach out via IRC or the mailing
>>> list.
>>>
>>> On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse 
>>> wrote:
>>>
 There's been a slight change in schedule. Now we believe the lowest
 risk option is to merge today instead of tomorrow.

 We're finishing the latest rebase now, letting Travis tell us it's
 good, and then merging it. We'll send a final note to the list post merge.

 Thanks to everyone for helping out!

 On Mon, May 14, 2018 at 12:07 PM, Dana Walker 
 wrote:

> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
> review/testing, and blog post.
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
> On Wed, May 9, 2018 at 12:20 PM, David Davis 
> wrote:
>
>> Great work on this. Also, thanks for announcing this on pulp-dev well
>> in advance.
>>
>>
>> David
>>
>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan  wrote:
>>
>>> dalley has learned how to do some debugging already, so maybe he can
>>> look at doing a demo. Good suggestion, Kersom. It would be good to
>>> link to in a blog post - and  also point out the good demo @bmbouter
>>> put together for pulp 2.
>>>
>>> great job @dalley & @bmbouter on the blog post!
>>>
>>> On Wed, May 9, 2018 at 11:24 AM, Kersom  wrote:
>>> > At the proper time, a demo about the Pulp 3 task system will be
>>> very
>>> > beneficial. I am thinking about something similar what it was done
>>> for Pulp
>>> > 2.
>>> >
>>> > Looking forward for this.
>>> >
>>> > Regards,
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <
>>> bbout...@redhat.com> wrote:
>>> >>
>>> >> All PRs have Travis showing green and all necessary LGTMs. The
>>> plan is to
>>> >> merge next Tuesday the 15th, which means it will be in core Beta
>>> 4.
>>> >>
>>> >> Yesterday, @dalley and I published a blog post which outlines the
>>> change
>>> >> for users along with a porting guide for plugins to port onto RQ
>>> as well.
>>> >>
>>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>> >>
>>> >> Thank you to everyone for the help, collaboration, and energy on
>>> this
>>> >> significant change.
>>> >>
>>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley 
>>> wrote:
>>> >>>
>>> >>> I've finished my review and resolved all of the 'blocker' issues
>>> that
>>> >>> were uncovered during testing.  Overall, I'm highly confident
>>> that this is a
>>> >>> better path forwards than the continued use of Celery / Kombu.
>>> There are a
>>> >>> couple of outstanding edge cases to be resolved eventually,
>>> which I plan to
>>> >>> file as issues post-merge, but nothing serious or intractable.
>>> >>>
>>> >>> If there are no objections, I think it would be reasonable to
>>> merge this
>>> >>> code after this week's beta builds are published (after, in
>>> order to avoid
>>> >>> major changes during Summit / PyCon prep time).
>>> >>>
>>> >>> Thank you, Brian, for doing the planning and work needed to make
>>> this
>>> >>> happen.  It was a lot of effort and is very highly appreciated.
>>> >>>
>>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
>>> bbout...@redhat.com>
>>> >>> wrote:
>>> 
>>>  Through several rebases, now all PRs are showing the RQ PRs on
>>> Travis as
>>>  passing with pulp-smash. Several points of feedback 

Re: [Pulp-dev] Port Pulp3 to use RQ

[David Davis]

Is there any way to convert an existing dev environment to use rq?

Or do I just need to vagrant destroy and vagrant up?


David

On Mon, May 14, 2018 at 4:02 PM, Daniel Alley  wrote:

> And pulp_python
>
> On Mon, May 14, 2018 at 3:55 PM, Brian Bouterse 
> wrote:
>
>> RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported
>> and merged pulp_ansible. This will be released with beta 3 of core coming
>> out this Wednesday.
>>
>> If anyone runs into any issues please reach out via IRC or the mailing
>> list.
>>
>> On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse 
>> wrote:
>>
>>> There's been a slight change in schedule. Now we believe the lowest risk
>>> option is to merge today instead of tomorrow.
>>>
>>> We're finishing the latest rebase now, letting Travis tell us it's good,
>>> and then merging it. We'll send a final note to the list post merge.
>>>
>>> Thanks to everyone for helping out!
>>>
>>> On Mon, May 14, 2018 at 12:07 PM, Dana Walker 
>>> wrote:
>>>
 +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
 review/testing, and blog post.

 Dana Walker

 Associate Software Engineer

 Red Hat

 
 

 On Wed, May 9, 2018 at 12:20 PM, David Davis 
 wrote:

> Great work on this. Also, thanks for announcing this on pulp-dev well
> in advance.
>
>
> David
>
> On Wed, May 9, 2018 at 8:29 AM, Robin Chan  wrote:
>
>> dalley has learned how to do some debugging already, so maybe he can
>> look at doing a demo. Good suggestion, Kersom. It would be good to
>> link to in a blog post - and  also point out the good demo @bmbouter
>> put together for pulp 2.
>>
>> great job @dalley & @bmbouter on the blog post!
>>
>> On Wed, May 9, 2018 at 11:24 AM, Kersom  wrote:
>> > At the proper time, a demo about the Pulp 3 task system will be very
>> > beneficial. I am thinking about something similar what it was done
>> for Pulp
>> > 2.
>> >
>> > Looking forward for this.
>> >
>> > Regards,
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <
>> bbout...@redhat.com> wrote:
>> >>
>> >> All PRs have Travis showing green and all necessary LGTMs. The
>> plan is to
>> >> merge next Tuesday the 15th, which means it will be in core Beta 4.
>> >>
>> >> Yesterday, @dalley and I published a blog post which outlines the
>> change
>> >> for users along with a porting guide for plugins to port onto RQ
>> as well.
>> >>
>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>> >>
>> >> Thank you to everyone for the help, collaboration, and energy on
>> this
>> >> significant change.
>> >>
>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley 
>> wrote:
>> >>>
>> >>> I've finished my review and resolved all of the 'blocker' issues
>> that
>> >>> were uncovered during testing.  Overall, I'm highly confident
>> that this is a
>> >>> better path forwards than the continued use of Celery / Kombu.
>> There are a
>> >>> couple of outstanding edge cases to be resolved eventually, which
>> I plan to
>> >>> file as issues post-merge, but nothing serious or intractable.
>> >>>
>> >>> If there are no objections, I think it would be reasonable to
>> merge this
>> >>> code after this week's beta builds are published (after, in order
>> to avoid
>> >>> major changes during Summit / PyCon prep time).
>> >>>
>> >>> Thank you, Brian, for doing the planning and work needed to make
>> this
>> >>> happen.  It was a lot of effort and is very highly appreciated.
>> >>>
>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
>> bbout...@redhat.com>
>> >>> wrote:
>> 
>>  Through several rebases, now all PRs are showing the RQ PRs on
>> Travis as
>>  passing with pulp-smash. Several points of feedback have been
>> addressed.
>> 
>>  If you're interested in commenting on these PRs or trying them
>> out,
>>  please do. I hope to merge after the other taking system
>> maintainers @dalley
>>  and @daviddavis have finished their testing/review and barring
>> any other
>>  calls for delay or blocking concerns.
>> 
>>  If there are any questions, issues, or concerns, please reach
>> out, and
>>  we can talk through them.
>> 
>> 
>> 
>>  On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
>> bbout...@redhat.com>
>>  wrote:
>> >
>> > I put together a prototype and posted the PRs. I'm still

Re: [Pulp-dev] Port Pulp3 to use RQ

[Brian Bouterse]

RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported and
merged pulp_ansible. This will be released with beta 3 of core coming out
this Wednesday.

If anyone runs into any issues please reach out via IRC or the mailing list.

On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse 
wrote:

> There's been a slight change in schedule. Now we believe the lowest risk
> option is to merge today instead of tomorrow.
>
> We're finishing the latest rebase now, letting Travis tell us it's good,
> and then merging it. We'll send a final note to the list post merge.
>
> Thanks to everyone for helping out!
>
> On Mon, May 14, 2018 at 12:07 PM, Dana Walker  wrote:
>
>> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
>> review/testing, and blog post.
>>
>> Dana Walker
>>
>> Associate Software Engineer
>>
>> Red Hat
>>
>> 
>> 
>>
>> On Wed, May 9, 2018 at 12:20 PM, David Davis 
>> wrote:
>>
>>> Great work on this. Also, thanks for announcing this on pulp-dev well in
>>> advance.
>>>
>>>
>>> David
>>>
>>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan  wrote:
>>>
 dalley has learned how to do some debugging already, so maybe he can
 look at doing a demo. Good suggestion, Kersom. It would be good to
 link to in a blog post - and  also point out the good demo @bmbouter
 put together for pulp 2.

 great job @dalley & @bmbouter on the blog post!

 On Wed, May 9, 2018 at 11:24 AM, Kersom  wrote:
 > At the proper time, a demo about the Pulp 3 task system will be very
 > beneficial. I am thinking about something similar what it was done
 for Pulp
 > 2.
 >
 > Looking forward for this.
 >
 > Regards,
 >
 >
 >
 >
 >
 >
 > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse 
 wrote:
 >>
 >> All PRs have Travis showing green and all necessary LGTMs. The plan
 is to
 >> merge next Tuesday the 15th, which means it will be in core Beta 4.
 >>
 >> Yesterday, @dalley and I published a blog post which outlines the
 change
 >> for users along with a porting guide for plugins to port onto RQ as
 well.
 >>
 >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
 >>
 >> Thank you to everyone for the help, collaboration, and energy on this
 >> significant change.
 >>
 >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley 
 wrote:
 >>>
 >>> I've finished my review and resolved all of the 'blocker' issues
 that
 >>> were uncovered during testing.  Overall, I'm highly confident that
 this is a
 >>> better path forwards than the continued use of Celery / Kombu.
 There are a
 >>> couple of outstanding edge cases to be resolved eventually, which I
 plan to
 >>> file as issues post-merge, but nothing serious or intractable.
 >>>
 >>> If there are no objections, I think it would be reasonable to merge
 this
 >>> code after this week's beta builds are published (after, in order
 to avoid
 >>> major changes during Summit / PyCon prep time).
 >>>
 >>> Thank you, Brian, for doing the planning and work needed to make
 this
 >>> happen.  It was a lot of effort and is very highly appreciated.
 >>>
 >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
 bbout...@redhat.com>
 >>> wrote:
 
  Through several rebases, now all PRs are showing the RQ PRs on
 Travis as
  passing with pulp-smash. Several points of feedback have been
 addressed.
 
  If you're interested in commenting on these PRs or trying them out,
  please do. I hope to merge after the other taking system
 maintainers @dalley
  and @daviddavis have finished their testing/review and barring any
 other
  calls for delay or blocking concerns.
 
  If there are any questions, issues, or concerns, please reach out,
 and
  we can talk through them.
 
 
 
  On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
 bbout...@redhat.com>
  wrote:
 >
 > I put together a prototype and posted the PRs. I'm still working
 to get
 > Travis happy, but locally 100% of smash tests using these
 branches. It's
 > worked very reliably for me so far. There are no gaps in the pulp
 feature
 > set on top of RQ.
 >
 > I hope people test it out and give some feedback. See the commit
 > messages for details on what was done. Here are the PRs:
 >
 > https://github.com/pulp/pulp/pull/3454
 > https://github.com/pulp/pulp_file/pull/72
 > https://github.com/pulp/devel/pull/146
 > 

[Pulp-dev] Composed Repositories

[Jeff Ortel]

Let's brainstorm on something.

Pulp needs to deal with remote repositories that are composed of 
multiple content types which may span the domain of a single plugin.  
Here are a few examples.  Some Red Hat RPM repositories are composed of: 
RPMs, DRPMs, , ISOs and Kickstart Trees.  Some OSTree repositories are 
composed of OSTrees & Kickstart Trees. This raises a question:


How can pulp3 best support syncing with remote repositories that are 
composed of multiple (unrelated) content types in a way that doesn't 
result in plugins duplicating support for content types?


Few approaches come to mind:

1. Multiple plugins (Remotes) participate in the sync flow to produce a 
new repository version.
2. Multiple plugins (Remotes) are sync'd successively each producing a 
new version of a repository.  Only the last version contains the fully 
sync'd composition.

3. Plugins share code.
4. Other?


Option #1: Sync would be orchestrated by core or the user so that 
multiple plugins (Remotes) participate in populating a new repository 
version.  For example: the RPM plugin (Remote) and the Kickstart Tree 
plugin (Remote) would both be sync'd against the same remote repository 
that is composed of both types.  The new repository version would be 
composed of the result of both plugin (Remote) syncs.  To support this, 
we'd need to provide a way for each plugin to operate seamlessly on the 
same (new) repository version.  Perhaps something internal to the 
RepositoryVersion. The repository version would not be marked "complete" 
until the last plugin (Remote) sync has succeeded.  More complicated 
than #2 but results in only creating truly complete versions or nothing. 
No idea how this would work with current REST API whereby plugins 
provide sync endpoints.


Option #2: Sync would be orchestrated by core or the user so that 
multiple plugins (Remotes) create successive repository versions.  For 
example: the RPM plugin (Remote) and the Kickstart Tree plugin (Remote) 
would both be sync'd against the same remote repository that is a 
composition including both types.  The intermediate versions would be 
incomplete. Only the last version contains the fully sync'd 
composition.  This approach can be supported by core today :) but will 
produce incomplete repository versions that are marked complete=True.  
This /seems/ undesirable, right?  This may not be a problem for 
distribution since I would imaging that only the last (fully composed) 
version would be published.  But what about other usages of the 
repository's "latest" version?


Option #3: requires a plugin to be aware of specific repository 
composition(s); other plugins and creates a code dependency between 
plugins.  For example, the RPM plugin could delegate ISOs to the File 
plugin and Kickstart Trees to the KickStart Tree plugin.


For all options, plugins (Remotes) need to limit sync to affect only 
those content types within their domain.  For example, the RPM (Remote) 
sync cannot add/remove ISO or KS Trees.


I am an advocate of some from of options #1 or #2.  Combining plugins 
(Remotes) as needed to deal with arbitrary combinations within remote 
repositories seems very powerful; does not impose complexity on plugin 
writers; and does not introduce code dependencies between plugins.


Thoughts?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PUP5 -- Adopting the "Common Cure Rights Commitment" for Pulp Core

[Dana Walker]

Other than the noted point that it takes time, is there any reason why Pulp
should stay on the current license instead of moving to GPLv3 (one of the
stated alternatives in this PUP)?  I don't know much about the differences
currently, but it strikes me that our new Pulp 3 using Python 3 would be a
good fit for moving to a new license as well that has taken various things
such as this enforcement issue into account and evolved over time.

Thoughts?

--Dana

Dana Walker

Associate Software Engineer

Red Hat




On Mon, May 14, 2018 at 6:28 AM, Ina Panova  wrote:

> *understanding
>
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Mon, May 14, 2018 at 12:27 PM, Ina Panova  wrote:
>
>> To make a concrete example to prove my understating:
>>
>> Since pulp_rpm is maintained by core team we could adopt this change,
>> meanwhile pulp_deb is beyond our control and we( core team) cannot enforce
>> or influence this change.
>> Yes?
>>
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>> On Tue, May 8, 2018 at 5:55 PM, Brian Bouterse 
>> wrote:
>>
>>> A Pulp Update Proposal (PUP) pull request has been opened by the
>>> go-to-lawyer for the Pulp community, Richard Fontana. The PUP is PUP5 [0].
>>> I don't want to paraphrase it here, so please read it [0] if you are
>>> interested to understand what it does.
>>>
>>> I am proposing a period of questions/discussion via the list/PR and then
>>> a call for a vote according to the process. All questions are welcome,
>>> please ask.
>>>
>>>
>>> # Timeline
>>>
>>> Today - May 18th mailing list and PR discussion
>>> May 18th - formally call for a vote which would end 12 calendar days
>>> from then May 30th
>>> May 30th - Merge or reject
>>>
>>>
>>> # FAQs
>>>
>>> Is this relicensing Pulp?
>>> No. It's still GPLv2. This adopts a procedural enforment approach within
>>> the existing license. See @rfontana's response here:
>>> https://github.com/pulp/pups/pull/9#issuecomment-384523020
>>>
>>> Do all prior contributors need to sign off on this change?
>>> No, because it's not a relicensing.
>>>
>>> Does this affect core, plugins, or both?
>>> This PR is only scoped to affect the GPLv2 codebases maintained by the
>>> core team. Plugins make their own decisions without PUPs. Initially this
>>> would be pulp/pulp, and as other GPLv2 repositories are maintained by the
>>> core team, it would apply to this in the future as well.
>>>
>>>
>>> [0]: https://github.com/pulp/pups/pull/9/files
>>>
>>> Thanks,
>>> Brian
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] MODIFIED redmine issues for 3.0 beta

[Dennis Kliban]

Historically we would transition issues in Pulp's issue tracker from
MODIFIED to CLOSED - CURRENTRELEASE when a GA build went out the door. The
same approach is working against us for the duration of the Pulp 3.0 beta.
We have a large number of issues in MODIFIED state, but are considered
released.

I propose that we transition issues to CLOSED - CURRENTRELEASE when they
have been shipped with a 3.0 beta.

Could we add automation to do this at release time?

What do you all think?


Thanks,
Dennis
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Port Pulp3 to use RQ

[Brian Bouterse]

There's been a slight change in schedule. Now we believe the lowest risk
option is to merge today instead of tomorrow.

We're finishing the latest rebase now, letting Travis tell us it's good,
and then merging it. We'll send a final note to the list post merge.

Thanks to everyone for helping out!

On Mon, May 14, 2018 at 12:07 PM, Dana Walker  wrote:

> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
> review/testing, and blog post.
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
> On Wed, May 9, 2018 at 12:20 PM, David Davis 
> wrote:
>
>> Great work on this. Also, thanks for announcing this on pulp-dev well in
>> advance.
>>
>>
>> David
>>
>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan  wrote:
>>
>>> dalley has learned how to do some debugging already, so maybe he can
>>> look at doing a demo. Good suggestion, Kersom. It would be good to
>>> link to in a blog post - and  also point out the good demo @bmbouter
>>> put together for pulp 2.
>>>
>>> great job @dalley & @bmbouter on the blog post!
>>>
>>> On Wed, May 9, 2018 at 11:24 AM, Kersom  wrote:
>>> > At the proper time, a demo about the Pulp 3 task system will be very
>>> > beneficial. I am thinking about something similar what it was done for
>>> Pulp
>>> > 2.
>>> >
>>> > Looking forward for this.
>>> >
>>> > Regards,
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse 
>>> wrote:
>>> >>
>>> >> All PRs have Travis showing green and all necessary LGTMs. The plan
>>> is to
>>> >> merge next Tuesday the 15th, which means it will be in core Beta 4.
>>> >>
>>> >> Yesterday, @dalley and I published a blog post which outlines the
>>> change
>>> >> for users along with a porting guide for plugins to port onto RQ as
>>> well.
>>> >>
>>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>> >>
>>> >> Thank you to everyone for the help, collaboration, and energy on this
>>> >> significant change.
>>> >>
>>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley 
>>> wrote:
>>> >>>
>>> >>> I've finished my review and resolved all of the 'blocker' issues that
>>> >>> were uncovered during testing.  Overall, I'm highly confident that
>>> this is a
>>> >>> better path forwards than the continued use of Celery / Kombu.
>>> There are a
>>> >>> couple of outstanding edge cases to be resolved eventually, which I
>>> plan to
>>> >>> file as issues post-merge, but nothing serious or intractable.
>>> >>>
>>> >>> If there are no objections, I think it would be reasonable to merge
>>> this
>>> >>> code after this week's beta builds are published (after, in order to
>>> avoid
>>> >>> major changes during Summit / PyCon prep time).
>>> >>>
>>> >>> Thank you, Brian, for doing the planning and work needed to make this
>>> >>> happen.  It was a lot of effort and is very highly appreciated.
>>> >>>
>>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse >> >
>>> >>> wrote:
>>> 
>>>  Through several rebases, now all PRs are showing the RQ PRs on
>>> Travis as
>>>  passing with pulp-smash. Several points of feedback have been
>>> addressed.
>>> 
>>>  If you're interested in commenting on these PRs or trying them out,
>>>  please do. I hope to merge after the other taking system
>>> maintainers @dalley
>>>  and @daviddavis have finished their testing/review and barring any
>>> other
>>>  calls for delay or blocking concerns.
>>> 
>>>  If there are any questions, issues, or concerns, please reach out,
>>> and
>>>  we can talk through them.
>>> 
>>> 
>>> 
>>>  On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
>>> bbout...@redhat.com>
>>>  wrote:
>>> >
>>> > I put together a prototype and posted the PRs. I'm still working
>>> to get
>>> > Travis happy, but locally 100% of smash tests using these
>>> branches. It's
>>> > worked very reliably for me so far. There are no gaps in the pulp
>>> feature
>>> > set on top of RQ.
>>> >
>>> > I hope people test it out and give some feedback. See the commit
>>> > messages for details on what was done. Here are the PRs:
>>> >
>>> > https://github.com/pulp/pulp/pull/3454
>>> > https://github.com/pulp/pulp_file/pull/72
>>> > https://github.com/pulp/devel/pull/146
>>> > https://github.com/PulpQE/pulp-smash/pull/960
>>> >
>>> > Feel free to send questions here or to the PR. Any feedback is
>>> welcome.
>>> >
>>> > -Brian
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <
>>> mkova...@redhat.com>
>>> > wrote:
>>> >>
>>> >> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
>>> >> there's Fakeredis ;)
>>> >>
>>> >>
>>> >> --
>>> >> milan
>>> >>
>>> >>
>>> >> 

Re: [Pulp-dev] Pulp3 Docs Builder Issues

[Dana Walker]

This seems like a reasonable approach.  +1

Dana Walker

Associate Software Engineer

Red Hat




On Thu, May 10, 2018 at 4:35 PM, Brian Bouterse  wrote:

> To have docs to align with the 4 phases of development we would have 4
> release streams: 'nightly', 'beta', 'rc', and 'stable'. The url structure
> of the site would be something like:
>
> docs.pulpproject.org/en/{x.y}/{beta
>  | nightly | rc}/{tag}/
>
> The {beta | nightly | rc} is literally 'beta' if beta, 'nightly' if
> nightly, 'rc' if rc, and that part is left out if it's stable. Travis will
> know if {tag} has a 'b' in it then it's a beta publish, 'rc' for release
> candidate, and nothing for stable. I believe the nightly one could run via
> cron which would know it was run by cron (and therefore nightly).
>
> {tag} would be the tag the release is tagged as e.g. 3.0.1b2, 3.1.3rc1,
> 3.0.0 (the ga).
>
> The docs pusher in Travis will also ensure that leaving off the tag gives
> you the "latest" from that release stream so:
> docs.pulpproject.org/en/3.0/beta/ would give you the latest beta and
> docs.pulpproject.org/en/3.0/beta/3.0.0b3/ would give you 3.0.0 beta 3.
>
> What about ^ approach?
>
>
>
> On Thu, May 10, 2018 at 2:31 PM, David Davis 
> wrote:
>
>> +1 to moving to Travis. This fits in with the push-to-PyPI stuff I think.
>>
>> What does the 'beta' come from in the URL and what would the URLs for say
>> a tag like 3.0.1 look like?
>>
>>
>> David
>>
>> On Thu, May 10, 2018 at 8:29 AM, Dennis Kliban 
>> wrote:
>>
>>> On Wed, May 9, 2018 at 1:37 PM, Brian Bouterse 
>>> wrote:
>>>
 We have several problems w/ the Pulp3 docs currently, in decreasing
 order of severity.

 1. We don't have docs per beta, we only have nightly docs [0]. This is
 a problem because during the beta cycle, when we make a backwards
 incompatible change in core's documentation, e.g. merging RQ work, plugins
 can't point there users to the core documentation that pairs wit the
 version of core they tested against. For example if we had core docs for
 beta1, beta2, beta3 ... and pulp_ansible was tested against beta2 only,
 pulp_ansible users should be able to browse the core docs at version beta
 2. We currently can't do that.

 2. The docs builder is uber complicated[1][2], strangely overlays
 content[3] from pulp-ci, and is still shared with pulp2. This is a
 hold-over from the pulp2 docs builders and we never cleaned them up. This
 has cost us some time recently.

 3. The docs builder is on Jenkins which is not community accessible.

 I think we should address this soon, but we need some ideas/discussion
 first.

 I've thought of two ideas: move to Read The Docs or Move to Travis.
 After looking into it, RTD is a non-starter AIUI because RTD won't be able
 to install and start all of the services necessary like Travis can.

 So to move to Travis we could: Enable a conditional publish so that any
 tag which publishes a build to PyPI also publishes the built docs to
 docs.pulpproject.org/ Additionally, we could configure Travis to
 publish with each merged commit as well so we would have continuous docs.


>>> +1
>>>
>>>
 I want to check in on the urls. We need Travis to be able to build it
 from the tag or branch info. Would the url be like:

 https://docs.pulpproject.org/en/3.0/beta/3/

 Or like:

 https://docs.pulpproject.org/en/3.0/beta/3.0.0b3/


>>> I prefer the latter URL.
>>>
>>>
 Feedback and ideas are welcome. After some discussion I can incorporate
 all the ideas into a Redmine ticket.

>>>
>>>
>>>
>>>

 -Brian


 [0]: http://docs.pulpproject.org/en/3.0/nightly/
 [1]: https://github.com/pulp/pulp-ci/blob/master/ci/jjb/jobs/docs
 .yaml#L72-L180
 [2]: https://github.com/pulp/pulp-ci/blob/master/ci/docs-builder.py
 [3]: https://github.com/pulp/pulp-ci/tree/master/ci/docs

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Replicate metadata

[Bryan Kearney]

@Justin, I think that makes sense for Pulp->Pulp. For Matthias, I think
he needs Native->Pulp which would not have a publication.

-- bk

On 05/14/2018 11:42 AM, Matthias Dellweg wrote:
> Mirroring the metedata exactly is also very important for Debian
> Repositories, because of the way the metadata is signed in lieu of the
> whole content. So it would be very beneficial, if this could be
> provided as a 'service' of the pulp core platform somehow.
> 
> Matthias
> 
> On Mon, 14 May 2018 11:31:52 -0400
> Justin Sherrill  wrote:
> 
>>  From my understanding of pulp 3, this would maybe involve the
>> ability to 'import' a publication.  Would that make sense?
>>
>> Justin
>>
>>
>> On 05/09/2018 08:22 AM, Bryan Kearney wrote:
>>> One of the themes I heard yesterday at the Red Hat Summit was around
>>> having a pulp server mirror the upstream RPM repo metadata exactly.
>>> The use case is that two pulp servers are behind a load balancer
>>> mirroring the same repo. The users would like to be able to flip a
>>> yum client acrross the two servers. Running createrepo to make
>>> unique repos causes issues for the clients that appear to be
>>> errors. I assume this pattern would not be unique for other package
>>> clients that cache metadata.
>>>
>>> So, when looking ahead to pulp 3 I would ask that this be taken into
>>> consideration. I can provide more info / use cases if necessary.
>>>
>>> -- bk
>>>
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev  
>>
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev




signature.asc
Description: OpenPGP digital signature
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp CLI MVP User Stories

[Dana Walker]

+1

Dana Walker

Associate Software Engineer

Red Hat




On Tue, May 8, 2018 at 10:31 AM, Jeremy Audet  wrote:

> A configuration file in the user's home dir, right?
>>>
>>
>> Yes, exactly.
>>
>
> Can we make sure to avoid placing configuration files directly in users
> home directories, and instead place them into directories like ~/.config?
> This is in line with the XDG Base Directory Specification
> .
> The spec is pretty straightforward, but Pulp Smash uses pyxdg
>  to avoid mistakes.
> There's two big benefits to doing this:
>
>- Less clutter in home directories.
>- Guidance on what to do with other types of files, such as cached
>files and runtime files.
>
> Projects such as git, htop, lftp, mpd, neovim, tmuxinator, boybo, and more
> do this.
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Replicate metadata

[Matthias Dellweg]

Mirroring the metedata exactly is also very important for Debian
Repositories, because of the way the metadata is signed in lieu of the
whole content. So it would be very beneficial, if this could be
provided as a 'service' of the pulp core platform somehow.

Matthias

On Mon, 14 May 2018 11:31:52 -0400
Justin Sherrill  wrote:

>  From my understanding of pulp 3, this would maybe involve the
> ability to 'import' a publication.  Would that make sense?
> 
> Justin
> 
> 
> On 05/09/2018 08:22 AM, Bryan Kearney wrote:
> > One of the themes I heard yesterday at the Red Hat Summit was around
> > having a pulp server mirror the upstream RPM repo metadata exactly.
> > The use case is that two pulp servers are behind a load balancer
> > mirroring the same repo. The users would like to be able to flip a
> > yum client acrross the two servers. Running createrepo to make
> > unique repos causes issues for the clients that appear to be
> > errors. I assume this pattern would not be unique for other package
> > clients that cache metadata.
> >
> > So, when looking ahead to pulp 3 I would ask that this be taken into
> > consideration. I can provide more info / use cases if necessary.
> >
> > -- bk
> >
> >
> >
> > ___
> > Pulp-dev mailing list
> > Pulp-dev@redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev  
> 


pgpMVLOMzUTHL.pgp
Description: OpenPGP digital signature
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Replicate metadata

[Justin Sherrill]

From my understanding of pulp 3, this would maybe involve the ability 
to 'import' a publication.  Would that make sense?


Justin


On 05/09/2018 08:22 AM, Bryan Kearney wrote:

One of the themes I heard yesterday at the Red Hat Summit was around
having a pulp server mirror the upstream RPM repo metadata exactly. The
use case is that two pulp servers are behind a load balancer mirroring
the same repo. The users would like to be able to flip a yum client
acrross the two servers. Running createrepo to make unique repos causes
issues for the clients that appear to be errors. I assume this pattern
would not be unique for other package clients that cache metadata.

So, when looking ahead to pulp 3 I would ask that this be taken into
consideration. I can provide more info / use cases if necessary.

-- bk



___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Request for feedback about rich dependencies implementation design

[Milan Kovacik]

Folks,

I'd like to ask Your feedback for $Subj, ideally on the
https://pulp.plan.io/issues/3528 issue.
Warning: long read ahead...

Thanks!
milan

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PUP5 -- Adopting the "Common Cure Rights Commitment" for Pulp Core

[Ina Panova]

*understanding




Regards,

Ina Panova
Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."

On Mon, May 14, 2018 at 12:27 PM, Ina Panova  wrote:

> To make a concrete example to prove my understating:
>
> Since pulp_rpm is maintained by core team we could adopt this change,
> meanwhile pulp_deb is beyond our control and we( core team) cannot enforce
> or influence this change.
> Yes?
>
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Tue, May 8, 2018 at 5:55 PM, Brian Bouterse 
> wrote:
>
>> A Pulp Update Proposal (PUP) pull request has been opened by the
>> go-to-lawyer for the Pulp community, Richard Fontana. The PUP is PUP5 [0].
>> I don't want to paraphrase it here, so please read it [0] if you are
>> interested to understand what it does.
>>
>> I am proposing a period of questions/discussion via the list/PR and then
>> a call for a vote according to the process. All questions are welcome,
>> please ask.
>>
>>
>> # Timeline
>>
>> Today - May 18th mailing list and PR discussion
>> May 18th - formally call for a vote which would end 12 calendar days from
>> then May 30th
>> May 30th - Merge or reject
>>
>>
>> # FAQs
>>
>> Is this relicensing Pulp?
>> No. It's still GPLv2. This adopts a procedural enforment approach within
>> the existing license. See @rfontana's response here:
>> https://github.com/pulp/pups/pull/9#issuecomment-384523020
>>
>> Do all prior contributors need to sign off on this change?
>> No, because it's not a relicensing.
>>
>> Does this affect core, plugins, or both?
>> This PR is only scoped to affect the GPLv2 codebases maintained by the
>> core team. Plugins make their own decisions without PUPs. Initially this
>> would be pulp/pulp, and as other GPLv2 repositories are maintained by the
>> core team, it would apply to this in the future as well.
>>
>>
>> [0]: https://github.com/pulp/pups/pull/9/files
>>
>> Thanks,
>> Brian
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] PUP5 -- Adopting the "Common Cure Rights Commitment" for Pulp Core

[Ina Panova]

To make a concrete example to prove my understating:

Since pulp_rpm is maintained by core team we could adopt this change,
meanwhile pulp_deb is beyond our control and we( core team) cannot enforce
or influence this change.
Yes?




Regards,

Ina Panova
Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."

On Tue, May 8, 2018 at 5:55 PM, Brian Bouterse  wrote:

> A Pulp Update Proposal (PUP) pull request has been opened by the
> go-to-lawyer for the Pulp community, Richard Fontana. The PUP is PUP5 [0].
> I don't want to paraphrase it here, so please read it [0] if you are
> interested to understand what it does.
>
> I am proposing a period of questions/discussion via the list/PR and then a
> call for a vote according to the process. All questions are welcome, please
> ask.
>
>
> # Timeline
>
> Today - May 18th mailing list and PR discussion
> May 18th - formally call for a vote which would end 12 calendar days from
> then May 30th
> May 30th - Merge or reject
>
>
> # FAQs
>
> Is this relicensing Pulp?
> No. It's still GPLv2. This adopts a procedural enforment approach within
> the existing license. See @rfontana's response here:
> https://github.com/pulp/pups/pull/9#issuecomment-384523020
>
> Do all prior contributors need to sign off on this change?
> No, because it's not a relicensing.
>
> Does this affect core, plugins, or both?
> This PR is only scoped to affect the GPLv2 codebases maintained by the
> core team. Plugins make their own decisions without PUPs. Initially this
> would be pulp/pulp, and as other GPLv2 repositories are maintained by the
> core team, it would apply to this in the future as well.
>
>
> [0]: https://github.com/pulp/pups/pull/9/files
>
> Thanks,
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev