Re: [Pulp-dev] Content Signing for Pulp 3.y -- Use Cases

[Dennis Kliban]

Thank you everyone who participated in the discussion earlier today. I have
moved the content to a google doc[0]. You can find the summary of today's
discussion below the use cases. The full log of the discussion is here[1].

I would like to continue the discussion at 9 AM EST (in your time zone
)
on Thursday, December 5, 2019 in #pulp-meeting on IRC. Your

[0]
https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit#
[1]
https://pulpadmin.fedorapeople.org/triage/pulp-dev/2019/pulp-dev.2019-11-26-14.30.log.html

On Thu, Nov 21, 2019 at 8:27 AM Dennis Kliban  wrote:

> On Thu, Nov 21, 2019 at 7:59 AM Dennis Kliban  wrote:
>
>> I will be hosting a chat meeting to discuss use cases for signing in Pulp
>> 3.y.
>> The meeting details and agenda link are below. Both users and plugin
>> developers are invited. Please join if you're interested!
>>
>> When: Wednesday, November 26 9:30 – 10:30am EST or in your time zone
>> 
>> .
>>
>
> Correction: this will take place on Tuesday. The link is correct.
>
>
>> Where: #pulp-dev on freenode
>> Agenda: https://etherpad.net/p/Pulp_-_Signing_Use_Cases
>>
>> Questions and feedback are also welcome ahead of time.
>>
>> Thanks!
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Memory consumption on RPM sync

[Daniel Alley]

Fabricio, this is great work!

One thing that stands out is that a very large amount of time is being
spent in remove_duplicates(), 65% of the total runtime of the sync.

   - 13% of the total runtime spent on this inside cast()
   
https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/repo_version_utils.py#L31
   - 31% of the total runtime spent in these lines
   
https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/repo_version_utils.py#L37-L38
   - 20% of the total runtime spent on this line
   
https://github.com/pulp/pulpcore/blob/master/pulpcore/plugin/repo_version_utils.py#L42

There's a couple of suggestions on how to improve this code written up here
https://pulp.plan.io/issues/5701

The memory usage seems to be coming from the Django ORM backend.  My guess
is that lines 37 & 38 (linked above) are generating some extremely long
SQL, because it's basically doing OR with a vast number of individual
queries. And I think that is also not great for performance.
https://www.cybertec-postgresql.com/en/avoid-or-for-better-performance/

Another hint that that might be the case is that the memory usage grows and
grows and then collapses all at once.  That would make sense if the SQL
compiler's data structure is growing and growing and then being destroyed
after the query executes.

I would imagine that if we fix our queries, we'll both fix the memory
consumption and improve performance at the same time.

On Tue, Nov 26, 2019 at 11:57 AM Fabricio Aguiar 
wrote:

> Hi everyone,
>
> I've been investigating memory consumption for syncing on RPM plugin - issue
> 5688 , and I want to share what I've
> found so far.
>
> First, it seems that a new process is created when we started to sync, and
> this new process is responsible for the large amount of memory consumption
> observed.
>
> As I shared on some notes here: https://pulp.plan.io/issues/5688
> The problem seems to affect only RHEL, I did not see a large consumption
> of memory for the following repos:
> - CentOS 6: http://mirror.centos.org/centos-6/6.10/os/x86_64/
> - CentOS 7: http://mirror.linux.duke.edu/pub/centos/7/os/x86_64/
> - CentOS 8 AppStrem:
> http://mirror.linux.duke.edu/pub/centos/8/AppStream/x86_64/os/
> - CentOS 8 BaseOS:
> http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/
> - EPEL 7: https://dl.fedoraproject.org/pub/epel/7/x86_64/
>
> As stated on the issue, with RHEL 7 we can observe the problem, for my
> tests, I used the following repo:
> http://cdn.stage.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/
>
> For my tests, I used pulp3-source-fedora30 vagrant box, initially set with
> 4GB, then I increased to 16GB, and it was not enough. I only was able to
> sync RHEL 7 when I increased the memory up to 23GB.
>
> Utilizing py-spy I got those SVG profiles attached, you can find them
> hosted here:
> Sync - https://sendeyo.com/up/d/90a8ae4c8f
> Re-sync - https://sendeyo.com/up/d/4c855bcce3
>
> With the following branch, I was able to take some tracemalloc
>  snapshots (every 30
> seconds):
>
> https://github.com/pulp/pulp_rpm/commit/f3f079010cfe81b7f5cf3ef94c2402b1ccf7d90c
>
> *Tracemalloc for Sync at peak of memory consumption:*
>
> #1: sql/compiler.py:1512: 7825340.4 KiB
> #2: tasks/synchronizing.py:154: 4062651.9 KiB
> #3: stages/declarative_version.py:149: 4062601.9 KiB
> #4: models/repository.py:631: 4062130.0 KiB
> #5: models/repository.py:96: 4062127.7 KiB
> #6: rq/job.py:611: 4060459.2 KiB
> #7: rq/job.py:605: 4058273.8 KiB
> #8: rq/worker.py:822: 4053883.3 KiB
> #9: tasking/worker.py:100: 4053875.9 KiB
> #10: rq/worker.py:684: 4053849.9 KiB
> #11: rq/worker.py:610: 4051645.7 KiB
> #12: plugin/repo_version_utils.py:31: 4028192.9 KiB
> #13: models/base.py:124: 4028191.2 KiB
> #14: fields/related_descriptors.py:401: 4028165.2 KiB
> #15: models/query.py:1242: 4021212.9 KiB
> #16: models/query.py:402: 3995284.5 KiB
> #17: models/query.py:256: 3995284.5 KiB
> #18: models/query.py:55: 3952718.2 KiB
> #19: sql/compiler.py:1133: 3919417.7 KiB
> #20: db/utils.py:96: 3912674.4 KiB
> #21: psycopg2/_json.py:166: 3837534.7 KiB
> #22: json/decoder.py:337: 3837533.2 KiB
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> #23: json/__init__.py:348: 3837533.2 KiB
> return _default_decoder.decode(s)
> #24: json/decoder.py:353: 3837532.7 KiB
> obj, end = self.scan_once(s, idx)
> #25: rq/worker.py:670: 211891.5 KiB
> #26: tasking/worker.py:69: 198267.0 KiB
> #27: rq/worker.py:477: 191261.5 KiB
> #28: cli/cli.py:252: 186110.9 KiB
> #29: click/core.py:555: 185957.8 KiB
> #30: cli/cli.py:78: 104822.7 KiB
> #31: models/query.py:73: 48350.9 KiB
> #32: models/base.py:513: 48350.9 KiB
> #33: sql/compiler.py:405: 38126.1 KiB
> #34: sql/compiler.py:1087: 33355.6 KiB
> #35: click/core.py:956: 29580.3 KiB
> #36: click/core.py:1137: 29498.3 KiB
> #37: models/base.py:430: 29340.9 KiB
> #38: models/query.py:274: 25946.8 KiB
> #39: plugin

Re: [Pulp-dev] Memory consumption on RPM sync

[Dennis Kliban]

The only difference between CentOS and RHEL repos is the Errata. So my
guess is that the memory consumption problem is introduced by how we keep
track of Errata and relationships to packages. Not sure how to fix it
though.

On Tue, Nov 26, 2019 at 1:21 PM Fabricio Aguiar 
wrote:

> I forgot to mention on my initial email, it was not an FYI,
> my intention in bringing these data was getting some discussion on what we
> should do to improve the performance and file some specific issues.
>
> Best regards,
> Fabricio Aguiar
> Software Engineer, Pulp Project
> Red Hat Brazil - Latam 
> +55 11 999652368
>
>
> On Tue, Nov 26, 2019 at 3:03 PM Lubos Mjachky  wrote:
>
>> Great results!
>>
>> I have tried to apply the same procedure like you, as I am currently
>> working on a similar issue (https://pulp.plan.io/issues/5689), but there
>> is one problem. I could not run performance tests with additional profilers
>> because my workstation run out of memory after a couple of minutes. I have
>> got only 16GB of RAM. Also, as you noticed, the synchronization takes a
>> huge amount of time. I think that we should not run those tests and
>> profilers on our personal machines.
>>
>> For my tests, I used pulp3-source-fedora30 vagrant box, initially set
>>> with 4GB, then I increased to 16GB, and it was not enough. I only was able
>>> to sync RHEL 7 when I increased the memory up to 23GB.
>>>
>>
>> What about using some allocated openstack instances? Do we have access to
>> some powerful machines as a team? I have managed to spin up some instances
>> but when I tried to run pulp services I always stumbled upon some issues.
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Memory consumption on RPM sync

[Fabricio Aguiar]

I forgot to mention on my initial email, it was not an FYI,
my intention in bringing these data was getting some discussion on what we
should do to improve the performance and file some specific issues.

Best regards,
Fabricio Aguiar
Software Engineer, Pulp Project
Red Hat Brazil - Latam 
+55 11 999652368


On Tue, Nov 26, 2019 at 3:03 PM Lubos Mjachky  wrote:

> Great results!
>
> I have tried to apply the same procedure like you, as I am currently
> working on a similar issue (https://pulp.plan.io/issues/5689), but there
> is one problem. I could not run performance tests with additional profilers
> because my workstation run out of memory after a couple of minutes. I have
> got only 16GB of RAM. Also, as you noticed, the synchronization takes a
> huge amount of time. I think that we should not run those tests and
> profilers on our personal machines.
>
> For my tests, I used pulp3-source-fedora30 vagrant box, initially set with
>> 4GB, then I increased to 16GB, and it was not enough. I only was able to
>> sync RHEL 7 when I increased the memory up to 23GB.
>>
>
> What about using some allocated openstack instances? Do we have access to
> some powerful machines as a team? I have managed to spin up some instances
> but when I tried to run pulp services I always stumbled upon some issues.
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Memory consumption on RPM sync

[Lubos Mjachky]

Great results!

I have tried to apply the same procedure like you, as I am currently
working on a similar issue (https://pulp.plan.io/issues/5689), but there is
one problem. I could not run performance tests with additional profilers
because my workstation run out of memory after a couple of minutes. I have
got only 16GB of RAM. Also, as you noticed, the synchronization takes a
huge amount of time. I think that we should not run those tests and
profilers on our personal machines.

For my tests, I used pulp3-source-fedora30 vagrant box, initially set with
> 4GB, then I increased to 16GB, and it was not enough. I only was able to
> sync RHEL 7 when I increased the memory up to 23GB.
>

What about using some allocated openstack instances? Do we have access to
some powerful machines as a team? I have managed to spin up some instances
but when I tried to run pulp services I always stumbled upon some issues.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3.0.0 GA Date -- Dec 12

[Brian Bouterse]

title correction: Dec 12th

On Tue, Nov 26, 2019 at 11:08 AM Brian Bouterse  wrote:

> 
>
> We're planning one final release candidate 3.0.0rc9 for Dec 3rd, and a
> generally available release with improved documentation on Dec 12th. This
> will include pulp_rpm, pulp_container, and pulp_file as well. This is a
> delay from the initial GA release date of Dec 3rd.
>
> Read the blog post for more details on the schedule change.
> https://pulpproject.org/2019/11/25/pulp-3-GA-update/
>
> Any questions or concerns are welcome! Please share.
>
> -Brian
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp 3.0.0 GA Date -- Dec 13

[Brian Bouterse]

We're planning one final release candidate 3.0.0rc9 for Dec 3rd, and a
generally available release with improved documentation on Dec 12th. This
will include pulp_rpm, pulp_container, and pulp_file as well. This is a
delay from the initial GA release date of Dec 3rd.

Read the blog post for more details on the schedule change.
https://pulpproject.org/2019/11/25/pulp-3-GA-update/

Any questions or concerns are welcome! Please share.

-Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Cherry pick process

[David Davis]

I've filed an issue here: https://pulp.plan.io/issues/5808. Feedback on the
issue is welcome.

David


On Tue, Nov 26, 2019 at 8:51 AM Brian Bouterse  wrote:

> Filing more issues sounds good. Email notifications sound good. Can we
> start with the bare minimum so we can get something basic going soon?
>
> On Tue, Nov 26, 2019 at 7:55 AM David Davis  wrote:
>
>> Yes but it's complicated. Travis does have a setting for email
>> notifications[0]. However, I don't think you can configure it specifically
>> to notify on failed cron jobs and we'd have to expose this setting via the
>> plugin_template. There's also the problem that Travis will notify you for
>> forks that are running in Travis[1] although it looks like you can work
>> around this by encrypting a notification setting such as an email[2].
>>
>> If all of that sounds acceptable, I can open a separate issue.
>>
>> [0]
>> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
>> [1] https://github.com/travis-ci/travis-ci/issues/329
>> [2] https://github.com/travis-ci/travis-ci/issues/5063
>>
>> David
>>
>>
>> On Tue, Nov 26, 2019 at 4:07 AM Tatiana Tereshchenko 
>> wrote:
>>
>>> +1 to the automation of the process
>>>
>>> Can we configure Travis to send an e-mail if the job fails and not rely
>>> on human checking it every time?
>>>
>>> Tanya
>>>
>>> On Mon, Nov 25, 2019 at 6:14 PM David Davis 
>>> wrote:
>>>
 I opened an issue to outline the design we're discussing:

 https://pulp.plan.io/issues/5795

 David


 On Mon, Nov 25, 2019 at 11:33 AM Brian Bouterse 
 wrote:

>
>
> On Fri, Nov 22, 2019 at 3:15 PM David Davis 
> wrote:
>
>> I think there are a couple caveats with this approach. One thing is I
>> think Travis will need to do is remove the "Needs Cherry Pick" label 
>> since
>> it's stateless and it won't be able to keep track of what's already been
>> cherry picked.
>>
>> +1, then it's stateless
>
> Also, there's the question of what to do when the cherry pick fails.
>> Travis could fail the job (this would probably be the easiest thing). I
>> imagine the release engineer will need to monitor the job and manually
>> merge any failed cherry picks if the job fails.
>>
> +1 to Travis failing the job, leaving the labels unchanged, and having
> a human look at it
>
>>
>> David
>>
>>
>> On Fri, Nov 22, 2019 at 10:35 AM Brian Bouterse 
>> wrote:
>>
>>> I was thinking we could make this happen quicker if we simplify it a
>>> bit, then we can add on more later. What if we:
>>>
>>> 1) Have humans apply the "Cherry Pick" label and not integrate with
>>> Redmine
>>> 2) Have Travis run once a day to create the PR of all merged PRs
>>> with the cherry pick label that have not yet been cherry picked.
>>> 3) Have a human merge the "cherry pick" PR daily
>>>
>>> We could make ^ run as a cron job on Travis, and then this would be
>>> available to all plugins who configure it to be enabled.
>>>
>>>
>>> On Fri, Nov 22, 2019 at 8:46 AM Ina Panova 
>>> wrote:
>>>
 +1 to automate cherry-picking. I like that depending on the label
 set, commit will be cherry-picked or not.


 
 Regards,

 Ina Panova
 Senior Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Thu, Nov 21, 2019 at 10:43 PM Brian Bouterse <
 bmbou...@redhat.com> wrote:

> After cherry picking a whole bunch today, I am very in favor of
> automation to do it.
>
> I had hoped we could avoid writing/maintaining a service, but I
> looked around, and I don't see a hosted "cherry-pick bot" like 
> dependabot
> for example. Can we set it up to run next to pulpbot? Or maybe we run 
> it as
> a cron job on Travis (see below). Overall though big +1.
>
>
> On Thu, Nov 21, 2019 at 12:07 PM Mike DePaulo <
> mikedep...@redhat.com> wrote:
>
>>
>> On Thu, Nov 21, 2019 at 9:01 AM David Davis <
>> davidda...@redhat.com> wrote:
>>
>>> On our scrum with Katello yesterday, they raised an issue that
>>> since they are developing against our release branches, they need 
>>> bug
>>> fixes to be cherry picked to release branches asap. Currently this 
>>> is up to
>>> release leads, but we've seen this week that this is problematic as 
>>> two of
>>> our release leads have been out/unavailable.
>>>
>>> One suggestion is to automate the cherry picking process. I
>>> think we could develop a PR bot similar to the one Foretello 
>>> us

Re: [Pulp-dev] Cherry pick process

[Brian Bouterse]

Filing more issues sounds good. Email notifications sound good. Can we
start with the bare minimum so we can get something basic going soon?

On Tue, Nov 26, 2019 at 7:55 AM David Davis  wrote:

> Yes but it's complicated. Travis does have a setting for email
> notifications[0]. However, I don't think you can configure it specifically
> to notify on failed cron jobs and we'd have to expose this setting via the
> plugin_template. There's also the problem that Travis will notify you for
> forks that are running in Travis[1] although it looks like you can work
> around this by encrypting a notification setting such as an email[2].
>
> If all of that sounds acceptable, I can open a separate issue.
>
> [0]
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> [1] https://github.com/travis-ci/travis-ci/issues/329
> [2] https://github.com/travis-ci/travis-ci/issues/5063
>
> David
>
>
> On Tue, Nov 26, 2019 at 4:07 AM Tatiana Tereshchenko 
> wrote:
>
>> +1 to the automation of the process
>>
>> Can we configure Travis to send an e-mail if the job fails and not rely
>> on human checking it every time?
>>
>> Tanya
>>
>> On Mon, Nov 25, 2019 at 6:14 PM David Davis 
>> wrote:
>>
>>> I opened an issue to outline the design we're discussing:
>>>
>>> https://pulp.plan.io/issues/5795
>>>
>>> David
>>>
>>>
>>> On Mon, Nov 25, 2019 at 11:33 AM Brian Bouterse 
>>> wrote:
>>>


 On Fri, Nov 22, 2019 at 3:15 PM David Davis 
 wrote:

> I think there are a couple caveats with this approach. One thing is I
> think Travis will need to do is remove the "Needs Cherry Pick" label since
> it's stateless and it won't be able to keep track of what's already been
> cherry picked.
>
> +1, then it's stateless

 Also, there's the question of what to do when the cherry pick fails.
> Travis could fail the job (this would probably be the easiest thing). I
> imagine the release engineer will need to monitor the job and manually
> merge any failed cherry picks if the job fails.
>
 +1 to Travis failing the job, leaving the labels unchanged, and having
 a human look at it

>
> David
>
>
> On Fri, Nov 22, 2019 at 10:35 AM Brian Bouterse 
> wrote:
>
>> I was thinking we could make this happen quicker if we simplify it a
>> bit, then we can add on more later. What if we:
>>
>> 1) Have humans apply the "Cherry Pick" label and not integrate with
>> Redmine
>> 2) Have Travis run once a day to create the PR of all merged PRs with
>> the cherry pick label that have not yet been cherry picked.
>> 3) Have a human merge the "cherry pick" PR daily
>>
>> We could make ^ run as a cron job on Travis, and then this would be
>> available to all plugins who configure it to be enabled.
>>
>>
>> On Fri, Nov 22, 2019 at 8:46 AM Ina Panova 
>> wrote:
>>
>>> +1 to automate cherry-picking. I like that depending on the label
>>> set, commit will be cherry-picked or not.
>>>
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Thu, Nov 21, 2019 at 10:43 PM Brian Bouterse 
>>> wrote:
>>>
 After cherry picking a whole bunch today, I am very in favor of
 automation to do it.

 I had hoped we could avoid writing/maintaining a service, but I
 looked around, and I don't see a hosted "cherry-pick bot" like 
 dependabot
 for example. Can we set it up to run next to pulpbot? Or maybe we run 
 it as
 a cron job on Travis (see below). Overall though big +1.


 On Thu, Nov 21, 2019 at 12:07 PM Mike DePaulo <
 mikedep...@redhat.com> wrote:

>
> On Thu, Nov 21, 2019 at 9:01 AM David Davis 
> wrote:
>
>> On our scrum with Katello yesterday, they raised an issue that
>> since they are developing against our release branches, they need bug
>> fixes to be cherry picked to release branches asap. Currently this 
>> is up to
>> release leads, but we've seen this week that this is problematic as 
>> two of
>> our release leads have been out/unavailable.
>>
>> One suggestion is to automate the cherry picking process. I think
>> we could develop a PR bot similar to the one Foretello uses[1] (see 
>> this
>> PR[2] as an example). I think the basic workflow for this bot would 
>> be:
>>
>> - If a PR is created with no issue attached or [noissue], it
>> would loudly warn that no issue is attached
>> - If the PR has a redmine issue attached, it would:
>>   - Attach the PR to the redmine issue a

Re: [Pulp-dev] Cherry pick process

[David Davis]

Yes but it's complicated. Travis does have a setting for email
notifications[0]. However, I don't think you can configure it specifically
to notify on failed cron jobs and we'd have to expose this setting via the
plugin_template. There's also the problem that Travis will notify you for
forks that are running in Travis[1] although it looks like you can work
around this by encrypting a notification setting such as an email[2].

If all of that sounds acceptable, I can open a separate issue.

[0]
https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
[1] https://github.com/travis-ci/travis-ci/issues/329
[2] https://github.com/travis-ci/travis-ci/issues/5063

David


On Tue, Nov 26, 2019 at 4:07 AM Tatiana Tereshchenko 
wrote:

> +1 to the automation of the process
>
> Can we configure Travis to send an e-mail if the job fails and not rely on
> human checking it every time?
>
> Tanya
>
> On Mon, Nov 25, 2019 at 6:14 PM David Davis  wrote:
>
>> I opened an issue to outline the design we're discussing:
>>
>> https://pulp.plan.io/issues/5795
>>
>> David
>>
>>
>> On Mon, Nov 25, 2019 at 11:33 AM Brian Bouterse 
>> wrote:
>>
>>>
>>>
>>> On Fri, Nov 22, 2019 at 3:15 PM David Davis 
>>> wrote:
>>>
 I think there are a couple caveats with this approach. One thing is I
 think Travis will need to do is remove the "Needs Cherry Pick" label since
 it's stateless and it won't be able to keep track of what's already been
 cherry picked.

 +1, then it's stateless
>>>
>>> Also, there's the question of what to do when the cherry pick fails.
 Travis could fail the job (this would probably be the easiest thing). I
 imagine the release engineer will need to monitor the job and manually
 merge any failed cherry picks if the job fails.

>>> +1 to Travis failing the job, leaving the labels unchanged, and having a
>>> human look at it
>>>

 David


 On Fri, Nov 22, 2019 at 10:35 AM Brian Bouterse 
 wrote:

> I was thinking we could make this happen quicker if we simplify it a
> bit, then we can add on more later. What if we:
>
> 1) Have humans apply the "Cherry Pick" label and not integrate with
> Redmine
> 2) Have Travis run once a day to create the PR of all merged PRs with
> the cherry pick label that have not yet been cherry picked.
> 3) Have a human merge the "cherry pick" PR daily
>
> We could make ^ run as a cron job on Travis, and then this would be
> available to all plugins who configure it to be enabled.
>
>
> On Fri, Nov 22, 2019 at 8:46 AM Ina Panova  wrote:
>
>> +1 to automate cherry-picking. I like that depending on the label
>> set, commit will be cherry-picked or not.
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Thu, Nov 21, 2019 at 10:43 PM Brian Bouterse 
>> wrote:
>>
>>> After cherry picking a whole bunch today, I am very in favor of
>>> automation to do it.
>>>
>>> I had hoped we could avoid writing/maintaining a service, but I
>>> looked around, and I don't see a hosted "cherry-pick bot" like 
>>> dependabot
>>> for example. Can we set it up to run next to pulpbot? Or maybe we run 
>>> it as
>>> a cron job on Travis (see below). Overall though big +1.
>>>
>>>
>>> On Thu, Nov 21, 2019 at 12:07 PM Mike DePaulo 
>>> wrote:
>>>

 On Thu, Nov 21, 2019 at 9:01 AM David Davis 
 wrote:

> On our scrum with Katello yesterday, they raised an issue that
> since they are developing against our release branches, they need bug
> fixes to be cherry picked to release branches asap. Currently this is 
> up to
> release leads, but we've seen this week that this is problematic as 
> two of
> our release leads have been out/unavailable.
>
> One suggestion is to automate the cherry picking process. I think
> we could develop a PR bot similar to the one Foretello uses[1] (see 
> this
> PR[2] as an example). I think the basic workflow for this bot would 
> be:
>
> - If a PR is created with no issue attached or [noissue], it would
> loudly warn that no issue is attached
> - If the PR has a redmine issue attached, it would:
>   - Attach the PR to the redmine issue and set the status to
> POST[3]
>   - Set the PR labels depending on the issue type. One of the
> labels would be 'Needs Cherrypick' if the issue type is a bug. This 
> label
> can be removed before merge for things we don't want cherry picked.
> - When any PR is merged with 'Needs Cherrypick', it could either
> automati

Re: [Pulp-dev] Cherry pick process

[Tatiana Tereshchenko]

+1 to the automation of the process

Can we configure Travis to send an e-mail if the job fails and not rely on
human checking it every time?

Tanya

On Mon, Nov 25, 2019 at 6:14 PM David Davis  wrote:

> I opened an issue to outline the design we're discussing:
>
> https://pulp.plan.io/issues/5795
>
> David
>
>
> On Mon, Nov 25, 2019 at 11:33 AM Brian Bouterse 
> wrote:
>
>>
>>
>> On Fri, Nov 22, 2019 at 3:15 PM David Davis 
>> wrote:
>>
>>> I think there are a couple caveats with this approach. One thing is I
>>> think Travis will need to do is remove the "Needs Cherry Pick" label since
>>> it's stateless and it won't be able to keep track of what's already been
>>> cherry picked.
>>>
>>> +1, then it's stateless
>>
>> Also, there's the question of what to do when the cherry pick fails.
>>> Travis could fail the job (this would probably be the easiest thing). I
>>> imagine the release engineer will need to monitor the job and manually
>>> merge any failed cherry picks if the job fails.
>>>
>> +1 to Travis failing the job, leaving the labels unchanged, and having a
>> human look at it
>>
>>>
>>> David
>>>
>>>
>>> On Fri, Nov 22, 2019 at 10:35 AM Brian Bouterse 
>>> wrote:
>>>
 I was thinking we could make this happen quicker if we simplify it a
 bit, then we can add on more later. What if we:

 1) Have humans apply the "Cherry Pick" label and not integrate with
 Redmine
 2) Have Travis run once a day to create the PR of all merged PRs with
 the cherry pick label that have not yet been cherry picked.
 3) Have a human merge the "cherry pick" PR daily

 We could make ^ run as a cron job on Travis, and then this would be
 available to all plugins who configure it to be enabled.


 On Fri, Nov 22, 2019 at 8:46 AM Ina Panova  wrote:

> +1 to automate cherry-picking. I like that depending on the label set,
> commit will be cherry-picked or not.
>
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Thu, Nov 21, 2019 at 10:43 PM Brian Bouterse 
> wrote:
>
>> After cherry picking a whole bunch today, I am very in favor of
>> automation to do it.
>>
>> I had hoped we could avoid writing/maintaining a service, but I
>> looked around, and I don't see a hosted "cherry-pick bot" like dependabot
>> for example. Can we set it up to run next to pulpbot? Or maybe we run it 
>> as
>> a cron job on Travis (see below). Overall though big +1.
>>
>>
>> On Thu, Nov 21, 2019 at 12:07 PM Mike DePaulo 
>> wrote:
>>
>>>
>>> On Thu, Nov 21, 2019 at 9:01 AM David Davis 
>>> wrote:
>>>
 On our scrum with Katello yesterday, they raised an issue that
 since they are developing against our release branches, they need bug
 fixes to be cherry picked to release branches asap. Currently this is 
 up to
 release leads, but we've seen this week that this is problematic as 
 two of
 our release leads have been out/unavailable.

 One suggestion is to automate the cherry picking process. I think
 we could develop a PR bot similar to the one Foretello uses[1] (see 
 this
 PR[2] as an example). I think the basic workflow for this bot would be:

 - If a PR is created with no issue attached or [noissue], it would
 loudly warn that no issue is attached
 - If the PR has a redmine issue attached, it would:
   - Attach the PR to the redmine issue and set the status to POST[3]
   - Set the PR labels depending on the issue type. One of the
 labels would be 'Needs Cherrypick' if the issue type is a bug. This 
 label
 can be removed before merge for things we don't want cherry picked.
 - When any PR is merged with 'Needs Cherrypick', it could either
 automatically open a cherrypick PR or actually do the cherrypick 
 (falling
 back to a PR if the merge fails due to conflicts).

>>> I think it's good to put the cherry-pick back through the PR process
>> so having it open a PR would be good so that the tests run. Maybe it 
>> should
>> run daily though so we don't increase the travis load and we can test a
>> group of cherry picks together? Note travis could maybe run this as a 
>> cron
>> job and run it there instead?
>>
>>
 Thoughts?

 [0]
 https://pulpproject.org/2019/11/04/pulp-3-GA-update/#cherry-picking
 [1] https://github.com/theforeman/prprocessor
 [2] https://github.com/Katello/katello/pull/8441
 [3] https://pulp.plan.io/issues/4365

 David
 ___

>>>
>>