CPE Weekly: 2019-10-11

2019-10-11 Thread Aoife Moloney
Hi everyone, Welcome to the CPE team weekly project update mail! Background: The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other

Re: FBR : update robosignatory

2019-10-11 Thread Kevin Fenzi
On Fri, Oct 11, 2019 at 05:47:42PM +0200, Aurelien Bompard wrote: > > Patch looks good and you have a plan of action. +1 > > > > Thanks. I've pushed the Ansible change and moved the build from > epel7-infra-stg to epel7-infra, but now I need someone in sysadmin-main to > update the RPM and run

Re: Emergency FBR: Lower concurrency on postgres to help load

2019-10-11 Thread Stephen John Smoogen
On Fri, 11 Oct 2019 at 12:10, Kevin Fenzi wrote: > > On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen wrote: > > Currently the load on db-koji is 140+ and it is slow and > > non-responsive to users. I believe it is the below change which has > > affected this. I would like to lower

Re: Emergency FBR: Lower concurrency on postgres to help load

2019-10-11 Thread Kevin Fenzi
On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen wrote: > Currently the load on db-koji is 140+ and it is slow and > non-responsive to users. I believe it is the below change which has > affected this. I would like to lower down io to 32 to see if that > affects things and tune

Re: FBR : update robosignatory

2019-10-11 Thread Aurelien Bompard
> Patch looks good and you have a plan of action. +1 > Thanks. I've pushed the Ansible change and moved the build from epel7-infra-stg to epel7-infra, but now I need someone in sysadmin-main to update the RPM and run the playbook on autosign01, since I don't have the permissions for that. I'm on

Re: [PATCH] postgresql_server / db-koji01: Adjust a bunch more for performance.

2019-10-11 Thread Kevin Fenzi
On Fri, Oct 11, 2019 at 09:20:59AM -0400, Stephen John Smoogen wrote: > > OK I think we have a problem with the above value. the db servers > seems to be running at a load average of over 140 consistently and we The load has actually always been nearly that high when the db dump is happening. >

Re: FBR : update robosignatory

2019-10-11 Thread Peter Robinson
On Fri, Oct 11, 2019 at 3:52 PM Pierre-Yves Chibon wrote: > > On Fri, Oct 11, 2019 at 03:47:00PM +0100, Peter Robinson wrote: > > On Fri, Oct 11, 2019 at 2:41 PM Aurelien Bompard > > wrote: > > > > > > Hey folks, > > > > > > Last Monday, before the freeze, we updated Robosignatory in prod with a

Re: FBR : update robosignatory

2019-10-11 Thread Pierre-Yves Chibon
On Fri, Oct 11, 2019 at 03:47:00PM +0100, Peter Robinson wrote: > On Fri, Oct 11, 2019 at 2:41 PM Aurelien Bompard > wrote: > > > > Hey folks, > > > > Last Monday, before the freeze, we updated Robosignatory in prod with a few > > new features, some of which could not be tested in staging as

Re: FBR : update robosignatory

2019-10-11 Thread Peter Robinson
On Fri, Oct 11, 2019 at 2:41 PM Aurelien Bompard wrote: > > Hey folks, > > Last Monday, before the freeze, we updated Robosignatory in prod with a few > new features, some of which could not be tested in staging as thoroughly as > we wanted to. As a result, the version currently in prod has

Re: Emergency FBR: Lower concurrency on postgres to help load

2019-10-11 Thread Clement Verna
On Fri, 11 Oct 2019 at 16:07, Pierre-Yves Chibon wrote: > On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen wrote: > > Currently the load on db-koji is 140+ and it is slow and > > non-responsive to users. I believe it is the below change which has > > affected this. I would like to

Re: FBR : update robosignatory

2019-10-11 Thread Stephen John Smoogen
Patch looks good and you have a plan of action. +1 On Fri, 11 Oct 2019 at 09:41, Aurelien Bompard wrote: > > Hey folks, > > Last Monday, before the freeze, we updated Robosignatory in prod with a few > new features, some of which could not be tested in staging as thoroughly as > we wanted to.

Re: Emergency FBR: Lower concurrency on postgres to help load

2019-10-11 Thread Pierre-Yves Chibon
On Fri, Oct 11, 2019 at 09:52:20AM -0400, Stephen John Smoogen wrote: > Currently the load on db-koji is 140+ and it is slow and > non-responsive to users. I believe it is the below change which has > affected this. I would like to lower down io to 32 to see if that > affects things and tune

Re: FBR : update robosignatory

2019-10-11 Thread Pierre-Yves Chibon
On Fri, Oct 11, 2019 at 03:41:28PM +0200, Aurelien Bompard wrote: >Hey folks, >Last Monday, before the freeze, we updated Robosignatory in prod with a >few new features, some of which could not be tested in staging as >thoroughly as we wanted to. As a result, the version currently

Emergency FBR: Lower concurrency on postgres to help load

2019-10-11 Thread Stephen John Smoogen
Currently the load on db-koji is 140+ and it is slow and non-responsive to users. I believe it is the below change which has affected this. I would like to lower down io to 32 to see if that affects things and tune further if needed. I am not doing this until Kevin or Patrick are around as it will

FBR : update robosignatory

2019-10-11 Thread Aurelien Bompard
Hey folks, Last Monday, before the freeze, we updated Robosignatory in prod with a few new features, some of which could not be tested in staging as thoroughly as we wanted to. As a result, the version currently in prod has issues with the CoreOS artifacts. We've worked on that and our tests in

Re: [PATCH] postgresql_server / db-koji01: Adjust a bunch more for performance.

2019-10-11 Thread Stephen John Smoogen
On Thu, 10 Oct 2019 at 12:02, wrote: > > From: Kevin Fenzi > > The current settings cause database dumps to drive the load way up > and make the entire application slow, so we need to adjust. > Using pgtune, these values might well be better. > > shared_buffers + effective_cache_size should =

Re: New mirrorlist server implementation

2019-10-11 Thread Adrian Reber
On Tue, Oct 08, 2019 at 08:38:18AM -0700, Kevin Fenzi wrote: > On Tue, Oct 08, 2019 at 08:42:13AM +0200, Adrian Reber wrote: > > After the Fedora 31 freeze I would like to introduce this new mirrorlist > > server implementation on the proxies. I already verified that I can run > > this mirrorlist