Re: Freeze break: db01 tuning

[Ralph Bean]

On Tue, Mar 15, 2016 at 09:21:52AM -0600, Kevin Fenzi wrote:
> Greetings. 
> 
> I was going to wait until after freeze to mess with the other
> postgresql hosts, but then mattdm pointed out this morning that
> hyperkitty is really slow currently. I'm not sure this will help, but
> it really make db-koji01 a great deal happier and it should be pretty
> easy to revert if needed. 
> 
> So, I'd like to make the same changes that made db-koji01 happy to
> db01:

+1!


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: do not redirect staging websites

[Robert Mayr]

2016-03-15 16:19 GMT+01:00 Ralph Bean :

> On Mon, Mar 14, 2016 at 12:22:29PM +0100, Robert Mayr wrote:
> > Ok this is for staging websites, but it's a change in ansible and we
> change
> > some pre_tasks, so I ask for +1s to be sure.
> >
> > Staging websites should not redirect prerelease pages, but show them
> > normally (while production must still redirect them to the main page). We
> > are just before readiness and should at least test them out.
> > I'd comment out the rules for staging as in the diff below, if we need to
> > drop also the config files on the proxies, I attach a diff also for
> > playbooks/groups/proxies.yml.
> > Thanks.
>
> +1 to the idea.  However, I think I found a bug:
>
> > diff --git a/playbooks/groups/proxies.yml b/playbooks/groups/proxies.yml
> > index 90d5001..1b46198 100644
> > --- a/playbooks/groups/proxies.yml
> > +++ b/playbooks/groups/proxies.yml
> > @@ -74,17 +74,33 @@
> ># When we have a prerelease we also need to drop the
> ># config files.
> >
> > -  - name: Remove prerelease-to-final-spins
> > -file: path=/etc/httpd/conf.d/
> spins.fedoraproject.org/prerelease-to-final-spins.conf state=absent
> > +  - name: Remove prerelease-to-final-spins-1
> > +file: path=/etc/httpd/conf.d/
> spins.fedoraproject.org/prerelease-to-final-spins-1.conf state=absent
> > +when: env == 'staging'
>
> Look in /etc/httpd/conf.d/blah.fp.o/ on the proxy01.stg node and see
> that the file name you're trying to remove here has the role name
> appended to it.
>
> So, it's not
>
> prerelease-to-final-spins.conf
>
> but instead is:
>
> prerelease-to-final-spins-redirectmatch.conf
>
> Longer term, it is kind of annoying that when we comment out one of
> those roles, we have to go back and add pre_tasks to clean up.
> Perhaps we should add a state=absent argument to the role itself so
> that it knows how to clean itself up.  (Let's save that until after
> freeze.)
>
> ___
> infrastructure mailing list
> infrastructure@lists.fedoraproject.org
>
> http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org
>
>

Thank you, fixed and pushed.

-- 
Robert Mayr
(robyduck)
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: db01 tuning

[Peter Robinson]

On Tue, Mar 15, 2016 at 4:01 PM, Kevin Fenzi  wrote:
> On Tue, 15 Mar 2016 15:32:37 +
> Peter Robinson  wrote:
>
>> +1 (and can we do the other koji instances that are in ansible too -
>> IE arm and s390 if they weren't done last time?)
>
> We could, but I would need to adjust for their memory/etc...
>
> can look at that, or we could just wait until after freeze for them?

TBH happy to wait post freeze, they can wait, and we can the do PPC
there too all at once. I've added it to my reminder list.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: db01 tuning

[Kevin Fenzi]

On Tue, 15 Mar 2016 15:32:37 +
Peter Robinson  wrote:

> +1 (and can we do the other koji instances that are in ansible too -
> IE arm and s390 if they weren't done last time?)

We could, but I would need to adjust for their memory/etc... 

can look at that, or we could just wait until after freeze for them?

kevin


pgpgGzbzLly7x.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: db01 tuning

[Peter Robinson]

> I was going to wait until after freeze to mess with the other
> postgresql hosts, but then mattdm pointed out this morning that
> hyperkitty is really slow currently. I'm not sure this will help, but
> it really make db-koji01 a great deal happier and it should be pretty
> easy to revert if needed.
>
> So, I'd like to make the same changes that made db-koji01 happy to
> db01:
>
> diff --git a/roles/postgresql_server/templates/postgresql.conf 
> b/roles/postgresql_server/templates/postgresql.conf
> index 4a8bdfb..de9f1f0 100644
> --- a/roles/postgresql_server/templates/postgresql.conf
> +++ b/roles/postgresql_server/templates/postgresql.conf
> @@ -124,6 +124,9 @@ shared_buffers = {{ shared_buffers }}   # min 128kB or 
> max_connections*16kB
>  {% if ansible_hostname.startswith("db-koji01") %}
>  work_mem = 4MB  # min 64kB
>  maintenance_work_mem = 16MB # min 1MB
> +{% elif ansible_hostname.startswith("db01") %}
> +work_mem = 4MB  # min 64kB
> +maintenance_work_mem = 1024MB # min 1MB
>  {% else %}
>  work_mem = 2MB  # min 64kB
>  maintenance_work_mem = 1024MB   # min 1MB
> @@ -222,8 +225,10 @@ random_page_cost = 3.0  # same scale as 
> above
>  #cpu_operator_cost = 0.0025 # same scale as above
>  {% if ansible_hostname.startswith("db-koji01") %}
>  effective_cache_size = 24GB
> +{% elif ansible_hostname.startswith("db01") %}
> +effective_cache_size = 12GB
>  {% else %}
> -effective_cache_size = 5GB
> +effective_cache_size = 5GB
>  {% endif %}
>
>  # - Genetic Query Optimizer -
>
> +1s?


+1 (and can we do the other koji instances that are in ansible too -
IE arm and s390 if they weren't done last time?)
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Freeze break: db01 tuning

[Kevin Fenzi]

Greetings. 

I was going to wait until after freeze to mess with the other
postgresql hosts, but then mattdm pointed out this morning that
hyperkitty is really slow currently. I'm not sure this will help, but
it really make db-koji01 a great deal happier and it should be pretty
easy to revert if needed. 

So, I'd like to make the same changes that made db-koji01 happy to
db01:

diff --git a/roles/postgresql_server/templates/postgresql.conf 
b/roles/postgresql_server/templates/postgresql.conf
index 4a8bdfb..de9f1f0 100644
--- a/roles/postgresql_server/templates/postgresql.conf
+++ b/roles/postgresql_server/templates/postgresql.conf
@@ -124,6 +124,9 @@ shared_buffers = {{ shared_buffers }}   # min 128kB or 
max_connections*16kB
 {% if ansible_hostname.startswith("db-koji01") %}
 work_mem = 4MB  # min 64kB
 maintenance_work_mem = 16MB # min 1MB
+{% elif ansible_hostname.startswith("db01") %}
+work_mem = 4MB  # min 64kB
+maintenance_work_mem = 1024MB # min 1MB
 {% else %}
 work_mem = 2MB  # min 64kB
 maintenance_work_mem = 1024MB   # min 1MB
@@ -222,8 +225,10 @@ random_page_cost = 3.0  # same scale as 
above
 #cpu_operator_cost = 0.0025 # same scale as above   
 {% if ansible_hostname.startswith("db-koji01") %}
 effective_cache_size = 24GB
+{% elif ansible_hostname.startswith("db01") %}
+effective_cache_size = 12GB
 {% else %}
-effective_cache_size = 5GB  
+effective_cache_size = 5GB
 {% endif %}
 
 # - Genetic Query Optimizer -

+1s?

kevin


pgplmWQjgNrZJ.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: do not redirect staging websites

[Ralph Bean]

On Mon, Mar 14, 2016 at 12:22:29PM +0100, Robert Mayr wrote:
> Ok this is for staging websites, but it's a change in ansible and we change
> some pre_tasks, so I ask for +1s to be sure.
> 
> Staging websites should not redirect prerelease pages, but show them
> normally (while production must still redirect them to the main page). We
> are just before readiness and should at least test them out.
> I'd comment out the rules for staging as in the diff below, if we need to
> drop also the config files on the proxies, I attach a diff also for
> playbooks/groups/proxies.yml.
> Thanks.

+1 to the idea.  However, I think I found a bug:

> diff --git a/playbooks/groups/proxies.yml b/playbooks/groups/proxies.yml
> index 90d5001..1b46198 100644
> --- a/playbooks/groups/proxies.yml
> +++ b/playbooks/groups/proxies.yml
> @@ -74,17 +74,33 @@
># When we have a prerelease we also need to drop the
># config files.
> 
> -  - name: Remove prerelease-to-final-spins
> -file: 
> path=/etc/httpd/conf.d/spins.fedoraproject.org/prerelease-to-final-spins.conf 
> state=absent
> +  - name: Remove prerelease-to-final-spins-1
> +file: 
> path=/etc/httpd/conf.d/spins.fedoraproject.org/prerelease-to-final-spins-1.conf
>  state=absent
> +when: env == 'staging'

Look in /etc/httpd/conf.d/blah.fp.o/ on the proxy01.stg node and see
that the file name you're trying to remove here has the role name
appended to it.

So, it's not

prerelease-to-final-spins.conf

but instead is:

prerelease-to-final-spins-redirectmatch.conf

Longer term, it is kind of annoying that when we comment out one of
those roles, we have to go back and add pre_tasks to clean up.
Perhaps we should add a state=absent argument to the role itself so
that it knows how to clean itself up.  (Let's save that until after
freeze.)


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: db-koji01 tuning

[Kevin Fenzi]

On Tue, 15 Mar 2016 13:35:59 +
Peter Robinson  wrote:

> > -work_mem = 2MB  # min 64kB
> > -maintenance_work_mem = 1024MB   # min 1MB
> > +{% if ansible_hostname.startswith("db-koji01") %}
> > +work_mem = 4MB  # min 64kB
> > +maintenance_work_mem = 16MB # min 1MB  
> 
> Was it intended to drop maintenance_work_mem from 1024MB to 16MB?

Perhaps not. I guess we can look and see if there's any problems with
autovacuum and reincrease it if needed. 

Good eye. ;) 

kevin



pgppAKsyAMNRm.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

can we add a link to hyperkitty archives at top of pipermail pages?

[Matthew Miller]

Pages like https://lists.fedoraproject.org/pipermail/ambassadors/

Part of the problem is that the More Info link there goes to
https://admin.fedoraproject.org/mailman/listinfo/ambassadors rather
than 
https://lists.fedoraproject.org/admin/lists/ambassadors.lists.fedoraproject.org/

but it'd also be nice to have a direct click-through to
https://lists.fedoraproject.org/archives/list/ambassad...@lists.fedoraproject.org/
so it doesn't just look like all our lists died in November 2015.

-- 
Matthew Miller

Fedora Project Leader
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: db-koji01 tuning

[Peter Robinson]

> -work_mem = 2MB  # min 64kB
> -maintenance_work_mem = 1024MB   # min 1MB
> +{% if ansible_hostname.startswith("db-koji01") %}
> +work_mem = 4MB  # min 64kB
> +maintenance_work_mem = 16MB # min 1MB

Was it intended to drop maintenance_work_mem from 1024MB to 16MB?
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Setting up development environment for Fedora Web Portal

[Dimuthu Lakmal]

Thank you Vit. I'll contact them :)
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: COPR builds/services for other distros?

[Miroslav Suchý]

Dne 14.3.2016 v 23:58 Patrick Uiterwijk napsal(a):
> I would suggest you to look at our Request For Resource procedure.

OK. I created:
https://fedorahosted.org/fedora-infrastructure/ticket/5166

> - How well would at least the frontend and package download server work behind
>   a load balanced/HA setup (so that at least the directly user-facing 
> components
>   can be made highly available, the backend doesn't need to be HA)
> - What kind of support are we aiming for here? Fully supported core infra 
> app, or
>   still somewhat in the middle?

As we previously discussed, it will be mixed. Backend with builders can stay in 
Fedora Cloud and we will declare that
builders will have different SLA and it can take longer to fix it. Critical 
part is frontend and package repositories.
Dist-git and keygen can stay in cloud as well, as backend is the only one who 
use it.
So fully supported should be just frontend and yum/dnf repositories.

> - Are components still "randomly" crashing? I haven't had to restart services
>   a lot recently, so I think it's in a lot better state than previously?

:)

It never randomly crashed. We always take care to introduce very well hidden 
bugs and we are proud of it. And we always
rejected ideas for just easy-to-solve bugs.

*nod* recently the situation is quite stable. But Copr is still under heavy 
development so from time to time there is
always some bug. But the services itself are pretty stable now.

> - What amount of disk space are you currently using, and what are you 
> expecting
>   to be using over the short to medium term (next year or so, just 
> approximate is
>   fine)?

I put it in RFR ticket.

> - Who is/are going to be the main point of contact? If the previous answer was
>   "core infra app", who do we page at 3AM when everything falls over and we 
> don't
>   understand any of it? (not likely since I now know quite a bit about it, 
> but still
>   I'd like some names for fallback).

I put it in ticket.
Originally it was me and Valentin. Valentin left RH in December and does not 
have time for Fedora any more (I will
likely propose to remove him from sysadmin-cloud in few months). Clime is the 
replacement, and I'm slowly introducing
him into the setup and fedora infrastructure design and tools.
I have two other guys familar with the setup and code base. Adam has already 
root access to prod instance, Jakub not
yet, but he had access to dev instance for long time and in fact he is now 
mature enough that I would propose him to get
access to production instance as well.
So I have 4 contacts and at least one of them should be available at 3AM.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Setting up development environment for Fedora Web Portal

[Vít Ondruch]

You probably want to contact developer portal folks at their ML:

developer-por...@lists.fedoraproject.org

But I adding some of them on BCC, so they might get back to you.


Vít



Dne 15.3.2016 v 08:40 Dimuthu Lakmal napsal(a):
> Hi, 
> I was setting up development environment for fedora web-portal
> I followed guidlines provided in 
> https://github.com/developer-portal/website/blob/master/DEVELOPMENT.md and 
> used Vagrant method. When i execute jekyll serve --force_polling -H 0.0.0.0 
> command, it says there are some missing files in content folder. But actually 
> files are in there. any solution?
> ___
> infrastructure mailing list
> infrastructure@lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Setting up development environment for Fedora Web Portal

[Dimuthu Lakmal]

Hi, 
I was setting up development environment for fedora web-portal
I followed guidlines provided in 
https://github.com/developer-portal/website/blob/master/DEVELOPMENT.md and used 
Vagrant method. When i execute jekyll serve --force_polling -H 0.0.0.0 command, 
it says there are some missing files in content folder. But actually files are 
in there. any solution?
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org