Freeze Broken by Accident: pagure02.fedoraproject.org

2023-02-28 Thread Stephen John Smoogen
I ran `dnf update` on pagure02.fedoraproject.org thinking I was working on a different system. The box has been updated to the latest packages. Once I realized my mistake I reviewed the changes and did not see anything that I felt would affect pagure's operation. -- Stephen J Smoogen. Let us be k

Re: Sourcegraph and exploded sources

2021-11-20 Thread Stephen John Smoogen
On Sat, 20 Nov 2021 at 15:05, Matthew Miller wrote: > > I mentioned on devel list that Sourcegraph is going to be indexing > source.fedoraproject.org. > > So I guess, first — heads-up! if you see their traffic, it's not malicious. > (I asked them to provide us with the user agent they use, and I'l

Re: Fedorapeople webspace

2021-11-05 Thread Stephen John Smoogen
On Fri, 5 Nov 2021 at 09:22, James Smith wrote: > > That is an old account, the new account is osiris2600 > OK I found the problem, and my apologies for missing the obvious. Your account was created in /home/osiris2600 and I was looking where all the other accounts are (/home/fedora//). Apache is

Re: Fedorapeople webspace

2021-11-05 Thread Stephen John Smoogen
ectory [root@people02 fedora][PROD]# ls -l smittix total 0 drwxrwxr-x. 2 smittix smittix 21 Jan 13 2014 public_html The account you set permissions to was smittix. > -Original Message- > From: Stephen John Smoogen > Sent: 05 November 2021 12:58 > To: Fedora Infrastructure >

Re: Fedorapeople webspace

2021-11-05 Thread Stephen John Smoogen
On Fri, 5 Nov 2021 at 06:54, James Smith wrote: > > Good morning, all, > > I'm having a small issue with the fedorapeople webspace. I've created a > public_html directory with permissions set to 755 however I'm still getting a > forbidden message when navigating to https://osiris2600.fedorapeopl

Re: FBR: fix download cron on download-cc-rdu

2021-10-19 Thread Stephen John Smoogen
Merging and deploying. On Mon, 18 Oct 2021 at 16:53, Kevin Fenzi wrote: > > +1 here. > > kevin > -- > On Mon, Oct 18, 2021 at 03:12:44PM -0400, Stephen John Smoogen wrote: > > https://pagure.io/fedora-infra/ansible/pull-request/834 > > > > -- Forward

FBR: fix download cron on download-cc-rdu

2021-10-18 Thread Stephen John Smoogen
https://pagure.io/fedora-infra/ansible/pull-request/834 -- Forwarded message - From: (Cron Daemon) Date: Mon, 18 Oct 2021 at 15:00 Subject: Cron /usr/local/bin/lock-wrapper sync-up-downloads "/usr/local/bin/sync-up-centos" To: /usr/local/bin/lock-wrapper: line 50: /usr/local/b

Re: Freeze break request: revert docs redirect

2021-10-11 Thread Stephen John Smoogen
+1 to fix. On Mon, 11 Oct 2021 at 13:06, Kevin Fenzi wrote: > > I'd like to revert this commit and remove > /etc/httpd/conf.d/docs.fedoraproject.org/workstation-main on all proxies > and reload httpd on them. > > This redirect never worked, and they are going to fix it by just > renaming the bran

Re: Retroactive freeze break: httpd update

2021-10-05 Thread Stephen John Smoogen
On Tue, 5 Oct 2021 at 18:04, Kevin Fenzi wrote: > > I've updated all our fedora 34 hosts to httpd-2.4.50-1.fc34 > in order to protect us from CVE-2021-41773. > > Since this was a security issue, I just went ahead and did it. > I would have done the same thing but forgotten to say I did.. so +1 >

Re: Freeze break request: disable fedmsg-irc bots in most channels

2021-09-23 Thread Stephen John Smoogen
I am ok with this. +1 On Thu, 23 Sept 2021 at 17:09, Kevin Fenzi wrote: > > Currently, the fedmsg-irc role connects a number of IRC bots to > libera.chat IRC. (Usually 2 per channel, one for production and one for > staging). These bots then send fedmsgs that match a specific regex to > the chann

Re: (retroactive) Freeze break: Proxy adjustments

2021-09-22 Thread Stephen John Smoogen
On Wed, 22 Sept 2021 at 15:19, Kevin Fenzi wrote: > > Yesterday we were having lots of issues with proxy01/10 in IAD2. > They would stop processing connections. Restarting httpd seemed to clear > it up for a while, then it would get stuck again. > > My current theory is that we were hitting the li

Re: Freeze break request: Add second datanommer server dns

2021-09-15 Thread Stephen John Smoogen
On Wed, 15 Sept 2021 at 12:53, Stephen John Smoogen wrote: > > +1 from sysadmin-main smoogen thanks for doing reverse DNS. > > On Wed, 15 Sept 2021 at 12:13, Mark O'Brien wrote: > > > > Hi All, > > > > I have a dns patch for review. > > > &g

Re: Freeze break request: Add second datanommer server dns

2021-09-15 Thread Stephen John Smoogen
+1 from sysadmin-main smoogen On Wed, 15 Sept 2021 at 12:13, Mark O'Brien wrote: > > Hi All, > > I have a dns patch for review. > > We would like to add a second datanommer db server > so that we can migrate the current database to it and > make some changes such as adding the timescaledb > plugi

Re: Freeze Break Request: Update rbac permissions for openshift apps

2021-09-14 Thread Stephen John Smoogen
On Tue, 14 Sept 2021 at 15:58, Mark O'Brien wrote: > > Hi All, > > This is a request to run the batcave playbook to update rbac permissions to > allow the sysadmin-openshift group to run all the openshift playbooks in > ansible. > > The group currently contains the people who are putting up the

Re: another ipsilon freeze break

2021-09-13 Thread Stephen John Smoogen
On Mon, 13 Sept 2021 at 15:06, Stephen John Smoogen wrote: > > On Mon, 13 Sept 2021 at 14:58, Kevin Fenzi wrote: > > > > This time I just want +1s to run the ipsilon playbook. > > > > This will deploy the OIDC client data for the production ocp cluster we >

Re: another ipsilon freeze break

2021-09-13 Thread Stephen John Smoogen
On Mon, 13 Sept 2021 at 14:58, Kevin Fenzi wrote: > > This time I just want +1s to run the ipsilon playbook. > > This will deploy the OIDC client data for the production ocp cluster we > are working on bringing up. It shouldn't cause any problems and woud be > easy to back out. > > +1s? > +1 -

Re: Freeze Break Request: more ipsilon/sssd debugging

2021-09-13 Thread Stephen John Smoogen
On Mon, 13 Sept 2021 at 13:34, Kevin Fenzi wrote: > > So, it turns out the additional debugging we setup in that sssd package > on the ipsilon servers isn't sufficent to show what the problem is. ;( > > sssd developers would like us to crank up debugging to try and get logs > of whats happening. U

Re: Freeze Break Request: debugging sssd on ipsilon

2021-09-09 Thread Stephen John Smoogen
+1 as the alternative is still outage On Thu, 9 Sept 2021 at 15:28, Kevin Fenzi wrote: > > Greetings everyone. > > We have been having a sporadic issue for a while now where sssd on > ipsilon01 and/or ipsilon02 goes into a error state and starts rejecting > everyone's logins. ;( See https://fedo

Re: FBR: add fedora-messaging queues for CentOS Stream robosignatory

2021-09-01 Thread Stephen John Smoogen
On Wed, 1 Sept 2021 at 14:40, kevin wrote: > > On Wed, Sep 01, 2021 at 09:25:02AM -0500, Brian Stinson wrote: > > Hi Folks, > > > > We'd like to have a separate fedora-messaging queue for our robosignatory > > installation in CentOS Stream: > > https://pagure.io/fedora-infra/ansible/pull-request/

Re: Freeze Break Request: increase memory on s390x kvm builders

2021-08-27 Thread Stephen John Smoogen
On Fri, 27 Aug 2021 at 11:57, Kevin Fenzi wrote: > > Not long ago we were given more memory and disk space for our s390x kvm > lpar (logical partition). I have on my list to redo those builders and > figure out the best number to have and what resources, etc... > I still plan to do this after beta

Re: Questions about the Fedora Release Life Cycle

2021-07-22 Thread Stephen John Smoogen
On Thu, 22 Jul 2021 at 10:48, Beatriz Michelson Reichert wrote: > > Hi, I'm Beatriz and I'm a student at the Santa Catarina State University. > Hi Beatriz and welcome to Fedora Infrastructure. > Currently, I'm studying the Fedora Release Life Cycle, and would like to know > if anyone could hel

Re: DNS patch to rename ocp4 clusters, and revert vhmost-ocp changes

2021-07-15 Thread Stephen John Smoogen
ox gets turned off and then comes back on somehow. Then we may have an ip conflict. So until the hardware is physically removed from the racks.. keep its mgmt ip in DNS. Maybe change the name to oldhost or retired .. but keep it there until then. > On Thu, 15 Jul 2021 at 09:22, Stephen John Sm

Re: DNS patch to rename ocp4 clusters, and revert vhmost-ocp changes

2021-07-15 Thread Stephen John Smoogen
On Thu, 15 Jul 2021 at 04:18, David Kirwan wrote: > > https://gist.github.com/davidkirwan/ecc1c135b6f2c82b1ef337ebdcd8414b > > Hi folks if someone would be so kind as to take a look over our changes here, > we want to revert some vhmost-ocp4 changes and rename the ocp hosts. > > Just two things t

Re: RHEL8 for Copr builders

2021-05-28 Thread Stephen John Smoogen
On Fri, 28 May 2021 at 07:49, Miroslav Suchý wrote: > With the sunset of CentOS8 we want to allow builds on top real RHEL in > Copr. Now that the RH Developer program is known, > what are the option? Do we already use it somewhere? Or I will be the > first to use it? :) > > I need it accessible f

Re: status.fp.org concept

2021-05-03 Thread Stephen John Smoogen
On Mon, 3 May 2021 at 10:53, Vít Ondruch wrote: > I like it. Nevertheless, I'll be missing the current page, which lists the > various services Fedora provides. > > > The reason we are changing is because of that list. Too many people expect us to know instantly that something is broken, somethin

Re: Host DOWN alert for vmhost-x86-copr01.rdu-cc.fedoraproject.org!

2021-04-26 Thread Stephen John Smoogen
On Mon, 26 Apr 2021 at 16:44, Stephen John Smoogen wrote: > > > On Mon, 26 Apr 2021 at 16:22, Pavel Raiskup wrote: > >> This is (probably) my fault. I played a bit with ipv6 && libvirt, and I >> rebooted the machine, and it doesn't boot up now :-( sorry

Re: Host DOWN alert for vmhost-x86-copr01.rdu-cc.fedoraproject.org!

2021-04-26 Thread Stephen John Smoogen
On Mon, 26 Apr 2021 at 16:22, Pavel Raiskup wrote: > This is (probably) my fault. I played a bit with ipv6 && libvirt, and I > rebooted the machine, and it doesn't boot up now :-( sorry for the noise. > > If there's some console (VPN?) access that we could use to fix such > problems > within our

Re: FREEZE BREAK: Remove sysadmin-gnome to fix playbooks

2021-04-21 Thread Stephen John Smoogen
+1 On Wed, 21 Apr 2021 at 16:59, Nick Bebout wrote: > [nebebout@GLFW5M2 fedora-ansible]$ git diff HEAD^ > diff --git a/inventory/group_vars/batcave b/inventory/group_vars/batcave > index 40c2b393a..5af3b70f7 100644 > --- a/inventory/group_vars/batcave > +++ b/inventory/group_vars/batcave > @@ -2

Re: About JS framework

2021-04-16 Thread Stephen John Smoogen
It is spam. I am blocking/removing On Fri, 16 Apr 2021 at 09:44, Vipul Siddharth wrote: > On Wed, Apr 14, 2021 at 6:15 PM Tom Jenner > wrote: > > > > Hi! Check out this article to find description of Angular and React: > https://itechcraft.com/angular-vs-reactjs/ . It describes pros, cons and >

Re: otp resets

2021-04-10 Thread Stephen John Smoogen
On Sat, 10 Apr 2021 at 02:46, Aurelien Bompard wrote: > > > > * If they are Red Hat employees we can use the internal verify thing > > > > Yes. Is there a way we could extend something similar to non-RHers? > > That would be interesting, how does it work? Can we replicate it in some > way? > > Th

Re: Congrats to Nick Bebout

2021-04-05 Thread Stephen John Smoogen
Congratulations Nick. On Mon, 5 Apr 2021 at 11:48, Kevin Fenzi wrote: > Greetings. > > I'm happy to announce that Nick Bebout(fas: nb, irc: nb) > has been added to our sysadmin-main group. > > This is the core group of trusted folks that high level access to most > everything in fedora infrastru

Re: Postgresql server in aws.f.o

2021-03-15 Thread Stephen John Smoogen
On Mon, 15 Mar 2021 at 12:00, Miroslav Suchý wrote: > Hi. > In Copr we use postgresql on copr-frontend. I decide that it is time to > split it off to separate server. I am going to > create new machine in AWS and deploy postgresql there. I have several open > questions: > > * is there interest

Re: Infra technos

2021-03-09 Thread Stephen John Smoogen
rs 2021 à 21:15, Stephen John Smoogen a écrit : > >  > > OK I am assigning the ticket to you. I don't know what is causing this not > to work. > > On Tue, 9 Mar 2021 at 14:46, Patrick Vavrina > wrote: > >> I haven’t “Take” after “None”. >> >> >>

Re: Infra technos

2021-03-09 Thread Stephen John Smoogen
n the right hand column something like *Assignee* None — Take click on Take and it should assign it to you. > > Le 9 mars 2021 à 18:41, Stephen John Smoogen a écrit : > >  > > > On Tue, 9 Mar 2021 at 10:37, Patrick Vavrina > wrote: > >> Hi Kevin, >> >

Re: Infra technos

2021-03-09 Thread Stephen John Smoogen
On Tue, 9 Mar 2021 at 10:37, Patrick Vavrina wrote: > Hi Kevin, > > Thanks a lot for your response. > > I’m interested in participating to package Ansible. > > As Fedora apprentice I saw that ticket concerns me: > https://pagure.io/fedora-infrastructure/issue/9693 > > Could I fix it? > > Have a n

Re: ssh git access to src.fedoraproject.org feedback

2021-03-04 Thread Stephen John Smoogen
On Wed, 3 Mar 2021 at 17:13, Matthew Miller wrote: > On Wed, Mar 03, 2021 at 01:53:28PM -0800, Kevin Fenzi wrote: > > 4) We could add some kind of GSSAPI/Kerberos support to pagure, so > > people could use https and a kerberos ticket. > > What's amount of effort required for this option? Because

Re: server updates

2021-02-16 Thread Stephen John Smoogen
On Mon, 15 Feb 2021 at 20:56, Kevin Fenzi wrote: > Greetings all. I thought I would let everyone know that I have applied > updates all around on our servers, excepting a few: > > copr* > retrace* > os* > notifs* > > I have also rebooted stg hosts. > > I am going to be doing some more (non outage

Re: ancient backups of the moinmoin wiki?

2020-11-16 Thread Stephen John Smoogen
On Mon, 16 Nov 2020 at 12:58, Matthew Miller wrote: > This is a long shot, I know. When we converted to MediaWiki from MoinMoin > in > 2008, pages were imported, but history was lost. I often like to go > spelunking in the past, and it's often frustrating to hit this barrier. > > As I recall, Moi

Re: best place to run an email-forwarding bot account?

2020-11-12 Thread Stephen John Smoogen
On Thu, 12 Nov 2020 at 18:13, Matthew Miller wrote: > On Thu, Nov 12, 2020 at 02:56:10PM -0800, Kevin Fenzi wrote: > > I'm not sure I like this from a security point of view... is there any > > way we could use the discourse api and pull a digest from a script and > > have it post to the list? (g

Re: Planned Outage: 2020-11-11 21:00 UTC Fedora Services

2020-11-09 Thread Stephen John Smoogen
On Mon, 9 Nov 2020 at 17:31, Stephen John Smoogen wrote: > > There will be an outage starting at 2020-11-11 21:00UTC, > which will last approximately 6 hours. > > To convert UTC to your local time, take a look at > http://fedoraproject.org/wiki/Infrastructure/UTCHowto > or r

Planned Outage: 2020-11-11 21:00 UTC Fedora Services

2020-11-09 Thread Stephen John Smoogen
There will be an outage starting at 2020-11-11 21:00UTC, which will last approximately 6 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2017-05-03 21:00UTC' Reason for outage: Apply updates and new kernels for all

Re: Another Rust MirrorManager experiment

2020-10-06 Thread Stephen John Smoogen
On Tue, 6 Oct 2020 at 03:46, Adrian Reber wrote: > On Mon, Oct 05, 2020 at 08:30:12AM -0400, Stephen John Smoogen wrote: > > On Mon, 5 Oct 2020 at 02:24, Adrian Reber wrote: > > > > > We are not wanting to deploy new EL7 systems but would probably install > an >

Re: Another Rust MirrorManager experiment

2020-10-05 Thread Stephen John Smoogen
On Mon, 5 Oct 2020 at 02:24, Adrian Reber wrote: > On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote: > > On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote: > > > On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote: > > > > On Wed, Jun 10, 2020 at 11:09:49AM +0200, Ad

Re: FBR: put new proxies in action

2020-09-28 Thread Stephen John Smoogen
+1 after review. On Mon, 28 Sep 2020 at 09:22, Mark O'Brien wrote: > Hi All, > > Attached is a DNS patch to put some new proxy servers in Europe, Asia > Pacific and Africa. The servers are up and running and passing all nagios > checks. This patch would start directing the regional traffic towar

Re: FBR

2020-09-24 Thread Stephen John Smoogen
On Thu, 24 Sep 2020 at 10:24, Pierre-Yves Chibon wrote: > On Thu, Sep 24, 2020 at 10:11:11AM -0400, Stephen John Smoogen wrote: > > Need to fix opendkim on bastion > > > > I merged a fix in https://pagure.io/fedora-infrastructure/issue/9250 but > > have not run the p

FBR

2020-09-24 Thread Stephen John Smoogen
Need to fix opendkim on bastion I merged a fix in https://pagure.io/fedora-infrastructure/issue/9250 but have not run the playbook for bastion to push this out yet. I would like to get a +1 or -1 on this before doing that. [I can remove the merge also if that is better.] -- Stephen J Smoogen. __

Re: FBR: increase koji ram to 32 GB

2020-09-21 Thread Stephen John Smoogen
This has been done. On Mon, 21 Sep 2020 at 16:02, Kevin Fenzi wrote: > On Mon, Sep 21, 2020 at 03:51:31PM -0400, Stephen John Smoogen wrote: > > Currently koji01 and koji02 are using 16 GB and need 32 GB . Bump each > > guest in ansible and via virsh commands. > > +1.

Re: FBR: increase koji ram to 32 GB

2020-09-21 Thread Stephen John Smoogen
On Mon, 21 Sep 2020 at 15:51, Stephen John Smoogen wrote: > > Currently koji01 and koji02 are using 16 GB and need 32 GB . Bump each > guest in ansible and via virsh commands. > > -- > Stephen J Smoogen. > > diff --git a/inventory/group_vars/koji b/inventory/group_v

FBR: increase koji ram to 32 GB

2020-09-21 Thread Stephen John Smoogen
Currently koji01 and koji02 are using 16 GB and need 32 GB . Bump each guest in ansible and via virsh commands. -- Stephen J Smoogen. ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructur

FBR: Fix DKIM signing for aws and fedorahosted

2020-09-02 Thread Stephen John Smoogen
diff --git a/roles/opendkim/files/SigningTable b/roles/opendkim/files/SigningTable index 84cb8f78f..81cad91f3 100644 --- a/roles/opendkim/files/SigningTable +++ b/roles/opendkim/files/SigningTable @@ -15,6 +15,10 @@ *@fedoraproject.org bastion-iad._domainkey.fedoraproject.org *@lists.fedoraprojec

Re: IPAM/DCIM for Fedora Infrastructure

2020-09-01 Thread Stephen John Smoogen
On Tue, 1 Sep 2020 at 02:36, Manu Hernandez wrote: > Hi! > > I was going to suggest NetBox as well. > > NetBox is also a data center management infrastructure tool (DCIM) too, > so it can be used to document racks, circuits, power, etc. > > https://netbox.readthedocs.io/en/stable/ > > This is a u

Freeze Break Request (Fix 2factor on builders)

2020-08-28 Thread Stephen John Smoogen
Currently the builders are trying to contact os-node boxes on port 8443 but that is not where fas-all lives. diff --git a/roles/base/templates/iptables/iptables.kojibuilder b/roles/base/templates/iptables/iptables.kojibuilder index a3819777c..805cf735f 100644 --- a/roles/base/templates/iptables/ip

Re: Freeze Break Request: koji update for all builders

2020-08-27 Thread Stephen John Smoogen
+1 On Thu, 27 Aug 2020 at 16:40, Mohan Boddu wrote: > On Thu, Aug 27, 2020 at 2:36 PM kevin wrote: > > > > The koji 1.22.0 update added handling for the '--extra-boot-args' > > kickstart option. UNfortunately, this support wasn't working right and > > causes all our images specifying such optio

Re: Hosting Options for QA Projects

2020-08-17 Thread Stephen John Smoogen
On Mon, 17 Aug 2020 at 13:17, Frantisek Zatloukal wrote: > > > > On Mon, Aug 17, 2020 at 4:36 PM Pierre-Yves Chibon > wrote: >> >> What about email addresses? I figure they are needed for the bugzilla >> integration, no? > > > No, we don't store/need any email addresses. We store the package lis

Re: s390x for EPEL

2020-07-29 Thread Stephen John Smoogen
On Wed, 29 Jul 2020 at 08:03, Pavel Raiskup wrote: > > On Tuesday, July 28, 2020 3:17:56 PM CEST Stephen John Smoogen wrote: > > On Mon, 27 Jul 2020 at 15:46, Kevin Fenzi wrote: > > > > > > On Mon, Jul 27, 2020 at 04:45:58PM +0200, Miroslav Suchý wrote: > >

Re: s390x for EPEL

2020-07-28 Thread Stephen John Smoogen
On Mon, 27 Jul 2020 at 15:46, Kevin Fenzi wrote: > > On Mon, Jul 27, 2020 at 04:45:58PM +0200, Miroslav Suchý wrote: > > What is using Koji to build EPEL builds for s390x. > > We are using 2 LPARS on a z13 mainframe in a Red Hat lab in Westford, MA. > > > We want to enable s390x in Copr. It is wit

Re: s390x for EPEL

2020-07-27 Thread Stephen John Smoogen
On Mon, 27 Jul 2020 at 10:46, Miroslav Suchý wrote: > > What is using Koji to build EPEL builds for s390x. > EPEL uses RHEL for all its builds. The problem is that the license we have for RHEL is meant just for this and we have not been able to extend it to COPR in the past. The issues are legal,

Re: What is our technical debt?

2020-07-07 Thread Stephen John Smoogen
On Tue, 7 Jul 2020 at 08:51, Eduard Lucena wrote: > > Maybe I'm far out, but it's interesting to me that nobody had said > ELK+Kibana+Logtash for monitoring. AFAIK there are ready to use containers > with it, the configuration it's easy to do and the work needs to be done > entirely for the das

Re: What is our technical debt? (fedora integration service)

2020-07-07 Thread Stephen John Smoogen
On Tue, 7 Jul 2020 at 05:23, Pavel Raiskup wrote: > > Hi all, > > as a Copr contributor, I am missing a _standard_ design for integrating > our cool infrastructure into the _upstream_ work-flows. We have a lot of > teams trying to implement the same thing: > I think we need to work out what tech

Re: What is our technical debt?

2020-07-01 Thread Stephen John Smoogen
On Wed, 1 Jul 2020 at 14:46, Miroslav Suchý wrote: > > Dne 30. 06. 20 v 9:44 Pierre-Yves Chibon napsal(a): > > What are you talking about here? The Fedora release process? The > > mass-branching > > in dist-git? > > This. And creating new gpg keys, new mock configs, new tags in Koji, add > relea

Re: What is our technical debt?

2020-06-28 Thread Stephen John Smoogen
On Fri, 26 Jun 2020 at 07:51, Stephen John Smoogen wrote: > > On Thu, 25 Jun 2020 at 15:27, Pierre-Yves Chibon wrote: > > > > Good Morning Everyone, > > > > Just like every team we have technical debt in our work. > > I would like your help to try to defi

Re: What is our technical debt?

2020-06-27 Thread Stephen John Smoogen
On Sat, 27 Jun 2020 at 08:05, Peter Robinson wrote: > > > > 10. Our deployment infrastructure of kickstarts/pxe/tftp falls under > > > technical debt. It is based off of what we have been doing for 10+ > > > years and it has broken a lot in this transition. When it works its > > > fine, and when i

Re: What is our technical debt?

2020-06-26 Thread Stephen John Smoogen
On Thu, 25 Jun 2020 at 15:27, Pierre-Yves Chibon wrote: > > Good Morning Everyone, > > Just like every team we have technical debt in our work. > I would like your help to try to define what it is for us. > > So far, I've come up with the following: > - python3 support/migration > - fedora-messagi

Re: Another Rust MirrorManager experiment

2020-06-10 Thread Stephen John Smoogen
On Wed, 10 Jun 2020 at 05:10, Adrian Reber wrote: > On Thu, Jun 04, 2020 at 07:02:41PM -0700, Kevin Fenzi wrote: > > On Mon, Jun 01, 2020 at 04:18:35PM +0200, Adrian Reber wrote: > > > Our MirrorManager setup exports the current state of all mirrors every > > > hour at :30 to a protobuf based fil

Re: RFC: How to deal with account creation

2020-05-15 Thread Stephen John Smoogen
On Fri, 15 May 2020 at 08:34, Patrick Uiterwijk wrote: > On Fri, May 15, 2020 at 1:58 PM Stephen John Smoogen > wrote: > > > > > > Currently I am dealing with 1-3 failed account creations a day due to > our spam checking tool, basset. > > > > Basset is th

RFC: How to deal with account creation

2020-05-15 Thread Stephen John Smoogen
Currently I am dealing with 1-3 failed account creations a day due to our spam checking tool, basset. Basset is the tool which sits in the account system creation path and tries to check to see if an account is semi-valid or not. This was written by Patrick Uiterwijk about 4 to 5 years ago to deal

Meeting Agenda for 2020-04-30 1500 UTC

2020-04-29 Thread Stephen John Smoogen
## Preamble The infrastructure team will be having its weekly meeting tomorrow, 2020-04-30 at 15:00 UTC in #fedora-meeting-1 on the freenode network. We have a document at https://board.net/p/fedora-infra Please try and review and edit that document before the meeting and we will use it to have our

Re: FBR: Fix DNS to match closer zones

2020-04-26 Thread Stephen John Smoogen
On Sun, 26 Apr 2020 at 14:40, Kevin Fenzi wrote: > On Sun, Apr 26, 2020 at 11:33:45AM -0400, Stephen John Smoogen wrote: > > Currently several countries are getting routed to proxies which are > timing > > out for them. This should 'fix' some though I expect we will

FBR: Fix DNS to match closer zones

2020-04-26 Thread Stephen John Smoogen
Currently several countries are getting routed to proxies which are timing out for them. This should 'fix' some though I expect we will need to break this out further per continental region. diff --git a/roles/dns/files/named.conf b/roles/dns/files/named.conf index c8b95b9..135ec69 100644 --- a/ro

Re: FBR: reboot ppc9-02

2020-04-24 Thread Stephen John Smoogen
+! On Fri, 24 Apr 2020 at 18:24, Kevin Fenzi wrote: > ppc9-02 is in a odd state. Guests are running ok, but the host itself is > hanging any ansible runs against it. :( Usually this is a sign of some > stuck hardware. > > I've disabled the builders on it and once builds finish on them I would >

FBR: Reboot of ppc09-02

2020-04-24 Thread Stephen John Smoogen
Server has some problems with services and guests. We would like to reboot the system to clear some issues being seen. -- Stephen J Smoogen. ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infras

Re: FBR: Pre GA tasks

2020-04-24 Thread Stephen John Smoogen
Reviewed updated diff. +1 On Fri, 24 Apr 2020 at 12:41, Mohan Boddu wrote: > Hi, > > On Fri, Apr 24, 2020 at 12:33 PM Kevin Fenzi wrote: > > > > On Fri, Apr 24, 2020 at 12:26:06PM -0400, Mohan Boddu wrote: > > > Hi, > > > > > > The following changes are needed for pre ga tasks, please review t

Re: Mirror the ansible repo from pagure in the batcave

2020-04-24 Thread Stephen John Smoogen
On Thu, 23 Apr 2020 at 16:37, Kevin Fenzi wrote: > On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote: > > Good Morning Everyone, > > > > Kevin and I had a discussion on IRC the other day about mirroring the > ansible > > repo. > > The options we considered are: > ...snip... > > >

Re: FBR: Allow RHEL8 systems to know about code-ready

2020-04-21 Thread Stephen John Smoogen
There were worries it might pull in things we do not want. I am going to let that be dealt with outside of FBR season. On Tue, 21 Apr 2020 at 11:05, Neal Gompa wrote: > On Tue, Apr 21, 2020 at 11:03 AM Stephen John Smoogen > wrote: > > > > > > diff --git a/files/c

FBR: Allow RHEL8 systems to know about code-ready

2020-04-21 Thread Stephen John Smoogen
diff --git a/files/common/rhel8.repo b/files/common/rhel8.repo index 162848d..9686500 100644 --- a/files/common/rhel8.repo +++ b/files/common/rhel8.repo @@ -25,3 +25,10 @@ baseurl= https://infrastructure.fedoraproject.org/repo/rhel/rhel8/$basearch/rhel- gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY

FBR: add more ips to batcave allows

2020-04-21 Thread Stephen John Smoogen
This is for the RDU-CC expansion. We have these "zones" but these are not subroutes (aka 8.43.85.40 is live and not a network high) Fedora 68.43.85.40/29 Fedora 78.43.85.48/29 Fedora 88.43.85.56/29 Fedora 98.43.85.64/29 Fedora 10 8.43.85.72/29 Fedora 11 8.43.85.80/29

FBR: add sysadmin-analysis to bastion

2020-04-20 Thread Stephen John Smoogen
The sysadmin-analysis group is used on data-analysis01 for people to log in and work there. Because everyone in that group was in an existing group, I forgot to add it to bastion. However a new person joined and they don't have access. [smooge@batcave01 ansible (master)]$ git diff diff --git a/inv

Re: FBR: To update ostree on bodhi-backend01, branched-composer, rawhide-composer

2020-04-20 Thread Stephen John Smoogen
+1 On Mon, 20 Apr 2020 at 17:05, Kevin Fenzi wrote: > On Mon, Apr 20, 2020 at 04:23:39PM -0400, Mohan Boddu wrote: > > Hi, > > > > Randomly we hit an issue in "ostree pull-local" command that it wont > > sync and throws out a traceback. The new ostree build adds more > > logging and should help

Re: [PATCH] fas_client: fix template to correctly apply on pkgs02 and add people02

2020-04-20 Thread Stephen John Smoogen
On Mon, 20 Apr 2020 at 11:13, Pierre-Yves Chibon wrote: > On Mon, Apr 20, 2020 at 02:41:51PM +, Kevin Fenzi wrote: > > The ansible_hostname variable is actually the short name of the host, > > not the fqdn, so this conditional didn't match before. Switch it to use > > startswith and also add

Re: FBR: Update all systems to nrpe-4.0.4

2020-04-20 Thread Stephen John Smoogen
On Sun, 19 Apr 2020 at 14:46, Kevin Fenzi wrote: > On Sun, Apr 19, 2020 at 01:55:56PM -0400, Stephen John Smoogen wrote: > > NRPE for Fedora was updated to 4.0.4 which got auto-updated on many of > the > > infrastructure systems. However, the noc servers are still runnin

FBR: Update all systems to nrpe-4.0.4

2020-04-19 Thread Stephen John Smoogen
NRPE for Fedora was updated to 4.0.4 which got auto-updated on many of the infrastructure systems. However, the noc servers are still running a much older version of nrpe which is causing some issues with monitoring. Plan: run on batcave01 sudo ansible -i noc:batcave:bastion:people -m shell -a 'y

Re: FBR: To enable signing on eln builds

2020-04-15 Thread Stephen John Smoogen
On Wed, 15 Apr 2020 at 16:21, Mohan Boddu wrote: > Hi, > > The following patch will enable signing on eln builds and for now it > will be used only in stg. > > For more info: https://pagure.io/releng/issue/9154 > > The patch comes with different spacing so not sure if that makes a difference...

Re: FBR: remove elections from inventory and run noc.yml playbook

2020-04-14 Thread Stephen John Smoogen
On Tue, 14 Apr 2020 at 15:49, Kevin Fenzi wrote: > On Tue, Apr 14, 2020 at 03:08:25PM -0400, Stephen John Smoogen wrote: > > The following patch should do a minimal removal of the elections app from > > ansible inventory so that a run of noc.yml will remove the systems from >

FBR: remove elections from inventory and run noc.yml playbook

2020-04-14 Thread Stephen John Smoogen
The following patch should do a minimal removal of the elections app from ansible inventory so that a run of noc.yml will remove the systems from being monitored. Then we can turn off the elections systems. -- Stephen J Smoogen. From acd374d1ba95aba044fc3417bb8a1ce191fe510b Mon Sep 17 00:00:00 20

FBR: Big patch to dhcpd to move all mgmt interfaces to dhcp for move

2020-04-10 Thread Stephen John Smoogen
Kevin and I found all the mac addresses for the mgmt boxes and are putting them onto dhcp so when they show up in new location.. they should come up without trying to use 'dead' networks. -- Stephen J Smoogen. From a2bc61f49793893d9e771c64b777047a53d6063f Mon Sep 17 00:00:00 2001 From: Stephen

Re: Issue with Fedora GeoIP service

2020-04-09 Thread Stephen John Smoogen
On Thu, 9 Apr 2020 at 12:35, Tomasz Torcz wrote: > On Thu, Apr 09, 2020 at 04:55:20PM +0200, Adrian Reber wrote: > > > > We finally pushed it to Fedora's staging environment. Please try it > > using https://geoip.stg.fedoraproject.org/city if you have a chance. > > It seems to have only IPv4 ad

Re: Update geoip-city

2020-03-31 Thread Stephen John Smoogen
Let me know when you want to run the playbook. On Tue, 31 Mar 2020 at 05:45, Adrian Reber wrote: > Attached is the patch to take the upstream update of geoip-city to > Fedora infrastructure. > > I would like to try this first in staging and need some help to run the > corresponding playbook to s

Re: [PATCH] s3-mirror: fix missing trailing /

2020-03-16 Thread Stephen John Smoogen
This is an FBR and I +1 it On Mon, 16 Mar 2020 at 13:01, Kevin Fenzi wrote: > This is causing f31 updates to not be synced. The cron job reports: > > Subject: Cron /usr/local/bin/lock-wrapper > s3sync-updates-current "/usr/local/bin/s3-sync-path.sh > /pub/fedora/linux/updates/31/Everything/x86_

Re: Freeze break request: rabbitmq-cluster-adjustments

2020-03-12 Thread Stephen John Smoogen
At this point I think we are in full outage and this doesn't need +1's in order to make things go. But I have +1 it On Thu, 12 Mar 2020 at 15:32, Kevin Fenzi wrote: > I'd apply this later tonight when things were not very busy, > or early tomorrow morning. (given +1s) > _

Re: Weekly Backlog Refinement - Week Mar 09 2020

2020-03-09 Thread Stephen John Smoogen
On Mon, 9 Mar 2020 at 14:20, Mattia Verga wrote: > Just my thoughts: > > > Hi all, > > > > This is the first email with trying to better prioritize our backlog of > > ticket ( https://pagure.io/fedora-infrastructure/issues ). > > > > So let's look at 5 tickets and rate them using the following ca

Re: Freeze Break Request: s3-mirror adjustments

2020-03-09 Thread Stephen John Smoogen
+1 On Sat, 7 Mar 2020 at 05:17, Adrian Reber wrote: > Looks correct from my point of view, for both scripts we are now doing: > > 1. sync everything besides repodata > 2. sync everything including (or only) repodata > 3. invalidate repodata > 4. delete files > > The only problem I can see now is

Re: Freeze Break Request: s3-mirror adjustments

2020-03-05 Thread Stephen John Smoogen
+1 ok so the s3sync does allow for delete. I thought from previous conversations it didn't work that way so my view of what was needed was wrong. On Thu, 5 Mar 2020 at 16:06, Kevin Fenzi wrote: > First, I noticed we are running the full sync twice right now, at the > same time: > > [root@mm-back

Re: Freeze break request: koji autovacuum_freeze_max_age

2020-03-04 Thread Stephen John Smoogen
On Wed, 4 Mar 2020 at 04:00, Julen Landa Alustiza wrote: > Is there an ETA for the next vacuum process? If we can wait for some > There is no ETA as it depends on the number of builds that have been done since the last autovac. A large number means it is further into the future but takes much lo

Re: FBR: Change the retirement date to yesterday in pdc

2020-03-03 Thread Stephen John Smoogen
Reviewed. Covers the problems I could think of. +1 On Tue, 3 Mar 2020 at 14:37, Mohan Boddu wrote: > Currently when a package is retired, it sets the eol date to today's > date in pdc, but that enables people to revert the git retirement > commit until the end of the day and that causes a couple

Re: FBR: Add monitoring for website build fails

2020-02-27 Thread Stephen John Smoogen
On Thu, 27 Feb 2020 at 00:47, Rick Elrod wrote: > I'd like to apply the following which does: > - Adds a script I wrote for reading a timestamp from a file on disk > and alerting if the timestamp within it is NOT within a particular > delta to now. > - Applies this to sundries01 and uses it to ch

Freeze Break Request: update wildcard cert to new one

2020-02-27 Thread Stephen John Smoogen
This patch should make the changes to the appropriate files so that various playbooks will use the newer certificate wildcard-2020 -- Stephen J Smoogen. From 40a3b08e2120f7e43bd5352b16f8717da64a828f Mon Sep 17 00:00:00 2001 From: Stephen Smoogen Date: Thu, 27 Feb 2020 21:19:54 + Subject: [PA

Re: State of FMN (FedMSG Notifications) and Replacement

2020-02-26 Thread Stephen John Smoogen
On Wed, 26 Feb 2020 at 15:58, Kevin Fenzi wrote: > On Wed, Feb 26, 2020 at 08:26:09AM +0100, Clement Verna wrote: > > Hi all, > > > > FMN (https://apps.fedoraproject.org/notifications) is currently one of > the > > main blocking point for dropping fedmsg in favour of fedora-messaging. > > FMN is

Meeting agenda for 2020-02-20

2020-02-19 Thread Stephen John Smoogen
## Preamble The infrastructure team will be having its weekly meeting tomorrow, 2020-02-20 at 15:00 UTC in #fedora-meeting-1 on the freenode network. We have a document at https://board.net/p/fedora-infra Please try and review and edit that document before the meeting and we will use it to have our

Re: When adding hosts to ansible please first update DNS

2020-02-19 Thread Stephen John Smoogen
On Wed, 19 Feb 2020 at 06:59, Miroslav Suchý wrote: > > Dne 17. 02. 20 v 21:50 Stephen John Smoogen napsal(a): > > Because ip addresses can change, and we do not have time to fix it in > > multiple places, > > That was me. > We will be adding 9 machines. One by one

When adding hosts to ansible please first update DNS

2020-02-17 Thread Stephen John Smoogen
When bringing up a new host, 1. Edit the appropriate dns file in the Fedora Infrastructure DNS repository. 2. Commit the changes and wait for them to sync out to dns. 3. Test that the hostname and ip address match. 4. Add the hostname to the ansible inventory file in the appropriate group. 5. Cont

  1   2   3   4   5   6   7   8   9   10   >