Re: Bugzilla and Groups

[Tim Flink]

On 12/22/23 10:55, Kevin Fenzi wrote:

On Thu, Dec 21, 2023 at 01:19:46PM -0700, Tim Flink wrote:

What are the general rules around default bugzilla assignee for packages? I'm 
trying to set the default assignee for rocm-cmake to rocm-packagers-sig but 
keep getting an error:

Unable to update the bugzilla assignee(s): Invalid user or group name as 
fedora_assignee

As far as I know, rocm-packagers-sig is a pkgdb group. Do I need to request a 
change to the group for it to be the default bugzilla assignee? Is setting a 
group as the default assignee against some policy that I don't know about?

My search-fu has been failing me on this so I figured I would ask to see if 
someone here has an answer.


I think this is a interface confusion... when adding a group as bugzilla
asignee, you have to prefix it with @

If you do that does it work?

kevin


If I prefix the group with @, that does work.

Thanks,

Tim
--
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Bugzilla and Groups

[Tim Flink]

What are the general rules around default bugzilla assignee for packages? I'm 
trying to set the default assignee for rocm-cmake to rocm-packagers-sig but 
keep getting an error:

Unable to update the bugzilla assignee(s): Invalid user or group name as 
fedora_assignee

As far as I know, rocm-packagers-sig is a pkgdb group. Do I need to request a 
change to the group for it to be the default bugzilla assignee? Is setting a 
group as the default assignee against some policy that I don't know about?

My search-fu has been failing me on this so I figured I would ask to see if 
someone here has an answer.

Thanks,

Tim
--
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: GitLab Grouping and Naming

[Tim Flink]

On 8/7/23 13:04, Kevin Fenzi wrote:




This might have to be something that we have a meeting to discuss and
figure out a scheme?


Sure, or the scheme below seems good to me.



cheers,
ryanlerch


Hi Ryan,

We more or less discussed that with Kevin in the past and for CentOS
groups (all coming from same common IPA infra) I proposed that we used
something like :
--

Let me explain : Assuming that we need to grant the CentOS Automotive
SIG access to gitlab, the name in FAS/IPA is :
gitlab-centos-sig-automotive-developer
(https://accounts.fedoraproject.org/group/gitlab-centos-sig-automotive-developer/)

Same rule but for openshift/ocp : we need to grant the hyperscale sig
access to the openshift CI centos infra :
https://accounts.fedoraproject.org/group/ocp-cico-hyperscale/

It's then easier to identify which group has access to what
(gitlab/openshift/etc) *while* keeping the existing groups, as IPA
supports nested groups (so the ocp-cico-hyperscale group in fact
contains the sig-hyperscale group
(https://accounts.fedoraproject.org/group/sig-hyperscale/)

At least that's the naming convention we agreed on so that we can also
easily identify if that's a fedora/centos group (all the sig-* groups
weren't following that naming convention as they were coming from
previous FAS and so imported/merged with the fedora groups in IPA, but
there was no conflicting group back then)



Oh, i can also definitely get on board with a set scheme for Fedora
Accounts groups <-> Gitlab Groups naming conventions.


Yeah, +1


However, the one of the main issues i am noticing with our current
GitLab setup is that the groups that are being added are being done in
an adhoc setting.

For example, there are groups for Council and Mindshare (and not yet,
but i can imagine a FesCO group too) -- should these be grouped
together under, say a "Governance" Sub group?

cheers,
ryanlerch



Multiple solutions : one can always create new groups and reflect that at
gitlab level (same membership but different group name[s]) and IPA supports
multiple "nesting" levels so you can (in your Governance example) have one
groups containining/nesting multiple other ones


Yeah, or 'project' instead of 'govenance'?

We should write up a doc with whatever we do to document it and make
sure everything is on the same page.

kevin


Has there been a conclusion to this? The AI/ML SIG is looking to request a FAS 
group to manage access to the sigs/ai-ml project in GitLab but we're not sure 
what to request for a name.

Thanks,

Tim
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: SOP for adding externally hosted services?

[Tim Flink]

On 5/4/23 13:18, Kevin Fenzi wrote:

On Wed, May 03, 2023 at 06:54:33PM -0600, Tim Flink wrote:

We're looking to deploy an instance of ReportPortal [1] for displaying and 
analyzing the output of automated tests in Fedora.

This aligns with the goals of Testing Farm and instead of having it hosted with 
the other Fedora hosted apps, it looks to be easier to rely on their 
setup/infrastructure.


yeah, the council has specifically said it's ok to do this.


Are there requirements around having something like 
reportportal.fedoraproject.org point to the service once it's up? I couldn't 
find anything by searching through the lists but I have a vague recollection of 
hearing talk about requirements in the past.



I don't think we have anything written up.
Some of it gets handled by going though the process of them becoming a
Red Hat vendor (in order for them to be paid for providing the service).

On top of that off my head though:
* Make sure we have some support avenue to send problems/issues to.
If we don't have this, people will bug us and we won't be sure how to
address issues or downtime.

* Make sure we have some kind of admin contact email or phone or
whatever in case there's some issue thats urgent/sensitive.
To allow us to contact them in case of a security issue or their site
somehow breaking other things.

* Make sure they know who is authorized to ask for changes from our
side.

For setup, we may need to interact with them to setup logins/etc.
(Although perhaps not in this case).

Would you be willing to start a 'external services' SOP on
https://pagure.io/infra-docs-fpo/ ?
Otherwise I can try, but I'm currently swamped. ;)


I've submitted a PR for an externally hosted service SOP:

https://pagure.io/infra-docs-fpo/pull-request/207

Tim
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

SOP for adding externally hosted services?

[Tim Flink]

We're looking to deploy an instance of ReportPortal [1] for displaying and 
analyzing the output of automated tests in Fedora.

This aligns with the goals of Testing Farm and instead of having it hosted with 
the other Fedora hosted apps, it looks to be easier to rely on their 
setup/infrastructure.

Are there requirements around having something like 
reportportal.fedoraproject.org point to the service once it's up? I couldn't 
find anything by searching through the lists but I have a vague recollection of 
hearing talk about requirements in the past.

Thanks,

Tim

[1] https://reportportal.io/
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Freeze Break Request: Blockerbugs Hotfix to Deal With RHBZ Change

[Tim Flink]

I've applied the patch to blockerbugs production, thanks.

For now, I've kept the limit to 20 because I was having trouble getting more 
than 20 to work for me. We can re-visit what the limit should be set to once 
beta freeze lifts.

Tim

On 9/13/21 12:55 PM, Pierre-Yves Chibon wrote:

On Mon, Sep 13, 2021 at 12:49:21PM -0600, Tim Flink wrote:

It turns out that the configuration for rhbz was changed without an 
announcement and now the max number of bugs returned for any query is 20. Some 
of the queries that we use in blockerbugs return more than 20 bugs so I have a 
hotfix to deal with the new pagination requirements. I've attached the patch to 
this email.

I'd like to apply this patch to production. Unfortunately, we're in the middle 
of a non-trivial rewrite that's currently on stg so testing the patch isn't 
really an option and that code isn't ready for production use right now.

Thanks,

Tim



diff --git a/blockerbugs/util/bz_interface.py b/blockerbugs/util/bz_interface.py
index 471140f..a9f90d7 100644
--- a/blockerbugs/util/bz_interface.py
+++ b/blockerbugs/util/bz_interface.py
@@ -29,11 +29,14 @@ from xmlrpc.client import Fault
  
  from blockerbugs import app
  
+# rhbz has been updated to have a max of 20 results returned

+BUGZILLA_QUERY_LIMIT = 20


If I understood correctly, you can request more than 20 results, but if you
don't specify how many you want, you'll only get 20.
I believe if you set it to 0 you may get everything, but this needs to be
confirmed empirically.

Anyway, I would check first if you can't retrieve 100 or 200 results in one go
instead of doing iterations of 20 ;-)


  base_query = {'o1': 'anywords',
'f1': 'blocked',
'query_format': 'advanced',
-  'extra_fields': ['flags']}
-
+  'extra_fields': ['flags'],
+  'limit': BUGZILLA_QUERY_LIMIT}
  
  class BZInterfaceError(Exception):

  """A custom wrapper for XMLRPC errors from Bugzilla"""
@@ -77,7 +80,8 @@ class BlockerBugs():
  # 
https://bugzilla.stage.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED
  # 
&bug_status=POST&bug_status=MODIFIED&classification=Fedora&component=anaconda&f1=component
  # 
&o1=changedafter&product=Fedora&query_format=advanced&v1=2013-03-21%2012%3A25&version=19
-def get_bz_query(self, tracker: int, last_update: datetime.datetime = None) 
-> dict[str, Any]:
+def get_bz_query(self, tracker: int, last_update: datetime.datetime = 
None, offset: int = 0
+ ) -> dict[str, Any]:


If you can set the limit to 0 to retrieve everything, you may want to default to
None rather than 0.


  """Build a Bugzilla query to retrieve all necessary info about all 
bugs which block the
  `tracker` bug.
  
@@ -129,6 +133,9 @@ class BlockerBugs():

  'f10': 'CP'
  })
  
+if offset > 0:


And if you default to None, this will need to be tweaked.


+query.update({'offset': offset})
+
  return query
  
  def query_tracker(self, tracker: int, last_update: Optional[datetime.datetime] = None

@@ -139,8 +146,21 @@ class BlockerBugs():
  :param last_update: If provided, the query is modified to ask only 
about bugs which have
  recent modifications; otherwise asks about all 
bugs.
  """
-query = self.get_bz_query(tracker, last_update)
-buglist = self.bz.query(query)
+
+buglist = []
+last_query_len = BUGZILLA_QUERY_LIMIT
+
+
+# this is a hotfix hack to work around the sudden config change in 
rhbz where the max
+# number of bugs returned for a query is 20
+# it seems to be working for now but may need more work going forward
+while last_query_len == BUGZILLA_QUERY_LIMIT:
+
+new_query = self.get_bz_query(tracker, last_update, 
offset=len(buglist))
+new_buglist = self.bz.query(new_query)
+buglist.extend(new_buglist)
+last_query_len = len(new_buglist)
+


Looks alright to me otherwise.


Pierre
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure


___
in

Freeze Break Request: Blockerbugs Hotfix to Deal With RHBZ Change

[Tim Flink]

It turns out that the configuration for rhbz was changed without an 
announcement and now the max number of bugs returned for any query is 20. Some 
of the queries that we use in blockerbugs return more than 20 bugs so I have a 
hotfix to deal with the new pagination requirements. I've attached the patch to 
this email.

I'd like to apply this patch to production. Unfortunately, we're in the middle 
of a non-trivial rewrite that's currently on stg so testing the patch isn't 
really an option and that code isn't ready for production use right now.

Thanks,

Tim
diff --git a/blockerbugs/util/bz_interface.py b/blockerbugs/util/bz_interface.py
index 471140f..a9f90d7 100644
--- a/blockerbugs/util/bz_interface.py
+++ b/blockerbugs/util/bz_interface.py
@@ -29,11 +29,14 @@ from xmlrpc.client import Fault
 
 from blockerbugs import app
 
+# rhbz has been updated to have a max of 20 results returned
+BUGZILLA_QUERY_LIMIT = 20
+
 base_query = {'o1': 'anywords',
   'f1': 'blocked',
   'query_format': 'advanced',
-  'extra_fields': ['flags']}
-
+  'extra_fields': ['flags'],
+  'limit': BUGZILLA_QUERY_LIMIT}
 
 class BZInterfaceError(Exception):
 """A custom wrapper for XMLRPC errors from Bugzilla"""
@@ -77,7 +80,8 @@ class BlockerBugs():
 # https://bugzilla.stage.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED
 # &bug_status=POST&bug_status=MODIFIED&classification=Fedora&component=anaconda&f1=component
 # &o1=changedafter&product=Fedora&query_format=advanced&v1=2013-03-21%2012%3A25&version=19
-def get_bz_query(self, tracker: int, last_update: datetime.datetime = None) -> dict[str, Any]:
+def get_bz_query(self, tracker: int, last_update: datetime.datetime = None, offset: int = 0
+ ) -> dict[str, Any]:
 """Build a Bugzilla query to retrieve all necessary info about all bugs which block the
 `tracker` bug.
 
@@ -129,6 +133,9 @@ class BlockerBugs():
 'f10': 'CP'
 })
 
+if offset > 0:
+query.update({'offset': offset})
+
 return query
 
 def query_tracker(self, tracker: int, last_update: Optional[datetime.datetime] = None
@@ -139,8 +146,21 @@ class BlockerBugs():
 :param last_update: If provided, the query is modified to ask only about bugs which have
 recent modifications; otherwise asks about all bugs.
 """
-query = self.get_bz_query(tracker, last_update)
-buglist = self.bz.query(query)
+
+buglist = []
+last_query_len = BUGZILLA_QUERY_LIMIT
+
+
+# this is a hotfix hack to work around the sudden config change in rhbz where the max
+# number of bugs returned for a query is 20
+# it seems to be working for now but may need more work going forward
+while last_query_len == BUGZILLA_QUERY_LIMIT:
+
+new_query = self.get_bz_query(tracker, last_update, offset=len(buglist))
+new_buglist = self.bz.query(new_query)
+buglist.extend(new_buglist)
+last_query_len = len(new_buglist)
+
 return buglist
 
 def query_prioritized(self) -> list[bzBug]:
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Hosting Options for QA Projects

[Tim Flink]

On Thu, 13 Aug 2020 14:25:28 -0700
Kevin Fenzi  wrote:

> On Thu, Aug 13, 2020 at 03:04:51PM -0600, Tim Flink wrote:
> > From what I understand, communishift as it was originally planned is
> > not likely to be a thing going forward. We had been planning to use
> >  
> 
> Yeah, it's not really clear. It's complicated by the fact that the
> datacenter where it's at needs a networking revamp in order to bring
> it back up. :( 
> 
> > communishift to host a number of QA projects and we're looking for
> > new options going forward. Those projects are:
> > 
> > 
> > Testdays [1] is an app that we use to help run test days. It used to
> > live on the old cloud and is currently hosted outside Fedora's
> > infra on a machine we have access to; the plan was to move it to
> > communishift once that was available.
> > 
> > The packager dashboard [2] was recently announced for testing on
> > devel@. It and its backend service, oraculum, live on the same
> > external machine as the testdays app.  
> 
> I think both of the above are things that we could run in our prod/stg
> openshift instances under the 'we provide the platform, you handle the
> day to day app and development issues'. 

That kind of a setup would work for us.

> I think both of these are pretty small resource wise too?

Testdays is very small, yes.

Oraculum, on the other hand, is less small because it's looking for and
storing a bunch of data. That being said, I'm not sure what you would
consider large.

> > We're also looking into a TCMS to replace our mediawiki-based setup
> > - this isn't an immediate need but our plan was to host test
> > instances on communishift for our evaluation.
> > 
> > 
> > Do folks here have suggestions on what we could do about hosting for
> > projects like these?  
> 
> Thats harder, but I think if you narrowed it down to one or two and
> wanted to get wider testing with them, we could probibly set you up
> with some aws instances? This would be completely seperate from the
> rest of infra and you will be responsible for everything with them
> (configuration, updates, etc). But that might be fine for a short term
> testing thing. 

At the moment, we're only looking at one option - kiwitcms [1].
Unfortunately, there isn't much out there that comes close to doing
what we need it to do.

[1] https://kiwitcms.org/

What would be the best way to go about requesting an AWS instance?
Infra ticket, I assume? Would the setup for the test instance need to
be in the ansible repo?

Thanks,

Tim


pgpOeGms6nZrr.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Hosting Options for QA Projects

[Tim Flink]

>From what I understand, communishift as it was originally planned is
not likely to be a thing going forward. We had been planning to use
communishift to host a number of QA projects and we're looking for new
options going forward. Those projects are:


Testdays [1] is an app that we use to help run test days. It used to
live on the old cloud and is currently hosted outside Fedora's infra on
a machine we have access to; the plan was to move it to communishift
once that was available.

The packager dashboard [2] was recently announced for testing on devel@.
It and its backend service, oraculum, live on the same external machine
as the testdays app.

We're also looking into a TCMS to replace our mediawiki-based setup -
this isn't an immediate need but our plan was to host test instances on
communishift for our evaluation.


Do folks here have suggestions on what we could do about hosting for
projects like these?

Thanks,

Tim

[1] https://testdays.fedorainfracloud.org/events
[2] https://packager.fedorainfracloud.org/


pgpEvkTprP2FB.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Bot Account for Bodhi Karma

[Tim Flink]

After the Flock talk from the Facebook folks, they approached QA about
the idea of submitting karma and/or test results from their internal
testing as a bot account.

Is this something that is possible? I know that we have done bot
accounts at various points in time but I don't remember if automated
karma is allowed or what the other restrictions on that were.

Thanks,

Tim


pgpJe3JMCYI4P.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: FBR: update libtaskotron on production taskotron workers

[Tim Flink]

I've updated the libtaskotron packages for production taskotron, thanks
for the +1s

Tim

On Fri, 26 Apr 2019 14:10:36 -0600
Tim Flink  wrote:

> It turns out that we've been getting rpmlintrc files from the cgit
> interface over dist-git which went away earlier this week. As a
> consequence, package-specific rpmlintrc files are not being used
> anymore.
> 
> The fix itself is simple:
> https://pagure.io/taskotron/libtaskotron/c/0e0c12a5dd6255ad44a123e1ed1287425743fd8d?branch=develop
> 
> We've already done a release and made builds in koji. I'd like to
> upgrade the production workers so that per-package rpmlintrc files
> start working again.
> 
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1254705
> 
> Tim



pgpWrmLyKrans.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

FBR: update libtaskotron on production taskotron workers

[Tim Flink]

It turns out that we've been getting rpmlintrc files from the cgit
interface over dist-git which went away earlier this week. As a
consequence, package-specific rpmlintrc files are not being used
anymore.

The fix itself is simple:
https://pagure.io/taskotron/libtaskotron/c/0e0c12a5dd6255ad44a123e1ed1287425743fd8d?branch=develop

We've already done a release and made builds in koji. I'd like to
upgrade the production workers so that per-package rpmlintrc files
start working again.

https://koji.fedoraproject.org/koji/buildinfo?buildID=1254705

Tim


pgpeJUxYTtoAa.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Planned Outage: Taskotron 2019-04-09 16:00 UTC

[Tim Flink]

There will be an outage starting at 2019-04-09 16:00 UTC, which will
last approximately 4 hours.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run:

date -d '2019-04-09 16:00UTC'

Reason for outage:

Major upgrade to Taskotron

Affected Services:

Taskotron
Bodhi (displaying results)
Greenwave

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/7686

Please join #fedora-admin or #fedora-noc on irc.freenode.net or add
comments to the ticket for this outage above.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [FBR] proxies: fix logic for websockets in reversepassproxy

[Tim Flink]

There was a bit of a miscommunication and a fix for this was pushed to
ansible while I was waiting for +1s on this and it's no longer needed.

I'll cleanup the template once freeze lifts since it's working now.

Tim



On Thu, 28 Mar 2019 15:07:55 -0600
Tim Flink  wrote:

> The last patch I did for this added some bits that didn't need to be
> there and made a bad assumption about the default value for
> remotepath for the reversepassproxy.conf template.
> 
> This ended up with unneccesarry complication in the ws balancers and a
> unintended RewriteCond for any declared reversepassproxy that didn't
> redefine remotepath.
> 
> This patch fixes the bad assumptions and removes the cruft that didn't
> actually do or fix anything
> ---
>  roles/httpd/reverseproxy/templates/reversepassproxy.conf | 10
> +- 1 file changed, 1 insertion(+), 9 deletions(-)
> 
> diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.conf
> b/roles/httpd/reverseproxy/templates/reversepassproxy.conf index
> 1e4afe0..1ea0b97 100644 ---
> a/roles/httpd/reverseproxy/templates/reversepassproxy.conf +++
> b/roles/httpd/reverseproxy/templates/reversepassproxy.conf @@ -29,25
> +29,17 @@ SSLProxyEngine On  "balancer://{{balancer_name}}-websocket"> {% for member in
> balancer_members %} {% if
> http_not_https_yes_this_is_insecure_and_i_feel_bad %}
> -{% if remotepath is defined %}
> -BalancerMember "ws://{{ member }}{{ remotepath }}
> -{% else %}
>  BalancerMember "ws://{{ member }}"
> -{% endif %}
> -{% else %}
> -{% if remotepath is defined %}
> -BalancerMember "wss://{{ member }}{{ remotepath }}
>  {% else %}
>  BalancerMember "wss://{{ member }}"
>  {% endif %}
> -{% endif %}
>{% endfor %}
>  
>  
>  RewriteEngine on
>  RewriteCond %{HTTP:Upgrade} ^WebSocket$ [NC]
>  RewriteCond %{HTTP:Connection} Upgrade [NC]
> -{% if remotepath is defined %}
> +{% if remotepath != "/" %}
>  RewriteCond %{REQUEST_URI} ^{{ remotepath }}/(.)*
>  {% endif %}
>  RewriteRule .*
> "balancer://{{ balancer_name }}-websocket%{REQUEST_URI}" [P]
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [FBR] proxies: fix logic for websockets in reversepassproxy (small change from first one)

[Tim Flink]

I forgot to make a change in the regexp for the RewriteCond - new patch
follows:

The last patch I did for this added some bits that didn't need to be there
and made a bad assumption about the default value for remotepath for the
reversepassproxy.conf template.

This ended up with unneccesarry complication in the ws balancers and a
unintended RewriteCond for any declared reversepassproxy that didn't
redefine remotepath.

This patch fixes the bad assumptions and removes the cruft that didn't
actually do or fix anything
---
 roles/httpd/reverseproxy/templates/reversepassproxy.conf | 12 ++--
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.conf 
b/roles/httpd/reverseproxy/templates/reversepassproxy.conf
index 1e4afe0..93baefd 100644
--- a/roles/httpd/reverseproxy/templates/reversepassproxy.conf
+++ b/roles/httpd/reverseproxy/templates/reversepassproxy.conf
@@ -29,26 +29,18 @@ SSLProxyEngine On
 
   {% for member in balancer_members %}
 {% if http_not_https_yes_this_is_insecure_and_i_feel_bad %}
-{% if remotepath is defined %}
-BalancerMember "ws://{{ member }}{{ remotepath }}
-{% else %}
 BalancerMember "ws://{{ member }}"
-{% endif %}
-{% else %}
-{% if remotepath is defined %}
-BalancerMember "wss://{{ member }}{{ remotepath }}
 {% else %}
 BalancerMember "wss://{{ member }}"
 {% endif %}
-{% endif %}
   {% endfor %}
 
 
 RewriteEngine on
 RewriteCond %{HTTP:Upgrade} ^WebSocket$ [NC]
 RewriteCond %{HTTP:Connection} Upgrade [NC]
-{% if remotepath is defined %}
-RewriteCond %{REQUEST_URI} ^{{ remotepath }}/(.)*
+{% if remotepath != "/" %}
+RewriteCond %{REQUEST_URI} ^{{ remotepath }}/(.*)
 {% endif %}
 RewriteRule .* "balancer://{{ balancer_name }}-websocket%{REQUEST_URI}" [P]
 
-- 
1.8.3.1
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

[FBR] proxies: fix logic for websockets in reversepassproxy

[Tim Flink]

The last patch I did for this added some bits that didn't need to be there
and made a bad assumption about the default value for remotepath for the
reversepassproxy.conf template.

This ended up with unneccesarry complication in the ws balancers and a
unintended RewriteCond for any declared reversepassproxy that didn't
redefine remotepath.

This patch fixes the bad assumptions and removes the cruft that didn't
actually do or fix anything
---
 roles/httpd/reverseproxy/templates/reversepassproxy.conf | 10 +-
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.conf 
b/roles/httpd/reverseproxy/templates/reversepassproxy.conf
index 1e4afe0..1ea0b97 100644
--- a/roles/httpd/reverseproxy/templates/reversepassproxy.conf
+++ b/roles/httpd/reverseproxy/templates/reversepassproxy.conf
@@ -29,25 +29,17 @@ SSLProxyEngine On
 
   {% for member in balancer_members %}
 {% if http_not_https_yes_this_is_insecure_and_i_feel_bad %}
-{% if remotepath is defined %}
-BalancerMember "ws://{{ member }}{{ remotepath }}
-{% else %}
 BalancerMember "ws://{{ member }}"
-{% endif %}
-{% else %}
-{% if remotepath is defined %}
-BalancerMember "wss://{{ member }}{{ remotepath }}
 {% else %}
 BalancerMember "wss://{{ member }}"
 {% endif %}
-{% endif %}
   {% endfor %}
 
 
 RewriteEngine on
 RewriteCond %{HTTP:Upgrade} ^WebSocket$ [NC]
 RewriteCond %{HTTP:Connection} Upgrade [NC]
-{% if remotepath is defined %}
+{% if remotepath != "/" %}
 RewriteCond %{REQUEST_URI} ^{{ remotepath }}/(.)*
 {% endif %}
 RewriteRule .* "balancer://{{ balancer_name }}-websocket%{REQUEST_URI}" [P]
-- 
1.8.3.1
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [FBR] proxies: adding rewritecond to reverseproxy for ws if remotepath exists

[Tim Flink]

Thanks for the +1s. Pushed to ansible repo and has been run on
proxies-stg.

Tim

On Fri, 22 Mar 2019 14:04:37 -0600
Tim Flink  wrote:

> I was hitting an issue where there were multiple reverseproxy
> instances configured for a single host and some of the rewrite rules
> were changing the request when they shouldn't be.
> 
> This patch adds a rewritecond to the websocket rewrite rule to make
> sure that the REQUEST_URI starts with $remotepath before it's
> rewritten. ---
>  roles/httpd/reverseproxy/templates/reversepassproxy.conf | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.conf
> b/roles/httpd/reverseproxy/templates/reversepassproxy.conf index
> 38950e4..1e4afe0 100644 ---
> a/roles/httpd/reverseproxy/templates/reversepassproxy.conf +++
> b/roles/httpd/reverseproxy/templates/reversepassproxy.conf @@ -47,6
> +47,9 @@ SSLProxyEngine On RewriteEngine on
>  RewriteCond %{HTTP:Upgrade} ^WebSocket$ [NC]
>  RewriteCond %{HTTP:Connection} Upgrade [NC]
> +{% if remotepath is defined %}
> +RewriteCond %{REQUEST_URI} ^{{ remotepath }}/(.)*
> +{% endif %}
>  RewriteRule .*
> "balancer://{{ balancer_name }}-websocket%{REQUEST_URI}" [P] 
>  
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

[FBR] proxies: adding rewritecond to reverseproxy for ws if remotepath exists

[Tim Flink]

I was hitting an issue where there were multiple reverseproxy instances
configured for a single host and some of the rewrite rules were changing
the request when they shouldn't be.

This patch adds a rewritecond to the websocket rewrite rule to make
sure that the REQUEST_URI starts with $remotepath before it's rewritten.
---
 roles/httpd/reverseproxy/templates/reversepassproxy.conf | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.conf
b/roles/httpd/reverseproxy/templates/reversepassproxy.conf index
38950e4..1e4afe0 100644 ---
a/roles/httpd/reverseproxy/templates/reversepassproxy.conf +++
b/roles/httpd/reverseproxy/templates/reversepassproxy.conf @@ -47,6
+47,9 @@ SSLProxyEngine On RewriteEngine on
 RewriteCond %{HTTP:Upgrade} ^WebSocket$ [NC]
 RewriteCond %{HTTP:Connection} Upgrade [NC]
+{% if remotepath is defined %}
+RewriteCond %{REQUEST_URI} ^{{ remotepath }}/(.)*
+{% endif %}
 RewriteRule .*
"balancer://{{ balancer_name }}-websocket%{REQUEST_URI}" [P] 
 
-- 
1.8.3.1
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: FBR: Adding remotepath to websocket balancers

[Tim Flink]

On Fri, 22 Mar 2019 16:17:55 +0100
Pierre-Yves Chibon  wrote:

> On Tue, Mar 19, 2019 at 04:48:11PM -0600, Tim Flink wrote:
> > The current template assumes that websockets are at the base of a
> > URL but that is not true for our buildmaster. This patch adds
> > remotepath to the end of the websocket url if remotepath is defined.
> > ---
> >  roles/httpd/reverseproxy/templates/reversepassproxy.conf | 8
> >  1 file changed, 8 insertions(+)
> > 
> > diff --git
> > a/roles/httpd/reverseproxy/templates/reversepassproxy.conf
> > b/roles/httpd/reverseproxy/templates/reversepassproxy.conf index
> > deec40b..6131b5d 100644 ---
> > a/roles/httpd/reverseproxy/templates/reversepassproxy.conf +++
> > b/roles/httpd/reverseproxy/templates/reversepassproxy.conf @@
> > -29,10 +29,18 @@ SSLProxyEngine On  > "balancer://{{balancer_name}}-websocket"> {% for member in
> > balancer_members %} {% if
> > http_not_https_yes_this_is_insecure_and_i_feel_bad %}
> > +{% if remotepath is defined %}
> > +BalancerMember "ws://{{ member }}{{ remotepath }}
> > +{% else %}
> >  BalancerMember "ws://{{ member }}"
> > +{% endif %}
> > +{% else %}
> > +{% if remotepath is defined %}
> > +BalancerMember "ws://{{ member }}{{ remotepath }}  
> 
> You're missing on 's' here: wss:// (the first if is about unsecure:
> ws, we're in the else where about secure ws, so wss).
> 
> >  {% else %}
> >  BalancerMember "wss://{{ member }}"
> >  {% endif %}
> > +{% endif %}
> >{% endfor %}
> >
> 
> One typo, otherwise +1 for me
> 
> 
> Pierre

Thanks for catching that and the +1. Patch is applied and I'm running
it on the stg proxy.

Tim
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

FBR: Adding remotepath to websocket balancers

[Tim Flink]

The current template assumes that websockets are at the base of a URL
but that is not true for our buildmaster. This patch adds remotepath
to the end of the websocket url if remotepath is defined.
---
 roles/httpd/reverseproxy/templates/reversepassproxy.conf | 8 
 1 file changed, 8 insertions(+)

diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.conf
b/roles/httpd/reverseproxy/templates/reversepassproxy.conf index
deec40b..6131b5d 100644 ---
a/roles/httpd/reverseproxy/templates/reversepassproxy.conf +++
b/roles/httpd/reverseproxy/templates/reversepassproxy.conf @@ -29,10
+29,18 @@ SSLProxyEngine On  {% for member in
balancer_members %} {% if
http_not_https_yes_this_is_insecure_and_i_feel_bad %}
+{% if remotepath is defined %}
+BalancerMember "ws://{{ member }}{{ remotepath }}
+{% else %}
 BalancerMember "ws://{{ member }}"
+{% endif %}
+{% else %}
+{% if remotepath is defined %}
+BalancerMember "ws://{{ member }}{{ remotepath }}
 {% else %}
 BalancerMember "wss://{{ member }}"
 {% endif %}
+{% endif %}
   {% endfor %}
 
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: How to get invite for joining Fi-apprentice group?

[Tim Flink]

On Tue, 21 Nov 2017 05:43:34 -
"Marut Pandya"  wrote:

> How do i join  fi-apprentice group?
> My login name is- pandyamarut 
> please send me invite. 

The getting started page is the best place to get started:

https://fedoraproject.org/wiki/Infrastructure/GettingStarted

Please keep in mind that many folks are busy and may not respond
immediately - patience is a virtue, especially since it's a holiday
week in the USA and some folk are away from the computer for more of
this week.

Thanks for your interest, I look forward to working with you.

Tim


pgprAnnHLWHO0.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: Add CSS handling to Taskotron artifacts serving

[Tim Flink]

On Fri, 2 Jun 2017 10:06:56 -0600
Tim Flink  wrote:

> The modularity-testing-framework tests use avocado to output nice HTML
> reports but due to the way that we store and serve the artifacts, the
> css doesn't load properly. An example (that should work post-fix) is:
> 
> https://taskotron.fedoraproject.org/artifacts/all/dedb65bc-4793-11e7-a421-5254008e42f6/task_output/avocado-result/html/results.html
> 
> I've fixed this in dev and stg already. I'd like to apply the fix to
> prod. I'd be removing the if statements from this commit.
> 
> https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles/taskotron/taskotron-master/templates/artifacts.conf.j2?id=5fb1ed978b83240b97f011d9160d6df2cffa29ac

I got distracted but this has now been applied to taskotron prod and
the avocado reports seem to be displaying properly now.

Tim


pgpG_xLnkI6Cc.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

FBR: Add CSS handling to Taskotron artifacts serving

[Tim Flink]

The modularity-testing-framework tests use avocado to output nice HTML
reports but due to the way that we store and serve the artifacts, the
css doesn't load properly. An example (that should work post-fix) is:

https://taskotron.fedoraproject.org/artifacts/all/dedb65bc-4793-11e7-a421-5254008e42f6/task_output/avocado-result/html/results.html

I've fixed this in dev and stg already. I'd like to apply the fix to
prod. I'd be removing the if statements from this commit.

https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles/taskotron/taskotron-master/templates/artifacts.conf.j2?id=5fb1ed978b83240b97f011d9160d6df2cffa29ac

+1s?

Tim


pgpeSZ223s5W2.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

[FBR] Add upstreamfirst.fedorainfracloud.org to DNS

[Tim Flink]

I'm planning to set up a pagure instance in the cloud as a place to
collect testcases that Red Hat will be releasing for upstream
consumption and collaborate on the tests as they change into a form that
could be accepted by Fedora package maintaners. The request is tracked
in the following ticket:

https://pagure.io/fedora-infrastructure/issue/6066


I'd like to add upstreamfirst.fedorainfracloud.org to DNS, build the
config and push it out to the DNS servers. That hostname would point to
one of the floating IPs we've allocated in openstack.

+1s?

Tim



pgpIqJ9Hneuj2.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Freeze Break Request: increase threads for resultsdb_frontend wsgi daemon

[Tim Flink]

Production resultsdb_frontend is also really slow and I'd like to
bump it's resources at the same time I bump resultsdb from the
other FBR.

+1s?

Tim

From 9757a01edfe14a2455f523d0ac7cb8cfb4e92f8e Mon Sep 17 00:00:00 2001
From: Tim Flink 
Date: Fri, 17 Mar 2017 16:23:12 +
Subject: [PATCH 2/2] bump resources for resultsdb_frontend as well

---
 .../taskotron/resultsdb-frontend/templates/resultsdb_frontend.conf.j2 | 4 
 1 file changed, 4 insertions(+)

diff --git 
a/roles/taskotron/resultsdb-frontend/templates/resultsdb_frontend.conf.j2 
b/roles/taskotron/resultsdb-frontend/templates/resultsdb_frontend.conf.j2
index 16fbc89..e71cf58 100644
--- a/roles/taskotron/resultsdb-frontend/templates/resultsdb_frontend.conf.j2
+++ b/roles/taskotron/resultsdb-frontend/templates/resultsdb_frontend.conf.j2
@@ -1,4 +1,8 @@
+{% if deployment_type in ['stg', 'prod'] %}
+WSGIDaemonProcess resultsdb_frontend user=apache group=apache threads=100 
processes=10
+{% else %}
 WSGIDaemonProcess resultsdb_frontend user=apache group=apache threads=5
+{% endif %}
 WSGIScriptAlias /{{ resultsdb_fe_endpoint }} 
/usr/share/resultsdb_frontend/resultsdb_frontend.wsgi
 WSGISocketPrefix run/wsgi
 
-- 
1.8.3.1


pgphtDWHZ5hvW.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Freeze Break Request: increase threads for resultsdb wsgi daemon

[Tim Flink]

Production resultsdb is still really slow and we're still seeing the
occasional error on result posting so I'd like to bump the resources
allocated to the wsgi app again.

+1s?

Tim

From 3d45155959cbdcde39c1a98a584a100b590761db Mon Sep 17 00:00:00 2001
From: Tim Flink 
Date: Fri, 17 Mar 2017 15:33:49 +
Subject: [PATCH] bumping resultsdb wsgi resources again

---
 roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git
a/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2
b/roles/taskotron/resultsdb-ba index 97e73b9..c3f4d5c 100644 ---
a/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2 +++
b/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2 @@ -1,5
+1,5 @@ {% if deployment_type in ['stg', 'prod'] %}
-WSGIDaemonProcess resultsdb user=apache group=apache threads=100
processes=5 +WSGIDaemonProcess resultsdb user=apache group=apache
threads=200 processes=20 {% else %}
 WSGIDaemonProcess resultsdb user=apache group=apache threads=5
 {% endif %}
-- 
1.8.3.1


pgp8_x67paMUo.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: Freeze Break Request: enable package-specific tasks for Taskotron production

[Tim Flink]

On Fri, 17 Mar 2017 09:37:07 +0100
Pierre-Yves Chibon  wrote:

> On Thu, Mar 16, 2017 at 12:44:37PM -0600, Tim Flink wrote:
> > This has been pending for a long time and now that the bureaucratic
> > stuff has been figured out and our dev/stg instances are running
> > well, we'd like to enable package-specific tasks in Taskotron
> > production.


This was pushed to production last night and everything seems to be
running fine.

Thanks,

Tim


pgpsZrRI7KyMv.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: Fix koji monitoring from noc02

[Tim Flink]

On Fri, 17 Mar 2017 13:28:31 +
Patrick Uiterwijk  wrote:

> Hi all,
> 
> So, our noc02 monitoring of Koji build targets broke because the
> modularity stuff pushed rawhide to the second page :).
> Can I get +1s to start testing for "infra" instead, which should match
> $release-infra and such?
> 
> Patrick


+1

Tim


pgpKhdwZf8J6z.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Freeze Break Request: Enabling Nested Virt for Production Taskotron Virthosts

[Tim Flink]

This is less important than the other FBR that I submitted but I'd like
to do it while things are paused for the other change.

I'd like to enable nested virt on the virthosts that run the Taskotron
production clients. What I'd be doing is:

 - adding the 'virthost' role in ansible
 - running the playbook for the Taskotron client virthosts
 - rebooting the machines (qa12.qa and qa13.qa)

The change is small and since jobs are already stopped, there should be
little impact other than nagios warnings if I forget to schedule
downtime :)

If something goes wrong, we can remove the nested virt enablement and
reboot the hosts again. If all else fails, we can sub in the dev and
stg virthosts because all of the Taskotron virthosts are pretty much
identical. I think that this is incredibly unlikely, though.

+1s?

Tim


pgpIsmKaVJg6I.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: Freeze Break Request: increase threads on resultsdb wsgi daemon

[Tim Flink]

On Wed, 15 Mar 2017 06:47:40 -0600
Tim Flink  wrote:

> Production resultsdb has started having issues in the last 12 hours or
> so that appear to be associated with increased load
> (https://phab.qa.fedoraproject.org/T929).


Applied, thanks for the +1s

Tim


pgpDV7plcfW7U.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: Freeze Break Request: increase threads on resultsdb wsgi daemon

[Tim Flink]

On Wed, 15 Mar 2017 06:47:40 -0600
Tim Flink  wrote:

> Production resultsdb has started having issues in the last 12 hours or
> so that appear to be associated with increased load
> (https://phab.qa.fedoraproject.org/T929).
> 
> As a quick patch to keep things running while we investigate more, I
> changed the resultsdb config on the machine and want to make the
> corresponding change in git.

Turns out, that config change wasn't enough so I'm bumping it once more
before applying it in git and roping stg into the change so that it can
be tested before deploying to prod.

Sorry for the quick change like this, can I get +1s again?

Tim


From 727a6f75fbcf2ec0c4f2d4e4fa093c14bdde8069 Mon Sep 17 00:00:00 2001
From: Tim Flink 
Date: Wed, 15 Mar 2017 12:44:14 +
Subject: [PATCH] bumping wsgi daemon to 100 thread, 5 processes in  resultsdb
 prod, stg

---
 roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2 | 4 
 1 file changed, 4 insertions(+)

diff --git a/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2 
b/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2
index b8d6f3e..97e73b9 100644
--- a/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2
+++ b/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2
@@ -1,4 +1,8 @@
+{% if deployment_type in ['stg', 'prod'] %}
+WSGIDaemonProcess resultsdb user=apache group=apache threads=100 processes=5
+{% else %}
 WSGIDaemonProcess resultsdb user=apache group=apache threads=5
+{% endif %}
 WSGIScriptAlias /{{ resultsdb_endpoint }} /usr/share/resultsdb/resultsdb.wsgi
 WSGISocketPrefix run/wsgi
 
-- 
1.8.3.1




pgpkCOn6XlyWo.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Freeze Break Request: increase threads on resultsdb wsgi daemon

[Tim Flink]

Production resultsdb has started having issues in the last 12 hours or
so that appear to be associated with increased load
(https://phab.qa.fedoraproject.org/T929).

As a quick patch to keep things running while we investigate more, I
changed the resultsdb config on the machine and want to make the
corresponding change in git.

+1s?

Tim


From 7f3ba1b5294fd91ba60e8d4173f0c20af1e34bf2 Mon Sep 17 00:00:00 2001
From: Tim Flink 
Date: Wed, 15 Mar 2017 12:44:14 +
Subject: [PATCH] bumping wsgi daemon threads to 20 for resultsdb prod

---
 roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2 | 4 
 1 file changed, 4 insertions(+)

diff --git a/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2 
b/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2
index b8d6f3e..99175f7 100644
--- a/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2
+++ b/roles/taskotron/resultsdb-backend/templates/resultsdb.conf.j2
@@ -1,4 +1,8 @@
+{% if deployment_type in ['prod'] %}
+WSGIDaemonProcess resultsdb user=apache group=apache threads=20
+{% else %}
 WSGIDaemonProcess resultsdb user=apache group=apache threads=5
+{% endif %}
 WSGIScriptAlias /{{ resultsdb_endpoint }} /usr/share/resultsdb/resultsdb.wsgi
 WSGISocketPrefix run/wsgi
 
-- 
1.8.3.1



pgpNfmEZC9Sdk.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: Freeze Break: Add F24-Taskotron to the PXE menu on noc01

[Tim Flink]

On Tue, 8 Nov 2016 10:55:11 -0700
Tim Flink  wrote:

> We're starting to rebuild our dev instance (not under freeze) but I
> need to install F24 on one of the machines via PXE boot. The patch I
> would be applying and pushing out to noc01 is attached to this email.

Changed pushed to ansible.git and applied to noc01.

Thanks,

Tim


pgpfAod4_UD5H.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Freeze Break: Add F24-Taskotron to the PXE menu on noc01

[Tim Flink]

We're starting to rebuild our dev instance (not under freeze) but I
need to install F24 on one of the machines via PXE boot. The patch I
would be applying and pushing out to noc01 is attached to this email.

+1s?

Thanks,

Tim

From 6e08447feb34eabc9d01d4b752ede609de5db2b2 Mon Sep 17 00:00:00 2001
From: Tim Flink 
Date: Tue, 8 Nov 2016 17:52:21 +
Subject: [PATCH] adding F24 PXE menu entry for Taskotron hosts

---
 roles/tftp_server/files/default.noc01.phx2.fedoraproject.org | 5 +
 1 file changed, 5 insertions(+)

diff --git a/roles/tftp_server/files/default.noc01.phx2.fedoraproject.org b/roles/tftp_server/files/default.noc01.phx2.fedoraproject.org
index fc988eb..224dd84 100644
--- a/roles/tftp_server/files/default.noc01.phx2.fedoraproject.org
+++ b/roles/tftp_server/files/default.noc01.phx2.fedoraproject.org
@@ -83,4 +83,9 @@ LABEL Fed23-taskotron-x86_64
  KERNEL images/Fedora/23/x86_64/vmlinuz
  APPEND ks initrd=images/Fedora/23/x86_64/initrd.img method=http://10.5.126.23/pub/fedora/linux/releases/23/Server/x86_64/os/ ip=dhcp ks=http://10.5.126.23/repo/rhel/ks/hardware-f23-taskotron.cfg net.ifnames=0 ksdevice=eth0
 
+LABEL Fed24-taskotron-x86_64
+ MENU LABEL Fedora24-Taskotron-x86_64
+ KERNEL images/Fedora/24/x86_64/vmlinuz
+ APPEND ks initrd=images/Fedora/23/x86_64/initrd.img method=http://10.5.126.23/pub/fedora/linux/releases/24/Server/x86_64/os/ ip=dhcp ks=http://10.5.126.23/repo/rhel/ks/hardware-f24-taskotron.cfg net.ifnames=0 ksdevice=eth0
+
 MENU end
-- 
1.8.3.1



pgpukdRgUABGa.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: Freeze break request: Fix blocker bug proposal

[Tim Flink]

On Mon, 3 Oct 2016 11:29:12 -0600
Tim Flink  wrote:

> There was an issue reported last week about using the blockerbugs app
> to propose new blocker bugs.
> 
> It turned out to be a simple fix and a new build has been deployed to
> stg.
> 
> Can I get +1s to update production blockerbugs? The change to the app
> itself is:
> 
> https://git.fedorahosted.org/cgit/blockerbugs.git/commit/?id=f3fd68156be72bad79eb87b4e4380c2c355a40f3

The production blockerbugs instances have been updated to the new
version for this fix.

Thanks,

Tim


pgplb6ctfnTqC.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Freeze break request: Fix blocker bug proposal

[Tim Flink]

There was an issue reported last week about using the blockerbugs app
to propose new blocker bugs.

It turned out to be a simple fix and a new build has been deployed to
stg.

Can I get +1s to update production blockerbugs? The change to the app
itself is:

https://git.fedorahosted.org/cgit/blockerbugs.git/commit/?id=f3fd68156be72bad79eb87b4e4380c2c355a40f3


Tim


pgp2zlgnaPDki.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: Freeze Break Request (Retroactive): Adding new staging database host for testing in qa

[Tim Flink]

On Wed, 10 Aug 2016 09:10:48 -0600
Kevin Fenzi  wrote:

> On Wed, 10 Aug 2016 16:56:20 +0200
> Tim Flink  wrote:
> 
> > I completely forgot that we were in Alpha freeze earlier today and
> > made some changes without requesting a freeze break. The changes
> > shouldn't impact production in any meaningful way but I can back all
> > the changes out if need be.
> > 
> > In summary, I did the following:
> >  - created new dns entries for db-qa-stg01.qa
> >  - added a new db-qa-stg01.qa host to inventory
> >  - changed the postgres_server role to support dnf for f22+
> >  - added the new db-qa-stg01.qa host to the postgres-server playbook
> > 
> > I've attached a diff to this email for review.  
> 
> Most of that doesn't affect prod, but it will need a master run with
> iptables tag to update the staging blocking in prod phx2 hosts. ;) 
> 
> +1 here in any case, and I can run the iptables run if we get more +1s

That's not necessary in my mind. We primarially created the host to see
if a performance issue we were seeing in postgres was fixed in 9.5.

Now that the tests are done, it's not a problem to wait until after
freeze is over before we start using it for other things.

Tim


pgpvc1I0l0hER.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Freeze Break Request (Retroactive): Adding new staging database host for testing in qa

[Tim Flink]

I completely forgot that we were in Alpha freeze earlier today and made
some changes without requesting a freeze break. The changes shouldn't
impact production in any meaningful way but I can back all the changes
out if need be.

In summary, I did the following:
 - created new dns entries for db-qa-stg01.qa
 - added a new db-qa-stg01.qa host to inventory
 - changed the postgres_server role to support dnf for f22+
 - added the new db-qa-stg01.qa host to the postgres-server playbook

I've attached a diff to this email for review.

Tim

diff --git a/inventory/host_vars/db-qa-stg01.qa.fedoraproject.org b/inventory/host_vars/db-qa-stg01.qa.fedoraproject.org
new file mode 100644
index 000..3b82df2
--- /dev/null
+++ b/inventory/host_vars/db-qa-stg01.qa.fedoraproject.org
@@ -0,0 +1,75 @@
+---
+
+# general
+
+
+datacenter: phx2
+fas_client_groups: sysadmin-qa,sysadmin-noc,sysadmin-veteran
+
+
+
+# networking
+
+
+nm: 255.255.255.0
+gw: 10.5.124.254
+dns: 10.5.126.21
+eth0_ip: 10.5.124.145
+eth0_nm: 255.255.255.128
+
+
+
+# install
+
+
+ks_url: http://10.5.126.23/repo/rhel/ks/kvm-fedora-24
+ks_repo: http://10.5.126.23/pub/fedora/linux/releases/24/Server/x86_64/os/
+volgroup: /dev/vg_guests
+sudoers: "{{ private }}/files/sudo/qavirt-sudoers"
+vmhost: virthost-comm04.qa.fedoraproject.org
+
+
+
+# virtual machine
+
+
+# These are normally group variables, but in this case db servers are often different
+lvm_size: 30
+mem_size: 8192
+num_cpus: 2
+tcp_ports: [ 5432, 443 ]
+
+
+
+# database details
+
+
+# This is a generic list, monitored by collectd
+databases:
+- postgres
+- resultsdb
+
+# This is a more strict list, to be made publicly available
+dbs_to_backup:
+- postgres
+#- buildmaster
+#- buildmaster_dev
+#- buildmaster_stg
+#- execdb
+#- execdb_stg
+#- execdb_dev
+## these names are also stored as host vars 'openqa_dbname',
+## make sure to keep in sync
+#- openqa
+#- openqa-stg
+#- resultsdb
+#- resultsdb_stg
+#- resultsdb_dev
+
+
+# kernel SHMMAX value
+kernel_shmmax: 68719476736
+
+host_backup_targets: ['/backups']
+shared_buffers: "2GB"
+effective_cache_size: "6GB"
diff --git a/inventory/inventory b/inventory/inventory
index f520a28..25430a3 100644
--- a/inventory/inventory
+++ b/inventory/inventory
@@ -273,6 +273,7 @@ db-s390-koji01.qa.fedoraproject.org
 db-arm-koji01.qa.fedoraproject.org
 db-ppc-koji01.ppc.fedoraproject.org
 db-qa01.qa.fedoraproject.org
+db-qastg01.qa.fedoraproject.org
 
 [dbserver-stg]
 db-fas01.stg.phx2.fedoraproject.org
@@ -681,6 +682,7 @@ kojipkgs01.phx2.fedoraproject.org
 ns03.phx2.fedoraproject.org
 ns04.phx2.fedoraproject.org
 db-qa01.qa.fedoraproject.org
+db-qa-stg01.qa.fedoraproject.org
 proxy01.phx2.fedoraproject.org
 proxy10.phx2.fedoraproject.org
 openqa-stg01.qa.fedoraproject.org
diff --git a/playbooks/groups/postgresql-server.yml b/playbooks/groups/postgresql-server.yml
index 6cb74fe..49ed976 100644
--- a/playbooks/groups/postgresql-server.yml
+++ b/playbooks/groups/postgresql-server.yml
@@ -2,12 +2,12 @@
 # NOTE: should be used with --limit most of the time
 # NOTE: most of these vars_path come from group_vars/backup_server or from hostvars
 
-- include: "/srv/web/infra/ansible/playbooks/include/virt-create.yml myhosts=db-datanommer02.phx2.fedoraproject.org:db-qa01.qa.fedoraproject.org:db-koji01.phx2.fedoraproject.org:db-fas01.stg.phx2.fedoraproject.org:db-fas01.phx2.fedoraproject.org:db01.phx2.fedoraproject.org:db01.stg.phx2.fedoraproject.org:db-s390-koji01.qa.fedoraproject.org:db-arm-koji01.qa.fedoraproject.org:db-ppc-koji01.ppc.fedoraproject.org"
+- include: "/srv/web/infra/ansible/playbooks/include/virt-create.yml myhosts=db-datanommer02.phx2.fedoraproject.org:db-qa01.qa.fedoraproject.org:db-koji01.phx2.fedoraproject.org:db-fas01.stg.phx2.fedoraproject.org:db-fas01.phx2.fedoraproject.org:db01.phx2.fedoraproject.org:db01.stg.phx2.fedoraproject.org:db-s390-koji01.qa.fedoraproject.org:db-arm-koji01.qa.fedoraproject.org:db-ppc-koji01.ppc.fedoraproject.org:db-qa-stg01.qa.fedoraproject.org"
 
 # Once the instance exists, configure it.
 
 - name: configure postgresql server system
-  hosts: db-datanommer02.phx2.fedoraproject.org:db-qa01.qa.fedoraproject.org:db-koji01.phx2.fedoraproject.org:db-fas01.stg.phx2.fedoraproject.org:db-fas01.phx2.fedoraproject.org:db01.phx2.fedoraproject.org:db01.stg.phx2.fedoraproject.org:db-s390-koji01.qa.fedoraproject.org:db-arm-koji01.qa.fedoraproject.or

Re: Bugzilla+fedmsg, 2016-03-29

[Tim Flink]

On Tue, 29 Mar 2016 14:17:33 -0400
Ralph Bean  wrote:

> On Thu, Mar 24, 2016 at 10:33:16AM -0400, Ralph Bean wrote:
> > There's a Bugzilla outage scheduled for 00:00 2016-03-29 UTC.
> > 
> > If everything goes to plan, bugzilla should start transmitting
> > messages about changes to bug tickets shortly after this outage is
> > complete.  We have a mediator service running and ready that should
> > receive those and rebroadcast them to our fedmsg bus.
> > 
> > Please keep party streamers and confetti on standby.  
> 
> This is all in place and working now.
> 
> As a point for discussion (maybe we can take it up in the Thursday
> meeting or here on list):  now that we have the data stream, what
> would we like to do with it?
> 
> I know in the past we wanted it for awarding badges in badges.fp.o.
> It will be useful for preparing metrics for presentations at Flock,
> etc..  The fedora-packages webapp currently doesn't cache bugzilla
> information because it needs a fedmsg event to invalidate the cache --
> but now we have one!  Same goes for fedora-hubs in the future.
> 
> What else can we come up with?

QA has been planning to rework the blockerbugs app but figured it made
more sense to wait for bugzilla-fedmsg instead of having to do things
twice. I'd really like to see the app be a bit more reactive and
event-driven instead of just syncing every X minutes.

We don't have a timeline on this right now since Taskotron is a
higher immediate priority but it's definitely on our radar.

Tim


pgpWX40DxDewG.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Bugzilla+fedmsg, 2016-03-29

[Tim Flink]

On Thu, 24 Mar 2016 10:33:16 -0400
Ralph Bean  wrote:

> There's a Bugzilla outage scheduled for 00:00 2016-03-29 UTC.
> 
> If everything goes to plan, bugzilla should start transmitting
> messages about changes to bug tickets shortly after this outage is
> complete.  We have a mediator service running and ready that should
> receive those and rebroadcast them to our fedmsg bus.
> 
> Please keep party streamers and confetti on standby.

I'm a little worried that the amount of real streamers and confetti
that event deserves would be a huge environmental issue.

Time to start figuring out what "virtual" confetti means :)

Tim


pgpL2dKBexEQB.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze break: cachefilesd

[Tim Flink]

On Mon, 21 Mar 2016 16:08:41 -0600
Kevin Fenzi  wrote:

+1 sounds reasonable and something that will be very useful to have
sooner than later.

Tim

> Greetings. 
> 
> I'd like to take download05 out of dns and test out using cachefilesd
> on it. We tried this a number of years ago with rhel6 and ran into
> problems, but perhaps rhel7 might be more stable. 
> 
> Basically this would be setting up a 500GB cache on the server and
> then caching 500GB of stuff from nfs. In theory this would reduce
> load on the backend NFS server. There is a pending netapp upgrade and
> storage folks would need to throttle our access while the upgrade was
> happening, so the more load we can take off the filer, the better it
> will be when we have to do the upgrade. 
> 
> I'd take download05 out of dns (so no one should hit it), setup
> cachefilesd and test. If all goes well on testing we could re-add it
> and see if handles normal load. If not or if it cannot handle regular
> load we can just repave it and put it back the way it is now. 
> 
> 4 download machines should be fine for the normal load we have, and
> this testing should long be done before the end of the week when we
> might need to stage Alpha. 
> 
> Thoughts? +1s? 
> 
> kevin



pgprHsskRHyQ9.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: [PATCH] [wiki] Disable captcha for logged in users and on login

[Tim Flink]

On Thu, 17 Mar 2016 17:48:39 +
Patrick Uiterwijk  wrote:

> Signed-off-by: Patrick Uiterwijk 
> ---
>  roles/mediawiki/templates/LocalSettings.php.fp.j2 | 7 +++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/roles/mediawiki/templates/LocalSettings.php.fp.j2
> b/roles/mediawiki/templates/LocalSettings.php.fp.j2 index
> 91edefb..a6c10b4 100644 ---
> a/roles/mediawiki/templates/LocalSettings.php.fp.j2 +++
> b/roles/mediawiki/templates/LocalSettings.php.fp.j2 @@ -33,6 +33,12
> @@ $wgCaptchaClass = 'SimpleCaptcha'; #$wgCaptchaDirectoryLevels = 0;
>  #$wgCaptchaSecret = "{{ mediawikiCaptchaKey }}";
>  
> +$wgCaptchaTriggers['edit']  = true; 
> +$wgCaptchaTriggers['create']= true; 
> +$wgCaptchaTriggers['addurl']= true; 
> +$wgCaptchaTriggers['createaccount'] = true;
> +$wgCaptchaTriggers['badlogin']  = false;
> +
>  $wgRawHtml = false;
>  $wgProto = "https";
>  {% if env == "staging" %}
> @@ -76,6 +82,7 @@ $wgMimeDetectorCommand= "file -bi";
>  #$wgGroupPermissions['user' ]['delete']= true;
>  
>  $wgGroupPermissions['*']['createaccount'] = false;
> +$wgGroupPermissions['user']['skipcaptcha'] = true;
>  
>  # HNP Can't manage the interwiki right... - Nigel
>  $wgGroupPermissions['*']['interwiki'] = false;

+1 here also. Like pingou said, it looks fine and it's easy to revert
if something bad happens.

Tim


pgpj4cYQ8yvrJ.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze Break Request: Let openqa01.qa publish to the fedmsg bus

[Tim Flink]

On Thu, 10 Mar 2016 16:56:15 -0500
Ralph Bean  wrote:

+1 from me as well.

Tim



pgpxBH5KMFi9G.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Group vars for tasktotron- prod stg and dev

[Tim Flink]

On Wed, 2 Dec 2015 10:22:39 -0500
David Shier  wrote:

> Tim-
> 
> Thanks for the detailed reply! I will review and make the adjustments.
> 
> In the list there are entries for the below that are flagged as
> needing group vars updates:
> 
> taskotron-dev
> taskotron-dev-client-hosts
> taskotron-dev-clients
> taskotron-prod
> taskotron-prod-clients
> taskotron-stg
> taskotron-stg-clients
> 
> I planned on updating all of those, but figured I would start with
> this piece. If there are other pieces that are not in the list let me
> know and I will add them to my list of vars to update.

resultsdb-* are also part of taskotron.

I'd suggest skipping the taskotron-*-clients playbooks, they're not
long for this earth and unless there's a huge urgency in getting 100%
coverage with csi vars, I don't think it'd be worth the time.

Tim

> Dave Shier / odin2016
> 
> 
> On 12/02/2015 10:14 AM, Tim Flink wrote:
> > On Thu, 26 Nov 2015 17:17:46 -0500
> > David Shier  wrote:
> >
> > Comments inline for the dev patch, stg and prod have similar
> > comments.
> >
> > +csi_security_category: Moderate
> > +csi_primary_contact: #fedora-admin infrastruct...@fedoraproject.org
> > +csi_purpose: Run automated tasks, such as builds on fedora
> > components and report results of those task runs.
> >
> > Taskotron does not run builds. I suppose it could but that would
> > likely overlap with Koji and/or copr.
> >
> > A better way to put it would be:
> > Runs automated tasks against Fedora components triggered by fedmsgs
> > and reports results of those tasks to resultsdb.
> >
> > The difference between the three systems is:
> >
> >   * dev is where we try out new features, often before they're
> > ready for general consumption
> >
> >   * stg is very close to prod and is used to verify that changes
> > work before they're deployed to prod
> >
> >   * prod is the production system which most folks interface with
> >
> > +csi_relationship: |
> > +
> > +The system is made up of several components:
> > +
> > +- trigger
> > +
> > +- task execution system
> > +  this is a master/slave system, currently using buildbot
> > +
> > +- results storage (in resultsdb)
> > +
> > +- mirror task git repos
> >
> > I'm a little fuzzy on what's supposed to be in this field. The
> > taskotron-{dev,stg,prod} playbooks are responsible for the following
> > parts of the Taskotron system:
> >
> >   * run the trigger program which listens for fedmsgs which kick off
> > tasks
> >
> >   * run the buildmaster
> >
> >   * host the static landing page at taskotron.fp.o
> >
> > +- requires access to the git mirror(s), builders (koji, copr) and
> > +resultsdb
> >
> > This is not quite right. A more accurate description of the
> > dependencies would be:
> >
> > requires access to kojihub, bodhi, fedmsg hub, dist-git and the
> > upstream task repositories.
> >
> > Are we planning to fill in the CSI vars for the other Taskotron
> > related playbooks or is this meant to describe the system in its
> > entirety?
> >
> > Thanks for getting this started,
> >
> > Tim
> >
> >
> > ___
> > infrastructure mailing list
> > infrastructure@lists.fedoraproject.org
> > http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org
> 



pgp85XbWtfWp9.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Group vars for tasktotron- prod stg and dev

[Tim Flink]

On Thu, 26 Nov 2015 17:17:46 -0500
David Shier  wrote:

Comments inline for the dev patch, stg and prod have similar comments.

+csi_security_category: Moderate
+csi_primary_contact: #fedora-admin infrastruct...@fedoraproject.org
+csi_purpose: Run automated tasks, such as builds on fedora components
and report results of those task runs.

Taskotron does not run builds. I suppose it could but that would likely
overlap with Koji and/or copr.

A better way to put it would be:
Runs automated tasks against Fedora components triggered by fedmsgs and
reports results of those tasks to resultsdb.

The difference between the three systems is:

 * dev is where we try out new features, often before they're ready for
   general consumption

 * stg is very close to prod and is used to verify that changes work
   before they're deployed to prod

 * prod is the production system which most folks interface with

+csi_relationship: | 
+
+The system is made up of several components:
+ 
+- trigger
+
+- task execution system
+  this is a master/slave system, currently using buildbot
+
+- results storage (in resultsdb)
+
+- mirror task git repos

I'm a little fuzzy on what's supposed to be in this field. The
taskotron-{dev,stg,prod} playbooks are responsible for the following
parts of the Taskotron system:

 * run the trigger program which listens for fedmsgs which kick off
   tasks

 * run the buildmaster

 * host the static landing page at taskotron.fp.o

+- requires access to the git mirror(s), builders (koji, copr) and
+resultsdb

This is not quite right. A more accurate description of the
dependencies would be:

requires access to kojihub, bodhi, fedmsg hub, dist-git and the
upstream task repositories.

Are we planning to fill in the CSI vars for the other Taskotron related
playbooks or is this meant to describe the system in its entirety?

Thanks for getting this started,

Tim


pgpqeMvI3AZq0.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Freeze Break Request: Update Taskotron configuration

[Tim Flink]

This is a retroactive freeze break request - due to a misunderstanding
of the process, the change has already been pushed to the ansible repo.

Taskotron configuration needs an update as repo locations change once a
branched release is actually released. The change is to a single
configuration file and is easy to revert if it somehow causes problems.

+1s?

Tim

https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=53f80a2c80cde3f47e6adf6ea1327e05c6cc043c


pgpDP91G5TyCz.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Re: Freeze break request: sync out f23 atomic too

[Tim Flink]

On Mon, 2 Nov 2015 16:13:04 -0700
Kevin Fenzi  wrote:

> Looks like we are not syncing out the f23 updates/updates-testing
> repos yet.
> 
> +1s?

+1

Tim


pgpOlroWw2_8S.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Re: Freeze break request: rework developer rss feed updating

[Tim Flink]

On Mon, 2 Nov 2015 08:47:44 -0700
Kevin Fenzi  wrote:

> We ran into an issue today with the developer.fedoraproject.org site. 
> 
> Every once in a while it would show up with some really old blog
> content. 
> 
> Turns out we pull the site from github at :25 after the hour, and then
> only update the rss feeds at :45 after the hour. This results in 20min
> of having the old content from the repo and general confusion. 
> 
> This change removes the seperate cron job for this and just adds the
> update to the cron that pulls the site so they are always in sync. 
> 
> +1s?

+1

Tim




pgpPzIuN1xCvi.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Re: Freeze Break Request: add mirrorlist-ibiblio02

[Tim Flink]

On Sun, 25 Oct 2015 01:56:07 +0200
Patrick Uiterwijk  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> Hi all,
> 
> I would like to setup mirrorlist-ibiblio02, to add one more node that
> can take the mirrorlist load.
> Can I get +1s for the following patches (and the corresponding
> private repo patches)?

+1

Tim





pgpWfU9yCwf0Y.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Re: Freeze break requeest: add script to make OpenVPN always fix its routes

[Tim Flink]

On Thu, 22 Oct 2015 17:11:51 -0400 (EDT)
Patrick Uiterwijk  wrote:

> Can I get any +1s?
> This will guarantee that the routes will have been created when the
> OpenVPN link is up.

+1 - must protect the virtual kittens traveling over openvpn wires in
order to avoid the lava on the floor

Tim

> commit e8f63323b4e236629f438a082422d61a37cc95af
> Author: Patrick Uiterwijk 
> Date:   Thu Oct 22 21:06:38 2015 +
> 
> Add script to OpenVPN for VPN route fixing
> 
> This will make sure that always after a start/restart the
> VPN routes are created
> 
> Signed-off-by: Patrick Uiterwijk 
> 
> diff --git a/roles/openvpn/client/files/client.conf
> b/roles/openvpn/client/files/client.conf index abb5d03..704becb 100644
> --- a/roles/openvpn/client/files/client.conf
> +++ b/roles/openvpn/client/files/client.conf
> @@ -14,6 +14,9 @@ nobind
>  
>  persist-key
>  
> +up /etc/openvpn/fix-routes.sh
> +up-restart
> +
>  ca ca.crt
>  cert client.crt
>  key client.key
> diff --git a/roles/openvpn/client/files/fix-routes.sh
> b/roles/openvpn/client/files/fix-routes.sh new file mode 100644
> index 000..a08e519
> --- /dev/null
> +++ b/roles/openvpn/client/files/fix-routes.sh
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +# First check if this server is actually an OpenVPN client
> +if [ -f /etc/openvpn/client.crt ];
> +then
> +   # Now the magic line
> +   # This first checks whether there is a route, and if there
> isn't it will:
> +   # 1. Get the local machine's VPN IP (up to and including awk)
> +   # 2. Add a new route to 192.168.0.0/16 via that IP addres
> (from xargs on)
> +   # 3. Print "Fixed VPN" and exit with code 2 to indicate that
> it changed
> +   # Note: I've been told that the grep and awk can be in one
> command, and I believe that, but I find this clearer.
> +   (ip route show | grep '192.168.0.0/16') || ((ip route show |
> grep '192.168.0.' | awk '{print $1}' | xargs ip route add
> 192.168.0.0/16 via) && echo "Fixed VPN" && exit 2); +fi diff --git
> a/roles/openvpn/client/tasks/main.yml
> b/roles/openvpn/client/tasks/main.yml index 76817a2..67e44b1 100644
> --- a/roles/openvpn/client/tasks/main.yml +++
> b/roles/openvpn/client/tasks/main.yml @@ -17,6 +17,9 @@
>- { file: client.conf,
>dest: /etc/openvpn/openvpn.conf,
>mode: '0644' }
> +  - { file: fix-routes.sh,
> +  dest: /etc/openvpn/fix-routes.sh,
> +  mode: '0755' }
>- { file:
> "{{ private }}/files/vpn/openvpn/keys/{{ inventory_hostname }}.crt",
> dest: "/etc/openvpn/client.crt", mode: '0600' }
> 
> 
> 
> With kind regards,
> Patrick Uiterwijk
> Fedora Infra
> ___
> infrastructure mailing list
> infrastructure@lists.fedoraproject.org
> http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org



pgpjo9cDa6dPE.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Re: rbac_playbook fix for RHEL7

[Tim Flink]

On Tue, 29 Sep 2015 08:27:01 -0600
Tim Flink  wrote:

> Long story short, when the batcave upgrade happened on Friday we found
> out that rbac_playbook doesn't work on el7 due to an issue with
> nss-altfiles.
> 
> I figured out how to sidestep the issue by changing the approach that
> rbac_playbook takes. It used to get all the groups for the user
> running the script and check for an intersection between those groups
> and the required groups for the playbook being run.
> 
> The new version looks at the groups required for the playbook being
> run, gathers all the users in those groups and verifies that the user
> running rbac_playbook is in that list before proceeding.
> 
> I've included the changes below for security review before updating
> anything on batcave01

Thanks for the reviews. Code has been pushed to git, I've built a new
ansible_utils package and put that in the el7 infrastructure repo.

Tim



pgpXjgefbeofA.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/postorius/infrastructure@lists.fedoraproject.org

Re: rbac_playbook fix for RHEL7

[Tim Flink]

> Why the change here?  Why the addition of bash?

No changed, just a slightly botched diff. New, properly done diff at
the end of this email but that should be the only discrepancy.

> > +def can_run(acl, username, playbook_file):
> >  """ determines whether the current user is allowed to run a
> > playbook 
> >  :param acl: dictionary of acl information
> > -:param groups: groups of which the user is a member
> > +:param username: username of user running the utility
> >  :param playbook_file: playbook file that is being run
> >  :return: True if the user is authorized, False if unauthorized
> >  """
> > -# exact match quick route
> > +
> > +authorized_groups = acl[playbook_file]
> > +
> >  if playbook_file in acl:
> > -pb_groups = frozenset(acl[playbook_file])
> > -if groups.intersection(pb_groups):
> > +pb_authorized =
> > _get_playbook_authorized_users(authorized_groups)
> > +if username in pb_authorized:
> >  return True
> >  return False
> 
> It looks like this function used to return False if an invalid
> playbook_file was passed to it, but now it will raise a KeyError.  Is
> that intentional?  I see there is a test case for it, but the
> `rbac_playbook` function doesn't handle it.

KeyError now handled and test case added, thanks.

Tim


diff --git a/ansible_utils.spec b/ansible_utils.spec
index a340832..ba45a3d 100644
--- a/ansible_utils.spec
+++ b/ansible_utils.spec
@@ -20,12 +20,14 @@ Requires:   PyYAML
 BuildRequires:  python2-devel
 BuildRequires:  python-setuptools
 BuildRequires:  python-kitchen
+BuildRequires:  python-mock
 
 # python-dingus isn't built for el7, can't run tests
 %if 0%{?rhel} > 6
 BuildRequires:  pytest
 BuildRequires:  PyYAML
 BuildRequires:  python-dingus
+BuildRequires:  python-mock
 %endif
 
 # the cli uses argparse, which is not part of the standard libaray in
python2.6 diff --git a/ansible_utils/rbac_playbook.py
b/ansible_utils/rbac_playbook.py index c850b94..553225b 100755
--- a/ansible_utils/rbac_playbook.py
+++ b/ansible_utils/rbac_playbook.py
@@ -140,7 +140,7 @@ def generate_message(result, username,
playbook_name, args, checksum): :return: rendered string summarizing
rbac activity """
 subject = "[rbac-playbook] {0} {1} ran {2}".format(result,
-
username.pw_name,
+   username,
playbook_name)
 body = ['Details:']
 body.extend(['{0}: {1}'.format(key, value) for key, value in
args.iteritems()]) @@ -209,36 +209,60 @@ def
run_sudo_command(playbook_file, playbook_args): print "EXECV: %s %s" %
(executable, ' '.join(args)) os.execv(executable, args)
 
+def _get_username():
+"""Retrieve the username for the user which started execution of
rbac_playbook""" 
-def can_run(acl, groups, playbook_file):
+user = os.getlogin()
+username = pwd.getpwnam(user)
+return username.pw_name
+
+def _get_group_members(groupname):
+"""Find the members of a specific group
+
+:param groupname: name of group to find members of
+:return: list of usernames for members of the given group, empty
list if the group does not exist""" +
+group_data = None
+try:
+group_data = grp.getgrnam(groupname)
+except KeyError:
+return []
+
+return group_data.gr_mem
+
+def _get_playbook_authorized_users(grouplist):
+"""Retrieve a set of all users who are members of one or more
groups +
+:param grouplist: list of one or more group names
+:return: set of usernames corresponding to the union of members
for each group in the grouplist""" +
+userlist = []
+for groupname in grouplist:
+userlist.extend(_get_group_members(groupname))
+
+return set(userlist)
+
+def can_run(acl, username, playbook_file):
 """ determines whether the current user is allowed to run a
playbook 
 :param acl: dictionary of acl information
-:param groups: groups of which the user is a member
+:param username: username of user running the utility
 :param playbook_file: playbook file that is being run
-:return: True if the user is authorized, False if unauthorized
+:return: True if the user is authorized, False if unauthorized or
the
+ playbook filename is not in the acl information
 """
-# exact match quick route
+
+try:
+authorized_groups = acl[playbook_file]
+except KeyError:
+return False
+
 if playbook_file in acl:
-pb_groups = frozenset(acl[playbook_file])
-if groups.intersection(pb_groups):
+pb_authorized =
_get_playbook_authorized_users(authorized_groups)
+if username in pb_authorized:
 return True
 return False
 
-
-def get_groups():
-""" retrieve the groups of which the current user is currently a
member
-:return: (username,groups) where groups is a set of groups which
the current user is a member
-"""
-

rbac_playbook fix for RHEL7

[Tim Flink]

Long story short, when the batcave upgrade happened on Friday we found
out that rbac_playbook doesn't work on el7 due to an issue with
nss-altfiles.

I figured out how to sidestep the issue by changing the approach that
rbac_playbook takes. It used to get all the groups for the user running
the script and check for an intersection between those groups and the
required groups for the playbook being run.

The new version looks at the groups required for the playbook being
run, gathers all the users in those groups and verifies that the user
running rbac_playbook is in that list before proceeding.

I've included the changes below for security review before updating
anything on batcave01

Tim


diff --git a/ansible_utils.spec b/ansible_utils.spec
index 2e26a70..f097020 100644
--- a/ansible_utils.spec
+++ b/ansible_utils.spec
@@ -20,12 +20,14 @@ Requires:   PyYAML
 BuildRequires:  python2-devel
 BuildRequires:  python-setuptools
 BuildRequires:  python-kitchen
+BuildRequires:  python-mock
 
 # python-dingus isn't built for el7, can't run tests
 %if 0%{?rhel} > 6
 BuildRequires:  pytest
 BuildRequires:  PyYAML
 BuildRequires:  python-dingus
+BuildRequires:  python-mock
 %endif
 
 # the cli uses argparse, which is not part of the standard libaray in
python2.6 diff --git a/ansible_utils/rbac_playbook.py
b/ansible_utils/rbac_playbook.py index 42be23f..c6a952c 100755
--- a/ansible_utils/rbac_playbook.py
+++ b/ansible_utils/rbac_playbook.py
@@ -140,7 +140,7 @@ def generate_message(result, username,
playbook_name, args, checksum): :return: rendered string summarizing
rbac activity """
 subject = "[rbac-playbook] {0} {1} ran {2}".format(result,
-
username.pw_name,
+   username,
playbook_name)
 body = ['Details:']
 body.extend(['{0}: {1}'.format(key, value) for key, value in
args.iteritems()]) @@ -188,10 +188,13 @@ def
run_sudo_command(playbook_file, playbook_args): full_filename =
os.path.abspath(os.path.join(config['config']['PLAYBOOKS_DIR'],
playbook_file)) 
+python_args = ['cd', config['config']['ANSIBLE_DIR'], ';',
+   '/usr/bin/python2',
config['config']['ANSIBLE_PLAYBOOK'],
+   full_filename]
+python_args.extend(playbook_args)
 executable = '/usr/bin/sudo'
-args = ['-i', '/usr/bin/python2',
-config['config']['ANSIBLE_PLAYBOOK'], full_filename]
-args.extend(playbook_args)
+args = ['-i', '/bin/bash', '-i', '-c',
+' '.join(python_args)]
 
 # Note: (1) The function used here has to pass the environment to
 # ansible-playbook so that it can connect to the ssh-agent
correctly @@ -206,36 +209,56 @@ def run_sudo_command(playbook_file,
playbook_args): print "EXECV: %s %s" % (executable, ' '.join(args))
 os.execv(executable, args)
 
+def _get_username():
+"""Retrieve the username for the user which started execution of
rbac_playbook""" 
-def can_run(acl, groups, playbook_file):
+user = os.getlogin()
+username = pwd.getpwnam(user)
+return username.pw_name
+
+def _get_group_members(groupname):
+"""Find the members of a specific group
+
+:param groupname: name of group to find members of
+:return: list of usernames for members of the given group, empty
list if the group does not exist""" +
+group_data = None
+try:
+group_data = grp.getgrnam(groupname)
+except KeyError:
+return []
+
+return group_data.gr_mem
+
+def _get_playbook_authorized_users(grouplist):
+"""Retrieve a set of all users who are members of one or more
groups +
+:param grouplist: list of one or more group names
+:return: set of usernames corresponding to the union of members
for each group in the grouplist""" +
+userlist = []
+for groupname in grouplist:
+userlist.extend(_get_group_members(groupname))
+
+return set(userlist)
+
+def can_run(acl, username, playbook_file):
 """ determines whether the current user is allowed to run a
playbook 
 :param acl: dictionary of acl information
-:param groups: groups of which the user is a member
+:param username: username of user running the utility
 :param playbook_file: playbook file that is being run
 :return: True if the user is authorized, False if unauthorized
 """
-# exact match quick route
+
+authorized_groups = acl[playbook_file]
+
 if playbook_file in acl:
-pb_groups = frozenset(acl[playbook_file])
-if groups.intersection(pb_groups):
+pb_authorized =
_get_playbook_authorized_users(authorized_groups)
+if username in pb_authorized:
 return True
 return False
 
-
-def get_groups():
-""" retrieve the groups of which the current user is currently a
member
-:return: (username,groups) where groups is a set of groups which
the current user is a member
-"""
-username = os.getlogin()
-user = pwd.getpwnam(usern

Re: Freeze Break Request: add robots.txt to public facing taskotron instances

[Tim Flink]

On Wed, 5 Aug 2015 11:53:49 -0600
Tim Flink  wrote:



> I'd like to add a robots.txt to the public facing taskotron instances
> so that we can avoid this in the future. Rendered content of the
> robots.txt is at the end of this email.



New robots.txt applied to taskotron production and staging.

Thanks,

Tim


pgpNMI9iD6L96.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: add robots.txt to public facing taskotron instances

[Tim Flink]

For some reason, yahoo's bot has been hitting our taskotron instance
really hard all of a sudden - slowing it down enough to trigger nagios
alerts twice this week.

There is a robots.txt served by the buildmaster instance but since that
isn't at the webroot in our setup, bots may not read it.

I'd like to add a robots.txt to the public facing taskotron instances
so that we can avoid this in the future. Rendered content of the
robots.txt is at the end of this email.

I've already deployed the file to taskotron-dev which can be seen at:
http://taskotron-dev.fedoraproject.org/robots.txt

Proposed content would be at:
http://taskotron.stg.fedoraproject.org/robots.txt
http://taskotron.fedoraproject.org/robots.txt

Tim


User-agent: *
Disallow: /taskmaster/waterfall
Disallow: /taskmaster/builders
Disallow: /taskmaster/changes
Disallow: /taskmaster/buildslaves
Disallow: /taskmaster/schedulers
Disallow: /taskmaster/one_line_per_build
Disallow: /taskmaster/grid
Disallow: /taskmaster/tgrid
Disallow: /taskmaster/json


pgp5d1Dl6cQqg.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: bodhi2 plans

[Tim Flink]

On Mon, 3 Aug 2015 11:03:38 -0600
Kevin Fenzi  wrote:

> Greetings. 
> 
> So, in the releng meeting last week we discussed about landing bodhi2
> before the alpha freeze started, however there were just too many
> things still up in the air to do that, so we held off. 
> 
> However, I'd like to propose a concrete date to enable it and try and
> make sure we have everything we need lined up for making that
> work. :) 
> 
> My understanding of the current status: 
> (Luke: please correct me if anything is wrong or unclear here)
> 
> * We have bodhi2 web frontend up and running fine in staging: 
> https://admin.stg.fedoraproject.org/updates/
> (well, right now it's down as Luke is working on a update id bug there
> and is repopulating the db, but it was and will be up again soon). 
> 
> * We have a bodhi2 backend up in staging and it's successfully mashed
>   updates pushes. 
> 
> * bodhi2 isn't yet packaged in Fedora (it and it's deps are in a
> copr): http://copr.fedoraproject.org/coprs/lmacken/bodhi2/

I didn't think that there had been much testing of the APIs yet. Last I
heard, the API wasn't ready for testing - has that changed?


> So, to get that last bit we need: 
> 
> * API users. I thought we had a list somewhere, but I cannot seem to
>   find it. At least fedora-easy-karma and fedora-gooey-karma use it.
>   Are all of these using python-fedora for the api? Do we need to
>   update that and bodhi-client? 

libtaskotron, resultsdb and blockerbugs use the Bodhi API through
python-fedora. None of them have been tested against bodhi2.


> * We need bodhi2 in fedora/epel. Were we going to do it as a seperate
>   package, or just an update to the existing 'bodhi' package. I assume
>   maintainers would need the new client to do 'fedpkg update' ?
> 
> * Do we need any fedpkg changes? Or anything else? How do we want to
>   handle the switchover?
> 
> * Is taskotron all ready to go with bodhi2? or more changes there?

Not really. Preparing for bodhi2 has been on the back burner for us
since it didn't sound like production was going to be ready before F23
was released.

I'll try to take a look into what's left but the guy who has been doing
most of that work is on PTO until Flock.


> * We will need an outage to switch over. I assume the bodhi1->bodhi2
> db conversion shouldn't take too long? But once we convert and make
> live we are commmited. 
> 
> * Anything else I missed?
> 
> I'd like to propose we schedule the switchover/outage for 2015-08-19. 
> (Provided alpha doesn't slip). This will allow us a few days to
> recover from flock and get anything lined up we need to. 

How heavily has bodhi2 been tested by folks who aren't bodhi2
developers?  The idea of swapping out bodhi mid-release makes me a
little nervous since hiccups could affect the release process.

I realize there are ~3 weeks between the proposed date and F23 beta
freeze and that there are going to be bumps in the road no matter
when the switchover happens. However, the tester in me is thinking
more along the lines of what might go wrong in the near future if the
more corner-casey stuff (freeze etc.) is hit so soon after deployment,
especially if there's a bug that's taken down the frontend for at least
a couple hours in addition to needing a database wipe/repopulate 2
weeks before the proposed go-live date.

Tim


pgpz6PRb0MlOK.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break request: fix backups for qadevel.qa.fedoraproject.org

[Tim Flink]

On Mon, 3 Aug 2015 14:07:56 -0600
Kevin Fenzi  wrote:

> qadevel.qa.fedoraproject.org uses port 222 for ssh shell access. 
> 
> ansible is happy using port 222 just fine, but rdiff-backup isn't.
> It uses port 22 by default, and fails. 
> 
> The following patch should fix this issue. 
> 
> +1s? I would apply this, run the backup playbook and then watch
> tomorrow's backups to confirm they work. If not we could revert this. 

+1 from me

It's a local, easily revertible change and the backups aren't happening
as it is - I don't see much risk in the change.

Tim


pgp9oHcjmvgfs.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: Whitelist copr-be for fedmsg

[Tim Flink]

On Tue, 19 May 2015 11:18:05 -0400
Ralph Bean  wrote:

> copr-be.cloud.fedoraproject.org got a new IP in the upgrade going on
> today, so we need to change the whitelist listing for inbound fedmsg
> connections.
> 
> I've already pushed the change out, but can I get two retroactive +1s?

+1


pgpX46eyKvR8D.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break request: Update servers with new python-bugzilla

[Tim Flink]

On Mon, 13 Apr 2015 18:17:22 +0200
Patrick Uiterwijk  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> Hi,
> 
> I would like to ask for +1 to update all other servers running
> python-bugzilla. The list is below:
> 
> bvirthost08.phx2.fedoraproject.org
> osuosl01.fedoraproject.org
> fed-cloud13.cloud.fedoraproject.org
> busgateway01.phx2.fedoraproject.org
> resultsdb01.qa.fedoraproject.org
> kojipkgs01.phx2.fedoraproject.org
> qa05.qa.fedoraproject.org
> badges-backend01.stg.phx2.fedoraproject.org
> taskotron-client26.qa.fedoraproject.org
> db-fas01.stg.phx2.fedoraproject.org
> buildhw-01.phx2.fedoraproject.org
> buildvm-08.phx2.fedoraproject.org
> arm01-builder04.arm.fedoraproject.org
> arm02-builder04.arm.fedoraproject.org
> arm04-builder05.arm.fedoraproject.org
> hosted03.fedoraproject.org
> mailman02.phx2.fedoraproject.org
> taskotron-client13.qa.fedoraproject.org
> value01.phx2.fedoraproject.org
> download02.phx2.fedoraproject.org
> proxy04.fedoraproject.org
> taskotron-client23.qa.fedoraproject.org
> qa13.qa.fedoraproject.org
> branched-composer.phx2.fedoraproject.org
> buildhw-05.phx2.fedoraproject.org
> buildvm-13.phx2.fedoraproject.org
> arm01-builder03.arm.fedoraproject.org
> arm02-builder05.arm.fedoraproject.org
> arm04-builder06.arm.fedoraproject.org
> proxy05.fedoraproject.org
> sign-bridge01.phx2.fedoraproject.org
> nuancier02.phx2.fedoraproject.org
> taskotron-client20.qa.fedoraproject.org
> log01.phx2.fedoraproject.org
> fed-cloud09.cloud.fedoraproject.org
> coloamer01.fedoraproject.org
> wiki01.phx2.fedoraproject.org
> datagrepper01.stg.phx2.fedoraproject.org
> mailman01.stg.phx2.fedoraproject.org
> buildhw-12.phx2.fedoraproject.org
> buildvm-22.phx2.fedoraproject.org
> arm01-builder17.arm.fedoraproject.org
> arm02-builder16.arm.fedoraproject.org
> arm04-builder16.arm.fedoraproject.org
> smtp-mm-ib01.fedoraproject.org
> notifs-web02.phx2.fedoraproject.org
> copr-keygen.cloud.fedoraproject.org
> virthost-comm01.qa.fedoraproject.org
> virthost17.phx2.fedoraproject.org
> fedimg01.phx2.fedoraproject.org
> pkgdb02.phx2.fedoraproject.org
> sundries02.phx2.fedoraproject.org
> serverbeach09.fedoraproject.org
> packages03.phx2.fedoraproject.org
> taskotron-client29.qa.fedoraproject.org
> github2fedmsg01.stg.phx2.fedoraproject.org
> buildhw-03.phx2.fedoraproject.org
> buildvm-24.phx2.fedoraproject.org
> arm01-builder16.arm.fedoraproject.org
> arm02-builder18.arm.fedoraproject.org
> arm04-builder18.arm.fedoraproject.org
> smtp-mm-tummy01.fedoraproject.org
> elections02.phx2.fedoraproject.org
> taskotron-client19.qa.fedoraproject.org
> taskotron01.qa.fedoraproject.org
> hotness01.phx2.fedoraproject.org
> cloud-noc01.cloud.fedoraproject.org
> mirrorlist-phx2.phx2.fedoraproject.org
> dedicatedsolutions01.fedoraproject.org
> bastion01.phx2.fedoraproject.org
> lockbox-comm01.qa.fedoraproject.org
> fedoauth01.stg.phx2.fedoraproject.org
> proxy01.stg.phx2.fedoraproject.org
> buildvm-10.phx2.fedoraproject.org
> arm01-builder07.arm.fedoraproject.org
> arm02-builder08.arm.fedoraproject.org
> arm04-builder09.arm.fedoraproject.org
> ns02.fedoraproject.org
> paste01.phx2.fedoraproject.org
> lists-dev.cloud.fedoraproject.org
> taskotron-client12.qa.fedoraproject.org
> db-qa01.qa.fedoraproject.org
> fed-cloud11.cloud.fedoraproject.org
> tagger01.phx2.fedoraproject.org
> arm03-packager00.cloud.fedoraproject.org
> relepel01.phx2.fedoraproject.org
> buildhw-11.phx2.fedoraproject.org
> buildvm-21.phx2.fedoraproject.org
> arm01-builder18.arm.fedoraproject.org
> arm02-builder17.arm.fedoraproject.org
> arm04-builder17.arm.fedoraproject.org
> smtp-mm-coloamer01.fedoraproject.org
> copr-fe-dev.cloud.fedoraproject.org
> beaker01.qa.fedoraproject.org
> badges-web01.phx2.fedoraproject.org
> db-fas01.phx2.fedoraproject.org
> buildvmhost-11.phx2.fedoraproject.org
> fed-cloud01.cloud.fedoraproject.org
> virthost11.phx2.fedoraproject.org
> notifs-backend01.phx2.fedoraproject.org
> pagure-stg01.fedoraproject.org
> taskotron-client22.qa.fedoraproject.org
> qa07.qa.fedoraproject.org
> notifs-backend01.stg.phx2.fedoraproject.org
> buildhw-08.phx2.fedoraproject.org
> buildvm-18.phx2.fedoraproject.org
> arm01-builder13.arm.fedoraproject.org
> arm02-builder13.arm.fedoraproject.org
> arm04-builder13.arm.fedoraproject.org
> noc01.phx2.fedoraproject.org
> mailman01.phx2.fedoraproject.org
> taskotron-client14.qa.fedoraproject.org
> copr-fe.cloud.fedoraproject.org
> virthost08.phx2.fedoraproject.org
> download04.phx2.fedoraproject.org
> osuosl03.fedoraproject.org
> paste01.stg.phx2.fedoraproject.org
> host1plus01.fedoraproject.org
> download-rdu02.fedoraproject.org
> gallery01.stg.phx2.fedoraproject.org
> buildhw-02.phx2.fedoraproject.org
> buildvm-09.phx2.fedoraproject.org
> arm01-builder05.arm.fedoraproject.org
> arm02-builder07.arm.fedoraproject.org
> arm04-builder08.arm.fedoraproject.org
> proxy11.fedoraproject.org
> blockerbugs-dev.cloud.fedoraproject.org
> hosted04.fedoraproject.org
> ibiblio01.fedoraproject.org
> fed-cl

Re: Freeze break request: update python-bugzilla on bodhi*

[Tim Flink]

On Mon, 13 Apr 2015 15:44:42 +0200
Patrick Uiterwijk  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> Hi,
> 
> Red Hat Bugzilla got upgraded to a version that no longer
> has the Bug.get_bugs API call.
> There is a new python-bugzilla available that uses the new
> Bug.get call instead.
> Staging bodhi* has been upgraded last friday to use the new
> python-bugzilla.
> 
> I would like +1s to upgrade python-bugzilla in production,
> since Red Hat Bugzilla has been upgraded and as such attaching
> bugs to updates is currently broken.

+1 - the update been working fine in blockerbugs-stg since last
Wednesday

Is blockerbugs included in the list of systems to update to the new
version of python-bugzilla?

Tim


pgppbDMYm_Mot.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: add long query logging to db-qa01.qa

[Tim Flink]

On Mon, 6 Apr 2015 13:38:29 -0600
Tim Flink  wrote:

> On Mon, 6 Apr 2015 10:33:15 -0600
> Tim Flink  wrote:
>  
> > I'm not crazy about doing this during freeze but I'm worried that
> > the timeout problem will start affecting more than stg before long
> > and want to get this figured out before that happens.
> 
> As an update - I'm not ignoring the +1s, I just managed to stumble on
> something I can poke at outside of production when preparing to make
> the change on the db server.
> 
> I'm still hoping to avoid changing the db server during freeze, so I'm
> going to keep digging into the issue using my local setup now that I
> have something I can dig into.
> 
> For anyone who's interested in following along, the issue is being
> tracked as:
> 
> https://phab.qadevel.cloud.fedoraproject.org/T452

After much debugging and poking, I think that I've figured out why
Taskotron staging is having the issues that it is. The root is still
slow queries but instead of modifying the production database, there is
a way to change how our code is using those slow queries to keep
mod_wsgi from timing out and killing the requests.

Since we have a different approach to solving the problem (at least in
the short term), I won't be applying the patch to ansible or modifying
the database configuration during freeze.

Tim


pgp6cmrxDyvzZ.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: add long query logging to db-qa01.qa

[Tim Flink]

On Mon, 6 Apr 2015 10:33:15 -0600
Tim Flink  wrote:
 
> I'm not crazy about doing this during freeze but I'm worried that the
> timeout problem will start affecting more than stg before long and
> want to get this figured out before that happens.

As an update - I'm not ignoring the +1s, I just managed to stumble on
something I can poke at outside of production when preparing to make
the change on the db server.

I'm still hoping to avoid changing the db server during freeze, so I'm
going to keep digging into the issue using my local setup now that I
have something I can dig into.

For anyone who's interested in following along, the issue is being
tracked as:

https://phab.qadevel.cloud.fedoraproject.org/T452

Tim


pgpWmOqaD1TKD.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: add long query logging to db-qa01.qa

[Tim Flink]

We're continuing to have timeout issues in Taskotron staging and my
investigation so far seems to indicate the cause to be slow queries in
the database.

This change does require a database restart but due to the way that
Taskotron works, I can do this so that there are no lost jobs and no
significant downtime. I'd stop incoming jobs until all queues are empty
and record the jobs which would have been scheduled on a machine
outside of infra. Once those queues are empty, I'd shut down all the
db-using processes, apply the patch, restart the db, start everything
back up and enqueue the jobs which would have been scheduled.

This template change to the postgresql-server module will only affect
db-qa01.qa but will make it look like other postgres servers have a
pending change due to the way I've changed the postgresql.conf template.

I'm not crazy about doing this during freeze but I'm worried that the
timeout problem will start affecting more than stg before long and
want to get this figured out before that happens.

+1s?

Tim


diff --git a/roles/postgresql_server/templates/postgresql.conf 
b/roles/postgresql_server/templates/postgresql.conf
index 603f9ea..c9756b8 100644
--- a/roles/postgresql_server/templates/postgresql.conf
+++ b/roles/postgresql_server/templates/postgresql.conf
@@ -319,9 +319,15 @@ log_rotation_size = 0   # Automatic 
rotation of logfiles will
 #   fatal  
 #   panic (effectively off)
 
+{% if ansible_hostname.startswith("db-qa01") %}
+log_min_duration_statement = 500# -1 is disabled, 0 logs all statements
+# and their durations, > 0 logs only   
+# statements running at least this 
time.
+{% else %}
 #log_min_duration_statement = -1# -1 is disabled, 0 logs all statements
 # and their durations, > 0 logs only   
 # statements running at least this 
time.
+{% endif %}
 
 #silent_mode = off  # DO NOT USE without syslog or
 # logging_collector   


pgpZTgRLR3ebD.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: add db for execdb-stg to db-qa01.qa

[Tim Flink]

On Wed, 1 Apr 2015 08:47:43 -0600
Tim Flink  wrote:

> We had intended to do this before freeze but managed to miss it. One
> of the upcoming features for taskotron that we've deployed to dev and
> want to deploy to staging is execdb - task execution tracking for
> Taskotron.
> 
> This requires adding a database to db-qa01.qa which is the db host for
> the production Taskotron instance but no other changes to production
> or frozen systems.

Changes applied.


pgp14V4KQl6E_.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break: run some playbooks

[Tim Flink]

On Wed, 1 Apr 2015 10:34:40 -0600
Kevin Fenzi  wrote:

> I'd like to run a few playbooks to roll up changes made just before
> freeze. These should only change the indicated stuff thats already in
> git below. If it causes any issues, we should find out now, instead of
> later in the freeze, and also will clean up the check/diff report and
> make sure we are running exactly whats in git. 

+1


pgpDjL2el7kpm.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break: fix log sync script on log01

[Tim Flink]

On Wed, 1 Apr 2015 10:37:33 -0600
Kevin Fenzi  wrote:

> log01 pulls httpd logs from proxys. 
> 
> When it was added, proxy11 wasn't right, this corrects it and saves us
> a cron email every night with the error: 

+1


pgpwab4ehf8xy.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: FYI retroactive freeze break: koji ssl connection timeout increase hotfix

[Tim Flink]

On Wed, 1 Apr 2015 11:35:32 -0600
Kevin Fenzi  wrote:

> A few times in the last week we have hit a state where kojira on
> koji02 times out and doesn't run newrepo tasks for buildroots. 
> 
> Restarting the httpd on koji01 seems to unstick it, but this is not a
> good even stop gap as we would then have to manually do that and
> people would get small windows when koji was down. 
> 
> So, some investigation from koji developers at least for now the
> solution is to increase the ssl timeout. It's currently 60s, but I
> have increased it to 180 in a hotfix. 
> 
> http://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=ebb160dceef8ede2ce97b3b940987b798380dfea
> and
> http://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=f99e19b0244cd7ff8be570a6df6b78f494eceb57
> 
> This is already applied on koji02 to get us out of an outage situation
> (no new buildroots means no new fedora), but with +1s, I will apply to
> koji01 as well and make sure the playbooks sync with the hosts 

Seems to be a relatively low risk change to me

+1


pgphgn304oOYP.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: add db for execdb-stg to db-qa01.qa

[Tim Flink]

On Wed, 1 Apr 2015 17:16:27 +0200
Pierre-Yves Chibon  wrote:

> On Wed, Apr 01, 2015 at 08:47:43AM -0600, Tim Flink wrote:
> > We had intended to do this before freeze but managed to miss it.
> > One of the upcoming features for taskotron that we've deployed to
> > dev and want to deploy to staging is execdb - task execution
> > tracking for Taskotron.
> > 
> > This requires adding a database to db-qa01.qa which is the db host
> > for the production Taskotron instance but no other changes to
> > production or frozen systems.
>  
> So you want a stg application to run against a db located on a
> production environment?

That's how it's been setup since we first deployed Taskotron - all the
dev, stg and prod dbs are on the same host.

> Wouldn't it be more logical to have a db-qa01.qa.stg or so?

I can see the argument for doing that but I'm not sure that separating
them out would provide enough benefit to justify the overhead of (an)
additional database host(s) since the Taskotron bits don't really hit
the db very hard. I'm not against the idea of having separate db servers
for prod and dev/stg but I'd rather not muck around with that during
freeze.

Tim


pgp405qMXFw0G.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break: sudoers on koji01/02

[Tim Flink]

On Wed, 1 Apr 2015 09:24:28 -0600
Kevin Fenzi  wrote:

> I'd like to allow sysadmin-releng (which already has shell access)
> sudo also on the koji hubs. 
> 
> This will allow us to have mikem able to debug our kojiria issues. 
> 
> +1s?

+1 from me as well


pgpv_fl_UDOOL.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: add db for execdb-stg to db-qa01.qa

[Tim Flink]

We had intended to do this before freeze but managed to miss it. One of
the upcoming features for taskotron that we've deployed to dev and want
to deploy to staging is execdb - task execution tracking for Taskotron.

This requires adding a database to db-qa01.qa which is the db host for
the production Taskotron instance but no other changes to production or
frozen systems.

The only change in the frozen bits of ansible is:

+++ b/inventory/host_vars/db-qa01.qa.fedoraproject.org
@@ -22,6 +22,7 @@ dbs_to_backup:
 - fakefedorainfra
 - fakefedorainfra_stg
 - dev_fakefedorainfra
+- execdb_stg
 - execdb_dev
 - resultsdb
 - resultsdb_stg


pgppyyfggns12.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Filters in our ansible.git

[Tim Flink]

On Wed, 25 Mar 2015 12:49:42 -0400 (EDT)
Patrick Uiterwijk  wrote:

> Hi,
> 
> This is because ansible only looks in the cwd or in /usr/lib/ansible
> according to the documentation. I have submitted a PR to
> rbac-playbook to fix the cwd:
> https://bitbucket.org/tflink/ansible_utils/pull-request/3/set-the-correct-working-directory/diff

I've merged in the PR, built a new rpm, updated the infra repo and
lockbox01.

Tim


pgpHF2ld84Zvd.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Announcement: Replacing old rbac-playbook with ansible_utils

[Tim Flink]

While the rbac-playbook script helps people run playbooks, it's missing
some significant features that make ansible much easier to use. We're
replacing the old rbac-playbook script with a new piece of code that
does the same role-based access control but has some new features.

The usage of rbac-playbook will be pretty much the same as it has been
with some new options which should make make life easier. The old
script will be replaced with ansible_utils later today.

Tags:
  the '-t ' option will run only the parts of a playbook which
  are tagged with that  which allows single sections of a
  playbook to be run instead of running through everything, every time

Limit Targets:
  the '-l ' option will run a playbook but exclude all
  targets that don't match . This is very useful when
  you want to run a playbook without targeting all the hosts it was
  designed for

Start At Task:
  the '--start-at-task ' option runs the entire playbook but
  skips all tasks before  which is useful when re-running a
  longer playbook to correct a failure.

The new script is part of the ansible_utils package which can be found
at https://bitbucket.org/tflink/ansible_utils

If you find any issues, please let me know or file an issue in the
bitbucket project.

Tim



pgpOGICsCXHBx.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: retroactive freeze break request: kojipkgs01

[Tim Flink]

On Thu, 26 Feb 2015 08:11:50 -0700
Kevin Fenzi  wrote:

> We have been having some persistent issues with kojipkgs01 lately. 
> 
> kojipkgs01 is our squid proxy in front of koji builds. It allows users
> and builders to get fast access to packages. (When it's working). 
> 
> Lately, it's been working fine at first, then in a few days or so it
> starts getting really slow. Downloads go from 25M/s to 200k/sec and
> sometimes things even just timeout. 
> 
> Restarting squid seems to fix this... for a few more days.
> There is never any errors on the box, i/o, load and everything is
> fine. 
> 
> I looked this morning a bunch at options and adjusted the memory cache
> down in case we were hitting some kind of issue with memory cache. 
> 
> I'd like +1's for that change, and also to solicit ideas for what we
> can do to fix this once and for all (if these changes don't do so). 
> 
> diff --git a/roles/kojipkgs/files/squid.conf
> b/roles/kojipkgs/files/squid.conf index b011143..a0d5312 100644
> --- a/roles/kojipkgs/files/squid.conf
> +++ b/roles/kojipkgs/files/squid.conf
> @@ -6,8 +6,8 @@ hierarchy_stoplist cgi-bin ?
>  
>  cache_swap_low 98
>  cache_swap_high 99
> -cache_mem 50 GB
> -maximum_object_size 700 MB
> +cache_mem 10 GB
> +maximum_object_size 200 MB
>  minimum_object_size 0 KB
>  cache_replacement_policy heap LFUDA
>  maximum_object_size_in_memory 100 MB

+1 from me as well, hopefully this'll fix things but as pingou said,
it's easy to revert if it ends up causing problems


pgpvA2JtWWm2a.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [PATCH] Increase the max size allowed for upload to 25Kb

[Tim Flink]

On Wed, 25 Feb 2015 15:49:45 +0100
Pierre-Yves Chibon  wrote:

> On Wed, Feb 25, 2015 at 09:46:32AM -0500, Chuck Anderson wrote:
> > On Wed, Feb 25, 2015 at 03:44:27PM +0100, Pierre-Yves Chibon wrote:
> > > diff --git a/roles/kerneltest/templates/kerneltest.cfg
> > > b/roles/kerneltest/templates/kerneltest.cfg index
> > > 6552cb1..e15785e 100644 ---
> > > a/roles/kerneltest/templates/kerneltest.cfg +++
> > > b/roles/kerneltest/templates/kerneltest.cfg @@ -26,7 +26,7 @@
> > > ADMIN_GROUP = ['sysadmin-kernel', 'sysadmin-main']
> > > ALLOWED_MIMETYPES = ['text/plain', 'text/x-log'] 
> > >  # Restrict the size of content uploaded, this is 10Kb
> > > -MAX_CONTENT_LENGTH = 1024 * 10
> > > +MAX_CONTENT_LENGTH = 1024 * 25
> > 
> > You should probably also change the comment to match...
> 
> Good point, I'll do that in a second commit (that I'll push at the
> same time if I get 2 +1).

I assume this was also requested by the kernel folks, as well.

+1 on setting and comment change

Tim


pgpgiUbywG19l.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break: update iptables

[Tim Flink]

On Wed, 25 Feb 2015 06:59:16 -0700
Kevin Fenzi  wrote:

> So, currently our iptables config is generated by a template in
> ansible. In that template we add in all the ip's of staging hosts on
> the production hosts (to make sure we block them all from talking to
> production and possibly causing problems) (except for a small list of
> production hosts that allow staging for various reasons). 
> 
> So, the consequence of this is that when we add a new staging host
> (like we did yesterday with ipsilon01.stg) all the production hosts
> need to add that ip to their list to block. 
> 
> So, I'd like to run: 
> 
> ansible-playbook master -t iptables -l \*.phx2.\*
> 
> This will update the iptables config on phx2 hosts and restart
> iptables. It will add:  
> 
> +# ipsilon01.stg.phx2.fedoraproject.org
> +-A INPUT -s 10.5.126.35 -j REJECT --reject-with icmp-host-prohibited
> 
> This will have 2 effects: 
> 
> 1) Will make sure that ipsilon01.stg cannot talk to production and
> cause any issue (not that it normally would). 
> 
> 2) My ansible check/diff report will be a ton smaller and I can see if
> there's any real changes pending to hosts instead of being lost in the
> list of pending iptables changes. ;) 

Sounds like a good idea to me

+1


pgpIk8tBcLVlG.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: freeze break: update an alias

[Tim Flink]

On Tue, 24 Feb 2015 14:38:50 -0700
Kevin Fenzi  wrote:

> A trivial freeze break request. 
> 
> I'd apply then run bastion playbooks. 
> 
> Fixes ticket 4677
> 
> diff --git a/roles/fas_client/files/aliases.template
> b/roles/fas_client/files/aliases.template index 408ce61..165c351
> 100644 --- a/roles/fas_client/files/aliases.template
> +++ b/roles/fas_client/files/aliases.template
> @@ -151,7 +151,7 @@ security: security-priv...@lists.fedoraproject.org
>  secalert: security-priv...@lists.fedoraproject.org
>  
>  webmaster: websi...@lists.fedoraproject.org
> -logo: rle...@redhat.com
> +logo: rle...@redhat.com,du...@redhat.com
>  ham-radio-exams: nb,jbwillia,robertjw,gholms
>  
>  # Misc Aliases

+1


pgpGvK_hOJNDr.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: freeze break: drop python-bugzilla hotfix

[Tim Flink]

On Mon, 1 Dec 2014 12:31:58 -0700
Kevin Fenzi  wrote:

> We currently have a hotfix for python-bugzilla in puppet. 
> 
> This is breaking the newer version of python-bugzilla that
> reviewstatus uses. ie, if we update to the newer version it's got
> this hotfix against the old version it breaks. 
> 
> tibbs investigated and this hotfix is no longer needed against the
> newer version. 
> 
> So, I would like to drop the hotfix, and update to the new version and
> see if reviewstats is happy as I hope it will be. It shouldn't affect
> anything release blocking, but would be nice to fix. 
> 
> Related ticket:
> https://fedorahosted.org/fedora-infrastructure/ticket/4429
> 
> +1s?

Assuming that the patch would involve removing all the includes on the
python-bugzilla hotfix, +1 from me.

If it somehow does affect FAS, it doesn't sound like that could be
non-workaroundable breakage and it sounds really unlikely, anyways.

Tim

> kevin
> --
> diff --git a/modules/hotfix/manifests/init.pp
> b/modules/hotfix/manifests/init.pp index a12b063..ae7850f 100644
> --- a/modules/hotfix/manifests/init.pp
> +++ b/modules/hotfix/manifests/init.pp
> @@ -142,10 +142,6 @@ class hotfix::fas_client {
>  }
>  
>  class hotfix::python-bugzilla {
> -file { "/usr/lib/python2.6/site-packages/bugzilla/base.py":
> -source => 'puppet:///hotfix/python-bugzilla/base.py',
> -mode => '0644'
> -}
>  }
>  
>  class hotfix::python-logging {



pgpJAIU7DSvcc.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Code Search for Fedora

[Tim Flink]

On Tue, 18 Nov 2014 13:00:22 -0800
Michael Stapelberg  wrote:

> Hey,
> 
> Recently I’ve been talking to Hannes (cc'ed) about whether Fedora
> would be interested in having the equivalent of
> http://codesearch.debian.net/¹
> 
> The project came to live as my Bachelor of Science Thesis² and aims to
> provide fast regular expression search over a big corpus, in this case
> 140 GB of source code of all software included in the Debian main
> distribution (as opposed to non-free or contrib, which we excluded
> because of licensing concerns). It is based on the work Russ Cox
> published, which in turn resembles the work he did on Google Code
> Search when he was an intern there in 2006.
> 
> So, what’s this discussion about?
> 
> What I’m offering is setting up/running a public version of Code
> Search for Fedora. It needs to be public because I want the open
> source community as a whole profit from it, and also I’m told you have
> somewhat comparable tools internally anyway :).

Thanks for starting the conversation.



> I feel like this email is long enough already, so I’ll just ask a
> general: what do you think? Do you need any more information? Please
> just ask, and keep me CC'ed, since I’m not subscribed to this list.

I'm a little late to the discussion, but I think that code search
sounds like a cool idea if we can find the human/machine resources to
do it. I've only glanced through all the docs so far but I have a couple
of concerns (some of which have already been raised). I hope this
doesn't sound like I'm completely against the idea, though - I wouldn't
have spent the time to go through your thesis and respond to the
discussion if that were the case.

Tim


Single points of (human) failure


Kevin already brought this up but I'm a little worried about supporting
a large, complex application like this with only one person familiar
with it and few people around familiar with the language that its core
is written in. Speaking as someone who has been the single point of
failure in an application deployment before, I'd strongly suggest
finding someone to help. Finding out that something went down when you
were/are on vacation and there's nobody else that can fix it is not
fun :)


Node Failure Behavior
-

I'm not clear from the docs I went through how node failure is handled.
I don't see any explicit mention of it, so I'm assuming that the index
shards all have single copies of the index. How does the system handle
failure in one of the index nodes? My first guess is that there would
be missing results from queries but I haven't gotten into the actual
code yet.


Code resiliency
---

I think that this is lessened a bit since the code has been running in
production but it sounds like the base code for indexing was meant
somewhat as a "proof of concept" or small-scale deployment. It sounds
like you've made quite a few enhancements on top of the released google
codesearch but tried to leave the core code alone as much as possible.
Have you seen many problems in the index nodes for DCS?


Indexing


Have there been any complaints/comments about your chosen update delta
of 3 days? You assert that 3 days is a good balance between indexing
load and keeping fresh code in the index but I don't see a
justification in your thesis. How did you come to the conclusion that 3
days was an optimal choice?

Is the indexing process automated or does it need to be kicked off by a
human? The way it's described in your thesis ("after verifying that no
human mistake was made by confirming that Debian Code Search still
delivers results ..."), it sounds somewhat manual.

Is there a downtime when updating the inverted trigram index? If so,
how long is that? Does it happen for every re-index? It sounds like the
resources for the index nodes would be almost 100% utilized after
indexing, leaving no additional resources to handle the re-indexing
load. Am I misunderstanding something in the architecture?


Ranking
---

Is the result ranking code compatible with non-debian sources? We don't
have an equivalent to popcon and I assume that the reverse dependency
factor would need different code in for Fedora than in Debian. Or is
this part of the modifications you were planning for already?



pgpv8GuajAFiT.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: new depcheck on taskotron clients

[Tim Flink]

On Wed, 19 Nov 2014 13:57:15 -0700
Tim Flink  wrote:



> I'd like to update the depcheck task in production to fix some
> recurring issues that some devs have been complaining about. Diff is
> at the end of this email.

Depcheck in taskotron production has been updated to include those
fixes and everything seems to be running fine.

Tim


pgpx5M69QZxY5.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: new depcheck on taskotron clients

[Tim Flink]

I thought that I had updated the depcheck task on production taskotron
a while back but it turns out that never happened and depcheck is
hitting failures in production that it shouldn't be hitting. Original
issue is:

https://phab.qadevel.cloud.fedoraproject.org/T366

The fix was reviewed before updating dev/stg and has been running
there more than 2 weeks now without issue.

I'd like to update the depcheck task in production to fix some
recurring issues that some devs have been complaining about. Diff is at
the end of this email.

Tim


diff --git a/depcheck/__init__.py b/depcheck/__init__.py
index ccac8a4..609eb13 100755
--- a/depcheck/__init__.py
+++ b/depcheck/__init__.py
@@ -36,6 +36,7 @@ import sys
 import argparse
 import tempfile
 import platform
+import logging
 
 from . import depcheck
 from . import metadata
@@ -91,7 +92,7 @@ def run(rpms, arch, repos, report_format="updates",
workdir=None): arch = 'i686'
 
 # handle data formatting for repo metadata (list of dicts)
-log.logger.debug("Prepairing metadata download")
+log.logger.debug("Preparing metadata download")
 handled_repos = {}
 for repo, path in repos.items():
 log.logger.debug("  - %r: %r, %r", repo, arch, path)
@@ -121,7 +122,7 @@ def run(rpms, arch, repos, report_format="updates",
workdir=None): 
 # Sometimes, depcheck gets run with no rpms
 if not run_result:
-log.logger.info("No results")
+log.logger.warn("No results")
 
 output = squash_results.format_output(run_result, arch,
report_format) log.logger.info(output)
@@ -131,6 +132,7 @@ def run(rpms, arch, repos, report_format="updates",
workdir=None): def depcheck_cli():
 parser = argparse.ArgumentParser()
 parser.add_argument("rpm", nargs='+', help="RPMs to be tested")
+parser.add_argument("-d", "--debug", action="store_true",
help="enable debug logging for depcheck") parser.add_argument("-a",
"--arch", choices=["i386", "x86_64", "armhfp"], help=""" Architecture
to be tested. If omitted, defaults to the current machine's
architecture. @@ -143,9 +145,17 @@ def depcheck_cli():
 parser.add_argument("-f", "--format", choices=["rpms", "updates"],
help=""" Specify output format to override configuration.
 """)
-
+parser.add_argument("-w", "--workdir", help="workdir to use for
depcheck") 
 args = parser.parse_args()
+
+# initialize logging when entering depcheck via cli - this won't
be executed
+# otherwise but in that case, taskotron is responsible for
initialization
+logging.basicConfig()
+
+if args.debug:
+log.logger.setLevel(logging.DEBUG)
+
 # autodetect arch if none is passed in
 arch = args.arch
 if arch is None:
@@ -155,8 +165,8 @@ def depcheck_cli():
 repos = dict([(repo, repo) for repo in args.repos])
 
 try:
-print run(args.rpm, arch, repos, args.format)
-except KeyboardInterrupt, e:
+run(args.rpm, arch, repos, args.format, args.workdir)
+except KeyboardInterrupt:
 print >> sys.stderr, ("\n\nExiting on user cancel.")
 sys.exit(1)
 
diff --git a/depcheck/depcheck.py b/depcheck/depcheck.py
index bb0b37a..e5ba15c 100755
--- a/depcheck/depcheck.py
+++ b/depcheck/depcheck.py
@@ -30,7 +30,6 @@ import solv
 import tempfile
 
 from . import Pass, Fail
-from . import exc
 from . import log
 
 
@@ -77,6 +76,8 @@ class Depcheck(object):
 checking dependencies for the supplied repositories
 """
 
+log.logger.debug('preparing libsolv repos for depcheck run')
+
 for repo in self.repos:
 primary = open(self.repos[repo]['primary'], 'rb')
 filelist = open(self.repos[repo]['filelists'], 'rb')
@@ -85,37 +86,36 @@ class Depcheck(object):
 self.solvrepo.add_rpmmd(filelist, None,
 solv.Repo.REPO_EXTEND_SOLVABLES)
 
-def _check_rpm(self, rpmfile):
+def _check_rpm(self, rpmfile, solvable):
 """Check a single rpm file against prepared repodata
  :param rpmfile: filename of rpm file to check
  :type rpmfile: str
+ :param solvable: libsolv solvable from solvpool for the rpm
to check
+ :type solvable: libsolv.Solver.Solvable
 """
-solvable = self.solvrepo.add_rpm(rpmfile)
-self.solvpool.addfileprovides()
-self.solvpool.createwhatprovides()
 
+# the job setup here is mostly taken from libsolv's
installcheck tool
+# create a job using the predetermined solvable in the repo
 job = self.solvpool.Job(solv.Job.SOLVER_INSTALL |
solv.Job.SOLVER_SOLVABLE,
-solvable.id)
+solvable.id)
+
self.solvpool.set_flag(solv.Solver.SOLVER_FLAG_IGNORE_RECOMMENDED, 1) 
-# important note - make sure to keep the solver in the same
scope as
-# the problem analysis code or else python-solv segfults will
ensue
-solvpool_solver = self.solvpool.Solver()
-solv_problems = s

Re: Freeze Break Request: update libtaskotorn on taskotron production systems

[Tim Flink]

On Tue, 4 Nov 2014 18:34:14 +0100
Pierre-Yves Chibon  wrote:



> > +import os
> > +
> > +class Arches():
> > +'''
> > +Helper class for working with supported arches inside taskotron
> > +'''
> > +
> > +#: all supported architectures
> > +all = ['i386', 'i486', 'i586', 'i686',
> > +   'x86_64',
> > +   'armhfp', 'arm7hl',
> > +   'noarch', 'src']
> > +
> > +#: base architectures
> > +base = ['i386', 'x86_64', 'armhfp']
> > +
> > +#: meta architectures
> > +meta = ['noarch', 'src']
> > +
> > +def basearch(arch=None):
> > +'''
> > +This returns the 'base' architecture identifier for a
> > specified architecture
> > +(e.g. ``i386`` for ``i[3-6]86``), to be used by YUM etc.
> > +
> > +:param str arch: an architecture to be converted to a
> > basearch. If ``None``,
> > + then the arch of the current machine is used.
> > +:return: basearch, or ``arch`` if no basearch was found for it
> > +:rtype: str
> > +'''
> > +if arch is None:
> > +arch = os.uname()[4]
> > +if arch in ['i386', 'i486', 'i586', 'i686']:
> > +arch = 'i386'
> > +if arch in ['armhfp', 'arm7hl']:
> > +arch = 'armhfp'
> > +return arch
> 
> Not at all related to the freeze-break, but have you consider putting
> the info used by basearch/Arches() into the config file?
> It would allow adding a new arch w/o making a new release.

This is mostly a stopgap - we have other arch handling issues that we
want to fix but haven't gotten to it yet.

https://phab.qadevel.cloud.fedoraproject.org/T227

> >  return (build2update, failures)
> > diff --git a/libtaskotron/config_defaults.py
> > b/libtaskotron/config_defaults.py index fc05abd..fb77a7c 100644
> > --- a/libtaskotron/config_defaults.py
> > +++ b/libtaskotron/config_defaults.py
> > @@ -46,13 +46,13 @@ class Config(object):
> >  koji_url =
> > 'http://koji.fedoraproject.org/kojihub'  #:
> > pkg_url =
> > 'http://kojipkgs.fedoraproject.org/packages'  #:
> > bodhi_server =
> > 'https://admin.fedoraproject.org/updates/'   #:
> > -resultsdb_server = \
> > -
> > 'http://resultsdb.qa.fedoraproject.org/resultsdb/api/v1.0/'
> > #:
> > -taskotron_master =
> > 'http://taskotron.qa.fedoraproject.org/taskmaster/'  #:
> > +resultsdb_server =
> > 'http://127.0.0.1/resultsdb/api/v1.0/'   #:
> > +taskotron_master =
> > 'http://127.0.0.1/taskmaster/'   #:
> > buildbot_task_step =
> > 'runtask'  #:
> 
> Was this meant?

Yes, it was. We intend to have folks install libtaskotron in order to
run stuff locally, so pointing them at the production instances by
default isn't a great idea. Also, those values aren't aren't even valid
internal or external hostnames so it's not as big of a change as it
looks :)

https://phab.qadevel.cloud.fedoraproject.org/T344


> Assuming the answer to the question just above is: yes, then I am +1
> as well :)

Thanks

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: update libtaskotorn on taskotron production systems

[Tim Flink]

Long story short: bodhi has been having intermittent network issues for
a while now and those network issues tend to kill taskotron tasks. We
implemented some retries in order to mitigate the effects of those
network issues but I botched one of the last builds and it didn't
actually get deployed before freeze.

The new build has been running in stg and dev for almost 2 weeks now
without issue. I'd like to update the production taskotron clients with
this latest build since we've seen 30+ bodhi related failures already
today.

I've attached the diff at the end of this email, there are several
unrelated fixes here but I don't want to make a new build with the
cherry picked fix extracted because the fix was committed a while ago
and freeze ends tomorrow. If it causes problems, we can downgrade to
the existing build or fix it when freeze ends tomorrow.

Tim


diff --git a/libtaskotron/__init__.py b/libtaskotron/__init__.py
index 66923e1..2eed326 100644
--- a/libtaskotron/__init__.py
+++ b/libtaskotron/__init__.py
@@ -4,4 +4,4 @@
 # See the LICENSE file for more details on Licensing
 
 from __future__ import absolute_import
-__version__ = '0.3.7'
+__version__ = '0.3.10'
diff --git a/libtaskotron/arch_utils.py b/libtaskotron/arch_utils.py
new file mode 100644
index 000..a62171d
--- /dev/null
+++ b/libtaskotron/arch_utils.py
@@ -0,0 +1,41 @@
+# -*- coding: utf-8 -*-
+# Copyright 2009-2014, Red Hat, Inc.
+# License: GPL-2.0+ 
+# See the LICENSE file for more details on Licensing
+
+import os
+
+class Arches():
+'''
+Helper class for working with supported arches inside taskotron
+'''
+
+#: all supported architectures
+all = ['i386', 'i486', 'i586', 'i686',
+   'x86_64',
+   'armhfp', 'arm7hl',
+   'noarch', 'src']
+
+#: base architectures
+base = ['i386', 'x86_64', 'armhfp']
+
+#: meta architectures
+meta = ['noarch', 'src']
+
+def basearch(arch=None):
+'''
+This returns the 'base' architecture identifier for a specified 
architecture
+(e.g. ``i386`` for ``i[3-6]86``), to be used by YUM etc.
+
+:param str arch: an architecture to be converted to a basearch. If 
``None``,
+ then the arch of the current machine is used.
+:return: basearch, or ``arch`` if no basearch was found for it
+:rtype: str
+'''
+if arch is None:
+arch = os.uname()[4]
+if arch in ['i386', 'i486', 'i586', 'i686']:
+arch = 'i386'
+if arch in ['armhfp', 'arm7hl']:
+arch = 'armhfp'
+return arch
\ No newline at end of file
diff --git a/libtaskotron/bodhi_utils.py b/libtaskotron/bodhi_utils.py
index 71396d9..403d94f 100644
--- a/libtaskotron/bodhi_utils.py
+++ b/libtaskotron/bodhi_utils.py
@@ -27,10 +27,10 @@ class BodhiUtils(object):
default BodhiClient instance is used.
 '''
 
-taskotron_config = config.get_config()
-username = taskotron_config.fas_username
-password = taskotron_config.fas_password
-bodhi_url = taskotron_config.bodhi_server
+self.taskotron_config = config.get_config()
+username = self.taskotron_config.fas_username
+password = self.taskotron_config.fas_password
+bodhi_url = self.taskotron_config.bodhi_server
 
 if not client:
 log.debug('creating bodhi client to %s' % bodhi_url)
@@ -42,7 +42,8 @@ class BodhiUtils(object):
 
 
 def query_update(self, package):
-'''Get the last Bodhi update matching criteria.
+'''Get the last Bodhi update matching criteria. Retry
+``bodhi_request_max_retries``-times when server returns HTTP 5xx 
response.
 
:param str package: package NVR or package name or update title or
update ID
@@ -55,11 +56,23 @@ class BodhiUtils(object):
:rtype: :class:`bunch.Bunch`
 '''
 log.debug('Searching Bodhi updates for: %s', package)
-res = self.client.query(package=package, limit=1)
-if res['updates']:
-return res['updates'][0]
-else:
-return None
+max_retries = self.taskotron_config.bodhi_request_max_retries
+for retry_num in xrange(1, max_retries + 1):
+try:
+res = self.client.query(package=package, limit=1)
+if res['updates']:
+return res['updates'][0]
+else:
+return None
+except fedora.client.ServerError as e:
+if retry_num >= max_retries:
+log.critical('Maximum Bodhi server retries exceeded')
+raise
+elif e.code < 500 or e.code > 599:
+raise
+else:
+log.warning('Got HTTP %d response from bodhi, retrying \
+(%d retries remaining)...', e.code, max_retries - retry_num)
 
 
 def build2update(self, builds, strict=False):

Freeze break request: blockerbugs bugfix to correctly sync updates

[Tim Flink]

We found a bug in the blockerbugs app yesterday that is preventing
updates from being properly synced and leaving us with an empty page of
pending updates:

https://qa.fedoraproject.org/blockerbugs/milestone/21/alpha/updates

I'd like to deploy a new version of blockerbugs with the fix for that
bug and adding a convenience endpoint to the API. The added API
endpoint is of little to no risk because there are no current users of
that API and the change itself is very small.

I'd like to get some +1s if the change is acceptable during freeze.

Tim



Reviews of changes:
(bugfix) https://phab.qadevel.cloud.fedoraproject.org/D228
(endpoint) https://phab.qadevel.cloud.fedoraproject.org/D230


Full diff of code change:

diff --git a/blockerbugs/cli.py b/blockerbugs/cli.py
index 2ec4898..b164b7b 100644
--- a/blockerbugs/cli.py
+++ b/blockerbugs/cli.py
@@ -166,7 +166,7 @@ def sync_updates(fullsync=False):
 active_releases = Release.query.filter_by(active=True).all()
 update_sync = UpdateSync(db)
 for release in active_releases:
-update_sync.sync_updates(release)
+update_sync.sync_updates(release, fullsync)
 
 
 def minify():
diff --git a/blockerbugs/controllers/api/api.py
b/blockerbugs/controllers/api/api.py index 09f437f..721b442 100644
--- a/blockerbugs/controllers/api/api.py
+++ b/blockerbugs/controllers/api/api.py
@@ -233,4 +233,7 @@ def update_spin(rel_num, milestone_version,
spin_id): db.session.commit()
 return JsonResponse(status=200)
 
-
+@api_v0.route('/milestones/current')
+def get_current_milestone():
+current_milestone = get_or_404(Milestone, current=True)
+return JsonResponse(current_milestone.simple())
diff --git a/blockerbugs/models/milestone.py
b/blockerbugs/models/milestone.py index 68bbf0e..0b1739d 100644
--- a/blockerbugs/models/milestone.py
+++ b/blockerbugs/models/milestone.py
@@ -60,3 +60,13 @@ class Milestone(db.Model):
 def __repr__(self):
 number = self.release.number if self.release else -1
 return 'milestone: %d-%s' % (number, self.version)
+
+def simple(self):
+return {'id': self.id,
+'release': self.release.number,
+'version': self.version,
+'name': self.name,
+'blocker_tracker': self.blocker_tracker,
+'fe_tracker': self.fe_tracker,
+'current': self.current,
+}
diff --git a/blockerbugs/models/update.py b/blockerbugs/models/update.py
index 033e623..777beec 100644
--- a/blockerbugs/models/update.py
+++ b/blockerbugs/models/update.py
@@ -77,7 +77,7 @@ class Update(db.Model):
 def __str__(self):
 return 'update: %s' % (self.title)
 
-def sync(self, updateinfo, milestone):
+def sync(self, updateinfo, fullsync=False):
 self.title = updateinfo['title']
 self.url = updateinfo['url']
 self.karma = updateinfo['karma']
@@ -91,16 +91,18 @@ class Update(db.Model):
 self.pending = updateinfo['pending']
 current_bugids = [bug.bugid for bug in self.bugs]
 for bugid in updateinfo['bugs']:
-if not bugid in current_bugids:
+# if full sync is specified, we want to sync all of the
bugs
+if fullsync or not bugid in current_bugids:
 newbugs = Bug.query.filter_by(bugid=bugid).all()
 for bug in newbugs:
-self.bugs.append(bug)
-if not milestone in self.milestones:
-self.milestones.append(milestone)
+if not bugid in current_bugids:
+self.bugs.append(bug)
+if not bug.milestone in self.milestones:
+self.milestones.append(bug.milestone)
 
 @classmethod
-def from_data(cls, updateinfo, release, milestone):
+def from_data(cls, updateinfo, release):
 newupdate = Update(updateinfo['title'], '', 0, '', [],
release, None)
-newupdate.sync(updateinfo, milestone)
+newupdate.sync(updateinfo)
 return newupdate
 
diff --git a/blockerbugs/util/update_sync.py
b/blockerbugs/util/update_sync.py index af8b2bf..4587a8c 100644
--- a/blockerbugs/util/update_sync.py
+++ b/blockerbugs/util/update_sync.py
@@ -132,7 +132,7 @@ class UpdateSync(object):
 buglist.extend(milestone.bugs.filter_by(active=True).all())
 return buglist
 
-def sync_bug_updates(self, release, bugs):
+def sync_bug_updates(self, release, bugs, fullsync=False):
 starttime = datetime.utcnow()
 bugs_ids = [bug.bugid for bug in bugs]
 self.log.debug('searching for updates for bugs %s' %
str(bugs_ids)) @@ -154,11 +154,10 @@ class UpdateSync(object):
 if existing_update:
 self.log.debug(
 'syncing existing update %s' %
existing_update.title)
-existing_update.sync(update, bug.milestone)
+existing_update.sync(update, fullsync)
 else:
 se

Re: Moving Taskotron to Production

[Tim Flink]

On Tue, 22 Jul 2014 12:12:36 -0600
Kevin Fenzi  wrote:

> On Tue, 22 Jul 2014 10:56:12 -0600
> Tim Flink  wrote:
> 
> > Now that Taskotron staging is up and running, I'm not sure what's
> > generally required for services before moving to production.
> > 
> > I'm still working out a few kinks in the initial production systems
> > but I think that the only big thing left is the proxy settings,
> > which I'm leaving alone for the moment.
> > 
> > Is there any review process or checklist for moving things from stg
> > to prod?
> 
> http://infrastructure.fedoraproject.org/infra/docs/requestforresources.txt
> 
> I'd like monitoring added. What should we monitor to make sure it's
> working as expected? 
> 
> Also, do we need any SOP's written up? Anything special in
> stopping/starting things? updates order or anything?
> 
> Also, I need to make sure we have backups going on the db server at
> least. Anything else that needs backing up?
> 
> kevin

I've started creating tickets on our end to track what is needed before
Taskotron is moved into production:

https://phab.qadevel.cloud.fedoraproject.org/T323

I imagine that I'm missing some tickets but will add them as we figure
out what else will be needed.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Congrats to Tim Flink

[Tim Flink]

On Sat, 9 Aug 2014 06:49:46 -0600
Kevin Fenzi  wrote:

> I'm happy to announce that I have approved Tim into the sysadmin-main
> group. This is the core group of trusted folks that high level access
> to most things. 
> 
> Tim is part of the Fedora QA group and has been working very hard on
> taskotron and resultsdb and related deployment. Being in the main
> group will allow him to not block on other folks adding data or doing
> things he needs. Also, he may be able to help out with other tasks as
> his time permits. 
> 
> Congrats and use your powers for good! :) 

Only for good!? So much for my plan to create a taskotron-managed
cluster of dogecoin miners :'(

Thanks for the welcome. Hopefully I'll be able to help out some but at
least I won't be pestering you all quite so often once I learn how to
do a few things.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Moving Taskotron to Production

[Tim Flink]

On Tue, 22 Jul 2014 12:12:36 -0600
Kevin Fenzi  wrote:

> On Tue, 22 Jul 2014 10:56:12 -0600
> Tim Flink  wrote:
> 
> > Now that Taskotron staging is up and running, I'm not sure what's
> > generally required for services before moving to production.
> > 
> > I'm still working out a few kinks in the initial production systems
> > but I think that the only big thing left is the proxy settings,
> > which I'm leaving alone for the moment.
> > 
> > Is there any review process or checklist for moving things from stg
> > to prod?
> 
> http://infrastructure.fedoraproject.org/infra/docs/requestforresources.txt

Thanks, I don't think that I would have looked in that doc.

> I'd like monitoring added. What should we monitor to make sure it's
> working as expected? 

It depends on how complicated we want to get, I suppose. The most basic
things would be to check to make sure resultsdb, resultsdb_frontend and
buildbot are responding to http but there are json apis for
resultsdb and buildbot which would give more details. I'd like to
monitor free disk space if that's not already done by default.

Are most other services monitored with a GET to a url to see if they're
up?

> Also, do we need any SOP's written up? Anything special in
> stopping/starting things? updates order or anything?

Yeah, the triggered jobs need to be buffered during downtime for
playback after everything comes up. I've always used my local system
for that but I'm not sure what the best choice is for infra.

> Also, I need to make sure we have backups going on the db server at
> least. Anything else that needs backing up?

/home/buildmaster/master on the taskotron server (taskotron01.qa for
production) should be backed up as that's where all the logs are
stored. Everything else that needs to be backed up is in the databases.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Moving Taskotron to Production

[Tim Flink]

Now that Taskotron staging is up and running, I'm not sure what's
generally required for services before moving to production.

I'm still working out a few kinks in the initial production systems but
I think that the only big thing left is the proxy settings, which I'm
leaving alone for the moment.

Is there any review process or checklist for moving things from stg to
prod?

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Review for new rbac_playbook

[Tim Flink]

On Wed, 4 Jun 2014 19:45:23 -0600
Tim Flink  wrote:

> I've been working to rewrite and extend the script that we've been
> using to control playbook execution for folks who are not in
> sysadmin-main.
> 
> https://bitbucket.org/tflink/rbac-ansible

With the removal of the '-P' option, I think that all the concerns
raised have been addressed.

Any objections to putting the script into place?

If not, where should it live? I put it up on bitbucket because that was
easy for me but I realize that none of the infra stuff lives there.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Review for new rbac_playbook

[Tim Flink]

On Mon, 09 Jun 2014 17:37:06 +0200
Michael Scherer  wrote:

> Le lundi 09 juin 2014 à 08:44 -0600, Tim Flink a écrit :
> 
> > The QA devel folks use phabricator and phabricator supports git repo
> > hosting (through http(s) and ssh). In order to support git over ssh
> > while keeping user information in phabricator (username, ssh key for
> > git, repo permissions etc.), it uses a short-circuited ssh daemon
> > that uses phabricator for auth instead of system accounts
> > (restricted to git commands, though). Git repos on alternate ports
> > is a bit of a pain, so to support git+ssh on port 22 I change the
> > real ssh daemon (that can do more than git) to an alternate port.
> 
> What about having the real sshd listening on one ip ( if possible, a
> rfc1918 one in the VPN ) and git from phabricator on a second ?

I can't think of any reason why that wouldn't work but I don't see
what's wrong with just using an alternate port instead of adding a
second IP.

I don't have a strong opinion on the exact setup as long as
the external port 22 is handled by phabricator and the machine remains
manageable through ansible without too many odd workarounds.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Review for new rbac_playbook

[Tim Flink]

On Wed, 11 Jun 2014 16:27:15 -0600
Kevin Fenzi  wrote:



> > If that's the way that we want to go, we'll have some extra commits
> > to the ansible repo but it'll work.
> 
> Yeah, I think a bit of messyness in the repo is better than having to
> bother with -P in rbac-playbook... just to make it simplier. 

-P has been removed in the code and tests.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Review for new rbac_playbook

[Tim Flink]

On Mon, 9 Jun 2014 08:49:48 -0600
Kevin Fenzi  wrote:

> On Mon, 9 Jun 2014 08:44:38 -0600
> Tim Flink  wrote:
> 
> > I think that most of your concerns have been addressed or are being
> > discussed in other parts of this thread but I wanted to speak
> > towards the reason that -P is there at all.
> > 
> > You are correct in reading that it has ansible-playbook use an ssh
> > port other than 22. That is set using -e 'ansible_ssh_port= > port>' and giving direct access to the -e parameter would be
> > port>problematic at best,
> > so I added the -P parameter which is restricted to just that option
> > even though it's rendered as -e
> > 
> > The QA devel folks use phabricator and phabricator supports git repo
> > hosting (through http(s) and ssh). In order to support git over ssh
> > while keeping user information in phabricator (username, ssh key for
> > git, repo permissions etc.), it uses a short-circuited ssh daemon
> > that uses phabricator for auth instead of system accounts
> > (restricted to git commands, though). Git repos on alternate ports
> > is a bit of a pain, so to support git+ssh on port 22 I change the
> > real ssh daemon (that can do more than git) to an alternate port.
> 
> If those hosts always have ssh on the same different port, we could
> just add that to vars?
> 
> http://docs.ansible.com/faq.html#how-do-i-handle-different-machines-needing-different-user-accounts-or-ports-to-log-in-with

I've generally been using port 222 for real ssh on those hosts. We
could set the port in the inventory file. While that would work for many
cases, I've always used the -e directly for 2 reasons:

1) My understanding is that ansible convention discourages putting
   stuff like that in the inventory files

2) Hosts are listening for ssh on port 22 when initially deployed.
   Initial deployments would require changing the inventory information
   to use port 22 for initial deployment and then changing it back
   to the alternate port after running the playbook/role which sets up
   the alternate port for ssh.

If that's the way that we want to go, we'll have some extra commits to
the ansible repo but it'll work.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Review for new rbac_playbook

[Tim Flink]

On Sat, 07 Jun 2014 16:26:45 +0100
Michael Scherer  wrote:

> Le mercredi 04 juin 2014 à 19:45 -0600, Tim Flink a écrit :
> > I've been working to rewrite and extend the script that we've been
> > using to control playbook execution for folks who are not in
> > sysadmin-main.
> > 
> > https://bitbucket.org/tflink/rbac-ansible
> > 
> > I've been testing the script but before we actually start using it
> > on lockbox01, I'd appreciate a review of the code to make sure I
> > didn't miss any security holes.
> > 
> > Injection attacks shouldn't be an issue due to usage of os.execv -
> > all injection attempts are grouped as a single argument and will
> > not be broken up.
> 
> So, I just have one question. how does the option -P of the script is
> supposed to behave ?
> 
> Can i assume that I would be able to say "use this playbook, but
> instead of using the port 22, use port 1234" without changing the
> playbook ?

I think that most of your concerns have been addressed or are being
discussed in other parts of this thread but I wanted to speak towards
the reason that -P is there at all.

You are correct in reading that it has ansible-playbook use an ssh port
other than 22. That is set using -e 'ansible_ssh_port=' and
giving direct access to the -e parameter would be problematic at best,
so I added the -P parameter which is restricted to just that option
even though it's rendered as -e

The QA devel folks use phabricator and phabricator supports git repo
hosting (through http(s) and ssh). In order to support git over ssh
while keeping user information in phabricator (username, ssh key for
git, repo permissions etc.), it uses a short-circuited ssh daemon that
uses phabricator for auth instead of system accounts (restricted to git
commands, though). Git repos on alternate ports is a bit of a pain, so
to support git+ssh on port 22 I change the real ssh daemon (that can do
more than git) to an alternate port.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Review for new rbac_playbook

[Tim Flink]

I've been working to rewrite and extend the script that we've been
using to control playbook execution for folks who are not in
sysadmin-main.

https://bitbucket.org/tflink/rbac-ansible

I've been testing the script but before we actually start using it on
lockbox01, I'd appreciate a review of the code to make sure I didn't
miss any security holes.

Injection attacks shouldn't be an issue due to usage of os.execv - all
injection attempts are grouped as a single argument and will not be
broken up.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: QA Client Network Isolation and FAS Groups

[Tim Flink]

On Sat, 18 Jan 2014 13:26:30 -0700
Kevin Fenzi  wrote:

> On Wed, 15 Jan 2014 11:11:51 -0700
> Tim Flink  wrote:
> 
> > The one thing that isn't part of that diagram is some sort of
> > lockbox-ish host for managing and monitoring the clients. I'm still
> > not sure how we want to configure all that (we had talked about
> > putting everything but the clients in the infra playbook on
> > lockbox01 and managing the clients with a different playbook on a
> > different host but none of that has been done), though but I
> > suspect that is one of the _simpler_ parts of the setup.
> 
> Could be yeah. I am planning on making a lockbox01.qa after our
> conversation the other day...

Yeah, from that conversation, it seems like the lockbox01.qa is the
best choice for now.

> ...snip
> > > Could we just do it with a private libvirt network on the qa
> > > virthosts? ie, pick 172.31.17.0 and put them all in that and
> > > setup a bastion host as their gateway that does NAT for them out
> > > to the stuff they need. Or would NAT not work for this? They
> > > would still physically be on the qa network tho, so I guess we
> > > could try and request a real seperate one from RHIT. 
> > 
> > I'm not sure I understand this completely. Would this allow the
> > various "special" hosts (beaker server and bastion host in
> > particular) to connect to the clients without adding the step of
> > "connect to virthostX" before being able to ssh into the clients?
> 
> Sure, they would just use the bridge on the virthosts to send their
> private network stuff around, but it would be traveling on the same
> physical network as the 10.5.124.x qa network. 

As noted below, it sounds like I have some reading to do, but that
sounds as if it would work.

> > NAT should work for everything other than communication with the
> > beaker server and bastion hosts.
> 
> Right. 
>  
> > Another option might be to use ebtables. I haven't investigated this
> > fully yet but it sounds like it would allow the use of qa network
> > IPs but restrict the traffic running through the bridged network on
> > the virthosts - effectively creating a restricted network without
> > needing any physical changes. Of course, that would only work with
> > VM clients but that's all we're planning for at the moment.
> 
> Yeah, that was essentially what I was suggesting above actually. ;) 

Ah, at least we have similar solutions in mind :) I guess I'll have to
do some more reading on libvirt networks. When I looked at the docs, it
sounded like those were designed only for VMs on the same virthost -
not communication between VMs on different virthosts.

> > > There was also talk about redoing a lot of our network setup a
> > > while back, but not sure where that went. The thought was to
> > > completely seperate Fedora from anything else (which would be
> > > great), but would require rework on a bunch of things. Once it's
> > > done however, we could not have to care as much about adding new
> > > private nets, etc. 
> > 
> > That sounds like it would end up in a good place for isolating the
> > clients but it also sounds like a lot of work. On the other hand,
> > this "lull" between actively working on releases might be a "less
> > bad" time to do major changes like that but I'll refrain from
> > commenting more on it since I'd likely not be the one doing all the
> > work.
> 
> Yeah. Lots of other things in the mix and it's also something that
> would have to weigh on RHIT networking folks more, so we will see. 

OK, we'll plan on finding something that works without any physical
networking changes.

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: QA Client Network Isolation and FAS Groups

[Tim Flink]

On Tue, 14 Jan 2014 16:09:05 -0700
Kevin Fenzi  wrote:

> On Fri, 10 Jan 2014 11:09:16 -0700
> Tim Flink  wrote:
> 
> > For network isolation, I don't pretend to be an expert on networking
> > so I'll describe the functionality that we're looking for and what I
> > think might work for a solution, but I'll defer to the expertise
> > here on whether it's a good idea or not :)
> > 
> > The beaker and taskotron clients will need network access to several
> > Fedora systems in order to work.
> > 
> > Taskotron Clients:
> >  - Taskotron buildmaster
> >  - bodhi, koji, repos, dist-git, task-git (part of taskotron, not
> > yet created), resultsdb (also part of taskotron)
> > 
> > Beaker Clients:
> >  - Beaker server and lab controller (same system for now)
> >  - repos, maybe grabbing packages from koji/bodhi
> 
> ok, and to be clear the koji/bodhi/dist-git is all public stuff
> right? (ie, it could get it via public ip ok?)

Yeah, it's all access to the public apis and URLs.

> > I put together a quick diagram with the various network connections:
> > http://tflink.fedorapeople.org/taskotron/client-network-connections.png
> 
> Cool. All those arrows are bidirectional?

Anything outside the box is not - the clients need to be able to access
those resources but those resources do not need to be able to access
the clients.

> Are all the ones outside the box http/https?

Yes. The only stuff that's not http/https is the connections from
bastion, the beaker server and the taskotron master(s).

The one thing that isn't part of that diagram is some sort of
lockbox-ish host for managing and monitoring the clients. I'm still not
sure how we want to configure all that (we had talked about putting
everything but the clients in the infra playbook on lockbox01 and
managing the clients with a different playbook on a different host but
none of that has been done), though but I suspect that is one of the
_simpler_ parts of the setup.

> > From a few previous conversations, I think that a private network
> > for the clients could provide the isolation that we're looking for.
> > As far as getting network access to the systems needed to function,
> > I figured that the beaker server and taskotron master would have
> > network interfaces on this private network and a gateway could be
> > used to restrict outgoing traffic to only the resources required.
> 
> So, in some senses the 'qa' network is this. It's restricted from
> talking to other internal stuff with some exceptions. 
> 
> Sadly over time, we have grown the number of things in that network
> and of course all the stuff in that network can talk to each other
> (barring local firewalls). 

Isn't that what usually happens to stuff like the qa network over
time? :)

> > All of the clients would be hosted on the qa virthosts, which are
> > currently in the same rack. I was thinking that it would be possible
> > to use one of the network interfaces in these virthosts to create
> > this private network (assuming that the network switch capacity is
> > available) but I'm definitely open to other ideas.
> 
> Could we just do it with a private libvirt network on the qa
> virthosts? ie, pick 172.31.17.0 and put them all in that and setup a
> bastion host as their gateway that does NAT for them out to the stuff
> they need. Or would NAT not work for this? They would still
> physically be on the qa network tho, so I guess we could try and
> request a real seperate one from RHIT. 

I'm not sure I understand this completely. Would this allow the various
"special" hosts (beaker server and bastion host in particular) to
connect to the clients without adding the step of "connect to
virthostX" before being able to ssh into the clients?

NAT should work for everything other than communication with the beaker
server and bastion hosts.

Another option might be to use ebtables. I haven't investigated this
fully yet but it sounds like it would allow the use of qa network IPs
but restrict the traffic running through the bridged network on the
virthosts - effectively creating a restricted network without needing
any physical changes. Of course, that would only work with VM clients
but that's all we're planning for at the moment.

> > Does this idea for network access and isolation seem reasonable and
> > do-able? I figure that the network isolation/access part will
> > require more discussion and time for implementation after a
> > decision is reached. Our systems will work fine with the current
> > network configuration but I wanted to get this part of the
> > conversation started so that the implemen

Re: QA Client Network Isolation and FAS Groups

[Tim Flink]

On Fri, 10 Jan 2014 11:29:40 -0700
Kevin Fenzi  wrote:

> On Fri, 10 Jan 2014 11:09:16 -0700
> Tim Flink  wrote:
> 
> > One of the things that QA is focusing on ATM is getting better
> > automation support for Fedora. We're currently doing this in two
> > ways (that will eventually combine):
> > 
> >  - taskotron (replacing autoqa)
> >  - beaker (different type of system - used by Red Hat QE, who is
> >interested in running some of their tests upstream in Fedora)
> > 
> > Before we start granting access to the systems to Fedora
> > contributors, we want to a) make the process relatively easy and b)
> > isolate the client machines so that if something goes wrong or a
> > bad actor gains access to them, any potential problems can be
> > limited.
> 
> Absolutely. :) 
> 
> > To do this, I'd like to propose one change and ask for input on
> > another.
> > 
> > The first change is with FAS groups and shell access. To the best of
> > my knowledge, in order to ssh into the bastion hosts, a user needs
> > to be a member of the sysadmin group. While this works well for
> > sysadmin people, I'm not sure I see a benefit of sending all
> > sysadmin email to folks who are just looking to debug beaker or
> > taskotron clients. I'd like to propose the creation of a new FAS
> > shell group for qa automation and separate groups for beaker users,
> > beaker admins, taskotron users and taskotron admins (qa-automation,
> > beaker, beaker-admin, taskotron, taskotron-admin).
> 
> So, it's a bit more complex than that. ;) 
> 
> All 'sysadmin-*' groups require you to be a member of 'sysadmin'
> first. This means if you are being added to 'sysadmin-qa' or
> something, you have to first get added to sysadmin. sysadmin is a
> tracking group, it provides no shell access to anything, but it does
> provide other things: 
> 
> * nagios alerts
> * all commits from puppet/ansible/etc repos via email
> * Can push commits to puppet/ansible/etc repos (but need to be in some
>   other group that provides them shell access to bastion and then
>   lockbox01 where the repos are). 
> 
> So, the idea is that anyone is a sysadmin subgroup gets all the info
> and commits and can see whats going on, etc. 

Ha, I'm pretty sure I've been corrected on this misunderstanding
before and forgot. Thanks for the reminder :)


> > The qa-automation group wouldn't need access to bastion01, just
> > whichever bastion host has access to the automation clients
> > (bastion-comm01 for now). The other groups would be kind of like the
> > relationship between the various sysadmin FAS groups and the base
> > sysadmin group (ie, sysadmin access for all members, sysadmin-dns,
> > sysadmin-qa etc. for farther access).
> > 
> > Does this seem reasonable and do-able? If so, I'd like to get the
> > FAS group stuff done in the relatively near future - not "drop
> > everything and do it now" but ideally in the next couple of weeks
> > if possible.
> 
> I have no objection. When we setup the sysadmin-qa group we made it
> like the rest of the sysadmin groups, but mostly that was due to me
> not being sure there wasn't something else going on with sysadmin
> group. :) 
> 
> I'm not sure you need admin and users groups seperate? Could you just
> have one group and make some people admins in the group and key off
> that? 

For the moment, it doesn't matter much beyond shell access because we
don't have FAS integration for either taskotron or beaker. There are
admin tasks for beaker that and there will be admin tasks for
taskotron. I'd prefer not to have more interfaces for group/permission
management than we need to have but as long as we can manage shell
access and base group membership to the two systems, we'll make it work.
 
> I think yeah, bastion-comm01 should mostly work fine, except if people
> in those groups need to commit to ansible/puppet/etc repos. Would
> they? Or would that be something we keep with sysadmin-qa?

The taskotron user and admin groups would be separate from any sysadmin
groups - I'd like to keep sysadmin-qa separate. I don't want to be
giving shell access to all our virthosts, write access to the infra
playbooks etc. to every user of beaker and taskotron who needs shell
access to the clients.

> > For network isolation, I don't pretend to be an expert on networking
> > so I'll describe the functionality that we're looking for and what I
> > think might work for a solution, but I'll defer to the expertise
> > here on whether it's a good idea or not :)
> 

QA Client Network Isolation and FAS Groups

[Tim Flink]

One of the things that QA is focusing on ATM is getting better
automation support for Fedora. We're currently doing this in two ways
(that will eventually combine):

 - taskotron (replacing autoqa)
 - beaker (different type of system - used by Red Hat QE, who is
   interested in running some of their tests upstream in Fedora)

Before we start granting access to the systems to Fedora contributors,
we want to a) make the process relatively easy and b) isolate the
client machines so that if something goes wrong or a bad actor gains
access to them, any potential problems can be limited.

To do this, I'd like to propose one change and ask for input on another.

The first change is with FAS groups and shell access. To the best of my
knowledge, in order to ssh into the bastion hosts, a user needs to be a
member of the sysadmin group. While this works well for sysadmin
people, I'm not sure I see a benefit of sending all sysadmin email to
folks who are just looking to debug beaker or taskotron clients. I'd
like to propose the creation of a new FAS shell group for qa automation
and separate groups for beaker users, beaker admins, taskotron users
and taskotron admins (qa-automation, beaker, beaker-admin, taskotron,
taskotron-admin).

The qa-automation group wouldn't need access to bastion01, just
whichever bastion host has access to the automation clients
(bastion-comm01 for now). The other groups would be kind of like the
relationship between the various sysadmin FAS groups and the base
sysadmin group (ie, sysadmin access for all members, sysadmin-dns,
sysadmin-qa etc. for farther access).

Does this seem reasonable and do-able? If so, I'd like to get the FAS
group stuff done in the relatively near future - not "drop everything
and do it now" but ideally in the next couple of weeks if possible.

For network isolation, I don't pretend to be an expert on networking so
I'll describe the functionality that we're looking for and what I think
might work for a solution, but I'll defer to the expertise here on
whether it's a good idea or not :)

The beaker and taskotron clients will need network access to several
Fedora systems in order to work.

Taskotron Clients:
 - Taskotron buildmaster
 - bodhi, koji, repos, dist-git, task-git (part of taskotron, not yet
   created), resultsdb (also part of taskotron)

Beaker Clients:
 - Beaker server and lab controller (same system for now)
 - repos, maybe grabbing packages from koji/bodhi

I put together a quick diagram with the various network connections:
http://tflink.fedorapeople.org/taskotron/client-network-connections.png

From a few previous conversations, I think that a private network for
the clients could provide the isolation that we're looking for. As far
as getting network access to the systems needed to function, I figured
that the beaker server and taskotron master would have network
interfaces on this private network and a gateway could be used to
restrict outgoing traffic to only the resources required.

All of the clients would be hosted on the qa virthosts, which are
currently in the same rack. I was thinking that it would be possible to
use one of the network interfaces in these virthosts to create this
private network (assuming that the network switch capacity is
available) but I'm definitely open to other ideas.

Does this idea for network access and isolation seem reasonable and
do-able? I figure that the network isolation/access part will require
more discussion and time for implementation after a decision is
reached. Our systems will work fine with the current network
configuration but I wanted to get this part of the conversation started
so that the implementation could happen before we get too far with
automation development.

Thanks,

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: 2014 dreaming

[Tim Flink]

On Wed, 8 Jan 2014 11:18:29 -0500
Ralph Bean  wrote:



> One more for the list:
> 
> * Increased automatic QA/continuous integration:
>   - More tests posting back to bodhi.

Actually, one of my goals is to kill off the current method of posting
to bodhi. In my mind, it's the wrong way to be doing things and causes
problems on our end. I want to have a restful api for results querying
and use that as an interface instead of posting things directly to
bodhi.

I talked to Luke about this informally at either PyCon or Flock and
IIRC, he liked the idea. It'll take some work to make all that happen
and I'm not expecting it to come with the initial deployment of
taskotron, but it's on my mind for the future.

>   - Dependencies between packages evaluated (if a package is updated,
> its dependants should be evaluated against it).

If you're talking about rpm dependencies, that should already be taken
care of with depcheck. I've honestly not given much thought to any
attempts to get into "do these packages actually work with eachother"
beyond checking for ABI breakage.

>   - A clear way for contributors to submit tests.
>   - fedmsg messages all around.

These are both on our todo list. The fedmsg part is going to be an
interesting conversation, though - I'm not quite sure how to handle
reporting and notifications. Interestingly enough, starting that
conversation on qa-devel@ (for now, eventually moving to devel@) is on
my todo list today :)

Tim


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break request: Configuring backups for qadevel.cloud.fedoraproject.org

[Tim Flink]

On Fri, 1 Nov 2013 15:53:54 -0600
Tim Flink  wrote:

With that +2, I've committed and pushed the change.

Thanks,

Tim

> I've recently set up a phabricator instance to trial for qa work. Even
> though it's a dev host, we're still using it as our bug tracker etc.
> and I'd like to have it backed up.
> 
> This patch adds qadevel.cloud.fedoraproject.org as a host to be backed
> up and adds /srv/backup (where the mysqldump goes)
> and /var/lib/phabricator/files (where file attachments to issues go)
> as additional directories to be backed up
> 
> As it's still beta freeze, I'm requesting a freeze break to push this
> change into ansible.
> 
> Thanks,
> 
> Tim
> 
> 
> diff --git a/inventory/host_vars/qadevel.cloud.fedoraproject.org
> b/inventory/hos new file mode 100644
> index 000..6bf9e9d
> --- /dev/null
> +++ b/inventory/host_vars/qadevel.cloud.fedoraproject.org
> @@ -0,0 +1,2 @@
> +---
> +host_backup_targets: ['/var/lib/phabricator/files', '/srv/backup']
> diff --git a/inventory/inventory b/inventory/inventory
> index dc7675b..b68e44c 100644
> --- a/inventory/inventory
> +++ b/inventory/inventory
> @@ -60,6 +60,7 @@ people03.vpn.fedoraproject.org
>  pkgs01.phx2.fedoraproject.org
>  log02.phx2.fedoraproject.org
>  value03.phx2.fedoraproject.org
> +qadevel.cloud.fedoraproject.org
>  
>  [badges-backend]
>  badges-backend01.phx2.fedoraproject.org



signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze break request: Configuring backups for qadevel.cloud.fedoraproject.org

[Tim Flink]

I've recently set up a phabricator instance to trial for qa work. Even
though it's a dev host, we're still using it as our bug tracker etc.
and I'd like to have it backed up.

This patch adds qadevel.cloud.fedoraproject.org as a host to be backed
up and adds /srv/backup (where the mysqldump goes)
and /var/lib/phabricator/files (where file attachments to issues go) as
additional directories to be backed up

As it's still beta freeze, I'm requesting a freeze break to push this
change into ansible.

Thanks,

Tim


diff --git a/inventory/host_vars/qadevel.cloud.fedoraproject.org
b/inventory/hos new file mode 100644
index 000..6bf9e9d
--- /dev/null
+++ b/inventory/host_vars/qadevel.cloud.fedoraproject.org
@@ -0,0 +1,2 @@
+---
+host_backup_targets: ['/var/lib/phabricator/files', '/srv/backup']
diff --git a/inventory/inventory b/inventory/inventory
index dc7675b..b68e44c 100644
--- a/inventory/inventory
+++ b/inventory/inventory
@@ -60,6 +60,7 @@ people03.vpn.fedoraproject.org
 pkgs01.phx2.fedoraproject.org
 log02.phx2.fedoraproject.org
 value03.phx2.fedoraproject.org
+qadevel.cloud.fedoraproject.org
 
 [badges-backend]
 badges-backend01.phx2.fedoraproject.org


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure