List of needed package reviews for fedmsg to move forwards

[Ralph Bean]

Here is the list of needed package reviews for fedmsg.

 - 810335
 - 720818
 - 811689
 - 811732
 - 811739
 - 811750
 - 811759
 - 811769
 - 811782
 - 812030
 - 812059

These ones are already done

 - 810033
 - 810382
 - 810386

These two are not package reviews, but are tickets that need to be
resolved in order to move forwards with fedmsg in stg:

 - 813925
 - 813915

All of the above are dependencies of the latest major version bump of
Moksha.  I haven't yet submitted the review request for python-fedmsg itself,
but it's coming soon.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: List of needed package reviews for fedmsg to move forwards

[Ralph Bean]

On Thu, Apr 19, 2012 at 02:26:58PM -0400, Ralph Bean wrote:
> Here is the list of needed package reviews for fedmsg.
> 
>  - 810335
>  - 720818
>  - 811689
>  - 811732
>  - 811739
>  - 811750
>  - 811759
>  - 811769
>  - 811782
>  - 812030
>  - 812059

Sending out a plaintext list of the bug ids was probably not that helpful.
Here's a friendlier link to the dependency tree for python-fedmsg so you
can view them all at once:

 https://bugzilla.redhat.com/showdependencytree.cgi?id=818297&hide_resolved=1

Even better, here are two links to the best next two tickets for review:

 https://bugzilla.redhat.com/show_bug.cgi?id=811689
 https://bugzilla.redhat.com/show_bug.cgi?id=811759
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Scheduling new meeting time for Messaging SIG

[Ralph Bean]

The Messaging SIG has been meeting on Tuesdays at 16.00 UTC.  Turnout
has been a little low and we concluded that it's not the best time.

If you're interested in participating in meetings, please fill out the
following survey so I can get a good idea of when to place them:

  http://whenisgood.net/fedmsg

For the curious, here's a list of the archived meeting logs:

 https://github.com/ralphbean/fedmsg/blob/develop/doc/meetings.rst

-threebean

___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

fedmsg status, production, and help with testing?

[Ralph Bean]

I've been working on the Messaging SIG work for fedmsg for a few months now and
have been active in IRC, but silent on this list.  Here's a status report.

At this point, I've developed an API for sending and receiving messages and have
patched some existing services for only sending.  Nothing *depends* on fedmsg
messages arriving at this point, nor will anything so depend for a long while
out.  We want to be able to test and make sure that the fedmsg message bus is
reliable and secure before we do anything like that.

Status
--

In staging, the following services are emitting fedmsg messages.

 - bodhi
 - fas
 - mediawiki
 - tagger
 - scm/gitolite/fedpkg

Additionally, there is a new service running on app01.stg called "fedmsg-relay".
There are also two new commands: "fedmsg-logger" and "fedmsg-tail".

The messages are all signed by service-host specific RSA keys.  Documentation
for the CA and generating new cert/key pairs can be found at
http://infrastructure.fedoraproject.org/infra/docs/fedmsg-certs.txt

Motivation
--

I'd like to move most of the puppet modules I have from "modules-staging" to
"modules" sooner than later.  I'm worried about that directory bloating and
getting out of sync with the main modules directory over the long winter of the
freeze.  There's much more work to be done and the freeze offers a good
opportunity to assess and commit to a setup.  However, the freeze is set for
next Tuesday, so soon!

lmacken is preparing a bodhi1 release before the freeze which will include the
fedmsg work and relrod is working on a fas release.  That leaves mediawiki,
tagger, and scm/gitolite/fedpkg on which we'll need to coordinate a push to
production or decide that they stay in staging for the time being.

For anyone with the time/energy, here is a set of instructions for how
to test what I've set up.  I've tested it pretty thoroughly, but peer review is
best.

Grotesque Detail


For all the tests, login to app0[1-7].stg or packages[1-2].stg and run
"fedmsg-tail".  There will be spam from the busmon consumer we have
running in staging, so its probably best to run
"fedmsg-tail | grep -v busmon".

Message signing is turned on globally from puppet in /etc/fedmsg.d/ssl.py.
On a given host (say, app01.stg), check /etc/pki/fedmsg to see if the
permissions on public certs and private keys makes sense.

 - fedmsg-logger

   - $ echo "this is a test" | fedmsg-logger

 - bodhi

   - Login to https://admin.stg.fedoraproject.org/updates
   - Messages are sent when you:

 - Make any change to an update.

 - fas

   - Login to https://admin.stg.fedoraproject.org/accounts
   - Messages are sent when you:

 - create a new user
 - edit your profile
 - apply for a group.
 - sponsor someone for a group.
 - create a group.
 - update a group.
 - remove a member from a group.

 - mediawiki

   - Login to https://stg.fedoraproject.org/wiki/Fedora_Project_Wiki
   - Messages are sent when you:

 - edit an article
 - upload something

 - tagger

   - Login to https://apps.stg.fedoraproject.org/tagger/
   - Messages are sent when you:

 - Upvote/downvote a tag
 - Add a new tag
 - Login

 - scm/gitolite/fedpkg

   - Change your /etc/rpkg/fedpkg.conf to point to stg.fedoraproject.org.
   - Messages are sent when you:

 - Push a new commit

Post Freeze Plans
-

Whether or not we can vet and push these into production before the freeze, I
will be working on the following bits afterwards:

 - Documentation sprint:  Some documentation for developers exists, but it
   hasn't had careful attention in a few months.  It should include both a
   How-To for developers and some discussion of how the fedmsg internals work.
 - Start in on patches that add hooks to other services.
   http://fedmsg.readthedocs.org/en/latest/status.html is my running list.  I
   think pkgdb, meetbot, and elections are next.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze break for post-update hook on pkgs01?

[Ralph Bean]

When we were adding the fedmsg hooks to pkgs01 and pkgs01.stg, we decided it
would be a nice "extra" to run "git update server-info" on each repo for each
push.  This made cloning via http possible.

We ran a script "git check-perms /srv/git/rpms --check=post-update --fix" on
pkgs01 that added this hook for every repo, but I forgot to add it to
"setup_git_package" so that it would be added for every new repo.  Now there
are a handful of repos that do not have the hook, while most others do.

Other than the inconsistency, this doesn't matter all too much.  It is
annoying, however, due to a cron job that is checking for the new hook,
failing to find it, and bothering sysadmins about it over email.

I'm seeking +1s for a freeze break to make the following two changes:

  1) Run "git check-perms /srv/git/rpms --check=post-update --fix" on pkgs01
 again to fix the handful of repos that are out of sync.

  2) Apply the following patch in puppet which will add the hook for new repos.

diff --git a/modules/gitolite/files/distgit/setup_git_package 
b/modules/gitolite/files/distgit/setup_git_package
index bd42b95..eeaa16f 100755
--- a/modules/gitolite/files/distgit/setup_git_package
+++ b/modules/gitolite/files/distgit/setup_git_package
@@ -121,6 +121,9 @@ ln -s /usr/share/git-core/post-receive-fedmsg \
 ln -s /usr/share/git-core/post-receive-chained \
 $GITROOT/$PACKAGE.git/hooks/post-receive
 
+# This executes "git update-server-info" on each push for clone via http
+ln -s /usr/share/git-core/templates/hooks/post-update.sample \
+$GITROOT/$PACKAGE.git/hooks/post-update
 
 rm -rf $TMPDIR
 echo "Done."
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break for post-update hook on pkgs01?

[Ralph Bean]

On Thu, Aug 16, 2012 at 07:32:39AM -0700, Toshio Kuratomi wrote:
> On Thu, Aug 16, 2012 at 09:54:46AM -0400, Ralph Bean wrote:
> >   1) Run "git check-perms /srv/git/rpms --check=post-update --fix" on pkgs01
> >  again to fix the handful of repos that are out of sync.
> > 
> Does this cause any end-user visible load on the server?

It takes a significant amount of time to run (a few hours).  But it
does not appear to impact user performance:

 1) We received no complaints last time we ran it
 2) I just tested cloning a package 3 times from pkgs.stg with and
without "git check-perms" running on the server.  The times taken
to complete the operations were indistinguishable.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break for post-update hook on pkgs01?

[Ralph Bean]

On Thu, Aug 16, 2012 at 08:55:16AM -0600, Kevin Fenzi wrote:
> (That just needs to run on the list of newer ones? or over the entire
> packages?)

Yeah, I can run it on only the newer ones.  That should lighten the
load.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break for post-update hook on pkgs01?

[Ralph Bean]

Ok, this should be all fixed up now.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze break request: tweak mediawiki's fedmsg plugin

[Ralph Bean]

Good evening,

The httpd root process has been dieing seemingly at random on the
different app0* servers.  We're not sure what it is, but there's a
chance that it could be the fedmsg-emit.php plugin.  Here's a proposed
patch that's been sitting in staging for about a week:

  http://www.fpaste.org/VqpG/

The theory is that, before, if "fedmsg-config" failed (which it
shouldn't), then the script would call die().  die() shouldn't
terminate the root httpd process, but.. well.. we're not sure
what else could be happening.

The change goes:  instead of calling die(), if the plugin can't
setup its fedmsg socket, then it just won't register its callbacks
with mediawiki, instead of calling die();

I've tested that this is sane in staging, but there's no way to test
that it fixes the issue, since it happens randomishly.

This doesn't require a new fedmsg rpm.  The fedmsg-emit.php plugin
is kept in puppet.

Seeking +1s for freeze break.

Cheers!
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break request: tweak mediawiki's fedmsg plugin

[Ralph Bean]

Done.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Review Requests for moksha (fedmsg)

[Ralph Bean]

fedmsg was pulling in way too many dependencies, including (almost) a
whole web stack.  None of it was even being used.  All of it came from
moksha which grew as an all-in-one, batteries-included framework.

I split moksha upstream into three separate namespace packages to
solve the problem.  Now they're up for package review.  If anyone is
free to lend a review, it'd be much appreciated.

https://bugzilla.redhat.com/showdependencytree.cgi?id=854605&hide_resolved=1

Some notes:

The package "moksha" will still exist, but now as a kind of
meta-package that just included the other three.  Its subpackages like
moksha-server (httpd) and moksha-doc (sphinx) will continue to exist
as well.

My changes will break most projects that use moksha until they can be
patched to use the new namespace.  For us, this includes:

 - The packages webapp
 - fedmsg
 - busmon

I've already prepared and tested commits to the development trees of
all three projects that work with the new moksha.

-threebean
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Review Requests for moksha (fedmsg)

[Ralph Bean]

On Wed, Sep 05, 2012 at 03:13:35PM +0200, Patrick Uiterwijk wrote:
> Hey,
> 
> I have taken the three reviews, will do them soon.
> 
> - Patrick

Thanks, Patrick.  :)
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: announcing repository updates over Fedora Message Bus

[Ralph Bean]

On Mon, Sep 10, 2012 at 08:56:26AM -0400, Kamil Paral wrote:
> I wonder, is it possible for Fedora Infrastructure team to implement 
> announcements of new repository pushes over Fedora Message Bus[1]?
> 
> By new repository pushes I mean updates to 'fedora', 'updates' and 
> 'updates-testing' repositories for all Fedora releases as well as updates to 
> 'branched' and 'rawhide' trees.

Some of this should already be happening.  There is a
"org.fedoraproject.prod.bodhi.mashtask.sync.done" message that should
be getting emitted, but we don't have a way to check the history and
verify yet. https://bugzilla.redhat.com/show_bug.cgi?id=853252 should
take care of that.

"branched" and "rawhide" are another story.  We'd need to add
"fedmsg-logger" statements to some of the scripts in
http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure
where yet.

> The benefits of this solution:
> 1. Mirrors that download from dl.fp.o could start syncing very soon after an 
> update is pushed (after a random timeout to decrease the load).
> 2. Those mirrors wouldn't have to periodically query the server for updates.
> 3. The duration when those mirrors are outdated would be shorter.
> 
> I can just guess about other people, but in our office I believe the message 
> bus would help a lot. Currently we query the master server every 4 hours for 
> updates. That presents a lot of bandwidth/IO load on both sides. Using the 
> message bug there would be no useless queries. Also it would help us stay 
> up-to-date more often that we currently are (if 'branched' is updated an hour 
> after our last query, there are 3 hours wasted until we run rsync again).
> 
> This solution could be extended in the future to all Fedora public mirrors, 
> so that they could announce using our message bus when they have finished 
> syncing and users that sync from that particular mirror would know when to 
> start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my 
> private mirror, I would wait for their announcement on our message bus).
> 
> What do you think?

This sounds great for the future.  I've been working on docs this week
(not quite done yet).  I'll have docs out soon with examples on how to
do something like this.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: announcing repository updates over Fedora Message Bus

[Ralph Bean]

On Mon, Sep 10, 2012 at 12:07:12PM -0400, Kamil Paral wrote:
> Exciting! Please don't forget to share on planet.fp.o once you have 
> something. This is great stuff.

Updated docs:

 - http://fedmsg.rtfd.org/
 - http://fedmsg.readthedocs.org/en/latest/consuming.html

I'm going to hold off sharing on the planet until we can get a port
open for external message consumption of the production bus.  Right
now, only staging is available (with practically no activity).

After the freeze, hopefully we can get that in place.  An announcement
pointing folks to the docs will be more timely then.  :)
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: announcing repository updates over Fedora Message Bus

[Ralph Bean]

On Mon, Sep 10, 2012 at 11:30:15AM -0400, Ralph Bean wrote:
> "branched" and "rawhide" are another story.  We'd need to add
> "fedmsg-logger" statements to some of the scripts in
> http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure
> where yet.

Just an update:

I talked with dgilmore about this today and he'd like to wait until a
"composedb" app is written so we can have all the fedmsg statements in
a nice, centralized place (not strewn out across all the releng
scripts).
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: paste to production?

[Ralph Bean]

On Tue, Dec 04, 2012 at 11:23:09PM -0700, Clint Savage wrote:
> One thought here might be to implement a link shortener. Something
> like fed.io or fedora.io might be cool. I did also think about fpas.te
> which would work well. However, the thought here is these would be
> good only if having something that doesn't end in fedoraproject.org is
> possible.

I think a link shortener would be great for general use.  If we could
grab http://fed.io, then the "fpaste" cli tool could automatically
pass your paste.fedoraproject.org url through the shortener on each
run.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Fudcon lawerence stuff

[Ralph Bean]

On Tue, Jan 22, 2013 at 09:34:43AM -0700, Kevin Fenzi wrote:
> Greetings. 
> 
> We discussed lots of things at fudcon, and I figured I would try 
> (I know I will miss things) to summarize/note those things for folks on
> the list: 
> 
> 1) we got rid of the /etc/host alias for admin.fedoraproject.org inside
> phx2. This turned out to hit a few corner cases, but we got it all
> fixed. This means that proxy01 is NOT a SPOF for all fas access inside
> phx2 anymore. 
> 
> 2) Shut down insight hosts that are no longer being used. We still have
> some cleanup to do there... remove them from haproxy, close old tickets
> around that, make sure there's no where else they are mentioned. 
> 
> 3) ppc secondary arch folks might want us to help backup their koji db
> or the like. We need to figure out how to do that. 
> 
> 4) Talked about puppet to ansible transitioning. This would not be a
> hard transition/mass changeover, but rather a gradual one. We need need
> to lay some more groundwork. We should probibly whip up a wiki page on
> planning this. 
> 
> 5) Talked about external fedmsg messages. Looks like we need two
> interfaces: one that can do certs like our internal stuff (this would
> be from applications/things we trust), and a web posting thing that
> would be from end users (for badges, etc). 
> 
> 6) Allowed projects to use github for their SCM needs if they choose
> to. Note that this doesn't mean we will be using anything at all non
> free in infrastructure, it just means free projects we use can choose
> their own hosting. 
> 
> 7) mirrormanager 1.4 is coming to staging soon, and some new folks are
> interested in helping maintain it now too. ;) 
> 
> 8) Spot talked about cool ideas around app store and badges. Hopefully
> their will be some new resources to help work on these, but we already
> have a good start on badges at least. Need to be ready for that work to
> ramp up. 
> 
> 9) QA is testing out using our openstack cloud for their autoqa needs.
> Hopefully we can get feedback from that and setup things as best works
> for them. 
> 
> 10) Moved openid out of fas to it's own service as part of thinking
> about moving to a model where each service has it's own small container
> or vm instead of everything on one big app server. 
> 
> 11) Work on the websites to not pull translations on every build. More
> work needed to finish this up on my part. 
> 
> 12) Added a HOTFIX SOP. I want us to use one additional step: check in
> the orig file, then check in a diff on top of that. This allows us to
> see what things the hotfix actually changed. 
> 
> 13) Seth found some interesting trends in mirrormanager requests over
> the last year. Luke is importing all that data into some tools to try
> and get a better view of whats happening there. 
> 
> ...and I bet there's more stuff that I forgot. If you remember
> something you worked on, add it here in a reply. ;) 
> 
> Things I wanted to get done, but didn't: 
> 
> 1) Pick a FAD topic. We have no lack of topics for a FAD. If someone
> wants to push a particular one, please do shout out about it. 
> 
> Overall a very productive fudcon. :) 
> 
> Thanks to everyone who came and worked on things!
> 
> You all rock. 
> 
> kevin

Also:

- Worked out an API design for datagrepper with Ian Weller.

- Figured out a mechanism for end users to validate incoming fedmsg
  messages.  Available in fedmsg-0.6.6. 

- Toshio and I brought pkgdb into the fedmsg brood.

- We moved a new release of the /packages webapp to production which
  includes my dogpile cache work.  Those patches got accepted upstream which
  I'm pretty happy about.  It's hitting our memcached servers pretty hard and
  having trouble, which makes me sad.

- I updated fedmsg in our production environment to be using zeromq3's
  TCP_KEEPALIVE features (finally).  We'll see if this solves our message
  dropping problem.

- dgilmore and I talked about hooking koji up to fedmsg.  We have a plan and it
  shouldn't be too much longer.  Getting messages out of koji was the #1
  request I heard at FUDCon.

- Also talked with Dennis about getting fedmsg messages out from secondary arch
  composes and with Seth about getting messages in from coprs/private cloud
  (Kevin mentioned this).

- `Jordan Sissel `_ remotely hooked fedmsg up
  to `the logstash demo `_.


pgp9hmdt_nQcA.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Fedocal to prod?

[Ralph Bean]

On Tue, Feb 12, 2013 at 09:15:42PM +0100, Pierre-Yves Chibon wrote:
> On Tue, 2013-02-12 at 13:00 -0700, Kevin Fenzi wrote:
> > On Tue, 12 Feb 2013 19:42:56 +0100
> > Pierre-Yves Chibon  wrote:
> > 
> > > Hi all,
> > > 
> > > I am getting some pressure to move fedocal to production (which is
> > > pretty nice).

Good!  :)

> > I think admin.fedoraproject.org/calendar or perhaps
> > apps.fedoraproject.org/calendar would be fine. 
> 
> hm, since it needs login shouldn't it be admin?

I'm not sure what the original reason for the distinction was, but fyi
apps.fedoraproject.org/tagger requires login.

If it's just an aesthetic choice, I prefer the apps.fp.o/calendar one.
Either way, I can add this to the apps.fp.o/ landing page when its
ready.

-Ralph


pgpvtG6j4fuV6.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request - koji fedmsg plugin

[Ralph Bean]

dgilmore noted in channel that the koji fedmsg plugin is producing
tracebacks like the following:

 Error running postBuildStateChange callback from 
_koji_plugin__fedmsg-koji-plugin: Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/koji/plugin.py", line 165, in 
run_callbacks
 func(cbtype, *args, **kws)
   File "/usr/lib/koji-hub-plugins/fedmsg-koji-plugin.py", line 94, in 
send_message
 body = get_message_body(topic, *args, **kws)
   File "/usr/lib/koji-hub-plugins/fedmsg-koji-plugin.py", line 50, in 
get_message_body
 msg['owner'] = kojihub.get_user(info['owner_id'])['name']
 KeyError: 'owner_id'

The error is not occurring for *all* postBuildStateChange calls, only for
some.  We are not sure if this has been happening since the fedmsg plugin
was introduced or if it is a result of the koji upgrade that happened last
night.  It is not causing koji to fail, but the subset of build state change
messages that hit this traceback are not being published.  It is also adding
noise to the koji logs.

The koji fedmsg plugin is kept in puppet, so we can patch it there (it is
not in an rpm).  The following patch should do the trick:
http://ralph.fedorapeople.org/0001-Silence-intermittant-koji-tracebacks.patch

For some messages, a None (null in json) will be published for the first time
in the owner field.  The code that parses those messages on the consuming
side can be found here: http://bit.ly/16jqhtq  It looks like it can handle
a None without causing any further issues.  I can't say for certain about any
third-party code that may be listening to koji/buildsys messages and whether
or not it can handle a None/null.

I'm looking for two +1s to apply the patch above to clean up the koji logs.

-Ralph


pgppa0VQEJseC.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [Change Request] python-fedora two timeout fixes and a flask_fas_openid fix

[Ralph Bean]

On Thu, Apr 04, 2013 at 12:04:13PM -0700, Toshio Kuratomi wrote:
> Could I get two +1s ?

+1


pgp_v6wcqh9lL.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Self introduction: Ankur Sinha

[Ralph Bean]

On Sun, Apr 14, 2013 at 11:28:21AM -0600, Kevin Fenzi wrote:
> On Sat, 13 Apr 2013 19:28:44 +1000
> Ankur Sinha  wrote:
> 
> ...snip...
>  
> > I'd like to step up and join the infra team. My interests are limited
> > to askfedora[3] at the moment, where I'm a forum moderator. I've
> > filed a few tickets[4][5] which could be classified as easyfixes. I'd
> > like to take care of them myself if possible, and then help Ralph and
> > the rest of the team in maintaining/updating/.. askfedora, to start
> > with. I'd request someone to mentor me though. I'm really not
> > confident enough to work on these by myself. (I rather think I might
> > blow something up being a complete noob at sysadmin-ing!). I'd be
> > most grateful if I could be added to the fi-apperentice group[6], and
> > given a few instructions that would get me started with askfedora's
> > staging instance[7]?
> > 
> > My IRC handle is FranciscoD. My user page is here[8]. 
> 
> Welcome!

Agreed!

> we can get you started and access to ask to work on it. :) 
> 
> you are in sysadmin-ask, so that should get you access to login to
> lockbox01 and ask01/ask01.stg. ;) 

If you have any questions at all about the askbot setup, feel free to
ask me.  I'm 'threebean' in #fedora-apps and #fedora-admin.


pgpNWO12C3XcH.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break request: update mediawiki

[Ralph Bean]

On Mon, Apr 15, 2013 at 11:45:01PM +0200, Patrick Uiterwijk wrote:
> Hello,
> 
> So today a mediawiki security update was released (v1.19.5), with some fixes 
> being for remote script invocation.
> I would like to request a freeze break to update this on the app servers in 
> production.
> In staging, the update has been succesful, and no database changes are needed.
> 
> Thanks in advance,
> Patrick Uiterwijk

+1


pgp3iKVEjiLQI.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Idea for a webapp: "Fedora Oculum"

[Ralph Bean]

I started a new webapp over the weekend and Seth helped me get it set
up on a cloud node this morning at http://209.132.184.189/

The idea is that you'll be able to click through the user's avatars to
their profile page at, for instance, http://209.132.184.189/pingou.

The charts from their page are embeddable, so we can use it like
gravatar.com.  Pierre could embed
http://209.132.184.189/pingou/radar/?style=dark_solarized on his wiki
page or personal site.

It would need lots of cleaning work:

  - The charts' data gets swamped out by the mass branch.  Some
math.log can help take care of that.
  - The 'dots' chart needs a 'widescreen' aspect ratio.  Not yet sure
how to do that.
  - It would be cool to be able to click on the dots from the dot
chart and be taken to a page listing the actual events of that
day.
  - It needs good end-user docs.
  - I need to figure out the click through for that front page.
  - Packaging and more..

I think its neat but I'm looking for feedback.  Is this an idea I should
pursue?  Is it something we want?

The source is at http://github.com/ralphbean/oculum for now.

-Ralph


pgpde5iYE7Psk.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Ask Fedora status update

[Ralph Bean]

On Thu, Apr 25, 2013 at 05:50:36PM -0400, Rahul Sundaram wrote:
> Hi
> 
> (cc'ing Ankur since he has expressed interest in helping with the admin
> tasks)

This is a good task list to keep track of.  Thanks for compiling it,
Rahul.

Ankur, I'd be glad to help brainstorm about how to fix each of these.
Feel free to email me directly or get in touch on IRC if you'd like to
start working through them.


pgp7xZyTe9zV7.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Nagios checks for datanommer/fedmsg

[Ralph Bean]

I came up with a draft of a nagios/nrpe check for datanommer/fedmsg

https://gist.github.com/ralphbean/5482129

It queries the datanommer DB and asks for the time since the latest
message in a particular category (i.e., bodhi, buildsys/koji, askbot).

We could configure it to raise a warning if, for example:

- It hasn't seen a buildsys message in the last 30 minutes.
- It hasn't seen a bodhi message in the last 6 hours.
- It hasn't seen a fedoratagger message in the last 2 months.

Since nagios alerts affect lots of people, it should probably be
discussed here or in the infra meeting before being rolled out.

Pierre pointed out in channel that this approach assumes that there *must*
be bodhi activity for such and such amount of time or else something
is wrong.  This could be problematic.  There are times like the
holidays in December when fewer people are contributing to Fedora, in
which case this plugin could throw false positives.  Accordingly, we
would need to set the WARN and CRIT thresholds to be generously long.


pgpYCmnp0WzZb.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request - Fedora Tagger

[Ralph Bean]

I'd like to request a freeze break request for Fedora-Tagger to cut
a new release and push it to production.  I'm in the process of trying
it out in staging now.

The new release includes fixes to 5 bugs:

https://github.com/fedora-infra/fedora-tagger/issues/90
https://github.com/fedora-infra/fedora-tagger/issues/89
https://github.com/fedora-infra/fedora-tagger/issues/84
https://github.com/fedora-infra/fedora-tagger/issues/87
https://github.com/fedora-infra/fedora-tagger/issues/88

They are all frontend/UI/javascript fixes.


pgps7ftDfDV3x.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request - Fedora Tagger

[Ralph Bean]

On Wed, May 15, 2013 at 01:35:42PM -0700, Toshio Kuratomi wrote:
> Since tagger is now part of the compose process we probably want to start
> doing only small, targeted changes.
> 
> however, this is only the second day of beta freeze (not final).  I've
> reviewed the first four changes and they seem okay.  The last one is larger
> and more complex.  All have already had signoff to merge into the upstream
> repo.
> 
> These are all UI changes, so at worst people wouldn't be able to add and
> edit tags; it shouldn't affect the ability for tags to be pulled out of
> tagger for use in the compose process.
> 
> So I think updating ASAP would be okay but we should still try not to do
> this in the future.
> 
> -Toshio

Adding further comment: the change is 'high impact' in the sense that it
touches tagger which is now part of the compose process.  It is 'low
impact' in that it is mostly superficial UI changes.

It is also low-urgency.  It doesn't *need* to go in during the freeze.
It can wait until after.

I just made the announcement the other day and got an avalanche of bug
reports -- this knocks off the low hanging fruit.


pgpwyZZFdMpiR.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: Fedora Tagger Schema

[Ralph Bean]

This is a second freeze break request for Fedora Tagger.  I haven't
actually carried out the first one yet, but Kevin's mention of the
broken bodhi sqlitebuildtags url took us down a rabbit hole.

This is the traceback that tagger logs when the bodhi masher tries to
pull down its sqlitebuildtags:

   Traceback (most recent call last):
 File "/usr/lib/python2.6/site-packages/flask/app.py", line 1687, in 
wsgi_app
   response = self.full_dispatch_request()
 File "/usr/lib/python2.6/site-packages/flask/app.py", line 1360, in 
full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/lib/python2.6/site-packages/flask/app.py", line 1358, in 
full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/lib/python2.6/site-packages/flask/app.py", line 1344, in 
dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "/usr/lib/python2.6/site-packages/fedoratagger/api/api.py", line 461, 
in tag_pkg_sqlite
   return fedoratagger.lib.sqlitebuildtags()
 File "/usr/lib/python2.6/site-packages/fedoratagger/lib/sqlite_export.py", 
line 56, in sqlitebuildtags
   conn.executemany(insert_statement, rows)
   IntegrityError: columns name, tag are not unique

The problem is that the schema upgrade from tagger1 to tagger2 didn't
go smoothly in production, and we didn't notice.  There is supposed to
be a DB constraint that disallows duplicate tags on packages, but it is
absent.

We can't just apply that new constraint, because there are now two
duplicate tags in the DB.

We need to:

  1) Remove the duplicates
  2) Apply the constraint so it doesn't happen again

Here's the constraint we need to apply in postgres:

ALTER TABLE tag ADD CONSTRAINT unique_package_label UNIQUE (package_id, 
label);

Here's a script to remove the duplicates:

#!/usr/bin/env python

import os
os.environ['FEDORATAGGER_CONFIG'] = '/etc/fedoratagger/fedoratagger.cfg'

from sqlalchemy import func, and_
import fedoratagger
import fedoratagger.lib.model as m

print 'Looking now for package tag dupes.'
query = fedoratagger.SESSION.query(m.Tag).all()
results = {}

for tag in query:
results[tag.label] = results.get(tag.label, {})
results[tag.label][tag.package_id] = \
results[tag.label].get(tag.package_id, 0) + 1 

dupes = []
for label, packages in results.items():
for package, count in packages.items():
if count > 1:
dupes.append((label, package))

print "Found these package+tag dupes:", dupes

base_query = fedoratagger.SESSION.query(m.Tag)
for label, package_id in dupes:
query = base_query.filter(and_(
m.Tag.label==label,
m.Tag.package_id==package_id
)).all()

keep, rest = query[0], query[1:]
for dupe in rest:
fedoratagger.SESSION.delete(dupe)

fedoratagger.SESSION.commit()


pgprI3IY5mWst.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: Fedora Tagger Schema

[Ralph Bean]

On Thu, May 16, 2013 at 10:33:38AM -0600, Kevin Fenzi wrote:
> Can we test this in stg first? Then if all goes well do it in prod? 
> 
> +1 if we can do that to make sure there's no thinkos in the script,
> etc. 
> 
> (we should be able to dump/restore the db over to stg from prod so it
> has the same data too, right?)

Yup.  I already tried without dump/restore in stg and there were no
duplicates there.  (There are two duplicates in production).

I'll dump/restore and try it in stg and report if there are any issues.


pgp_KgzsqKI9h.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: areas where we can invest in automation?

[Ralph Bean]

On Tue, May 28, 2013 at 08:33:22AM -0400, Matthew Miller wrote:
> I was asked (with my Red Hat hat on) to put together a little report on
> areas in Fedora which could be improved with an investment in better
> automation. From what I'm working on myself, I'm aware of the gigantic need
> in the cloud images production process, and I've been keeping an eye on Tim
> Flink's autoqa revamp ideas. I expect there are others, because I know from
> my previous jobs that there's always a balance between building
> condiment-passing machines and just _passing the salt_. [1] Are there other
> things which could be made better if only someone came up with the spare
> time and resources to do the work?

Hi!

Some observations/ideas:

1) The packager workflow is pretty tedious.  There has been some
   improvement to it, but more can be done.  Things like
   fedora-review and fedora-create-review (and bodhi!) are a huge
   help.  But there are plenty of inefficient "blocking" points in the
   process.

   For instance, once a new package is approved, only then does the
   submitter declare what branches they want with an scm admin
   request.  They then wait for an scm admin to declare that they
   have created their branches, and then wait for a cronjob to run
   that gives them permission to push on those branches (manually).
   They then wait for their koji builds to finish to (manually) submit
   bodhi updates.

   It would be nice if we could automate that whole process -- once a
   package is approved, if there were a "make-it-so" button that
   required no further intervention from the packager (but still
   required the keen eye of an scm admin).

   There are further sequences down the pipeline like requesting that
   packages in testing be pushed to stable, but there are good
   arguments against automating that.

2) Continuous deployment for infrastructure.  It has been tossed
   around in IRC, possibly at FUDCon as well.  If application
   developers could "git push" on the develop branch and have those
   changes automatically roll out to our staging infrastructure --
   that would save a lot of time.  Packaging our apps, building rpms,
   signing them, copying them to our infra yum repos, rebuilding those
   repos, clearing the cache on the target machines, performing a yum
   update <-- that process is cumbersome.

   I suspect that the "release only when we have accumulated enough
   changes to warrant enduring the burdensome release process" mode
   of deployment (as opposed to "release early, release often") also
   poses somewhat of a barrier to new contributors.  They contribute
   a patch.. nice!  When does it go live?  When one of our
   overstretched sysadmin-mains can get around to it (it is required
   that one of them sign the package).

   Caveat #2.1:
   There are some ways around this.  Individuals can get around the
   requirement of having a sysadmin-main touch their test release by
   installing their rpm directly on the target machine.  They still
   have to jump through some hoops to make it happen.

   Caveat #2.2:
   This is one of the reasons we put so much work into our private
   cloud (dev nodes).  There is no barrier there for teams to set up
   their own continuous deployment mechanism.  This meets most needs,
   but we don't have a way to iterate rapidly on some of the more
   important pieces of our infrastructure.  Apps/services that
   interact with each other don't quite work out on isolated cloud
   nodes.  The bodhi masher?  Koji?  fedmsg?  mirror manager?  We
   can't necessarily test those on dev nodes (and some we can't test
   in staging -- resolving this down the road would save some
   headaches).


pgpGI2Jnx22XQ.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Pull Request Reviews

[Ralph Bean]

Hi infra team-

We have a handful of pull requests open on our github repo for which
I'd like to ask for help in reviewing:

1) fedbadges  4d PR#22  - Allow lambda expressions in the dat .. 
http://bit.ly/1aeY7qh
2) tahrir-api 4d PR#19  - Feature/ranking .. http://bit.ly/1dhjI0G
3) fedbadges  3d PR#23  - Publish a fedmsg message when a use .. 
http://bit.ly/14UUlkP
4) tahrir 3d PR#199 - Feature/publish message on rank cha .. 
http://bit.ly/14sVXfq
5) fedmsg 2d PR#173 - Feature/idempotent .. http://bit.ly/1643CQX

-

Number 1 is an independent change and is an enhancement to the badges
awarder that will make it so we can perform much more flexible
datanommer queries.  (It makes it so we can do flexible operations in
python after making the actual datanommer query but before we
conclude that a badge should or should not be awarded).

Number 2 is a requirement for numbers 3 and 4.  It does a number of things:
- it pulls the leaderboard calculation code from tahrir (the web
  frontend) into tahrir-api (the underlying lib used by both the
  badges awarder backend and the web frontend)
- it caches users' rank in the database which should improve
  performance on the webapp significantly.
- it adds a new `notification_callback` which we'll use to publish
  fedmsg messages about users' rank changing.  We want to award
  badges based on this kind of stuff, so its a nice bonus.

Number 3 simply makes use of Number 2.  It removes the old hardcoded
fedmsg stuff from the backend badges awarder.

Number 4 simply makes use of Number 2.  It removes the old leaderboard
code and the old hardcoded fedmsg stuff from the badges web frontend.

Number 5 is an independent change that will ultimately fix a bug in
the datagrepper web api.

-

Any help providing review would be much appreciated.

Cheers-
 -Ralph


pgp90mDnypJ2m.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Adding libravatar to trac

[Ralph Bean]

I gave a try at adding libravatar support to the tracvatar trac plugin
we currently use.  I just sent it upstream for review there:

  https://bitbucket.org/zzzeek/tracvatar/pull-request/1


pgpL_h20tjQZK.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Trac Housekeeping

[Ralph Bean]

On Thu, Aug 22, 2013 at 08:38:24PM -0500, Tim Krupinski wrote:
> Hello friends,
> 
> One thing i've noticed is that many of the tickets on the Trac haven't been
> updated in months.  Since i'm new, I thought a good warmup would be to get
> in touch with owners of tasks still not closed, with no activity in over a
> month.  So i'll be attempting to contact owners individually in the coming
> days.
> 
> I'm also open to any suggestions on this - if you think it's a waste of
> time or have some info that might help, let me know!
> 
> Thanks,
> 
> Tim

Tim, this sounds great to me.  Some tasks don't have owners -- in that
case you might try reaching out to the reporter instead to ask them if
whatever they reported is still an issue, or if there have been any
developments since they filed it.


pgpEmV4xXtNH_.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Stale Tasks Sheet

[Ralph Bean]

On Sat, Aug 24, 2013 at 11:08:49AM -0500, Tim Krupinski wrote:
> Hey all,
> 
> I've created a Google doc to track this.  I only have a few emails so far,
> but if anybody has info missing from the sheet I would appreciate updates!
>  Once I track down all the owner/reporter emails I'll start trying to get
> updates.
> 
> https://docs.google.com/spreadsheet/ccc?key=0AtlkXQqIZDb3dG0xSFBkQVBKUnVyUnI2THdidU8wNGc&usp=sharing
> 
> P.S. - I was looking around and didn't seem to find any type of member list
> with emails, so i've been using the mailing list archives to search for
> them.  Is there a better way?

Ah, there is a trick here.  You can use fasusern...@fedoraproject.org and
that should be able to reach him or her for most people.


pgp7U0w7fHXIl.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Pull Request Reviews

[Ralph Bean]

On Thu, Aug 22, 2013 at 09:04:03AM -0400, Ralph Bean wrote:
> Hi infra team-
> 
> We have a handful of pull requests open on our github repo for which
> I'd like to ask for help in reviewing:
> 
> 1) fedbadges  4d PR#22  - Allow lambda expressions in the dat .. 
> http://bit.ly/1aeY7qh
> 2) tahrir-api 4d PR#19  - Feature/ranking .. http://bit.ly/1dhjI0G
> 3) fedbadges  3d PR#23  - Publish a fedmsg message when a use .. 
> http://bit.ly/14UUlkP
> 4) tahrir 3d PR#199 - Feature/publish message on rank cha .. 
> http://bit.ly/14sVXfq
> 5) fedmsg 2d PR#173 - Feature/idempotent .. http://bit.ly/1643CQX

I'll wait another few days and then merge these even if they haven't
found a reviewer by then.  I'm pretty confident about them, but we can
sort out any bugs as we go.  (I'd rather we go than let them sit and
get stale.)


pgppzpXDsMMZc.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

freeze break request: take badges nodes out of the freeze list

[Ralph Bean]

By ansible's configuration, we have the badges frontend and backend nodes
frozen right now.  This seems like an oversight to me.  I have some new badge
definitions and art assets I'd like to push, but I thought I'd clear it with
the list first.



diff --git a/inventory/group_vars/badges-backend 
b/inventory/group_vars/badges-backend
index 723cf58..9ede836 100644
--- a/inventory/group_vars/badges-backend
+++ b/inventory/group_vars/badges-backend
@@ -2,6 +2,7 @@
 lvm_size: 2
 mem_size: 4096
 num_cpus: 2
+freezes: false
 
 # for systems that do not match the above - specify the same parameter in
 # the host_vars/$hostname file
diff --git a/inventory/group_vars/badges-web b/inventory/group_vars/badges-web
index eb40512..f70909f 100644
--- a/inventory/group_vars/badges-web
+++ b/inventory/group_vars/badges-web
@@ -2,6 +2,7 @@
 lvm_size: 2
 mem_size: 4096
 num_cpus: 2
+freezes: false
 
 # for systems that do not match the above - specify the same parameter in
 # the host_vars/$hostname file


pgpIxBoVLfwPP.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: freeze break request: take badges nodes out of the freeze list

[Ralph Bean]

On Wed, Sep 04, 2013 at 09:23:15AM -0600, Kevin Fenzi wrote:
> Yeah, this looks good. 
> 
> I wanted it to be that by default things assume frozen, and just failed
> to set it right on these. ;) 

Cool.  :)

Its pushed out now with new badges on the way!


pgpUKxStkoifH.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Intro to Infra Videos

[Ralph Bean]

On Fri, Sep 13, 2013 at 12:20:45PM -0400, Chris Roberts wrote:
> Ralph, Is it ok If I add these to the Magazine? We have made a
> section on the Fedora magazine site for videos, we already have 1
> which is Tatica's how to use Ask Fedora.

Yes, that sounds great.  But please do wait for Nitesh to have his
hand at editing them.  I promised them to Nitesh a month ago and I'd
rather not jump around him.


pgpZBzjq47cys.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Intro to Infra Videos

[Ralph Bean]

At Flock, there was some discussion about creating more videos
to introduce new contributors to different fedora teams.  I
volunteered there to make some for the infra team.  Nitesh has
volunteered to add nice intro/outro theming stuff and to do other
editing (like superimposing text and links where appropriate).

I came up with these two:

- http://threebean.org/videos/fedora-infra-introduction-general.ogv
- http://threebean.org/videos/fedora-infra-introduction-apps.ogv

Is this stuff off base?

I want to make a third introducing the -noc side of stuff.  I was
thinking it should include:

- How to apply for fi-apprentice
- How to log in to lockbox
- How to find your way around the puppet and ansible repos.
- How to find the list of SOPs

Any feedback?  Anything I should add/leave out/do differently?

Anyone else want to do the -noc one instead of me?  It would be better
to have a variety I think.


pgpVBM2Q8_jdp.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: freeze break request: simple alias update

[Ralph Bean]

On Fri, Sep 13, 2013 at 01:20:34PM -0600, Kevin Fenzi wrote:
> Here's another simple alias update: 
> 
> diff --git a/configs/system/smtp/aliases.template.erb
> b/configs/system/smtp/aliases.template.erb index ec44cec..337f08e 100644
> --- a/configs/system/smtp/aliases.template.erb
> +++ b/configs/system/smtp/aliases.template.erb
> @@ -173,7 +173,7 @@ census: npmccallum,kevin,toshio,ianweller,tflink
>  # User for openshift fedora-status instance
>  fedora-status: kevin,codeblock,puiterwijk
>  # User for openshift fedora magazine wordpress instance.
> -fedora-mag-admin: kevin,duffy,chrisroberts,mitzie
> +fedora-mag-admin: kevin,duffy,chrisroberts,mitzie,jzb
>  endoflife: tri...@lists.fedoraproject.org
>  
>  # Amazon cloud account, ticket #1903
> 
> +1s?

+1


pgpDMzQ26FgoZ.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [PATCH] Replace http link by https link as we deploy behind https

[Ralph Bean]

On Fri, Sep 13, 2013 at 06:24:44PM +, Pierre-Yves Chibon wrote:
> ---
> 
> I would like to request a freeze break to apply this change in the template 
> of easyfix.
> 
> It will avoid mixing http and https while we only use https.
> 
> +1?

+1.


pgp4MvLUUXihq.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Revoked fedmsg keys

[Ralph Bean]

Hi all-

This morning, Patrick and Pierre noticed that the fedmsg keys deployed
by ansible were being set on their systems as world readable (o+r).

Those keys have been revoked, regenerated, and redeployed with the correct mode
as of 15:00 UTC.

The following keys were affected:

bodhi-releng01.phx2.fedoraproject.org.key
bodhi-releng02.phx2.fedoraproject.org.key
copr-copr-be.cloud.fedoraproject.org.key
fedbadges-badges-backend01.phx2.fedoraproject.org.key
fedbadges-badges-backend01.stg.phx2.fedoraproject.org.key
nuancier-nuancier01.phx2.fedoraproject.org.key
nuancier-nuancier01.stg.phx2.fedoraproject.org.key
nuancier-nuancier02.phx2.fedoraproject.org.key
nuancier-nuancier02.stg.phx2.fedoraproject.org.key
shell-badges-backend01.phx2.fedoraproject.org.key
shell-badges-backend01.stg.phx2.fedoraproject.org.key
shell-badges-web01.phx2.fedoraproject.org.key
shell-badges-web01.stg.phx2.fedoraproject.org.key
shell-badges-web02.phx2.fedoraproject.org.key
shell-copr-be.cloud.fedoraproject.org.key
shell-nuancier01.phx2.fedoraproject.org.key
shell-nuancier01.stg.phx2.fedoraproject.org.key
shell-nuancier02.phx2.fedoraproject.org.key
shell-nuancier02.stg.phx2.fedoraproject.org.key
tahrir-badges-web01.phx2.fedoraproject.org.key
tahrir-badges-web01.stg.phx2.fedoraproject.org.key
tahrir-badges-web02.phx2.fedoraproject.org.key

The majority of our other keys deployed by puppet were not affected.

-Ralph


pgp3tDBQCsqsr.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: iptables rule for fedmsg+copr

[Ralph Bean]

We're looking to publish fedmsg messages from the copr backend.  The
node that lives on is in the cloud and so will need to follow the same
route as the secondary arch compose nodes.  We have an inbound fedmsg
relay running on busgateway01, port 9941, and haproxy proxies
connections to it from hub.fedoraproject.org, port 9941.

We have that external port firewalled to only allow connections from
the secondary arch compose nodes, and the copr backend!

I only discovered today, as we went to test this for the first time,
that the IP listed in manifests/services/proxy.pp is incorrect.

I'd like to make this change to set it straight:

diff --git a/manifests/services/proxy.pp b/manifests/services/proxy.pp
index 01b39d3..529b242 100644
--- a/manifests/services/proxy.pp
+++ b/manifests/services/proxy.pp
@@ -1012,7 +1012,7 @@ if $puppetEnvironment == 'staging'{
 tcpPorts => [ 80, 443, 873, 8080, 6081, 9939, 9940],
 custom => [
 # Allow copr-be.cloud to talk to the inbound relay.
-'-A INPUT -p tcp -m tcp --dport 9941 -s 209.132.184.142 -j ACCEPT',
+'-A INPUT -p tcp -m tcp --dport 9941 -s 209.132.184.131 -j ACCEPT',
 # Also, ppc-composer.qa.fedoraproject.org (secondary arch)
 '-A INPUT -p tcp -m tcp --dport 9941 -s 209.132.181.33 -j ACCEPT',
 # Also, s390-hub01.qa.fedoraproject.org (secondary arch)


This is potentially high impact in that it will be distributed to all of
our proxies (which everything depends on).  On the other hand, it is a
really simple change that only modifies the last chunk of that ip address.

Can I get two +1's?

-Ralph


pgplaiCd68bdZ.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request - supybot-fedmsg on value03

[Ralph Bean]

Backstory:  I released a new supybot-fedmsg about 5 weeks ago.  For
whatever reason, we didn't deploy it to production and try it out.
Last week with the updates and reboots it made its way on to value03.
There's a simple typo that's causing meetbot to malfunction.

Here's a patch that fixes the typo:
  https://github.com/fedora-infra/supybot-fedmsg/pull/7/files

Here's a new release of the rpm carrying that patch:
  http://koji.fedoraproject.org/koji/taskinfo?taskID=6230640

Since we just entered freeze, I'll need 2 +1s to add that rpm to our
repos and install it on value03.

-Ralph


pgpQAGzmdVGoA.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request - supybot-fedmsg on value03

[Ralph Bean]

On Tue, Nov 26, 2013 at 08:19:00PM -0500, Ralph Bean wrote:
> Backstory:  I released a new supybot-fedmsg about 5 weeks ago.  For
> whatever reason, we didn't deploy it to production and try it out.
> Last week with the updates and reboots it made its way on to value03.
> There's a simple typo that's causing meetbot to malfunction.
> 
> Here's a patch that fixes the typo:
>   https://github.com/fedora-infra/supybot-fedmsg/pull/7/files
> 
> Here's a new release of the rpm carrying that patch:
>   http://koji.fedoraproject.org/koji/taskinfo?taskID=6230640
> 
> Since we just entered freeze, I'll need 2 +1s to add that rpm to our
> repos and install it on value03.
> 
> -Ralph

Okay - the issue has been resolved.


pgpE9OIA7Y0qV.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request - fedmsg for freemedia fedorahosted instance

[Ralph Bean]

I just added a new badge this afternoon for people who complete
freemedia requests:

https://badges.fedoraproject.org/badge/free-the-fedora

It only needs the fedmsg hook to be enabled for the freemedia
fedorahosted instance.  There is a SOP for this:

http://infrastructure.fedoraproject.org/infra/docs/fedorahosted-fedmsg.txt

hosted03 is frozen right now, so I'm requesting a freeze break to
enable the hook.  It's a one-line change to the trac.ini for freemedia
and requires restarting apache.  It is of minor importance but would
have relatively low impact if things went wrong.

Can I get 2 +1s?

-Ralph


pgpYCLpsXPEPh.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request - fedmsg for freemedia fedorahosted instance

[Ralph Bean]

On Wed, Dec 04, 2013 at 10:13:16PM +0100, Pierre-Yves Chibon wrote:
> +1 for me as well

Ok, its done!


pgpclviyze1Pp.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: https://apps.fedoraproject.org/packages broken again?

[Ralph Bean]

On Wed, Dec 18, 2013 at 11:46:14PM +0800, Christopher Meng wrote:
> zmap

It should be back now - https://apps.fedoraproject.org/packages/zmap

A puppet upgrade caused a gluster issue that caused the indexer to
lose control of its lock files.  That's been resolved and it should be
indexing nightly again.


pgpnVVxdhnh6j.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: https://apps.fedoraproject.org/packages broken again?

[Ralph Bean]

On Wed, Dec 18, 2013 at 07:40:12PM +0330, james Abtahi wrote:
> 
> On 12/18/2013 04:46 AM, Christopher Meng wrote:
> >Hi,
> >
> >I can't find any information of packages recently added to pkgdb, I
> >think the data hasn't been synchronized, can someone have a look at
> >it?
> >
> 
> I'm experiencing the same issue too. Even though my package (yarock) has
> been pushed to fedora stable repos since two days ago I can't find it in
> apps.fedoraproject.org/packages. I thought perhaps it'd take some time to
> get synchronized but so far searching it would return nothing :(
> 
> Best,
> James

James,

It finally got indexed:
https://apps.fedoraproject.org/packages/yarock


pgpC4ijXwt460.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: https://apps.fedoraproject.org/packages broken again?

[Ralph Bean]

On Thu, Dec 19, 2013 at 12:17:16PM -0700, T.C. Hollingsworth wrote:
> On Thu, Dec 19, 2013 at 12:09 PM, Ralph Bean  wrote:
> > It should be back now
> 
> While you're messing around with it...is the sources tab supposed to work?

Yeah, there's a thousand of these little things :(  This particular
one is on my list for after the holidays.  I added a new nagios check
as of a week or so ago that checks for this functionality
specifically.

> The request it does has always returned a 500 for me.  :-(

Its working for me right now, but it goes out
intermittently/frequently:

https://apps.fedoraproject.org/packages/v8/sources/spec/

The issue is that an rpmdb gets corrupted somehow.  I *think* its due
to a race condition between the webapp, a cron job, and a cache
warming daemon.  We likely just need to introduce some locking in
those places to get it to stop.

Right now, if the cron job detects that the rpmdb is corrupt, it
rebuilds it and that sets it to working again until the next time it
melts down.


pgp6gHt17kOki.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: 2014 dreaming

[Ralph Bean]

On Thu, Dec 19, 2013 at 02:16:31PM -0700, Kevin Fenzi wrote:
> Greetings. 
> 
> I meant to send this out sooner, but with the Fedora 20 release things
> have been crazy. :) Now that we are nearing the end of 2013, I think it
> might be a nice time to think ahead to next year. 
> 
> What would you love to see happen? What would you like to work on
> making happen? 
> (even if it's not practical resource wise or logistically)
> 
> Here's a list of some of mine: 
> 
> * 0 downtime upgrades. By this I mean we can update and reboot all our
>   servers (probibly in some specific order with specific actions
>   between) and not have to schedule any downtime or notify users
>   anything is going on. This means at least that we have db
>   replication/failover working and 2 of everything. 
> 
> * Migration to ansible fully done, with all hosts moved over and
>   rebuilt and working. 
> 
> * Migration to RHEL7. :)
> 
> * Ticket queue down to a very small set. When I first started it was
>   gigantic, then I closed/fixed/redirected a lot of the ticket and we
>   started going down in number, but over the last year or so we have
>   hovered between 140-150. 
> 
> * Migration to hyperkitty/mailman3 complete with all lists moved. 
> 
> * All hosts selinux enforcing. :) 
> 
> * No "app" servers left. All applications split out to their own (at
>   least pair) of instances. 
> 
> * Logging from every app/server goes to a known place and we process
>   them all looking for problems. 
> 
> I'll probibly think of some more, will send them to this thread when I
> do. So how about you? Any big dreams for 2014? Hopefully we can make at
> least some of them real. 
> 
> kevin

One more for the list:

* Increased automatic QA/continuous integration:
  - More tests posting back to bodhi.
  - Dependencies between packages evaluated (if a package is updated,
its dependants should be evaluated against it).
  - A clear way for contributors to submit tests.
  - fedmsg messages all around.


pgpMGOf52jl_c.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: 2014 dreaming

[Ralph Bean]

On Wed, Jan 08, 2014 at 12:19:00PM -0500, Matthew Miller wrote:
> On Wed, Jan 08, 2014 at 11:18:29AM -0500, Ralph Bean wrote:
> > One more for the list:
> > 
> > * Increased automatic QA/continuous integration:
> >   - More tests posting back to bodhi.
> >   - Dependencies between packages evaluated (if a package is updated,
> > its dependants should be evaluated against it).
> >   - A clear way for contributors to submit tests.
> >   - fedmsg messages all around.
> 
> Yes! Have you talked to Tim Flink and the QA team about this?

Yes, briefly before the holidays.  We'll be in more touch this year.


pgpGiTMDj3HeO.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: FedoraHosted down?

[Ralph Bean]

On Sat, Jan 04, 2014 at 05:58:07PM +, Frank Murphy wrote:
> On Fri, 13 Dec 2013 14:14:58 +
> Frank Murphy  wrote:
> 
> > This is the infra ticket was adding to:
> > https://fedorahosted.org/freemedia/ticket/15351
> > 
> > 
> 
> Remember this intermittent problem was on freemedia trac.
> Disabling trac-fedmsg-plugin 
> has allowed it to work without problem so far.

I believe I have fixed the problem here:
https://github.com/fedora-infra/trac-fedmsg-plugin/issues/1

I just pushed it out to hosted03 this morning.  If you run into it
again, please let me know.


pgpFTR4FL2Bex.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: state of the infra ansible, cron job and roadmap

[Ralph Bean]

On Wed, Jan 08, 2014 at 01:10:40PM -0700, Kevin Fenzi wrote:
> That said, how do we want to run our non manual ansible jobs? 
> 
> a) run a --check --diff once a day and yell about unreachable or
> changed>0
> (I could commit this now)

:+1:

> b) just run them once a day and yell about anything that changes. 
> (I could commit this now)

+0, this could be fine.. but it would be a shame if it ran at a time
when we were all asleep, or when there happened to be load, or...

> c) Trigger them on git commits. 
> This would take work to figure out what was affected by the commit,
> or just fire off a run of everything. 

I think I saw you mention that a run of everything takes ~1 hour?
That's probably too much for a per-commit action.  A selective run per
commit could be cool though :)

> d) setup some file somewhere that can be created by sysadmin group and
> a cron job picks it up and runs the next time it runs. This would allow
> someone to commit something, schedule a run and give a bit of time for
> someone to notice a problem with it before it does. 

Push... and then wait for the doom.


pgpaGShDBuAsM.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: state of the askbot

[Ralph Bean]

On Mon, Jan 13, 2014 at 10:46:37PM +1100, Ankur Sinha wrote:
> Whoever wants to work on this will probably be better off setting up a
> local instance of askbot, making changes there and then applying them to
> our instances as a hotfix.
> 
> Here's a quick way of setting askbot up on openshift:
> http://www.pittrap.com/askbot-openshift/
> 
> I'm just giving it a whirl to see if it works.

Last week, I set ryan up with an ask instance on openshift to do theme
development so it wouldn't get wires crossed with our efforts in
staging.  http://askbot-threebean.rhcloud.com/


pgp5h7oJG_jVB.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: About cnucnu web

[Ralph Bean]

On Wed, Jan 22, 2014 at 09:51:00AM -0700, Luke Macken wrote:
> On Wed, Jan 22, 2014 at 09:30:59AM +0100, Pierre-Yves Chibon wrote:
> > - If so and if we go the OpenShift way, we would need to figure out how to
> >   set-up fedmsg as the cron job sends fedmsg notifications
> 
> I think that should work just fine, but Ralph would know for sure.
> You'll probably need to point it at a config for your endpoints, since
> /etc/fedmsg.d won't exist.

We'd need a static IP that we allow in with iptables.  That's the only
tricky part I can think of.  (We can put the config in $(pwd)/fedmsg.d/
to point it at our inbound relay)


pgpsGXv9iZXiE.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Porting bodhi to ansible

[Ralph Bean]

On Mon, Jan 27, 2014 at 08:44:32PM +0100, Janez Nemanič wrote:
> Hi,
> I am working on ticket
> https://fedorahosted.org/fedora-infrastructure/ticket/4193. So far I have
> managed to port just the puppet's basic bodhi class. I think that should be
> basic role for bodhi in ansible. Now I am a little uncertain what to do
> next. I am not sure how many other roles should create in ansible. The most
> simple solution would be to have the following roles:
> - bodhi_base
> - bodhi_app
> - bodhi_app_masher
> - bodhi_proxy
> - bodhi_app_epelmasher
> - bodhi_app_masher_jobrunner
> This is just mapping of puppet's bodhi classes to ansible roles. What I am
> asking myself is do I really need that many roles? Is it possible to do the
> job with fewer roles?
> Could other infra team members share their thoughts with me and help me a
> little bit?

I think its okay.  However, I've recently learned that roles can be
nested though, which might help with sprawl in our ansible/roles/
directory.

You can, for instance, have:

ansible/roles/bodhi/base
ansible/roles/bodhi/app/wsgi
ansible/roles/bodhi/app/masher

etc..

I plan to at some point namespace the badges and notifs roles like so:

ansible/roles/badges/frontend
ansible/roles/badges/backend
ansible/roles/notifs/frontend
ansible/roles/notifs/backend

Cheers, and thanks for looking into the bodhi port!
-Ralph


pgplA0K_jR6Jc.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Scheduling a sprint? github2fedmsg

[Ralph Bean]

Pierre and I were talking in #fedora-apps, and a tool we've been
wanting to build is a service that republishes select github events on
our fedmsg bus.

It will be a webapp where you can login to and register your github
repos.  Behind the scenes it will tell github to notify it of events.

When github pings our app, it will then publish a fedmsg message
saying such and such event has occurred over at
github.com/fedora-infra/python-fedora, or whatever.

We can then use this to:

- sync fedorahosted git repos
- watch upstreams that opt-in for statistics
- award badges for upstream development.

Two questions:

- What do people think about this idea?
- If you're interested in hacking on it, do you have any preference
  for a sprint date?  We would do it virtually in #fedora-apps.  We
  were thinking of March 17th, 18th and 19th.
  https://apps.fedoraproject.org/calendar/infrastructure/2014/3/17/

-Ralph


pgp2a7hkjVxQn.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: bug in fedora badges?

[Ralph Bean]

On Sat, Mar 15, 2014 at 10:25:04PM +1100, Ankur Sinha wrote:
> Tahrir should pick the IRC from the FAS correctly though, so it is kind
> of a bug I'd think. The one account is also marked as inactive. 
> 
> Might also be good to file a bug upstream:
> https://github.com/fedora-infra/tahrir/issues

I think we figured out what's going on:
https://github.com/fedora-infra/tahrir/issues/268

Mohan updated his irc nick in FAS just earlier in the day and the
fedbadges cache didn't have it yet.


pgpDXOmi2fAi5.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [ansible] Try out this conditional restart stuff.

[Ralph Bean]

On Mon, Mar 17, 2014 at 11:02:40AM +0100, Pierre-Yves Chibon wrote:
> On Mon, Mar 17, 2014 at 10:46:38AM +0100, Miroslav Suchý wrote:
> > So where I can get that file please?
> 
> It is part of the ansible repo:
> ./roles/base/files/common-scripts/conditional-restart.sh
> 
> You might be missing the part that installs it.

Yes, it is installed by the 'base' role.  Did I incorrectly assume
that we are including that in every playbook?


pgp1ovqPIAqu3.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Scheduling an IRC meeting on our private cloud setup

[Ralph Bean]

Our cloud went down today and we realized that only very few of us
actually know how it works and how to repair it.

To address this, Kevin has agreed to teach us what he knows about it.
We're looking to set up a #fedora-classroom meeting, have zodbot log
it, and afterwards turn the log into an infra-docs document.

If you're interested in attending and asking questions, please fill
out the following survey so we can find the best time:

http://whenisgood.net/cloud-halp

Please note that you need to set your timezone for that questionnaire
to work.

-Ralph


pgpN3hIZ4e9CI.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Scheduling an IRC meeting on our private cloud setup

[Ralph Bean]

On Tue, Mar 18, 2014 at 10:33:34AM -0400, Ralph Bean wrote:
> Our cloud went down today and we realized that only very few of us
> actually know how it works and how to repair it.
> 
> To address this, Kevin has agreed to teach us what he knows about it.
> We're looking to set up a #fedora-classroom meeting, have zodbot log
> it, and afterwards turn the log into an infra-docs document.
> 
> If you're interested in attending and asking questions, please fill
> out the following survey so we can find the best time:
> 
> http://whenisgood.net/cloud-halp
> 
> Please note that you need to set your timezone for that questionnaire
> to work.
> 
> -Ralph

Okay, based on the results of the survey: let's do Tuesday March 25th
at 18:00UTC in #fedora-classroom.  Not everyone can make it then, but
that's the time when the most people can.

I've marked it on the infrastructure calendar:
https://apps.fedoraproject.org/calendar/list/infrastructure/


pgp_jToi5Z8P5.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Plan for tomorrow's Fedora Infrastructure meeting (2014-03-20)

[Ralph Bean]

The infrastructure team will be having it's weekly meeting tomorrow, 
2014-03-13 at 18:00 UTC in #fedora-meeting on the freenode network.

Suggested topics:

#topic New folks introductions and Apprentice tasks.

If any new folks want to give a quick one line bio or any apprentices
would like to ask general questions, they can do so in this part of the
meeting. Don't be shy!

#topic Applications status / discussion

Check in on status of our applications: pkgdb, fas, bodhi, koji,
community, voting, tagger, packager, dpsearch, etc. 
If there's new releases, bugs we need to work around or things to note. 

#topic Sysadmin status / discussion

Here we talk about sysadmin related happenings from the previous week,
or things that are upcoming. 

#topic Upcoming Tasks/Items 

https://apps.fedoraproject.org/calendar/list/infrastructure/

#topic Open Floor

Submit your agenda items, as tickets in the trac instance and send a 
note replying to this thread.

More info here:

https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings

Thanks

ralph


pgpBhz_nqyKsa.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Plan for tomorrow's Fedora Infrastructure meeting (2014-03-20)

[Ralph Bean]

#fedora-meeting: Infrastructure (2014-03-20)



Meeting started by threebean at 18:00:05 UTC. The full logs are
available at
http://meetbot.fedoraproject.org/fedora-meeting/2014-03-20/infrastructure.2014-03-20-18.00.log.html
.



Meeting summary
---
* robot roll-call  (threebean, 18:00:18)

* New folks introductions and Apprentice tasks  (threebean, 18:02:36)

* Applications status / discussion  (threebean, 18:04:34)
  * pkgdb2 0.5 released to staging.  production is pending some final
work on pkgdb-cli  (threebean, 18:08:54)
  * switchover of the bugz.fp.o alias from pkgdb1 to fedora-packages
coming soon too.  (threebean, 18:09:46)
  * github2fedmsg sprint happened this week  (threebean, 18:12:35)
  * GSoC deadline is tomorrow  (threebean, 18:16:19)

* Sysadmin status / discussion  (threebean, 18:17:15)
  * LINK: http://kevin.fedorapeople.org/ansible-20140224.odp   (willo,
18:22:53)

* Upcoming Tasks/Items  (threebean, 18:29:10)
  * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
(threebean, 18:29:14)
  * LINK:

http://meetbot.fedoraproject.org/meetbot/fedora-meeting-1/2013-07-17/infrastructure-ansible-meetup.2013-07-17-19.00.html
(threebean, 18:32:35)

* Open Floor  (threebean, 18:40:36)

Meeting ended at 18:47:04 UTC.




Action Items






Action Items, by person
---
* **UNASSIGNED**
  * (none)




People Present (lines said)
---
* threebean (62)
* pingou (28)
* smooge (14)
* adimania (5)
* abadger1999 (5)
* axil42 (5)
* willo (4)
* zodbot (4)
* janeznemanic (3)
* jchristi (3)
* abompard (1)
* dgilmore (1)
* puiterwijk (0)
* nirik (0)
* lmacken (0)
* mdomsch (0)
* relrod (0)

--


18:00:05  #startmeeting Infrastructure (2014-03-20)
18:00:05  Meeting started Thu Mar 20 18:00:05 2014 UTC.  The chair is 
threebean. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:05  Useful Commands: #action #agreed #halp #info #idea #link 
#topic.
18:00:10  #meetingname infrastructure
18:00:10  The meeting name has been set to 'infrastructure'
18:00:17  hre
18:00:18  #topic robot roll-call
18:00:21 * pingou 
18:00:26  #chair smooge relrod nirik abadger1999 lmacken dgilmore 
mdomsch threebean pingou puiterwijk
18:00:26  Current chairs: abadger1999 dgilmore lmacken mdomsch nirik 
pingou puiterwijk relrod smooge threebean
18:00:50  hello -- anyone who's here for the infrastructure meeting, 
please chime in
18:01:06  chime (mostly lurking)
18:01:24 * willo waves hello
18:02:10  hola
18:02:36  #topic New folks introductions and Apprentice tasks
18:02:42  Any new folks like to introduce themselves? or apprentices 
with questions or comments?
18:03:18  if not, we can move along -- I'll wait a moment.
18:03:30  first timer here, mostly interested in the follow up 
meeting regarding GitLab
18:03:49  cool.  welcome, jchristi.
18:04:00  jchristi, welcome :)
18:04:06  hey axil :)
18:04:20  we'll talk later ;)
18:04:34  #topic Applications status / discussion
18:04:49  Any news on application development, maintenance?
18:05:23  I wrote a small tool to do daily build on copr for project 
hosted on git: https://github.com/pypingou/dgroc
18:05:46  otherwise, a new pkgdb2 release was tagged and pushed to stg: 
0.5
18:06:08  pingou: cool on both fronts.
18:06:15  Cool.
18:06:22  what needs to be done before pkgdb2 can be moved into 
production?
18:06:30  I also worked on something that's named progit (for the 
moment), eventually I'll have something to show a bit later
18:06:43  threebean: finish pkgdb-cli and the pkgdb2 module, that's 
almost there
18:07:16  threebean: then a lot of courage and few hours of no-pkgdb
18:07:18  :)
18:07:30  cool -- we'll need to be ready to switch the 
bugz.fedoraproject.org alias over from pkgdb to the fedora-packages app then 
too.
18:07:36  true
18:07:43  do we want to do that earlier?
18:07:46  or announce it?
18:08:10  sure, it can be done whenever I think.  We've sent 
numerous emails over the last year.  The latest was another blog post 
announcement back in january.
18:08:21  "In preparation for the landing of pkgdb2, we're pulling the 
switch to move bugz.fp.o to packages on Monday, feedbacks welcome"
18:08:44  something like that on devel-announce maybe?
18:08:54  #info pkgdb2 0.5 released to staging.  production is 
pending some final work on pkgdb-cli
18:08:59  pingou: sounds good, yeah
18:09:12  hola
18:09:22  +1
18:09:34  that will get us feedback if anything does ;-)
18:09:44  oh, and there is the points abadger1999 and I raised to 
dgilmore that we should figure out before releasing 1.0 :)
18:09:46  #info switchover of the bugz.fp.o alias from pkgdb1 to 
fedora-packages coming soon too.
18:09:57 * abompard tiptoes discretely into the back of the room
18:10:34  abadger1999: and the last things is going to be the cron jobs
18:11:24  other news -> we started work on a new app thi

Re: Scheduling an IRC meeting on our private cloud setup

[Ralph Bean]

==
#fedora-classroom: Infrastructure Private Cloud Class (2014-03-25)
==


Meeting started by nirik at 18:00:03 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-classroom/2014-03-25/infrastructure-private-cloud-class.2014-03-25-18.00.log.html


Meeting summary
---
* intro  (nirik, 18:00:03)

* History/current setup  (nirik, 18:02:51)
  * LINK: https://fed-cloud02.cloud.fedoraproject.org/dashboard/
(nirik, 18:10:48)
  * LINK:
http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/README
has a bunch of cloud specific info.  (nirik, 18:11:35)
  * IDEA: add openid support to openstack's horizon dashboard
(threebean, 18:13:19)
  * LINK: https://wiki.openstack.org/wiki/Nova_openid_service
(danofsatx-work, 18:17:06)

* Upcoming plans / TODO  (nirik, 18:26:33)
  * LINK: https://en.wikipedia.org/wiki/OpenStack#Components
(threebean, 18:27:00)

* Open Questions  (nirik, 18:40:57)
  * LINK: https://fedoraproject.org/wiki/Infrastructure_private_cloud
(nirik, 18:41:36)

Meeting ended at 19:02:55 UTC.

People Present (lines said)
---
* nirik (143)
* mirek-hm (29)
* danofsatx-work (21)
* threebean (19)
* tflink (16)
* smooge (5)
* zodbot (3)
* abadger1999 (3)
* webpigeon (3)
* jsmith (3)
* jamielinux (1)
* blob (1)
* relrod (1)
* janeznemanic (1)


18:00:03  #startmeeting Infrastructure Private Cloud Class (2014-03-25)
18:00:03  Meeting started Tue Mar 25 18:00:03 2014 UTC.  The chair is 
nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:03  Useful Commands: #action #agreed #halp #info #idea #link 
#topic.
18:00:03  #meetingname infrastructure-private-cloud-class
18:00:03  #topic intro
18:00:03  The meeting name has been set to 
'infrastructure-private-cloud-class'
18:00:22  hey everyone. who is around for a bit of talking about 
fedora's infrastructure private cloud?
18:00:26 * threebean is here
18:00:32  hello
18:00:38  aqui! aqui!
18:00:44 * mirek-hm is here
18:00:45 * tflink is here
18:00:53 * blob is here
18:00:57 * relrod here
18:01:01 * webpigeon is here
18:01:11 * jamielinux is here out of curiousity
18:01:53 * abadger1999 is here
18:01:57  hi
18:02:24  cool. ;) ok, I thought I would give a bit of 
background/history first, then a bit about ansible integration, then talk about 
plans...
18:02:51  #topic History/current setup
18:02:58  that sounds good.  can we interrupt you with questions?  
or should be hold them for later?
18:03:10  threebean: please do... questions are good. ;)
18:03:16  cool, cool.
18:03:20 * jsmith is late
18:03:23  so, our current setup is a openstack folsom cloud
18:03:31  It was mostly manually installed by me.
18:03:38  Ie, I installed rpms, ran setup, etc...
18:03:55  there are currently 7 nodes in use
18:04:07  Nodes = servers?
18:04:08  1 (fed-cloud02.cloud) is the 'head node'
18:04:15  6 are compute nodes
18:04:20  jsmith: yeah, physical boxes.
18:04:37  Perfect, just wanted to make sure I was getting the 
nomenclature right
18:04:53  The compute notes only run openstack-nova-compute, and one of 
them also runs cinder (the storage service will get to that in a few)
18:05:04  the head node runs all the other stuff.
18:05:18  and acts as the gateway to all the other things.
18:05:34  it runs network, the db (mysqld), amqp, etc, etc.
18:05:48  It also runs cinder for storage.
18:05:50  head node is fed-cloud02.cloud.fedoraproject.org BTW
18:06:29  when you fire off an instance openstack looks around and 
schedules it on a compute node.
18:06:34  outside of cinder, is there any shared storage?
18:07:07  when you ask for a persistent volume, it allocates it from one 
of the cinder servers... it makes a lv in a pool and shares it via iscsi
18:07:14  tflink: nope. Not currently.
18:07:45  aha, that are VG cinder-volumes in lvs output
18:07:46  all storage is either cinder from 02 or cinder from 08... or 
each compute node has a small amount of storage it uses to cache images locally.
18:07:56  mirek-hm: yeah.
18:08:04  is that a requirement of newer openstack? when I set up my 
dev/standalone system, I got the impression that something like gluster was 
highly reccomended for more than 1 mode
18:08:05  lets see...what elese.
18:08:24  tflink: we tried gluster, but it was really really really 
really slow at the time.
18:08:42  oh, networking:
18:08:44  no swift for image service?
18:09:01  nirik: you forget on one exception, Copr-be have one 
storage allocated on one compute node (800 GB)
18:09:02  we have a number of tenants (projects). Each one gets it's own 
vlan.
18:09:18  mirek-hm: thats from the 08 compute node via cinder there. ;)
18:09:44  we have a /24 network for external ips. Each instance gets one 
by default and can also allocated a static one.
18:10:08  danofsatx-work: yeah, we do have swift.
18:10:17  it uses storage on fed-cloud0

Re: L10n fedmsg integration?

[Ralph Bean]

On Wed, Jul 23, 2014 at 10:24:57AM -0400, Paul W. Frields wrote:
> Is there any way to attach an outside hosted service such as Zanata or
> Transifex to the Fedora's fedmsg bus?  Sorry if this is a naive
> question.

Yeah, I looked into it with Transifex.  They provide a
pubsubhubbub-like interface which would let us bridge nicely.
This is exactly how we currently interface with GitHub:
https://apps.fedoraproject.org/GitHub2fedmsg
Whenever an event would occur in GitHub or Transifex, they would send
an http POST to a listening web service on our side.  Our web service
would then re-broadcast that event as a fedmsg event.

The problem with Transifex, though, is that they do not
cryptographically sign their POSTs, which opens us up to.. anybody.
(GitHub actually signs each POST so we know its coming from them).

I haven't looked at Zanata yet, though.


pgp4k8XQEUJPe.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: L10n fedmsg integration?

[Ralph Bean]

On Wed, Jul 23, 2014 at 10:05:44PM +0300, Dimitris Glezos wrote:
> On Jul 23, 2014 5:51 PM, "Ralph Bean"  wrote:
> > The problem with Transifex, though, is that they do not
> > cryptographically sign their POSTs, which opens us up to.. anybody.
> > (GitHub actually signs each POST so we know its coming from them).
> 
> Ralph, can you please send an email to supp...@transifex.com with the info
> needed to get this on our roadmap?

Sure thing.  I asked a question about it previously, but the link
doesn't seem to work anymore:

http://support.transifex.com/customer/en/portal/questions/6100078-webhook-signature-

I'll email off more info shortly.


pgp3XMxsNYFo0.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Summary/Minutes from today's Fedora Infrastructure meeting (2014-07-24)

[Ralph Bean]

===
#fedora-meeting: Infrastructure
===


Meeting started by threebean at 18:00:43 UTC. The full logs are
available at
http://meetbot.fedoraproject.org/fedora-meeting/2014-07-24/infrastructure.2014-07-24-18.00.log.html
.



Meeting summary
---
* greetings  (threebean, 18:00:59)

* Any new people?  Apprentice tasks!  (threebean, 18:03:12)
  * progress made on the infrastructure map.  a preview will be
available in the coming weeks.  (threebean, 18:11:40)

* Applications status / discussion  (threebean, 18:12:33)
  * new openstack instance is coming together, but still lots of
problems  (threebean, 18:17:56)
  * LINK:

https://lists.fedoraproject.org/pipermail/infrastructure/2014-July/014621.html
(threebean, 18:17:57)
  * 2 new pkgdb2 releases this week  (threebean, 18:18:04)
  * 1 new fedocal release this week  (threebean, 18:18:08)
  * accidental upgrades of both fedoauth and fedmsg broke some
production services, but were downgraded and restored this morning.
(threebean, 18:19:22)

* Sysadmin status / discussion  (threebean, 18:21:57)

* Upcoming Tasks/Items  (threebean, 18:30:11)
  * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
(threebean, 18:30:17)

* Open Floor  (threebean, 18:32:47)

Meeting ended at 18:42:16 UTC.


People Present (lines said)
---
* threebean (62)
* pingou (13)
* ootbro (9)
* msuchy-hm (8)
* christianh814 (6)
* relrod (6)
* zodbot (5)
* dne0 (5)
* puiterwijk (4)
* nirik (4)
* lanica (3)
* michel_slm (2)
* bochecha (2)
* webpigeon (2)
* Xirrin (1)
* lmacken (1)
* bwood09 (1)
* smooge (0)
* abadger1999 (0)
* mdomsch (0)
* dgilmore (0)


Generated by `MeetBot`_ 0.1.4

.. _`MeetBot`: http://wiki.debian.org/MeetBot

18:00:43  #startmeeting Infrastructure
18:00:43  Meeting started Thu Jul 24 18:00:43 2014 UTC.  The chair is 
threebean. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:43  Useful Commands: #action #agreed #halp #info #idea #link 
#topic.
18:00:50  #meetingname infrastructure
18:00:50  The meeting name has been set to 'infrastructure'
18:00:57 * lanica is here for the infrastructure meeting.
18:00:59  #topic greetings
18:01:07  Hi lanica
18:01:14  anyone else around for the meeting today?
18:01:14 * michel_slm is here
18:01:14  o/
18:01:18 * lmacken 
18:01:31 * nirik is kinda sort of around, but running back and forth to racks, 
so distracted.
18:01:37 * puiterwijk is here.
18:01:38 * pingou 
18:01:50  threebean: just curious, you didn't set the chairs. any 
reason for that?
18:02:01  oh, lazy.
18:02:11  hah, okay
18:02:14 * bochecha is here
18:02:17  #chair smooge relrod nirik abadger1999 lmacken dgilmore 
mdomsch threebean pingou puiterwijk
18:02:17  Current chairs: abadger1999 dgilmore lmacken mdomsch nirik 
pingou puiterwijk relrod smooge threebean
18:02:29  I'm here
18:03:03  cool - welcome, everyone.
18:03:12  #topic Any new people?  Apprentice tasks!
18:03:21  Anybody new to the meeting today?
18:03:28  if so, could you introduce yourself?
18:03:47  Well I'm new-ish...this is my second meeting.
18:03:55 * msuchy-hm is here
18:03:57 * relrod is kinda sort of around too
18:04:32  christianh814: welcome back :)
18:04:54  christianh814: do you have any questions based on your 
first week hanging around?
18:04:59  anything we can try to answer or help with?
18:06:01  Just been looking through the tickets (easy fix) and 
working on learning ansible as I got (came from a chef background). No 
questions yet. I was able to get ssh-ed in
18:06:16  to lockbox and such
18:06:28  cool.  good stuff.
18:06:42  apprentice news    after getting a layout preference from 
nirik last night, started to build the final version of the fp.o network 
diagram  (very high-level picture)   discovering OpenOffice draw has some 
quirks. :p
18:07:40  ootbro: cool.  i'm looking forwards to it :)
18:08:21  I've almost got it ready to post to a temp spot (my own web 
server) and will send the URL to the infra list ...   then wait for the peanut 
gallery
18:08:38  ha!  great.  previews are good :)
18:08:45  ootbro: you could use fedorapeople.org too
18:09:17  second picture to create is roughly how the proxies work  
(from info I got from nirik)
18:09:19  ootbro: Sounds very cool!  Can we see a preview?
18:10:31  lanica: I'm battling Draw now, trying to get a quality .png 
out of it  will sent the URL to the infra list as soon as I have the .png 
posted for review / preview
18:11:16  michel_slm: I'll take a look at fedorapeople.org in a little 
bit, see how easliy I can get the .png posted
18:11:29  cool, thx!
18:11:40  #info progress made on the infrastructure map.  a preview 
will be available in the coming weeks.
18:11:46  thanks, ootbro
18:11:49  ootbro: if you are cla+1 it's as easy as scp ;-)
18:11:52 * webpigeon sneaks in
18:12:03  unless there's anything else in this sections, we can move 
on...?
18:12:33  #topic Applicatio

Re: L10n fedmsg integration?

[Ralph Bean]

On Thu, Jul 24, 2014 at 03:14:05PM -0400, Paul W. Frields wrote:
> On Thu, Jul 24, 2014 at 12:12:54PM +1000, Sean Flanigan wrote:
> > On 2014-07-24 03:02, Paul W. Frields wrote:
> > > On Wed, Jul 23, 2014 at 10:51:04AM -0400, Ralph Bean wrote:
> > >> On Wed, Jul 23, 2014 at 10:24:57AM -0400, Paul W. Frields wrote:
> > >>> Is there any way to attach an outside hosted service such as Zanata or
> > >>> Transifex to the Fedora's fedmsg bus?  Sorry if this is a naive
> > >>> question.
> > >>
> > >> Yeah, I looked into it with Transifex.  They provide a
> > >> pubsubhubbub-like interface which would let us bridge nicely.
> > >> This is exactly how we currently interface with GitHub:
> > >> https://apps.fedoraproject.org/GitHub2fedmsg
> > >> Whenever an event would occur in GitHub or Transifex, they would send
> > >> an http POST to a listening web service on our side.  Our web service
> > >> would then re-broadcast that event as a fedmsg event.
> > >>
> > >> The problem with Transifex, though, is that they do not
> > >> cryptographically sign their POSTs, which opens us up to.. anybody.
> > >> (GitHub actually signs each POST so we know its coming from them).
> > >>
> > >> I haven't looked at Zanata yet, though.
> > >
> > > ISTM we have an untapped stream of contributions we could recognize if
> > > the translation system (whichever it is) would sign those POSTs.  I'm
> > > not sure whether glezos is on this list, but I'll ask him whether this
> > > is an interesting RFE, and do the same for Zanata.
> > 
> > Zanata doesn't integrate with any buses yet, but a concrete use case
> > will help, so please feel free to submit an RFE.
> > 
> > https://bugzilla.redhat.com/enter_bug.cgi?product=Zanata
> 
> One use case would be for rewarding Fedora translation contributors
> with badges via the Fedora Badges app[1].  Having translation
> submissions trigger a message on the fedmsg bus allows the Badges app
> to record and award those.  Our experience has shown community members
> like receiving these badges and exhibiting them for the
> accomplishments and contributions they represent.
> 
> Ralph, would you be willing to file the RFE with details on signing
> the POSTs to prevent abuse?

Not at all.  I sent it up this morning:
https://bugzilla.redhat.com/show_bug.cgi?id=1122776


pgp6nAMHqpTD2.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

[PATCH] Point hub.fedoraproject.org at just proxy01.

[Ralph Bean]

Previously we had it doing round-robin over all the proxies.
We've been having issues over the last few weeks involving vpn saturation and
nirik tracked down one possible cause to busgateway01 which was and oddly high
amount of traffic.  This change will direct external scripts to consume messages
only directly from proxy01, skipping the vpn.  We'll see if this reduces our vpn
issues.

If this causes things to go haywire, we should just revert in git.
---
 fedoraproject.org.template |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fedoraproject.org.template b/fedoraproject.org.template
index 5593b28..b4fccb9 100644
--- a/fedoraproject.org.template
+++ b/fedoraproject.org.template
@@ -187,7 +187,7 @@ hosted-lists01  INA 85.236.55.8
 hosted02INA 66.135.62.187
 hosted03INA 140.211.169.199
 hosted04INA 152.19.134.149
-hub INCNAME wildcard
+hub INCNAME proxy01
 hub.stgINA 209.132.181.5
 ibiblio01   INA 152.19.134.138
 ibiblio01   IN  2610:28:3090:3001:dead:beef:cafe:fe01
-- 
1.7.2.1

___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: Point hub.fp.o dns entry at proxy01.

[Ralph Bean]

Seeking two +1s for a dns change that might help with our vpn issues.

___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request - Introduce fedmsg-gateway slaves on the proxies

[Ralph Bean]

I'm seeking two +1s to apply some work I did and tested in staging to
the production proxies.

The gist is that we currently proxy persistent tcp connections for the
fedmsg firehose through our proxies to busgateway01 (which these days
numbers around 700 connections but could scale up unexpectedly).  We
suspect that this has been causing our vpn issues since all fedmsg
traffic is sent to the proxies for every connection (700 times the
traffic normally on the bus).

The following patch 'turns on' a scheme where each proxy will run its
own fedmsg-gateway slave.  Haproxy connects remote requests to these
slaves, and the slaves in turn connect just once to the master
fedmsg-gateway on busgateway01.  Traffic should then only be sent once
to each proxy (over the vpn in most cases).

I tested it pretty thoroughly in stg (removed the daemon and its
config and re-ran puppet a few times to make sure things were put in
place in the correct order).

Patch attached.

-Ralph
From e88462f501ff79bf1f346f717b453d7a58d72517 Mon Sep 17 00:00:00 2001
From: Ralph Bean 
Date: Thu, 4 Sep 2014 20:02:01 +
Subject: [PATCH] Apply fedmsg-gateway-slave work to prod proxies.

---
 manifests/services/proxy.pp   |4 +---
 modules/haproxy/files/haproxy.cfg |7 +++
 2 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/manifests/services/proxy.pp b/manifests/services/proxy.pp
index 4f2c8d0..d73b957 100644
--- a/manifests/services/proxy.pp
+++ b/manifests/services/proxy.pp
@@ -20,9 +20,7 @@ class proxy {
 varnish::varnish{ 'proxy.vcl': }
 }
 
-if $environment == "staging" {
-include haproxy::fedmsg-gateway-slave
-}
+include haproxy::fedmsg-gateway-slave
 
 
 $fpo_ips = [
diff --git a/modules/haproxy/files/haproxy.cfg 
b/modules/haproxy/files/haproxy.cfg
index 76f05d3..981d366 100644
--- a/modules/haproxy/files/haproxy.cfg
+++ b/modules/haproxy/files/haproxy.cfg
@@ -202,9 +202,8 @@ listen fedmsg-websockets 0.0.0.0:9938
 
 # This, unlike the websockets entry just above, is listening directly to the
 # outside world with no stunnel inbetween.
-# Simply redirect tcp connections to busgateway01 on the same port.  There, an
-# instance of fedmsg-gateway will be publishing every message that comes across
-# the bus internally.  This is for consumption by desktop applications.
+# Simply redirect tcp connections to a local fedmsg-gateway slave.  It should 
be
+# forwarding messages from the master gateway on busgateway01.
 listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
 mode tcp
 balance roundrobin
@@ -212,7 +211,7 @@ listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
 timeout queue 5000
 timeout server 8640
 timeout connect 8640
-server  busgateway01 busgateway01:9940 weight 1 maxconn 16384
+server  localhost localhost:9942 weight 1 maxconn 16384
 
 # While the above fedmsg-raw-zmq-outbound forwards incoming connections to an
 # instance of the "fedmsg-gateway" daemon (which pushes internal messages out),
-- 
1.7.2.1



pgpn2B9k8lUe4.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: System Freeze Break Request: add sysadmin-ppc to gateway boxes

[Ralph Bean]

+1.  Looks good to me.


pgpeJXTdTb4r_.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: System Freeze Break Request Fwd: [ansible] add nagios items for virthost-comm03

[Ralph Bean]

On Mon, Sep 08, 2014 at 01:17:42PM -0600, Stephen John Smoogen wrote:
> I forgot to get a SFBR before pushing into ansible main repo. I have not
> run the job, so could I get a review please?

+1


pgp0F3lKeA_xa.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Service Break Request: Update dhcp for PPC systems

[Ralph Bean]

On Thu, Sep 11, 2014 at 02:12:31PM -0600, Stephen John Smoogen wrote:
> diff --git a/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org
> b/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org
> index 41b8626..27e918e 100644
> --- a/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org
> +++ b/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org
> @@ -1774,6 +1774,14 @@ shared-network qa {
>  option host-name "ppc-le-builder7.qa.fedoraproject.org";
>  filename "/boot/grub2/powerpc-ieee1275/core.elf";
>  }
> +
> +   host ppc-builder9 {
> +hardware ethernet 5C:F3:FC:89:BD:C5;
> +fixed-address 10.5.124.142;
> +next-server 10.5.124.136;
> +option host-name "ppc-builder9.qa.fedoraproject.org";
> +filename "/boot/grub2/powerpc-ieee1275/core.elf";
> +}
> }
> 
>  }
> 
> 
> -- 
> Stephen J Smoogen.


+1


pgpI_F77rAapg.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break request: adding FMN to the list of trusted trust roots

[Ralph Bean]

On Thu, Sep 11, 2014 at 05:15:04PM +0200, Patrick Uiterwijk wrote:
> Hi all,
> 
> Here I come again with yet another request for +1's.
> Basically, I want to add the trust root that FMN 
> (https://apps.fedoraproject.org/, FMN at 
> https://apps.fedoraproject.org/notifications/) to the list of trusted roots 
> in FedOAuth.
> 
> 
> 
> 
> From 0fd1ce2a4aef09119d9be2694d7b6475cdcc7a1c Mon Sep 17 00:00:00 2001
> From: Patrick Uiterwijk 
> Date: Thu, 11 Sep 2014 15:13:17 +
> Subject: [PATCH] Add FMN to the list of trusted trust roots
> 
> ---
>  roles/fedoauth/templates/fedoauth.cfg |3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/roles/fedoauth/templates/fedoauth.cfg 
> b/roles/fedoauth/templates/fedoauth.cfg
> index ffd1529..557891f 100644
> --- a/roles/fedoauth/templates/fedoauth.cfg
> +++ b/roles/fedoauth/templates/fedoauth.cfg
> @@ -80,7 +80,8 @@ AUTH_PROVIDER_CONFIGURATION = {
> 
> 'https://admin.fedoraproject.org/pkgdb/',
> 
> 'https://admin.fedoraproject.org/voting/',
> 
> 'https://apps.fedoraproject.org/github2fedmsg',
> -   
> 'https://admin.fedoraproject.org'],  # Nagios
> +   
> 'https://admin.fedoraproject.org',  # Nagios
> +   
> 'https://apps.fedoraproject.org/'],  # FMN
>  {% endif %}
>   'non_trusted_roots': [],
>   'handle_magic_groups_value': True
> -- 
> 1.7.2.1
> 
> 
> 
> 
> Thanks,
> Patrick

+1 from me.


pgpfUpp2YeRrW.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break: update exclude list for clamav scan on pkgs01

[Ralph Bean]

On Fri, Sep 12, 2014 at 10:57:33AM -0600, Kevin Fenzi wrote:
> Newer versions of the fwsnort package have a false positive with clamav
> virus scanning. 
> 
> I'd like to make the exception for that package more general so we
> don't have to keep updating versions.
> 
> I don't see any impact from this really... 
> 
> +1s?
> 
> kevin
> --
> diff --git a/manifests/services/distgit.pp b/manifests/services/distgit.pp
> index c9ec76a..8f3137b 100644
> --- a/manifests/services/distgit.pp
> +++ b/manifests/services/distgit.pp
> @@ -44,7 +44,7 @@ class distgit {
>  
>  clamav::clamscan { 'ad...@fedoraproject.org':
>  paths => '/srv/cache/lookaside/pkgs',
> -excludes => ['clamav-', 'amavisd-new-2.3.3.tar.gz', 
> 'bro-20080804.tgz', 'mailman-', 'sagator-', 'nicotine', 
> 'fwsnort-1.0.6.tar.gz', 'psad-2.1.7.tar.bz2', 'pymilter-', 'linkchecker-' ]
> +excludes => ['clamav-', 'amavisd-new-2.3.3.tar.gz', 
> 'bro-20080804.tgz', 'mailman-', 'sagator-', 'nicotine', 'fwsnort-*', 
> 'psad-2.1.7.tar.bz2', 'pymilter-', 'linkchecker-' ]
>  }
>  
>  git::git-server { '/srv/git/rpms': }
> diff --git a/modules/ansible/manifests/ini

+1


pgpwZQ7st2TOi.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break: install ansible-openstack-modules on lockbox01

[Ralph Bean]

On Fri, Sep 12, 2014 at 10:54:35AM -0600, Kevin Fenzi wrote:
> I'd like to install ansible-openstack-modules rpm on lockbox01. 
> 
> This is for ticket 4519 and will help us get our new cloud install up
> and running. 
> 
> +1s?
> 
> kevin
> --
> diff --git a/modules/ansible/manifests/init.pp 
> b/modules/ansible/manifests/init.pp
> index 0c2c81c..3db1ee3 100644
> --- a/modules/ansible/manifests/init.pp
> +++ b/modules/ansible/manifests/init.pp
> @@ -4,6 +4,10 @@ class ansible::ansible {
>ensure => present,
>  }
>  
> +package { ansible-openstack-modules:
> +  ensure => present,
> +}
> +
>  file { '/etc/ansible/ansible.cfg':
>  source => 'puppet:///ansible/ansible.cfg',
>  require => Package['ansible']

+1


pgp7GnezPakeR.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Entropy on VM for gpg

[Ralph Bean]

On Fri, Sep 12, 2014 at 05:01:50PM +0200, Valentin Gologuzov wrote:
> Hi,
> 
> I need to generate a lot of gpg keys for package signing in Copr,
> and VMs in cloud have low entropy for cryptography, that results
> in very-very slow key generation.
> 
> I've have solved it with `haveged` daemon, is it proper solution or
> i need something else?

This tool might be able to help:
https://pypi.python.org/pypi/quantumrandom


pgpcp0V1nFOmg.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break: adjust ppc dhcp

[Ralph Bean]

On Fri, Sep 12, 2014 at 02:11:24PM -0600, Kevin Fenzi wrote:
> We need another small adjustment in the ppc dhcp setup on noc01. 
> 
> +1s?
> 
> kevin
> --
> diff --git a/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org 
> b/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org
> index 27e918e..14ec365 100644
> --- a/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org
> +++ b/roles/dhcp_server/files/dhcpd.conf.noc01.phx2.fedoraproject.org
> @@ -1770,7 +1770,7 @@ shared-network qa {
> host ppc-le-builder7 {
>  hardware ethernet 52:54:00:4e:c0:de;
>  fixed-address 10.5.131.57;
> -   next-server 10.5.124.136;
> +   next-server 10.5.131.56;
>  option host-name "ppc-le-builder7.qa.fedoraproject.org";
>  filename "/boot/grub2/powerpc-ieee1275/core.elf";
>  }


+1


pgpLQ9h2Qd0er.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Something is polluting lockbox01 /

[Ralph Bean]

On Tue, Sep 16, 2014 at 02:14:06PM -0600, Stephen John Smoogen wrote:
> Anyone know what this is?

I think ansible leaves .retry files around (but I haven't been able to
find documentation directly about them).


pgpar0N43kGXR.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [RFR #4562] Koschei - continuous integration in Koji

[Ralph Bean]

On Wed, Oct 15, 2014 at 01:31:57PM -0600, Kevin Fenzi wrote:
> * How well does it keep up currently? I know you are careful not to
>   overload koji, but I wonder if that means things like perl builds are
>   often behind because there are so many of them? 

It would be great if there was a way to quantify and monitor this
during runtime with both nagios and collectd.

> * The notifications of failed builds currently are via fedmsg? We
>   should investigate adding this to FMN if it's not already there, so
>   anyone interested could be notified via that. 

The Koschei team submitted patches to FMN last week, so it should be
already ready.  :)

> * Are there any common sysadmin tasks we need to know about with the
>   instance? Is there any special process to start/stop/reinstall it? 

I got to check it out on the cloud node and it is 4 systemd-managed
services.  It was pretty straight forward to control for whatever
short amount of time I was trying to do so.


pgpFaXMe1D88o.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break request: switch nightly check/diff back to run each playbook

[Ralph Bean]

On Thu, Oct 16, 2014 at 09:22:31AM -0600, Kevin Fenzi wrote:
> Greetings. 
> 
> In puppet commit a9d2e61de5413edf297bd594051905e661760d0d I changed the
> nightly ansible check/diff cron job to just use the master playbook
> instead of doing each playbook on it's own. 
> 
> Turns out this has a few downsides: 
> 
> * If the execution fails somewhere, the run stops and it never runs on
>   the playbooks after the one that failed. 
> 
> * Our logging/reporting looks at the playbook name that was run, so it
>   lumps all of them into 'master.yml' and it's harder to see what
>   playbooks have changed or failed items in them.
> 
> I'd like to just revert this commit.
> 
> +1s?

+1 here



pgpBMWtXlBdUH.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break: add some playbooks to rbac-playbook

[Ralph Bean]

On Thu, Oct 16, 2014 at 09:17:52AM -0600, Kevin Fenzi wrote:
> Greetings. 
> 
> vgologuz has been reworking the copr ansible playbooks. Before we had
> some host playbooks that had all the logic in them. Now, we will have
> some group ones that use roles properly, etc. 
> 
> I'd like to add the new group playbooks to rbac-playbook so he can run
> them and test with them. 
> 
> copr is not frozen, but lockbox01 is, so thats why I ask. 
> 
> kevin
> --
> +'groups/copr-frontend.yml': ['sysadmin-cloud'],
> +'groups/copr-backend.yml': ['sysadmin-cloud'],
> +'groups/copr-keygen.yml': ['sysadmin-cloud'],
> 

Yup, yup.  +1 here.


pgpAZZ4CCGmy5.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: MM2/Ansible FAD Hotel rooms

[Ralph Bean]

On Sat, Oct 18, 2014 at 07:17:15PM +0200, Pierre-Yves Chibon wrote:
> The official dates are from Dec 4th to Dec 9th, as in, we start working on the
> 4th and finish on the 8th, thus departure on the 9th, so arrival dates should 
> be
> Dec 3rd.

Cool - put me down for the 3rd/9th for arrival/departure.


pgpS4vpNhPrLl.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [freeze break] Nuke old references to noc03

[Ralph Bean]

+1


pgpHv26SH22Mm.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [freeze break] add host1plus01 vpn ccd file

[Ralph Bean]

On Tue, Oct 21, 2014 at 11:59:11PM -0400, Ricky Elrod wrote:
> Adding the new box to the VPN, but this file technically affects bastion
> so sending a freeze break request.
> 
> 
> commit 5b7776d738628d1a57f95e777a0968f47bbc6e58
> Author: Ricky Elrod 
> Date:   Wed Oct 22 03:57:32 2014 +
> 
> add host1plus01 ccd
> 
> Signed-off-by: Ricky Elrod 
> 
> diff --git a/modules/openvpn/files/ccd/host1plus01.fedoraproject.org
> b/modules/openvpn/files/ccd/host1plus01.fedoraproject.org
> new file mode 100644
> index 000..68266e1
> --- /dev/null
> +++ b/modules/openvpn/files/ccd/host1plus01.fedoraproject.org
> @@ -0,0 +1,2 @@
> +# ifconfig-push actualIP PtPIP
> +ifconfig-push 192.168.1.138 192.168.0.138
> 


+1


pgpD33ihWSTPx.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [freeze break request] Make resolv.conf get installed before packages are installed in base.

[Ralph Bean]

On Tue, Oct 21, 2014 at 08:30:18PM -0400, Ricky Elrod wrote:
> In ansible's base role, we install packages before setting resolv.conf.
> But if resolv.conf isn't set, DNS queries don't necessarily work (e.g.
> when setting up a new virthost for the first time).
> 
> This patch moves the resolv.conf task up to the top of the base role to
> ensure it's in place before we do anything requiring DNS lookups.
> 
> +1's?

Agreed, +1.


pgpvnY3xnbbuh.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze break request: add taskotron01.qa to backups

[Ralph Bean]

On Mon, Oct 27, 2014 at 08:22:15AM -0600, Kevin Fenzi wrote:
> I'd like to add the production taskotron01.qa to backups per
> https://fedorahosted.org/fedora-infrastructure/ticket/4560
> 
> We couldn't add this before because it needed a RHIT firewall change to
> allow backup03 to talk to taskotron01.qa. This has been completed, so
> it should just be a matter of enabling it and running initial backups. 
> 
> +1s?

+1 here


pgphDxOu7CSC9.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze break request: disable nagios event_handler for fedora-tagger

[Ralph Bean]

Tagger started acting up recently and responding very slowly.  Longer
term, I think this is due to the database outgrowing our original
expectations for it -- a "select count(*) from votes;" takes quite a
while to return on the db server.

This slowness is causing nagios to time out every now and then which
makes it then try to restart httpd to get things working again (and
notify us about this).

To cut down on spam, I'd just like to remove the event_handler with
the following patch.  Any +1s?:

---
 .../files/nagios/services/websites.cfg |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/roles/nagios_server/files/nagios/services/websites.cfg 
b/roles/nagios_server/files/nagios/services/websites.cfg
index c52c167..9595b3d 100644
--- a/roles/nagios_server/files/nagios/services/websites.cfg
+++ b/roles/nagios_server/files/nagios/services/websites.cfg
@@ -95,7 +95,7 @@ define service {
   check_commandcheck_website!localhost!/tagger/!Tagger
   max_check_attempts   8
   use  internalwebsitetemplate
-  event_handler restart_httpd
+  #event_handler restart_httpd
 }
 
 define service {
-- 
1.7.2.1


pgpvvOng63nxI.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze break request: publish feodra-tagger db dump

[Ralph Bean]

As per https://fedorahosted.org/fedora-infrastructure/ticket/4578
I'd like to publish a regular db dump of fedora-tagger for
development.

The following patch should do it. Any +1s?

---
 scripts/public-db-copy |2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/scripts/public-db-copy b/scripts/public-db-copy
index 9126fd8..08ddbbd 100755
--- a/scripts/public-db-copy
+++ b/scripts/public-db-copy
@@ -8,3 +8,5 @@
 scp db-datanommer02:/backups/datanommer-$(date +%F).dump.xz 
/srv/web/infra/db-dumps/datanommer.dump.xz
 
 scp db01:/backups/db01.phx2.fedoraproject.org/pkgdb2.db 
/srv/web/infra/db-dumps/pkgdb2.db
+
+scp db01:/backups/db01.phx2.fedoraproject.org/fedoratagger.db 
/srv/web/infra/db-dumps/fedoratagger.db
-- 
1.7.2.1


pgpRM8dnduwDF.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: Add collectd monitoring for memcached

[Ralph Bean]

Hi all-

Memcached has been locking up on us lately.  If I remember correctly
it was Saturday morning, then again Monday morning, and it did it
again today (which manifested in a badges.fedoraproject.org outage for
users).

We currently don't have any collectd monitoring directly on memcached.
Here's an ansible patch that should add it.  Since we're in freeze,
this would need two +1s in order to be able to be applied.

If, for some reason it was broken or hosed up, I could revert it by
removing the /etc/collectd.d/memcached.conf file on both memcached
servers and subsequently restarting the memcached daemon.

Patch follows:

From 6535e0344f4db5b0eb00aeb37007c92f471d7224 Mon Sep 17 00:00:00 2001
From: Ralph Bean 
Date: Wed, 19 Nov 2014 14:37:50 +
Subject: [PATCH] Add collectd monitoring for the memcached daemon.

---
 playbooks/groups/memcached.yml|1 +
 roles/collectd/memcached/files/memcached.conf |6 ++
 roles/collectd/memcached/tasks/main.yml   |8 
 3 files changed, 15 insertions(+), 0 deletions(-)
 create mode 100644 roles/collectd/memcached/files/memcached.conf
 create mode 100644 roles/collectd/memcached/tasks/main.yml

diff --git a/playbooks/groups/memcached.yml b/playbooks/groups/memcached.yml
index 5fc955b..90d52b0 100644
--- a/playbooks/groups/memcached.yml
+++ b/playbooks/groups/memcached.yml
@@ -31,6 +31,7 @@
   - hosts
   - fas_client
   - collectd/base
+  - collectd/memcached
   - sudo
   - memcached
 
diff --git a/roles/collectd/memcached/files/memcached.conf 
b/roles/collectd/memcached/files/memcached.conf
new file mode 100644
index 000..d59815b
--- /dev/null
+++ b/roles/collectd/memcached/files/memcached.conf
@@ -0,0 +1,6 @@
+LoadPlugin memcached
+
+
+Host "localhost"
+Port "11211"
+
diff --git a/roles/collectd/memcached/tasks/main.yml 
b/roles/collectd/memcached/tasks/main.yml
new file mode 100644
index 000..3023af7
--- /dev/null
+++ b/roles/collectd/memcached/tasks/main.yml
@@ -0,0 +1,8 @@
+---
+
+- name: Copy in the memcached collectd config
+  copy: src=memcached.conf dest=/etc/collectd.d/memcached.conf
+  tags:
+  - collectd
+  - memcached
+  notify: restart collectd
-- 
1.7.2.1



pgp1bNdCsN603.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Further Freeze Break Request: Fix selinux for memcached+collectd

[Ralph Bean]

I applied the original patch, but selinux is blocking collectd from
connecting to the local memcached instance.

This followup patch should fix that.  Any add-on +1s?

diff --git a/roles/collectd/memcached/tasks/main.yml 
b/roles/collectd/memcached/tasks/main.yml
index 3023af7..fc0d04f 100644
--- a/roles/collectd/memcached/tasks/main.yml
+++ b/roles/collectd/memcached/tasks/main.yml
@@ -3,6 +3,23 @@
 - name: Copy in the memcached collectd config
   copy: src=memcached.conf dest=/etc/collectd.d/memcached.conf
   tags:
+  - collectd/memcached
   - collectd
   - memcached
   notify: restart collectd
+
+- name: Ensure that semanage is present
+  yum: name=policycoreutils-python state=present
+  tags:
+  - collectd/memcached
+  - collectd
+  - memcached
+  - selinux
+
+- name: lastly, set some selinux booleans
+  seboolean: name=collectd_tcp_network_connect persistent=yes state=yes
+  tags:
+  - collectd/memcached
+  - collectd
+  - memcached
+  - selinux


pgpT5z3mAkEF4.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Further Freeze Break Request: Fix selinux for memcached+collectd

[Ralph Bean]

All applied and it seems to be happy now:

Output can be seen here:

https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=memcached01.phx2.fedoraproject.org&hostname=memcached02.phx2.fedoraproject.org&plugin=memcached×pan=3600&action=show_selection&ok_button=OK

http://da.gd/d58d


pgpxO7It4Et20.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Freeze Break Request: Add nagios check for memcached connectivity

[Ralph Bean]

Another freeze break request - this one to add a nagios check for connectivity
to memcached.

This will attempt to call the daemon's stats command which, if broken, might
hang and cause nrpe to time out.  We want that, as it will give us a clue to
what might be causing some other app to fail.

This is in part what happened this morning.  The memcached process was present,
and it was even accepting tcp requests, but it would not respond to commands.

---
 .../files/scripts/check_memcache_connect   |   24 
 roles/nagios_client/tasks/main.yml |1 +
 .../nagios_client/templates/check_memcache.cfg.j2  |2 +-
 .../files/nagios/services/memcached.cfg|   16 +++-
 roles/nagios_server/files/nrpe.cfg |1 +
 5 files changed, 41 insertions(+), 3 deletions(-)
 create mode 100644 roles/nagios_client/files/scripts/check_memcache_connect

diff --git a/roles/nagios_client/files/scripts/check_memcache_connect 
b/roles/nagios_client/files/scripts/check_memcache_connect
new file mode 100644
index 000..7c472e3
--- /dev/null
+++ b/roles/nagios_client/files/scripts/check_memcache_connect
@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+# 2014-12-19
+# Author: Ralph Bean 
+
+# exit codes
+ok=0
+warn=1
+crit=2
+unkn=3
+
+# Right now we just check to see if we can even run this command without
+# hanging and timing out.  In the future, we could parse stdout for more
+# fine-grained information.
+echo stats | nc 127.0.0.1 11211 > /dev/null
+status=$?
+
+if [ $status -ne 0 ]; then
+echo "CRIT:  stats command got status code $status"
+exit $crit
+else
+echo "OK:  stats command got status code $status"
+exit $ok
+fi
diff --git a/roles/nagios_client/tasks/main.yml 
b/roles/nagios_client/tasks/main.yml
index 6c91dda..aa9b6c2 100644
--- a/roles/nagios_client/tasks/main.yml
+++ b/roles/nagios_client/tasks/main.yml
@@ -31,6 +31,7 @@
   - check_fedmsg_producers_consumers.py
   - check_supybot_plugin
   - check_datanommer_timesince.py
+  - check_memcache_connect
   when: not inventory_hostname.startswith('noc')
   tags:
   - nagios_client
diff --git a/roles/nagios_client/templates/check_memcache.cfg.j2 
b/roles/nagios_client/templates/check_memcache.cfg.j2
index b350a65..b0ec100 100644
--- a/roles/nagios_client/templates/check_memcache.cfg.j2
+++ b/roles/nagios_client/templates/check_memcache.cfg.j2
@@ -1,2 +1,2 @@
 command[check_memcache]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -a 
'/usr/bin/memcached' -u memcached
-
+command[check_memcache_connect]=/usr/lib64/nagios/plugins/check_memcache_connect
diff --git a/roles/nagios_server/files/nagios/services/memcached.cfg 
b/roles/nagios_server/files/nagios/services/memcached.cfg
index 9f497b5..814a5a8 100644
--- a/roles/nagios_server/files/nagios/services/memcached.cfg
+++ b/roles/nagios_server/files/nagios/services/memcached.cfg
@@ -1,12 +1,24 @@
 define service {
   host_name memcached01
-  service_description   Check memcached daemon
+  service_description   Check for the presence of the memcached daemon
   check_command check_by_nrpe!check_memcache
   use   defaulttemplate
 }
 define service {
   host_name memcached02
-  service_description   Check memcached daemon
+  service_description   Check for the presence of the memcached daemon
   check_command check_by_nrpe!check_memcache
   use   defaulttemplate
 }
+define service {
+  host_name memcached01
+  service_description   Check for connectivity to the memcached daemon
+  check_command check_by_nrpe!check_memcache_connect
+  use   defaulttemplate
+}
+define service {
+  host_name memcached02
+  service_description   Check for connectivity to the memcached daemon
+  check_command check_by_nrpe!check_memcache_connect
+  use   defaulttemplate
+}
diff --git a/roles/nagios_server/files/nrpe.cfg 
b/roles/nagios_server/files/nrpe.cfg
index 86af64b..4fb1cdb 100644
--- a/roles/nagios_server/files/nrpe.cfg
+++ b/roles/nagios_server/files/nrpe.cfg
@@ -238,6 +238,7 @@ 
command[check_fcomm_queue]=/usr/lib64/nagios/plugins/check_fcomm_queue
 command[check_redis_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 
'redis-server' -u redis
 command[check_openvpn_link]=/usr/lib64/nagios/plugins/check_ping -H 
192.168.1.58 -w 375.0,20% -c 500,60%
 command[check_memcache]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -a 
'/usr/bin/memcached' -u memcached
+command[check_memcache_connect]=/usr/lib64/nagios/plugins/check_memcache_connect
 
 # The following are fedmsg/datanommer checks to be run on busgateway01.
 # They check for the time since the latest message in any particular category.
-- 
1.7.2.1



pgp46PBeu2giu.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: Add nagios check for memcached connectivity

[Ralph Bean]

Good to go.  The patch is all applied and the check is reporting
status "OK".


pgpiy2eVyILWr.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: new depcheck on taskotron clients

[Ralph Bean]

+1 here too


pgpI1pnPN1BbY.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: Better rsync to bapp02::docs

[Ralph Bean]

On Wed, Nov 19, 2014 at 03:06:15PM -0700, Pete Travis wrote:
> The proxies are currently not purging deleted content when rsyncing
> docs.fp.o. I'd like to fix that:

+1 here as well.

-Ralph


pgpcqSamseGO1.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: point pkgdb2 memcached config to memcached01

[Ralph Bean]

On Thu, Nov 20, 2014 at 04:01:35PM +0100, Pierre-Yves Chibon wrote:
> After discussing with Ralph, he pointed out we should activate the
> distributed_lock as we have multiples processes querying the same memcached
> servers. In addition, Ralph suggested that we use both memcached01 and
> memcached02 servers instead of relying on one only.
> 
> The diff on the pkgdb2 config becomes:
> diff --git a/ roles/pkgdb2/templates/pkgdb2.cfg b/ 
> roles/pkgdb2/templates/pkgdb2.cfg
> index 828d41d..cd8e862 100644
> --- a/ roles/pkgdb2/templates/pkgdb2.cfg
> +++ b/ roles/pkgdb2/templates/pkgdb2.cfg
> @@ -29,7 +29,8 @@ ADMIN_GROUP = ['sysadmin-main', 'cvsadmin']
>  PKGDB2_CACHE_BACKEND = 'dogpile.cache.memcached'
>  PKGDB2_CACHE_KWARGS = {
>  'arguments': {
> -'url': "127.0.0.1:11211",
> +'url': ["memcached01:11211", "memcached02:11211"],
> +'distributed_lock': True,
>  }
>  }

+1 to the updated change.  Thanks!

-Ralph


pgpCPUOcGTuNX.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: Freeze Break Request: Add nagios check for memcached connectivity

[Ralph Bean]

On Sun, Nov 23, 2014 at 08:54:05PM +0200, Achilleas Pipinellis wrote:
> On 11/19/2014 08:50 PM, Ralph Bean wrote:
> > 
> > diff --git a/roles/nagios_client/files/scripts/check_memcache_connect 
> > b/roles/nagios_client/files/scripts/check_memcache_connect
> > new file mode 100644
> > index 000..7c472e3
> > --- /dev/null
> > +++ b/roles/nagios_client/files/scripts/check_memcache_connect
> > @@ -0,0 +1,24 @@
> > +#!/bin/bash
> > +#
> > +# 2014-12-19
> > +# Author: Ralph Bean 
> > +
> 
> 
> Hey Ralph, sorry for the hijack, I noticed that you mistyped the date in
> the script. It should read 2014-11-19 ;)

:facepalm:  Thanks!


pgpkIk4MR2PxQ.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure