[OpenStack-Infra] Announcing firehose.openstack.org

[Matthew Treinish]

Hi Everyone,

I wanted to announce the addition of a new infra service we started running
recently, the firehose. Running at firehose.openstack.org the firehose is an
MQTT based unified message bus for infra services. The concept behind is it that
there are a lot of infra services many of which emit events, however there
wasn't a single place to go to for anything. If you have a script or tool which
is listening for events from an infra service, or has a poll loop (like anything
using gerritlib) there is now a single place to go for consuming those messages.
We also have 2 interfaces to subscribe to topics on the firehose, either via the
MQTT protocol on the default port or via a websockets over port 80. The 
websocket
interface should enable easier consumption for people on networks with stricter
firewalls.

Right now the only things sending messages into the firehose is the gerrit event
stream (via germqtt) and launchpad bug events (via lpmqtt) but several other
efforts are in progress to add additional services to the firehose. The plan is
to expand what publishes events to include all the infra services. This way
there is a single location for anything that needs to consume events.

There is also an example on the consuming side with gerritbot, which now has 
support for subscribing to the gerrit event stream over MQTT. You can see the
patch here:

http://git.openstack.org/cgit/openstack-infra/gerritbot/commit/?id=7c6e57983d499b16b3fabb864cf3bd5cfea8ab63

For those interested the spec detailing all the pieces is here:

http://specs.openstack.org/openstack-infra/infra-specs/specs/firehose.html

and the docs are available here:

http://docs.openstack.org/infra/system-config/firehose.html

which contain details on how the services are setup and includes some basic
steps and examples on how to start consuming events from the firehose.

Thanks,

Matt Treinish


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] [Infra] Job failure notifications

[Matthew Treinish]

On Thu, Jun 16, 2016 at 08:23:25PM +0200, Andreas Jaeger wrote:
> On 06/16/2016 05:31 PM, Ivan Kolodyazhny wrote:
> > Hi team,
> > 
> > Is any way to subscribe to notifications to any failure on gate job? I'm
> > maintainer of  gate-tempest-dsvm-full-bdd-nv job in Cinder and would
> > like to know when it is failed.
> 
> 
> Use the OpenStack health to see yourself, for example:
> 
> http://status.openstack.org/openstack-health/#/g/build_queue/post?groupKey=build_queue=translation
> 
> There's only a mailing list for periodic jobs, no generic one,

openstack-health also supports rss feeds for failures. For example:

http://health.openstack.org/runs/key/build_name/gate-manila-tempest-dsvm-mysql-generic/recent/rss

is the rss feed of failures for one of the manila tempest plugin jobs. The rss
feeds are linked of the openstack-health home page:

http://status.openstack.org/openstack-health/#/

That being said we don't collect data for check and experimental jobs right now.
It's on the roadmap to add support for that, but it adds a lot of complexity to
the system and moving forward here is blocked on some deployment changes we need
to make to the mysql database we use for subunit2sql. Once that's addressed we
can start moving forward again on having a method to include that data.

-Matt Treinish


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates

[Matthew Treinish]

On Thu, May 26, 2016 at 05:55:34PM +, Jeremy Stanley wrote:
> On 2016-05-26 12:54:49 -0400 (-0400), Matthew Treinish wrote:
> > Just a quick follow-up I started running this on a throwaway server at
> > 15.184.138.236. So you can subscribe to events from gerrit.
> [...]
> 
> How resource-intensive is it? Curious whether it makes sense to run
> something like this directly on review.openstack.org. If zuul grew
> support for that mechanism, it might allow CI systems (third party
> or even our own) to wean off using SSH entirely since this is a
> problem in a lot of places from crazy enterprise firewall policies
> to systems running in mainland China.

It's eating like nothing on my server right now. This is all running on a single
cpu vm on a private cloud with a "Intel Xeon E312xx (Sandy Bridge)" (according
to /proc/cpuinfo) Mosquitto itself has a memory footprint < 1MB and I've seen it
spike up to a whopping 1% cpu utilization. Although, this might increase a as 
more
subscribers are added. This is the first time I've played with mosquitto and 
mqtt
so I don't know what it's scaling is like. But, I imagine it should handle a lot
of subscriptions well since it's supposed to be an IoT thing. germqtt is eating
a bit more with consuming about 1.5M of RAM and it uses about the same CPU as
Mosquitto.

-Matt Treinish


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates

[Matthew Treinish]

On Wed, May 25, 2016 at 10:08:57PM -0400, Matthew Treinish wrote:
> On Wed, May 25, 2016 at 02:43:36PM -0400, Sean Dague wrote:
> > On 05/25/2016 01:54 PM, Spencer Krum wrote:
> > > 
> > > When working on a previous project, Nikki wrote this tool as a general
> > > purpose hook into gerrit:
> > > 
> > > https://github.com/notnownikki/zoidberg
> > > 
> > > I don't think she is actively maintaining or using it right now though.
> > 
> > One thing I've been thinking a bit about is whether the event stream
> > could get into something like MQTT easily. In completely unrelated
> > activities (https://home-assistant.io/) I've been playing with mosquitto
> > (http://mosquitto.org/) quite a bit, and the ease of consumption of mqtt
> > is quite nice. (You can even do it straight in javascript for web based
> > things).
> 
> It looked like this would be a fun little side project to experiment with, so 
> I
> took a bit of time and wrote:
> 
> https://github.com/mtreinish/germqtt
> 
> It's still kinda rough and basic. For right now all it does is publishes a 
> json
> blob of every event from the gerrit stream over mqtt. But, if it's something 
> we
> wanted to continue with it's a good starting point.
> 

Just a quick follow-up I started running this on a throwaway server at
15.184.138.236. So you can subscribe to events from gerrit. Using the
mosquitto_sub command [1] is a simple way to play with it. For example:

 $ mosquitto_sub --topic 'gerrit/#' -h 15.184.138.236

will subscribe to all events

 $ mosquitto_sub --topic 'gerrit/openstack/nova/#' -h 15.184.138.236

will subscribe to all nova events

 $ mosquitto_sub --topic 'gerrit/+/+/comment-added' -h 15.184.138.236

will subscribe to all comment-added events. You can come up with any whatever
subscriptions you want using the builtin wildcards on the protocol. [2] 

I'll probably keep the server running for a while, so feel free to play around
with it.

-Matt Treinish

[1] http://mosquitto.org/man/mosquitto_sub-1.html
[2] http://mosquitto.org/man/mqtt-7.html


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] A tool for slurping gerrit changes in to bug updates

[Matthew Treinish]

On Wed, May 25, 2016 at 02:43:36PM -0400, Sean Dague wrote:
> On 05/25/2016 01:54 PM, Spencer Krum wrote:
> > 
> > When working on a previous project, Nikki wrote this tool as a general
> > purpose hook into gerrit:
> > 
> > https://github.com/notnownikki/zoidberg
> > 
> > I don't think she is actively maintaining or using it right now though.
> 
> One thing I've been thinking a bit about is whether the event stream
> could get into something like MQTT easily. In completely unrelated
> activities (https://home-assistant.io/) I've been playing with mosquitto
> (http://mosquitto.org/) quite a bit, and the ease of consumption of mqtt
> is quite nice. (You can even do it straight in javascript for web based
> things).

It looked like this would be a fun little side project to experiment with, so I
took a bit of time and wrote:

https://github.com/mtreinish/germqtt

It's still kinda rough and basic. For right now all it does is publishes a json
blob of every event from the gerrit stream over mqtt. But, if it's something we
wanted to continue with it's a good starting point.

-Matt Treinish


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] problems on periodic-neutron

[Matthew Treinish]

On Mon, Feb 08, 2016 at 08:13:38AM +0900, fumihiko kakuma wrote:
> Hi Jeremy,
> 
> On Fri, 5 Feb 2016 14:45:17 +
> Jeremy Stanley  wrote:
> 
> > On 2016-02-05 09:00:24 +0100 (+0100), Andreas Jaeger wrote:
> > > One idea: Remove the branch filters you created.
> > [...]
> > 
> > Correct. From Zuul's perspective periodic jobs are triggered simply
> > by a changeless/refless timer, and thus branchless since there's no
> > associated Gerrit change or Git ref from which it can infer a
> > branch. Asking Zuul to limit a periodic job to specific branches
> > means it will never be run, as that precondition can never be
> > satisfied.
> > -- 
> > Jeremy Stanley
> 
> I see. Thank you for the explanation of the mechanism of periodic
> pipeline.
> 
> BTW can you also explain the non-indication of periodic pipeline data
> by openstack-helth and tell the resolution to it?

There is nothing wrong with openstack-health it's behaving as currently
expected. There is a known limitation with the dashboard right now where
results aren't counted if the job failure occurs before devstack starts. If your
jobs are running but never even getting to devstack openstack-health (well
really the subunit2sql db) will not have any data on those runs. Once you fix
these jobs to actually start running devstack (or anything else which generates
a subunit stream in the expected place) it'll appear on the dashboard.

I wrote a brief blog post a little while ago on how openstack-health works and
some of the features it has:

http://blog.kortar.org/?p=279

The current limitations section explains this issue in a bit more detail.

-Matt Treinish


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] client manager setup in tempest

[Matthew Treinish]

Just for future reference most of the ML tempest discussions happen on the
openstack-dev list with a [QA] or a [tempest] tag.

On Thu, Aug 06, 2015 at 08:35:40PM -0400, John Warren wrote:
 
 Could somebody explain to me how this code in tempest/test.py can possibly
 work:
 
 @classmethod
 def setup_credentials(cls):
 Allocate credentials and the client managers from them.
 A test class that requires network resources must override
 setup_credentials and defined the required resources before super
 is invoked.
 
 for credentials_type in cls.credentials:
 # This may raise an exception in case credentials are not
 available
 # In that case we want to let the exception through and the test
 # fail accordingly
 if isinstance(credentials_type, six.string_types):
 manager = cls.get_client_manager(
 credential_type=credentials_type)
 setattr(cls, 'os_%s' % credentials_type, manager)
 # Setup some common aliases
 # TODO(andreaf) The aliases below are a temporary hack
 # to avoid changing too much code in one patch. They should
 # be removed eventually
 if credentials_type == 'primary':
 cls.os = cls.manager = cls.os_primary
 if credentials_type == 'admin':
 cls.os_adm = cls.admin_manager = cls.os_admin
 if credentials_type == 'alt':
 cls.alt_manager = cls.os_alt
 elif isinstance(credentials_type, list):
 manager = cls.get_client_manager(roles=credentials_type[1:],
  force_new=True)
 setattr(cls, 'os_roles_%s' % credentials_type[0], manager)
 
 I'm wondering about the following line:
 
 cls.os_adm = cls.admin_manager = cls.os_admin
 
 From inside the root directory I do a grep search:
 
 $ grep -Hr os_admin tempest
 Binary file tempest/test.pyc matches
 tempest/test.py:cls.os_adm = cls.admin_manager =
 cls.os_admin
 tempest/api/baremetal/admin/base.py:cls.client =
 cls.os_admin.baremetal_client
 Binary file tempest/api/baremetal/admin/base.pyc matches
 
 This indicates to me that os_admin isn't being assigned a value anywhere.
 Therefore, I would expect an AttributeError (class BaseTestCase has no
 attribute 'os_admin'), yet this class method seems to work.  What am I
 missing?

So a grep actually won't help here, because that attribute, and all of the other
manager object class attributes, get set dynamically by setup_credentials(). If
you look at:

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/test.py#n352

that setattr() call is setting the attribute for each manager on the test class.

The way this method is intended to be used is that each test class defines as an
attribute the list of isolated credential types which are required for the tests
in the class. setup_credentials(), which gets called as part of setUpClass, will
then take that list, allocate the credentials (either using tenant isolation or
the locking test accounts mechanism) and use that to init a client manager
object. That then gets set to as an attribute on the class so that tests can
get to the manager objects for each set of credentials.

The line you pointed out is only there because before we had a common
implementation for creating client manager objects each test class would do it
slightly differently. So the class attributes that tests were expecting vary
slightly. (hence the todo comment saying we need to change all the tests to
remove the aliases) That's actually a quick low hanging fruit kinda thing for
someone to pick up, to just go through all the tests and unify the use of the
manager object attrs so we can remove this alias.


-Matt Treinish


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Periodic jobs still failing

[Matthew Treinish]

On Tue, Jun 30, 2015 at 03:50:55PM +, Jeremy Stanley wrote:
 On 2015-06-15 16:52:09 -0400 (-0400), David Kranz wrote:
  Though the zuul problem seems to have been fixed, all the tempest periodic
  jobs are still failing with
  
  /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 480: cd:
  /opt/stack/new/devstack: No such file or directory
 [...]
 
 It looks to me from recent runs like this is resolved with our
 latest fixes (excepting of course the ones from today which didn't
 work because of issues with the logserver). Can you confirm?

Yes, the issue that David was reporting here looks like it's been addressed
since https://review.openstack.org/194302 took effect last week. I haven't seen
a periodic tempest job failure related to this since.

-Matt Treinish


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Orphaned repositories in openstack* namespace

[Matthew Treinish]

On Mon, Mar 30, 2015 at 10:53:51AM -0400, Monty Taylor wrote:
 On 03/30/2015 03:43 AM, Thierry Carrez wrote:
  Andreas Jaeger wrote:
  We noticed last week that bashate was not part of any project in the
  governance repo and wondered on IRC which other repositories are
  orphaned - meaning added in one of the openstack namespaces but not
  mentioned in governance/reference/.
  
  Yes, I usually periodically run a script to check consistency, but that
  run was long overdue! Thanks for looking into that.
  
  I did some quick checking and came up with the following list and
  added current proposed changes and my suggestions:
 
   openstack-dev/bashate
Proposed for QA:
https://review.openstack.org/#/c/168012/
 
   openstack-infra/ansible-puppet
Proposed for Infra:
https://review.openstack.org/#/c/166820/
 
   openstack/gantt
   openstack/python-ganttclient
Suggestion nova:
https://review.openstack.org/#/c/168682/
 
   openstack/os-client-config
Proposed for openstackclient:
https://review.openstack.org/#/c/167730/
 
   openstack-dev/ci-sandbox
   openstack-dev/sandbox
 
 These should be special-cased - they are repos that people use to test
 their CI systems or dev workflow.
 
   openstack-infra/vinz
   openstack-infra/vinz-webclient
 Suggestion: Infrastructure
 
 These don't really exist yet, but I don't see a problem with that.
 
   openstack/ossa
 Suggestion: Release cycle management
 
   openstack-dev/devstack-vagrant
 Suggestion: QA

At least in this case I think this is basically another orphaned project. It was
started by sdague in an effort to consolidate multiple nearly identical efforts
that were spinning up. (before docker became the new hawtness) I don't think
anyone has really been contributing to this in quite some time, it might be a
better candidate for the attic. That being said if we want to keep it around it
definitely would fit under QA's purview.

 
   openstack-dev/specs-cookiecutter
 Suggestion: Oslo
 
  Do you agree with my suggestions? If yes, I can sent patches for the
  governance repo.
  
  Yes. Thanks!
  
 

-Matt Treinish


pgp0txFWqUZ1S.pgp
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra