Re: [Openstack-operators] [openstack-dev] [Openstack-sigs] Dropping lazy translation support

[Rochelle Grober]

I seem to recall list discussion on this quite a ways back.  I think most of it 
happened on the Docs ml, though. Maybe Juno/Kilo timeframe?  If possible, it 
would be good to search over the code bases for places it was called to see its 
current footprint.  I'm pretty sure it was the docs folks working with the oslo 
folks to make it work.  But then the question was put to the ops folks about 
translations of logs (maybe the New York midcycle) and ops don't use 
translation.  The ops input was broadcast to dev and docs and most efforts 
stopped at that point.  But, I believe some projects had already done some work 
on lazy translation.  I suspect the amount done, though was pretty low.

Maybe the fastest way to get info would be to turn it off and see where the 
code barfs in a long run (to catch as many projects as possible)?

--rocky

> From: Ben Nemec > Sent: Monday, November 05, 2018 1:40 PM
> 
> On 11/5/18 3:13 PM, Matt Riedemann wrote:
> > On 11/5/2018 1:36 PM, Doug Hellmann wrote:
> >> I think the lazy stuff was all about the API responses. The log
> >> translations worked a completely different way.
> >
> > Yeah maybe. And if so, I came across this in one of the blueprints:
> >
> > https://etherpad.openstack.org/p/disable-lazy-translation
> >
> > Which says that because of a critical bug, the lazy translation was
> > disabled in Havana to be fixed in Icehouse but I don't think that ever
> > happened before IBM developers dropped it upstream, which is further
> > justification for nuking this code from the various projects.
> >
> 
> It was disabled last-minute, but I'm pretty sure it was turned back on (hence
> why we're hitting issues today). I still see coercion code in oslo.log that 
> was
> added to fix the problem[1] (I think). I could be wrong about that since this
> code has undergone significant changes over the years, but it looks to me like
> we're still forcing things to be unicode.[2]
> 
> 1: https://review.openstack.org/#/c/49230/3/openstack/common/log.py
> 2:
> https://github.com/openstack/oslo.log/blob/a9ba6c544cbbd4bd804dcd5e38
> d72106ea0b8b8f/oslo_log/formatters.py#L414
> 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Openstack-sigs] [openstack-dev] [goals][tc][ptl][uc] starting goal selection for T series

[Rochelle Grober]

Oh, very definitely +1000



--
Rochelle Grober Rochelle Grober
M: +1-6508889722(preferred)
E: rochelle.gro...@huawei.com<mailto:rochelle.gro...@huawei.com>
2012实验室-硅谷研究所技术规划及合作部
2012 Laboratories-Silicon Valley Technology Planning & 
Cooperation,Silicon Valley Research Center
From:Mathieu Gagné
To:openstack-s...@lists.openstack.org,
Cc:OpenStack Development Mailing List (not for usage questions),OpenStack 
Operators,
Date:2018-09-26 12:41:24
Subject:Re: [Openstack-sigs] [openstack-dev] [goals][tc][ptl][uc] starting goal 
selection for T series

+1 Yes please!

--
Mathieu

On Wed, Sep 26, 2018 at 2:56 PM Tim Bell  wrote:
>
>
> Doug,
>
> Thanks for raising this. I'd like to highlight the goal "Finish moving legacy 
> python-*client CLIs to python-openstackclient" from the etherpad and propose 
> this for a T/U series goal.
>
> To give it some context and the motivation:
>
> At CERN, we have more than 3000 users of the OpenStack cloud. We write an 
> extensive end user facing documentation which explains how to use the 
> OpenStack along with CERN specific features (such as workflows for requesting 
> projects/quotas/etc.).
>
> One regular problem we come across is that the end user experience is 
> inconsistent. In some cases, we find projects which are not covered by the 
> unified OpenStack client (e.g. Manila). In other cases, there are subsets of 
> the function which require the native project client.
>
> I would strongly support a goal which targets
>
> - All new projects should have the end user facing functionality fully 
> exposed via the unified client
> - Existing projects should aim to close the gap within 'N' cycles (N to be 
> defined)
> - Many administrator actions would also benefit from integration (reader 
> roles are end users too so list and show need to be covered too)
> - Users should be able to use a single openrc for all interactions with the 
> cloud (e.g. not switch between password for some CLIs and Kerberos for OSC)
>
> The end user perception of a solution will be greatly enhanced by a single 
> command line tool with consistent syntax and authentication framework.
>
> It may be a multi-release goal but it would really benefit the cloud 
> consumers and I feel that goals should include this audience also.
>
> Tim
>
> -Original Message-
> From: Doug Hellmann 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: Wednesday, 26 September 2018 at 18:00
> To: openstack-dev , openstack-operators 
> , openstack-sigs 
> 
> Subject: [openstack-dev] [goals][tc][ptl][uc] starting goal selection for T   
>   series
>
> It's time to start thinking about community-wide goals for the T series.
>
> We use community-wide goals to achieve visible common changes, push for
> basic levels of consistency and user experience, and efficiently improve
> certain areas where technical debt payments have become too high -
> across all OpenStack projects. Community input is important to ensure
> that the TC makes good decisions about the goals. We need to consider
> the timing, cycle length, priority, and feasibility of the suggested
> goals.
>
> If you are interested in proposing a goal, please make sure that before
> the summit it is described in the tracking etherpad [1] and that you
> have started a mailing list thread on the openstack-dev list about the
> proposal so that everyone in the forum session [2] has an opportunity to
> consider the details.  The forum session is only one step in the
> selection process. See [3] for more details.
>
> Doug
>
> [1] https://etherpad.openstack.org/p/community-goals
> [2] https://www.openstack.org/summit/berlin-2018/vote-for-speakers#/22814
> [3] https://governance.openstack.org/tc/goals/index.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> openstack-sigs mailing list
> openstack-s...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-sigs

___
openstack-sigs mailing list
openstack-s...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-sigs
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova][placement][upgrade][qa] Some upgrade-specific news on extraction

[Rochelle Grober]

Sounds like an important discussion to have with the operators in Denver. 
Should put this on the schedule for the Ops meetup.

--Rocky

> -Original Message-
> From: Matt Riedemann [mailto:mriede...@gmail.com]
> Sent: Thursday, September 06, 2018 1:59 PM
> To: OpenStack Development Mailing List (not for usage questions)
> ; openstack-
> operat...@lists.openstack.org
> Subject: [openstack-dev] [nova][placement][upgrade][qa] Some upgrade-
> specific news on extraction
> 
> I wanted to recap some upgrade-specific stuff from today outside of the
> other [1] technical extraction thread.
> 
> Chris has a change up for review [2] which prompted the discussion.
> 
> That change makes placement only work with placement.conf, not
> nova.conf, but does get a passing tempest run in the devstack patch [3].
> 
> The main issue here is upgrades. If you think of this like deprecating config
> options, the old config options continue to work for a release and then are
> dropped after a full release (or 3 months across boundaries for CDers) [4].
> Given that, Chris's patch would break the standard deprecation policy. Clearly
> one simple way outside of code to make that work is just copy and rename
> nova.conf to placement.conf and voila. But that depends on *all*
> deployment/config tooling to get that right out of the gate.
> 
> The other obvious thing is the database. The placement repo code as-is
> today still has the check for whether or not it should use the placement
> database but falls back to using the nova_api database [5]. So technically you
> could point the extracted placement at the same nova_api database and it
> should work. However, at some point deployers will clearly need to copy the
> placement-related tables out of the nova_api DB to a new placement DB and
> make sure the 'migrate_version' table is dropped so that placement DB
> schema versions can reset to 1.
> 
> With respect to grenade and making this work in our own upgrade CI testing,
> we have I think two options (which might not be mutually
> exclusive):
> 
> 1. Make placement support using nova.conf if placement.conf isn't found for
> Stein with lots of big warnings that it's going away in T. Then Rocky 
> nova.conf
> with the nova_api database configuration just continues to work for
> placement in Stein. I don't think we then have any grenade changes to make,
> at least in Stein for upgrading *from* Rocky. Assuming fresh devstack installs
> in Stein use placement.conf and a placement-specific database, then
> upgrades from Stein to T should also be OK with respect to grenade, but
> likely punts the cut-over issue for all other deployment projects (because we
> don't CI with grenade doing
> Rocky->Stein->T, or FFU in other words).
> 
> 2. If placement doesn't support nova.conf in Stein, then grenade will require
> an (exceptional) [6] from-rocky upgrade script which will (a) write out
> placement.conf fresh and (b) run a DB migration script, likely housed in the
> placement repo, to create the placement database and copy the placement-
> specific tables out of the nova_api database. Any script like this is likely
> needed regardless of what we do in grenade because deployers will need to
> eventually do this once placement would drop support for using nova.conf (if
> we went with option 1).
> 
> That's my attempt at a summary. It's going to be very important that
> operators and deployment project contributors weigh in here if they have
> strong preferences either way, and note that we can likely do both options
> above - grenade could do the fresh cutover from rocky to stein but we allow
> running with nova.conf and nova_api DB in placement in stein with plans to
> drop that support in T.
> 
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2018-
> September/subject.html#134184
> [2] https://review.openstack.org/#/c/600157/
> [3] https://review.openstack.org/#/c/600162/
> [4]
> https://governance.openstack.org/tc/reference/tags/assert_follows-
> standard-deprecation.html#requirements
> [5]
> https://github.com/openstack/placement/blob/fb7c1909/placement/db_api
> .py#L27
> [6] https://docs.openstack.org/grenade/latest/readme.html#theory-of-
> upgrade
> 
> --
> 
> Thanks,
> 
> Matt
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [Forum] [all] [Stable] OpenStack is "mature" -- time to get serious on Maintainers -- Session etherpad and food for thought for discussion

[Rochelle Grober]

Thanks, Lance!

Also, the more I think about it, the more I think Maintainer has too much 
baggage to use that term for this role.  It really is “continuity” that we are 
looking for.  Continuous important fixes, continuous updates of tools used to 
produce the SW.

Keep this in the back of your minds for the discussion.  And yes, this is a 
discussion to see if we are interested, and only if there is interest, how to 
move forward.

--Rocky

From: Lance Bragstad [mailto:lbrags...@gmail.com]
Sent: Friday, May 18, 2018 2:03 PM
To: Rochelle Grober <rochelle.gro...@huawei.com>; openstack-dev 
<openstack-...@lists.openstack.org>; openstack-operators 
<openstack-operators@lists.openstack.org>; user-committee 
<user-commit...@lists.openstack.org>
Subject: Re: [User-committee] [Forum] [all] [Stable] OpenStack is "mature" -- 
time to get serious on Maintainers -- Session etherpad and food for thought for 
discussion

Here is the link to the session in case you'd like to add it to your schedule 
[0].

[0] 
https://www.openstack.org/summit/vancouver-2018/summit-schedule/events/21759/openstack-is-mature-time-to-get-serious-on-maintainers
On 05/17/2018 07:55 PM, Rochelle Grober wrote:
Folks,

TL;DR
The last session related to extended releases is: OpenStack is "mature" -- time 
to get serious on Maintainers
It will be in room 220 at 11:00-11:40
The etherpad for the last session in the series on Extended releases is here:
https://etherpad.openstack.org/p/YVR-openstack-maintainers-maint-pt3

There are links to info on other communities’ maintainer 
process/role/responsibilities also, as reference material on how other have 
made it work (or not).

The nitty gritty details:

The upcoming Forum is filled with sessions that are focused on issues needed to 
improve and maintain the sustainability of OpenStack projects for the long 
term.  We have discussion on reducing technical debt, extended releases, fast 
forward installs, bringing Ops and User communities closer together, etc.  The 
community is showing it is now invested in activities that are often part of 
“Sustaining Engineering” teams (corporate speak) or “Maintainers (OSS speak).  
We are doing this; we are thinking about the moving parts to do this; let’s 
think about the contributors who want to do these and bring some clarity to 
their roles and the processes they need to be successful.  I am hoping you read 
this and keep these ideas in mind as you participate in the various Forum 
sessions.  Then you can bring the ideas generated during all these discussions 
to the Maintainers session near the end of the Summit to brainstorm how to 
visualize and define this new(ish) component of our technical community.

So, who has been doing the maintenance work so far?  Mostly (mostly) unsung 
heroes like the Stable Release team, Release team, Oslo team, project liaisons 
and the community goals champions (yes, moving to py3 is a 
sustaining/maintenance type of activity).  And some operators (Hi, mnaser!).  
We need to lean on their experience and what we think the community will need 
to reduce that technical debt to outline what the common tasks of maintainers 
should be, what else might fall in their purview, and how to partner with them 
to better serve them.

With API lower limits, new tool versions, placement, py3, and even projects 
reaching “code complete” or “maintenance mode,” there is a lot of work for 
maintainers to do (I really don’t like that term, but is there one that fits 
OpenStack’s community?).  It would be great if we could find a way to share the 
load such that we can have part time contributors here.  We know that operators 
know how to cherrypick, test in there clouds, do bug fixes.  How do we pair 
with them to get fixes upstreamed without requiring them to be full on 
developers?  We have a bunch of alumni who have stopped being “cores” and 
sometimes even developers, but who love our community and might be willing and 
able to put in a few hours a week, maybe reviewing small patches, providing 
help with user/ops submitted patch requests, or whatever.  They were trusted 
with +2 and +W in the past, so we should at least be able to trust they know 
what they know.  We  would need some way to identify them to Cores, since they 
would be sort of 1.5 on the voting scale, but……

So, burn out is high in other communities for maintainers.  We need to find a 
way to make sustaining the stable parts of OpenStack sustainable.

Hope you can make the talk, or add to the etherpad, or both.  The etherpad is 
very musch still a work in progress (trying to organize it to make sense).  If 
you want to jump in now, go for it, otherwise it should be in reasonable shape 
for use at the session.  I hope we get a good mix of community and a good 
collection of those who are already doing the job without title.

Thanks and see you next week.
--rocky




华为技术有限公司 Huawei Technologies Co., L

Re: [Openstack-operators] [Openstack-sigs] [QA] Proposal for a QA SIG

[Rochelle Grober]

First off, let me say I think this is a tremendous idea.  And, it's perfect for 
the SIG concept.

Next, see inline:

Thierry Carrez wrote:
> Andrea Frittoli wrote:
> > [...]
> > during the last summit in Sydney we discussed the possibility of
> > creating an OpenStack quality assurance special interest group (OpenStack
> QA SIG).
> > The proposal was discussed during the QA feedback session [0] and it
> > received positive feedback there; I would like to bring now the
> > proposal to a larger audience via the SIG, dev and operators mailing
> > lists.
> > [...]
> 
> I think this goes with the current trends of re-centering upstream "project
> teams" on the production of software, while using SIGs as communities of
> practice (beyond the governance boundaries), even if they happen to
> produce (some) software as the result of their work.
> 
> One question I have is whether we'd need to keep the "QA" project team at
> all. Personally I think it would create confusion to keep it around, for no 
> gain.
> SIGs code contributors get voting rights for the TC anyway, and SIGs are free
> to ask for space at the PTG... so there is really no reason (imho) to keep a
> "QA" project team in parallel to the SIG ?

Well, you can get rid of the "QA Project Team" but you would then need to 
replace it with something like the Tempest Project, or perhaps the Test 
Project.  You still need a PTL and cores to write, review and merge tempest 
fixes and upgrades, along with some of the tests.  The Interop Guideline tests 
are part of Tempest because being there provides oversight on the style and 
quality of the code of those tests.  We still need that.

--Rocky

> In the same vein we are looking into turning the Security project team into a
> SIG, and could consider turning other non-purely-upstream teams (like I18n)
> in the future.
> 
> --
> Thierry Carrez (ttx)
> 
> ___
> openstack-sigs mailing list
> openstack-s...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-sigs
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [LTS] Upstream LTS Releases

[Rochelle Grober]

Well, moving this discussion is easy.  All that takes is everyone posting 
responses to the openstack-...@lists.openstack.org mailing list instead of dev 
and ops lists.  I've cc'ed all here.  I've also added [LTS] to the subject 
(sorry to break all the threaders). So that the sig list knows what the general 
topic is.  Yeah.  It's not really a topic, but everyone is used to parsing 
those things, even if the mailserver sw isn't.

But, are the two groups willing to move this discussion to the sigs list?  If 
they are, great.  If not,  hmmm.

Anyway, here's my attempt to serve

--Rocky

> -Original Message-
> From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
> Sent: Tuesday, November 14, 2017 4:25 PM
> To: Rochelle Grober <rochelle.gro...@huawei.com>
> Cc: OpenStack Development Mailing List (not for usage questions)
> <openstack-...@lists.openstack.org>; openstack-oper.  operat...@lists.openstack.org>
> Subject: Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases
> 
> On Tue, Nov 14, 2017 at 4:10 PM, Rochelle Grober
> <rochelle.gro...@huawei.com> wrote:
> > Folks,
> >
> > This discussion and the people interested in it seem like a perfect
> application of the SIG process.  By turning LTS into a SIG, everyone can
> discuss the issues on the SIG mailing list and the discussion shouldn't end up
> split.  If it turns into a project, great.  If a solution is found that 
> doesn't need a
> new project, great.  Even once  there is a decision on how to move forward,
> there will still be implementation issues and enhancements, so the SIG could
> very well be long-lived.  But the important aspect of this is:  keeping the
> discussion in a place where both devs and ops can follow the whole thing and
> act on recommendations.
> >
> > Food for thought.
> >
> > --Rocky
> >
> Just to add more legs to the spider that is this thread: I think the SIG idea 
> is a
> good one. It may evolve into a project team some day, but for now it's a
> free-for-all polluting 2 mailing lists, and multiple etherpads. How do we go
> about creating one?
> 
> -Erik
> 
> >> -Original Message-
> >> From: Blair Bethwaite [mailto:blair.bethwa...@gmail.com]
> >> Sent: Tuesday, November 14, 2017 8:31 AM
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> <openstack-...@lists.openstack.org>; openstack-oper.  >> operat...@lists.openstack.org>
> >> Subject: Re: [openstack-dev] Upstream LTS Releases
> >>
> >> Hi all - please note this conversation has been split variously
> >> across -dev and - operators.
> >>
> >> One small observation from the discussion so far is that it seems as
> >> though there are two issues being discussed under the one banner:
> >> 1) maintain old releases for longer
> >> 2) do stable releases less frequently
> >>
> >> It would be interesting to understand if the people who want longer
> >> maintenance windows would be helped by #2.
> >>
> >> On 14 November 2017 at 09:25, Doug Hellmann
> <d...@doughellmann.com>
> >> wrote:
> >> > Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
> >> >> >> The concept, in general, is to create a new set of cores from
> >> >> >> these groups, and use 3rd party CI to validate patches. There
> >> >> >> are lots of details to be worked out yet, but our amazing UC
> >> >> >> (User
> >> >> >> Committee) will be begin working out the details.
> >> >> >
> >> >> > What is the most worrying is the exact "take over" process. Does
> >> >> > it mean that the teams will give away the +2 power to a
> >> >> > different team? Or will our (small) stable teams still be
> >> >> > responsible for landing changes? If so, will they have to learn
> >> >> > how to debug 3rd party CI
> >> jobs?
> >> >> >
> >> >> > Generally, I'm scared of both overloading the teams and losing
> >> >> > the control over quality at the same time :) Probably the final
> >> >> > proposal will
> >> clarify it..
> >> >>
> >> >> The quality of backported fixes is expected to be a direct (and
> >> >> only?) interest of those new teams of new cores, coming from users
> >> >> and operators and vendors. The more parties to establish their 3rd
> >> >> party
> >> >
> >> > We have an unhealthy focus on &

Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Rochelle Grober]

Folks,

This discussion and the people interested in it seem like a perfect application 
of the SIG process.  By turning LTS into a SIG, everyone can discuss the issues 
on the SIG mailing list and the discussion shouldn't end up split.  If it turns 
into a project, great.  If a solution is found that doesn't need a new project, 
great.  Even once  there is a decision on how to move forward, there will still 
be implementation issues and enhancements, so the SIG could very well be 
long-lived.  But the important aspect of this is:  keeping the discussion in a 
place where both devs and ops can follow the whole thing and act on 
recommendations.

Food for thought.

--Rocky

> -Original Message-
> From: Blair Bethwaite [mailto:blair.bethwa...@gmail.com]
> Sent: Tuesday, November 14, 2017 8:31 AM
> To: OpenStack Development Mailing List (not for usage questions)
> ; openstack-oper.  operat...@lists.openstack.org>
> Subject: Re: [openstack-dev] Upstream LTS Releases
> 
> Hi all - please note this conversation has been split variously across -dev 
> and -
> operators.
> 
> One small observation from the discussion so far is that it seems as though
> there are two issues being discussed under the one banner:
> 1) maintain old releases for longer
> 2) do stable releases less frequently
> 
> It would be interesting to understand if the people who want longer
> maintenance windows would be helped by #2.
> 
> On 14 November 2017 at 09:25, Doug Hellmann 
> wrote:
> > Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
> >> >> The concept, in general, is to create a new set of cores from
> >> >> these groups, and use 3rd party CI to validate patches. There are
> >> >> lots of details to be worked out yet, but our amazing UC (User
> >> >> Committee) will be begin working out the details.
> >> >
> >> > What is the most worrying is the exact "take over" process. Does it
> >> > mean that the teams will give away the +2 power to a different
> >> > team? Or will our (small) stable teams still be responsible for
> >> > landing changes? If so, will they have to learn how to debug 3rd party CI
> jobs?
> >> >
> >> > Generally, I'm scared of both overloading the teams and losing the
> >> > control over quality at the same time :) Probably the final proposal will
> clarify it..
> >>
> >> The quality of backported fixes is expected to be a direct (and
> >> only?) interest of those new teams of new cores, coming from users
> >> and operators and vendors. The more parties to establish their 3rd
> >> party
> >
> > We have an unhealthy focus on "3rd party" jobs in this discussion. We
> > should not assume that they are needed or will be present. They may
> > be, but we shouldn't build policy around the assumption that they
> > will. Why would we have third-party jobs on an old branch that we
> > don't have on master, for instance?
> >
> >> checking jobs, the better proposed changes communicated, which
> >> directly affects the quality in the end. I also suppose, contributors
> >> from ops world will likely be only struggling to see things getting
> >> fixed, and not new features adopted by legacy deployments they're used
> to maintain.
> >> So in theory, this works and as a mainstream developer and
> >> maintainer, you need no to fear of losing control over LTS code :)
> >>
> >> Another question is how to not block all on each over, and not push
> >> contributors away when things are getting awry, jobs failing and
> >> merging is blocked for a long time, or there is no consensus reached
> >> in a code review. I propose the LTS policy to enforce CI jobs be
> >> non-voting, as a first step on that way, and giving every LTS team
> >> member a core rights maybe? Not sure if that works though.
> >
> > I'm not sure what change you're proposing for CI jobs and their voting
> > status. Do you mean we should make the jobs non-voting as soon as the
> > branch passes out of the stable support period?
> >
> > Regarding the review team, anyone on the review team for a branch that
> > goes out of stable support will need to have +2 rights in that branch.
> > Otherwise there's no point in saying that they're maintaining the
> > branch.
> >
> > Doug
> >
> >
> __
> 
> >  OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> --
> Cheers,
> ~Blairo
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-operators mailing list

Re: [Openstack-operators] Fault Genes WG

[Rochelle Grober]

The meeting is an online video/voice/collaboration meeting.  By clicking the 
link: https://welink-meeting.zoom.us/j/317491860 you will go to a page that 
will download the zoom client installation package.  Install that, run it and 
put the meeting ID in where asked.  Zoom works all over the world.  Once you 
have the client, when you click on the link in the future, it will ask you 
whether you want the zoom client launched.

No IRC room for this one.  It seems that the user groups often are more 
comfortable and productive with interactive meetings.

Hope this helps.

--Rocky


From: randy.perry...@dell.com [mailto:randy.perry...@dell.com]
Sent: Thursday, July 20, 2017 9:06 AM
To: Nematollah Bidokhti ; 
user-commit...@lists.openstack.org; openstack-operators@lists.openstack.org
Subject: Re: [User-committee] [Openstack-operators] Fault Genes WG

Dell - Internal Use - Confidential
Hi,
Is this only on a weblink?  Is there a meeting room on IRC for this?

-Original Appointment-
From: Nematollah Bidokhti [mailto:nematollah.bidok...@huawei.com]
Sent: Wednesday, March 22, 2017 8:20 PM
To: Nematollah Bidokhti; 
user-commit...@lists.openstack.org; 
openstack-operators@lists.openstack.org
Subject: [Openstack-operators] Fault Genes WG
When: Thursday, July 20, 2017 9:00 AM-10:00 AM (UTC-08:00) Pacific Time (US & 
Canada).
Where: Using Zoom Conf. Service - Meeting ID is 317491860


When: Occurs every Thursday from 9:00 AM to 10:00 AM effective 3/23/2017. 
(UTC-08:00) Pacific Time (US & Canada)
Where: Using Zoom Conf. Service - Meeting ID is 317491860

*~*~*~*~*~*~*~*~*~*
Hi there,

nematollah.bidok...@huawei.com is 
inviting you to a scheduled Zoom meeting.

Topic: Fault Genes WG

Time: this is a recurring meeting Meet anytime

Join from PC, Mac, Linux, iOS or Android: 
https://welink-meeting.zoom.us/j/317491860

Or iPhone one-tap (US Toll):  +16465588656,317491860# or +14086380968,317491860#

Or Telephone:

Dial: +1 646 558 8656 (US Toll) or +1 408 638 0968 (US Toll)
Meeting ID: 317 491 860

International numbers available: 
https://welink-meeting.zoom.us/zoomconference?m=qqUZ1nX7Q2YCsoeZbbUf9Wf3EkBnmwWe


 << File: ATT1.txt >>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Enterprise][Product WG][LCOO][Scientific][Public Cloud][Massively Parallel][All][Interop]FW: [openstack-dev] [all] etcd3 as base service - update

[Rochelle Grober]

I wanted to make sure this got out to as much of the community as soon as 
possible.  This "new base service" will be part of Pike.  This means that etcd 
will be a requirement for Pike installations and beyond.

Most of you won't need to take any immediate actions, but knowing the plans for 
your future is always a good thing.  I'm trying to limit surprises here.

--Rocky

-Original Message-
From: Davanum Srinivas [mailto:dava...@gmail.com] 
Sent: Wednesday, June 07, 2017 3:48 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: [openstack-dev] [all] etcd3 as base service - update

Team,

Here's the update to the base services resolution from the TC:
https://governance.openstack.org/tc/reference/base-services.html

First request is to Distros, Packagers, Deployers, anyone who 
installs/configures OpenStack:
Please make sure you have latest etcd 3.x available in your environment for 
Services to use, Fedora already does, we need help in making sure all distros 
and architectures are covered.

Any project who want to use etcd v3 API via grpc, please use:
https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)

Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
https://pypi.python.org/pypi/etcd3gw

If you use tooz, there are 2 driver choices for you:
https://github.com/openstack/tooz/blob/master/setup.cfg#L29
https://github.com/openstack/tooz/blob/master/setup.cfg#L30

If you use oslo.cache, there is a driver for you:
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33

Devstack installs etcd3 by default and points cinder to it:
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356

Review in progress for keystone to use etcd3 for caching:
https://review.openstack.org/#/c/469621/

Doug is working on proposal(s) for oslo.config to store some configuration in 
etcd3:
https://review.openstack.org/#/c/454897/

So, feel free to turn on / test with etcd3 and report issues.

Thanks,
Dims

--
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] MIL Ops meetup -- Rabbitmq Pitfalls with HA etherpad

[Rochelle Grober]

Please add your comments, suggestions, info, etc. to the Rabbitmq with HA 
pitfalls etherpad.  There are a few things out there already that will 
hopefully induce more and better items to include.

See you tomorrow!

https://etherpad.openstack.org/p/MIL-ops-rabbitmq-pitfalls-ha

--Rocky


华为技术有限公司 Huawei Technologies Co., Ltd.
[Company_logo]
Rochelle Grober
Sr. Staff Architect, Open Source
Office Phone:408-330-5472
Email:rochelle.gro...@huawei.com

 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁
止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中
的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This e-mail and its attachments contain confidential information from HUAWEI, 
which
is intended only for the person or entity whose address is listed above. Any 
use of the
information contained herein in any way (including, but not limited to, total 
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender by
phone or email immediately and delete it!

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [keystone][tripleo][ansible][puppet][all] changing default token format

[Rochelle Grober]

a blog post on the OpenStack sore might be good. superuser? there are folks 
reading this who can help

Sent from HUAWEI AnyOffice
From:Lance Bragstad
To:OpenStack Development Mailing List (not for usage 
questions),openstack-operators@lists.openstack.org,
Date:2016-11-03 08:11:20
Subject:Re: [openstack-dev] [keystone][tripleo][ansible][puppet][all] changing 
default token format

I totally agree with communicating this the best we can. I'm adding the 
operator list to this thread to increase visibility.

If there are any other methods folks think of for getting the word out, outside 
of what we've already done (release notes, email threads, etc.), please let me 
know. I'd be happy to drive those communications.

On Thu, Nov 3, 2016 at 9:45 AM, Alex Schultz 
> wrote:
Hey Steve,

On Thu, Nov 3, 2016 at 8:29 AM, Steve Martinelli 
> wrote:
> Thanks Alex and Emilien for the quick answer. This was brought up at the
> summit by Adam, but I don't think we have to prevent keystone from changing
> the default. TripleO and Puppet can still specify UUID as their desired
> token format; it is not deprecated or slated for removal. Agreed?
>

My email was not to tell you to stop.I was just letting you know that
your change does not affect the puppet modules because we define our
default as UUID.  It was just as a heads up to others on this email
that this change should not affect anyone consuming the puppet modules
because our default is still UUID and will be even after keystone's
default changes.

Thanks,
-Alex

> On Thu, Nov 3, 2016 at 10:23 AM, Alex Schultz 
> > wrote:
>>
>> Hey Steve,
>>
>> On Thu, Nov 3, 2016 at 8:11 AM, Steve Martinelli 
>> >
>> wrote:
>> > As a heads up to some of keystone's consuming projects, we will be
>> > changing
>> > the default token format from UUID to Fernet. Many patches have merged
>> > to
>> > make this possible [1]. The last 2 that you probably want to look at are
>> > [2]
>> > and [3]. The first flips a switch in devstack to make fernet the
>> > selected
>> > token format, the second makes it default in Keystone itself.
>> >
>> > [1] https://review.openstack.org/#/q/topic:make-fernet-default
>> > [2] DevStack patch: https://review.openstack.org/#/c/367052/
>> > [3] Keystone patch: https://review.openstack.org/#/c/345688/
>> >
>>
>> Thanks for the heads up. In puppet openstack we had already
>> anticipated this and attempted to do the same for the
>> puppet-keystone[0] module as well.  Unfortunately after merging it, we
>> found that tripleo wasn't yet prepared to handle the HA implementation
>> of fernet tokens so we had to revert it[1].  This shouldn't impact
>> anyone currently consuming puppet-keystone as we define uuid as the
>> default for now. Our goal is to do something similar this cycle but
>> there needs to be some further work in the downstream consumers to
>> either define their expected default (of uuid) or support fernet key
>> generation correctly.
>>
>> Thanks,
>> -Alex
>>
>> [0] https://review.openstack.org/#/c/389322/
>> [1] https://review.openstack.org/#/c/392332/
>>
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] : Public cloud operators group in

[Rochelle Grober]

I've followed "cloud for at least as long as OpenStack has existed, but back 
then I followed whatever/whoever called themselves "cloud" or 
"cloud-{app|service|etc}" and at one point there was a heated discussion 
(mostly that the rest of the group agreed with) that you couldn't claim you ran 
in the/a cloud if you utilized your own equipment in your own data center.



So, yeah.  The rest of the world doesn't always see cloud the way we do.



--Rocky





From: Silence Dogood <m...@nycresistor.com>

I figure if you have entity Y's workloads running on entity X's hardware...

and that's 51% or greater portion of gross revenue... you are a public

cloud.



On Mon, Sep 26, 2016 at 11:35 AM, Kenny Johnston 
<ke...@kencjohnston.com<mailto:ke...@kencjohnston.com>>

wrote:



> That seems like a strange definition. It doesn't incorporate the usual

> multi-tenancy requirement that traditionally separates private from public

> clouds. By that definition, Rackspace's Private Cloud offer, where we

> design, deploy and operate a single-tenant cloud on behalf of customers (in

> their data-center or ours) would be considered a "public" cloud.

>

> On Fri, Sep 23, 2016 at 3:54 PM, Rochelle Grober <

> rochelle.gro...@huawei.com<mailto:rochelle.gro...@huawei.com>> wrote:

>

>> Hi Matt,

>>

>>

>>

>> At considerable risk of heading down a rabbit hole... how are you

>> defining "public" cloud for these purposes?

>>

>>

>>

>> Cheers,

>>

>> Blair

>>

>>

>>

>> Any cloud that provides a cloud to a thirdparty in exchange for money.

>> So, rent a VM, rent a collection of vms, lease a fully operational cloud

>> spec'ed to your requirements, lease a team and HW with your cloud on

>> them.

>>

>>

>>

>> So any cloud that provides offsite IAAS to lessees

>>

>>

>>
>> --Rocky
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Public cloud operators group in

[Rochelle Grober]

Hi Matt,



At considerable risk of heading down a rabbit hole... how are you defining 
"public" cloud for these purposes?



Cheers,

Blair

Any cloud that provides a cloud to a thirdparty in exchange for money.  So, 
rent a VM, rent a collection of vms, lease a fully operational cloud spec'ed to 
your requirements, lease a team and HW with your cloud on them.

So any cloud that provides offsite IAAS to lessees

--Rockyy
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [ALL] Looking for participants in the Interop Challenge for Barcelona

[Rochelle Grober]

Hey folks!

Do you know about the Interop Challenge?  Well, here's a way to learn about it 
and and participate if you like...

"The interop challenge was started in July 2016 to create a set of common 
workloads/tests to be executed across multiple OpenStack distributions and/or 
cloud deployment models. The participants in this challenge will work together 
to prove once and for all that OpenStack-Powered clouds are interoperable."

The WG (and lots of other interested folks) would like to see how many of our 
Cloud deployers, vendors, distros, etc can successfully deploy and run some 
very common apps to their clouds.  And we want to let the world know about how 
easy it is in Barcelona.

The WG started with a number of vendors to create some OpenStack apps that 
demonstrate the kind of workloads users want to run.  The team identified four 
or so and started implementing the apps.  Those apps are:

* LAMP Stack

* Docker Swarm

* NFV
The  Ansible LAMP stack code can be found here:
http://git.openstack.org/cgit/openstack/osops-tools-contrib/tree/ansible/lampstack

And a Terraform version is here:
http://git.openstack.org/cgit/openstack/osops-tools-contrib/tree/terraform/lampstack

The Heat LAMP stack code is here:
http://git.openstack.org/cgit/openstack/osops-tools-contrib/tree/heat

The LAMP stack code is ready for Operators and other deployers and developers 
to take for a spin.  There are still some bugs and we are wringing out issues 
in the app as more clouds attempt to deoploy it.  But, you can file bugs on it, 
or file fixes, etc.

Mailing List discussions are happening on the defcore-committee mailing list.
IRC discussions are in openstack-defcore


Here is a bunch moreinfo on the challenge:
https://wiki.openstack.org/wiki/Interop_Challenge

And here is how to share your results:
https://wiki.openstack.org/wiki/Interop_Challenge#RefStack_Testing_and_Results_Upload

Come to the IRC meeting in #openstack-meeting-cp on Wednesday at UTC 14:-00 to 
ask questions.

Show the world that the OpenStack community not only has lots of clouds out 
there, but that they play nice together!!
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] Seeking feedback: Active User Contributor (AUC) eligibility requirements

[Rochelle Grober]

Umm, I see a major contribution area not included here:

OpenStack community meetup organizers.  I know some sink a large amount of time 
scheduling and organizing at least one a month, some two or more.  These 
organizers are critical for getting info out on OpenStack and familiarizing 
their local tech communities with the OpenStack project.  I hope they are 
included somewhere for their contributions.

--Rocky

-Original Message-
From: Jonathan D. Proulx [mailto:j...@csail.mit.edu] 
Sent: Thursday, June 30, 2016 12:00 PM
To: Shamail Tahir
Cc: openstack-operators; user-committee
Subject: Re: [User-committee] Seeking feedback: Active User Contributor (AUC) 
eligibility requirements


I'm surprised this hasn't generated more feed back, though I'd
generally take that as positive.

List seems good to me.

The self nomintaion + confirm by UC is a good catch all especially in
the beginning where we're unlikely to have though of everything.  We
can always expand criterial later if the 'misc' of UC confirmantion
gets too big and we idnetify patterns.

Thanks all!
-Jon

On Wed, Jun 29, 2016 at 04:52:00PM -0400, Shamail Tahir wrote:
:Hi everyone,
:
:The AUC Recognition WG has been hard at work on milestone-4 of our plan
:which is to identify the eligibility criteria for each community
:contributor role that is covered by AUC.  We had a great mix of community
:people involved in defining these thresholds but we wanted to also open
:this up for broader community feedback before we propose them to the user
:committee.  AUC is a new concept and we hope to make iterative improvements
:going forward... you can consider the guidelines below as "version 1" and I
:am certain they will evolve as lessons are learned.  Thank you in advance
:for your feedback!
:
:*  Official User Group organizers
:
:o   Listed as an organizer or coordinator for an official OpenStack user
:group
:
:*  Active members of official UC Working Groups
:
:o   Attend 25% of the IRC meetings and have spoken more than 25 times OR
:have spoken more than 100 times regardless of attendance count over the
:last six months
:
:o   WG that do not use IRC for their meetings will depend on the meeting
:chair(s) to identify active participation from attendees
:
:*  Ops meetup moderators
:
:o   Moderate a session at the operators meetup over the last six
:months AND/OR
:
:o   Host the operators meetup (limit 2 people from the hosting
:organization) over the last six months
:
:*  Contributions to any repository under UC governance (ops
:repositories, user stories repository, etc.)
:
:o   Submitted two or more patches to a UC governed repository over the last
:six months
:
:*  Track chairs for OpenStack Summits
:
:o   Identified track chair for the upcoming OpenStack Summit (based on when
:data is gathered) [this is a forward-facing metric]
:
:*  Contributors to Superuser (articles, interviews, user stories, etc.)
:
:o   Listed as author in at least one publication at superuser.openstack.org
:over the last six months
:
:*  Submission for eligibility to AUC review panel
:
:o   No formal criteria, anyone can self-nominate, and nominations will be
:reviewed per guidance established in milestone-5
:
:*  Active moderators on ask.openstack
:
:o   Listed as moderator on Ask OpenStack and have over 500 karma
:
:There is additional information available in the etherpad[1] the AUC
:recognition WG has been using for this task which includes Q (question
:and answers) between team members.
:
:[1] https://etherpad.openstack.org/p/uc-recog-metrics
:
:-- 
:Thanks,
:Shamail Tahir
:t: @ShamailXD
:tz: Eastern Time

:___
:User-committee mailing list
:user-commit...@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee


-- 

___
User-committee mailing list
user-commit...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] Rabbit-mq 3.4 crashing (anyone else seen this?)

[Rochelle Grober]

repository is:  http://git.openstack.org/cgit/openstack/osops-tools-contrib/

FYI, there are also:  osops-tools-generic, osops-tools-logging, 
osops-tools-monitoring, osops-example-configs and osops-coda

Wish I could help more,

--Rocky

-Original Message-
From: Joshua Harlow [mailto:harlo...@fastmail.com] 
Sent: Tuesday, July 05, 2016 10:44 AM
To: Matt Fischer
Cc: openstack-...@lists.openstack.org; OpenStack Operators
Subject: Re: [openstack-dev] [Openstack-operators] [nova] Rabbit-mq 3.4 
crashing (anyone else seen this?)

Ah, those sets of command sound pretty nice to run periodically,

Sounds like a useful script that could be placed in the ops tools repo 
(I forget where this repo exists at, but pretty sure it does exist?).

Some other oddness though is that this issue seems to go away when we 
don't run cross-release; do you see that also?

Another hypothesis was that the following fix may be triggering part of 
this @ https://bugs.launchpad.net/oslo.messaging/+bug/1495568

So that if we have some queues being set up as auto-delete and some 
beign set up with expiry that perhaps the combination of these causes 
more work (and therefore eventually it falls behind and falls over) for 
the management database.

Matt Fischer wrote:
> Yes! This happens often but I'd not call it a crash, just the mgmt db
> gets behind then eats all the memory. We've started monitoring it and
> have runbooks on how to bounce just the mgmt db. Here are my notes on that:
>
> restart rabbitmq mgmt server - this seems to clear the memory usage.
>
> rabbitmqctl eval 'application:stop(rabbitmq_management).'
> rabbitmqctl eval 'application:start(rabbitmq_management).'
>
> run GC on rabbit_mgmt_db:
> rabbitmqctl eval
> '(erlang:garbage_collect(global:whereis_name(rabbit_mgmt_db)))'
>
> status of rabbit_mgmt_db:
> rabbitmqctl eval 'sys:get_status(global:whereis_name(rabbit_mgmt_db)).'
>
> Rabbitmq mgmt DB how much memory is used:
> /usr/sbin/rabbitmqctl status | grep mgmt_db
>
> Unfortunately I didn't see that an upgrade would fix for sure and any
> settings changes to reduce the number of monitored events also require a
> restart of the cluster. The other issue with an upgrade for us is the
> ancient version of erlang shipped with trusty. When we upgrade to Xenial
> we'll upgrade erlang and rabbit and hope it goes away. I'll also
> probably tweak the settings on retention of events then too.
>
> Also for the record the GC doesn't seem to help at all.
>
> On Jul 5, 2016 11:05 AM, "Joshua Harlow"  > wrote:
>
> Hi ops and dev-folks,
>
> We over at godaddy (running rabbitmq with openstack) have been
> hitting a issue that has been causing the `rabbit_mgmt_db` consuming
> nearly all the processes memory (after a given amount of time),
>
> We've been thinking that this bug (or bugs?) may have existed for a
> while and our dual-version-path (where we upgrade the control plane
> and then slowly/eventually upgrade the compute nodes to the same
> version) has somehow triggered this memory leaking bug/issue since
> it has happened most prominently on our cloud which was running
> nova-compute at kilo and the other services at liberty (thus using
> the versioned objects code path more frequently due to needing
> translations of objects).
>
> The rabbit we are running is 3.4.0 on CentOS Linux release 7.2.1511
> with kernel 3.10.0-327.4.4.el7.x86_64 (do note that upgrading to
> 3.6.2 seems to make the issue go away),
>
> # rpm -qa | grep rabbit
>
> rabbitmq-server-3.4.0-1.noarch
>
> The logs that seem relevant:
>
> ```
> **
> *** Publishers will be blocked until this alarm clears ***
> **
>
> =INFO REPORT 1-Jul-2016::16:37:46 ===
> accepting AMQP connection <0.23638.342> (127.0.0.1:51932
>  -> 127.0.0.1:5671 )
>
> =INFO REPORT 1-Jul-2016::16:37:47 ===
> vm_memory_high_watermark clear. Memory used:29910180640
> allowed:47126781542
> ```
>
> This happens quite often, the crashes have been affecting our cloud
> over the weekend (which made some dev/ops not so happy especially
> due to the july 4th mini-vacation),
>
> Looking to see if anyone else has seen anything similar?
>
> For those interested this is the upstream bug/mail that I'm also
> seeing about getting confirmation from the upstream users/devs
> (which also has erlang crash dumps attached/linked),
>
> https://groups.google.com/forum/#!topic/rabbitmq-users/FeBK7iXUcLg
>
> Thanks,
>
> -Josh
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> 
> 

[Openstack-operators] FW: [openstack-dev] Collecting our wiki use cases

[Rochelle Grober]

FYI.

If you don't comment, you can't complain when they don't address your needs.

--Rocky

-Original Message-
From: Thierry Carrez [mailto:thie...@openstack.org] 
Sent: Wednesday, May 11, 2016 8:28 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] Collecting our wiki use cases

(posting as separate thread upon request)

Hi everyone,

At the beginning of OpenStack we have been using wiki.openstack.org as 
our default community lightweight information publication platform. 
There were/are lots of things in there, reference information that a 
couple people struggled to keep up to date (and non-vandalized), old 
pages referencing processes or projects long gone, meeting agendas and 
minutes, etc. That created a mix of up-to-date reference information and 
completely outdated random information that is confusing to use 
(especially for newcomers to our community) and that Google search 
indiscriminately provides links to.

Over the last two years, various groups have worked to push reference 
information out of the wiki, to proper documentation guides (like the 
infra guide or the project team guide) or to peer-reviewed specific 
reference websites (like security.o.o or releases.o.o). These efforts to 
move reference content to other tools and websites where appropriate 
will continue.

There are still a lot of use cases for which the wiki is a good 
solution, though, and it is likely that we'll need a lightweight 
publication platform like a wiki to cover those use cases indefinitely. 
Since it's difficult to have a complete view of what the wiki is 
currently used for, I'd like to collect information from current wiki 
users, to make sure we have the complete picture of wiki use cases as we 
discuss adding new tools to our toolbelt.

So, if you use the wiki as part of your OpenStack work, make sure to 
check if your use case is mentioned on the etherpad at:

https://etherpad.openstack.org/p/wiki-use-cases

If not, please add your use case, together with the URL of such a wiki 
page to that etherpad.

Thanks in advance,

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Austin Ops meetup session: Taxonomy of Failure -- please comment!

[Rochelle Grober]

The Ops session on the taxonomy of failure  can use some input even before the 
session itself!

The etherpad is here:

https://etherpad.openstack.org/p/AUS-ops-Taxonomy-of-Failures

And has a brief outline on some of info we'd like to gather.  There is also a 
google spreadsheet here:

https://docs.google.com/spreadsheets/d/1sekKLp7C8lsTh-niPHNa2QLk5kzEC_2w_UsG6ifC-Pw/edit?usp=sharing

that has a couple of examples already in it that would be a great way to 
collect data.

Please go out to the etherpad, read and think about the issue if you are 
interested, and comment ahead if possible.  It will provide more material for 
the discussion.  It also is a way to provide your input even if you can't  be 
at the summit.

Thanks for your time and consideration and see some of you there!

--Rocky
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Config] Cross project spec that organizes/changes how configuration options are provided at deploy/upgrade

[Rochelle Grober]

There is currently a cross project specification under review that changes how 
and where the config files are written for an OpenStack installation.  If 
config options and how they are presented to you are important to you, I 
strongly suggest you go out , read the spec/review and comment on both the 
aspects that work for you and the aspects that don't work for you.  Be clear on 
explaining why something doesn't work, as the developers believe what they are 
proposing is what you would want.  You also might want to read the comments on 
version 6 of this review.

--Rocky

The review is here:
https://review.openstack.org/#/c/295543/

The commit message is:
This spec looks at how to get a better configuration option experience
for operators and (to a lesser extent) developers.
The summary is:
Operators currently have to grapple with hundreds of configuration options
across tens of OpenStack services across multiple projects. In many cases
they also need to be aware of the relationships between the options.
In particular, this makes the initial deployment a very daunting task.

This spec proposes a way projects can use oslo.config in a way that leads
to better experience for operators trying to understand all the configuration
options, and helps developers better express the intent of each configuration
option.

This spec re-uses and builds on the ideas in this recent Nova specification:
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/centralize-config-options.html

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] Floating IPs and Public IPs are not equivalent

[Rochelle Grober]

Cross posting to the Ops ML as one/some of them might have a test cloud like 
this.

Operators:

If you respond to this thread, please only respond to the openstack-dev list?

They could use your input;-)

--Rocky

-Original Message-
From: Sean Dague [mailto:s...@dague.net] 
Sent: Thursday, March 31, 2016 12:58 PM
To: openstack-...@lists.openstack.org
Subject: Re: [openstack-dev] Floating IPs and Public IPs are not equivalent

On 03/31/2016 01:23 PM, Monty Taylor wrote:
> Just a friendly reminder to everyone - floating IPs are not synonymous
> with Public IPs in OpenStack.
> 
> The most common (and growing, thank you to the beta of the new
> Dreamcompute cloud) configuration for Public Clouds is directly assign
> public IPs to VMs without requiring a user to create a floating IP.
> 
> I have heard that the require-floating-ip model is very common for
> private clouds. While I find that even stranger, as the need to run NAT
> inside of another NAT is bizarre, it is what it is.
> 
> Both models are common enough that pretty much anything that wants to
> consume OpenStack VMs needs to account for both possibilities.
> 
> It would be really great if we could get the default config in devstack
> to be to have a shared direct-attached network that can also have a
> router attached to it and provider floating ips, since that scenario
> actually allows interacting with both models (and is actually the most
> common config across the OpenStack public clouds)

If someone has the the pattern for what that config looks like,
especially if it could work on single interface machines, that would be
great.

The current defaults in devstack are mostly there for legacy reasons
(and because they work everywhere), and for activation energy to getting
a new robust work everywhere setup.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] Announcing the Non-ATC Recognition Working Group

[Rochelle Grober]

Left a comment on the doodle.

Could you double-check the dates?  Most of the days and times listed are 
already past.

--Rocky

From: Shamail Tahir [mailto:itzsham...@gmail.com]
Sent: Wednesday, March 30, 2016 10:36 AM
To: openstack-operators; commun...@lists.openstack.org; 
user-commit...@lists.openstack.org
Subject: Re: [User-committee] Announcing the Non-ATC Recognition Working Group

Hi everyone,

Friendly reminder:  The doodle poll[1] for the Non-ATC recognition WG meeting 
time will be closing at March 31st, 2100 UTC (tomorrow).

Please vote if you plan to join us!

[1] http://doodle.com/poll/7r5khrefyx7wysi4

Thanks,
Shamail

On Mon, Mar 28, 2016 at 3:33 PM, Shamail Tahir 
> wrote:
Hi everyone,

We had some great discussion on the mailing lists on how to recognize operators 
and other contributors in the OpenStack community who don't qualify for ATC 
(Active Technical Contributor) status earlier this month[1].  We revisited this 
topic at the last User Committee meeting[2] and decided to start a short-term 
working group with one objective: to help define the User Committee 
constituency.  This will help us determine who should be included in the {not 
yet named} contributor program by the User Committee and how to qualify.

Maish and I volunteered to help start the conversations, document the 
decisions, and provide updates on the outcome to the User Committee.  We have 
created a wiki[3] for this new working group (which will be disbanded once we 
have reached our objective) and there is an active doodle poll[4] to determine 
the time for our weekly meeting, which will be held in an IRC channel.  The 
poll shows the start dates as this week but the please just vote for the 
day/time (not date) since we will not begin our meeting until next week (week 
of 4/4).

Please volunteer if you want to help us develop the membership definition for 
this new designation.  The user committee will have a lot more work ahead of 
them to fully build out a new charter and/or program but this activity will be 
extremely helpful as a foundational building block.

[1] http://lists.openstack.org/pipermail/community/2016-March/001422.html
[2] http://eavesdrop.openstack.org/meetings/uc/2016/uc.2016-03-14-19.00.log.html
[3] https://wiki.openstack.org/wiki/NonATCRecognition
[4] http://doodle.com/poll/7r5khrefyx7wysi4

--
Thanks,
Shamail Tahir
t: @ShamailXD
tz: Eastern Time

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][neutron] What are your cells

[Rochelle Grober]

There is also a bit of info on cells from the Manchester meetup in the Large 
Deployment team ether pad:

https://etherpad.openstack.org/p/MAN-ops-Large-Deployment-Team

I've added the link to the cells etherpad.

--Rocky
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Draft Agenda for MAN Ops Meetup (Feb 15, 16)

[Rochelle Grober]

Hey, folks.  Two things about the agenda.

I can handle the "OSOps - what is it, where is it going, what you can do" 
presentation.  There might be one other from the group, but if not, it's mine.

Also, instead of the OSOps working session, I'd like to propose:

A session on config opts and what the Ops guys need/want with regard to 
consolidation/distribution.  Also give a briefing on what the oslo project is 
pushing for them wrt to documentation and what Nova opts working group is 
doing.  This is an important precursor for some of the notifications, logging 
and metrics work that still needs to be addressed.  And I can present/moderate 
this session.

--Rocky
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Log] Tomorrow's Log WG meeting is cancelled

[Rochelle Grober]

Getting ready to eat lots of turkey

--Rocky
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [all]Can we get some sanity in the Neutron logs please?

[Rochelle Grober]

Thanks both Armando and Matt!

I am cross posting this to the operators' list (as the main post -- operators, 
simple reply and no spam to dev).

The logging tag should be a tag in all projects.  I'm glad it's already there 
in Neutron.  And, this sort of feedback is exactly what we need operators and 
gate debuggers to provide when they hit stuff in the logs that is either 
classified at the wrong level (e.g., warning instead of info), is too verbose, 
generates a traceback rather than an error, or is too chatty, happens too often.

Please file these bugs to the appropriate projects and tag them "logging".  
Generally, these are easy fixes, or are "while you're in there" fixes while a 
bug is being addressed, so reporting them will most often get them fixed 
quickly.  I know that doesn't help the Ops folks as much as the developers, but 
it will help the developers when they upgrade to releases the logging has been 
fixed in.

These sorts of issues are major maintainability issues for operators, so a 
focus on this for the Mitaka release will help make it a release operators want 
to upgrade to :-)

Thanks for the ear and for the help with debugging OpenStack issues by 
improving the signal to noise ratio.

--Rocky

From: Armando M. [mailto:arma...@gmail.com]
Sent: Tuesday, November 10, 2015 11:39 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Can we get some sanity in the Neutron logs please?



On 10 November 2015 at 11:12, Matt Riedemann 
> wrote:


On 11/10/2015 1:10 PM, Matt Riedemann wrote:


On 11/10/2015 12:51 PM, Armando M. wrote:


On 10 November 2015 at 10:33, Matt Riedemann 

>> wrote:

Let me qualify by saying I'm not a Neutron person.

We know that gate-tempest-dsvm-neutron-full is failing hard as of
the last 24 hours [1].

An error that's been showing up in tempest runs with neutron a lot
is:

"AssertionError: 0 == 0 : No IPv4 addresses found in: []"

So checking logstash [2] it's hitting a lot. It's only recent
because that failure message is new to Tempest in the last day or
so, but it has a lot of hits, so whatever it is, it's failing a lot.

So the next step is usually digging into service logs looking for
errors. I check the q-svc logs first. Not many errors but a
bazillion warnings for things not found (networks and devices). [3]

For example:

2015-11-10 17:13:02.542 WARNING neutron.plugins.ml2.rpc
[req-15a73753-1512-4689-9404-9658a0cd0c09 None None] Device
aaa525be-14eb-44a5-beb0-ed722896be93 requested by agent
ovs-agent-devstack-trusty-rax-iad-5785199 not found in database

2015-11-10 17:14:17.754 WARNING neutron.api.rpc.handlers.dhcp_rpc
[req-3d7e9848-6151-4780-907f-43f11a2a8545 None None] Network
b07ad9b2-e63e-4459-879d-3721074704e5 could not be found, it might
have been deleted concurrently.

Are several hundred of these warnings useful to an operator trying
to debug a problem? The point of the CI gate testing is to try and
simulate a production cloud environment. When something goes wrong,
you check the logs. With the amount of warning/error level logging
that is in the neutron logs, finding a real problem is like looking
for a needle in a haystack. Since everything is async, 404s are
expected when racing to delete a resource and they should be handled
gracefully.

Anyway, the server log isn't useful so I go digging in the agent
logs and stacktraces there are aplenty. [4]

Particularly this:

"Exception: Port tapcea51630-e1 is not ready, resync needed"

That's due to a new change landing in the last 24 hours [5]. But the
trace shows up over 16K times since it landed [6].

Checking the code, it's basically a loop processing events and when
it hits an event it can't handle, it punts (breaking the loop so you
don't process the other events after it - which is a bug), and the
code that eventually handles it is just catching all Exception and
tracing them out assuming they are really bad.

At this point, as a non-neutron person, i.e. not well versed in the
operations of neutron or how to debug it in great detail, I assume
something is bad here but I don't really know - and the logs are so
full of noise that I can't distinguish real failures.

I don't mean to pick on this particular change, but it's a good
example of a recent thing.

I'd like to know if this is all known issue or WIP type stuff. I've
complained about excessively noisey neutron logs in channel before
and I'm usually told that they are either necessary (for whatever
reason) or that rather than complain about the verbosity, I should
fix the race that is causing it - which is not 

Re: [Openstack-operators] [Scale][Performance] / compute_nodes ratio experience

[Rochelle Grober]

Sorry this doesn't thread properly, but cut and pasted out of the digest...



> As providing OpenStack community with understandable recommendations

> and instructions on performant OpenStack cloud deployments is part of

> Performance Team mission, I'm kindly asking you to share your

> experience on safe cloud deployment ratio between various types of

> nodes you're having right now and the possible issues you observed (as

> an example: discussed GoDaddy's cloud is having 3 conductor boxes vs

> 250 computes in the cell, and there was an opinion that it's simply

> not enough and that's it).



That was my opinion, and it was based on an apparently incorrect assumption 
that they had a lot of things coming and going on their cloud. I think they've 
demonstrated at this point that (other issues

aside) three is enough for them, given their environment, workload, and 
configuration.



This information is great for building rules of thumb, so to speak.  GoDaddy 
has an example configuraton that is adequate for low frequency 
construct/destruct (low number of vm create/destroy) cloud architectures.  This 
provides a lower bounds and might be representative of a lot of enterprise 
cloud deployments.



The problem with coming up with any sort of metric that will apply to everyone 
is that it's highly variable. If you have 250 compute nodes and never create or 
destroy any instances, you'll be able to get away with

*many* fewer conductors than if you have a very active cloud. Similarly, during 
a live upgrade (or following any upgrade where we do some online migration of 
data), your conductor load will be higher than normal. Of course, 4-core and 
96-core conductor nodes aren't equal either.



And here we have another rule of thumb, but no numbers put to it yet.  If you 
have a low frequency construct/destruct cloud model, you will need to 
temporarily increase your number of conductors by {x amount OR x%} when 
performing OpenStack live upgrades.



So, by all means, we should gather information on what people are doing 
successfully, but keep in mind that it depends *a lot* on what sort of 
workloads the cloud is supporting.



Right, but we can start applying fuzzy logic (the human kind, not machine) and 
get a better understanding of working configurations and *why* they work, then 
start examining where the transition states between configurations are.   You 
need data before you can create information ;-)



--Rocky


--Dan
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [all] [api] [log] Erring is Caring

[Rochelle Grober]

Hi all,

The first draft of the spec for Log Message Error Codes (from the Log Working 
Group) is out for review:

https://review.openstack.org/#/c/172552/

Please comment, and please look for the way forward so that the different 
places, ways and reasons errors get reported provide consistency across the 
projects the make up the Openstack ecosystem.  Consistency across uses will 
speed problem solving and will provide a common language across the diversity 
of users of the OpenStack code and environments.

This cross project spec is focused on just a part of the Log Message header, 
but it is the start of where log messages need to go.  It dovetails in with 
developer focused API errors, but is aimed at the log files the operators rely 
on to keep their clouds running. Over the next couple of days, I will also 
specifically add reviewers to the list if you haven't already commented on the 
spec;-).

Thanks for your patience waiting for this.  It *is* a work in progress, but I 
think the meat is there for discussion.

Also, please look at and comment on: Return Request ID to caller
https://review.openstack.org/#/c/156508/
as this is also critical to get right for logging and developer efforts.

--Rocky


 

-Original Message-
From: Everett Toews [mailto:everett.to...@rackspace.com] 
Sent: Tuesday, March 31, 2015 14:36
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [all] [api] Erring is Caring

Hi All,

An API Working Group Guideline for Errors

https://review.openstack.org/#/c/167793/

Errors are a crucial part of the developer experience when using an API. As 
developers learn the API they inevitably run into errors. The quality and 
consistency of the error messages returned to them will play a large part in 
how quickly they can learn the API, how they can be more effective with the 
API, and how much they enjoy using the API.

We need consistency across all services for the error format returned in the 
response body.


The Way Forward

I did a bit of research into the current state of consistency in errors across 
OpenStack services [1]. Since no services seem to respond with a top-level 
errors key, it's possible that they could just include this key in the 
response body along with their usual response and the 2 can live side-by-side 
for some deprecation period. Hopefully those services with unstructured errors 
should okay with adding some structure. That said, the current error formats 
aren't documented anywhere that I've seen so this all feels fair game anyway.

How this would get implemented in code is up to you. It could eventually be 
implemented in all projects individually or perhaps a Oslo utility is called 
for. However, this discussion is not about the implementation. This discussion 
is about the error format.


The Review

I've explicitly added all of the API WG and Logging WG CPLs as reviewers to 
that patch but feedback from all is welcome. You can find a more readable 
version of patch set 4 at [2]. I see the id and code fields as the 
connection point to what the logging working group is doing.


Thanks,
Everett


[1] https://wiki.openstack.org/wiki/API_Working_Group/Current_Design/Errors
[2] 
http://docs-draft.openstack.org/93/167793/4/check/gate-api-wg-docs/e2f5b6e//doc/build/html/guidelines/errors.html


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [all][log][api] Reminder: Log Working Group meeting 3/4/2015 -- Wednesday 20:00UTC

[Rochelle Grober]

Log Working Group is a cross project (horizontal) and community group that is 
working to rationalize log messages and logging practices across the OpenStack 
ecosystem.  This has been identified as a big concern within the user 
communities.  Anyone with an interest in improving logs, logging and 
documentation around this is encouraged to join in the discussion.

First meeting:
3/4/2015 20:00UTC in #openstack-meeting-4

https://wiki.openstack.org/wiki/Meetings/log-wg 

We'll discuss:


* Intro/Getting organized/ Agenda
* Error Codes (from the mailing list, and from Kilo Summit)
* Action Items for Ops Midcycle meetup
* Priorities
* General Discussion

https://wiki.openstack.org/wiki/LogWorkingGroup

Thanks,
--Rocky Grober

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [Product] [all][log] Openstack HTTP error codes

[Rochelle Grober]

What I see in this conversation is that we are talking about multiple different 
user classes.

Infra-operator needs as much info as possible, so if it is a vendor driver that 
is erring out, the dev-ops can see it in the log.

Tenant-operator is a totally different class of user.  These guys need VM based 
logs and virtual network based logs, etc., but should never see as far under 
the covers as the infra-ops *has* to see.

So, sounds like a security policy issue of what makes it to tenant logs and 
what stays in the data center thing.  

There are *lots* of logs that are being generated.  It sounds like we need 
standards on what goes into which logs along with error codes, 
logging/reporting levels, criticality, etc.

--Rocky

(bcc'ing the ops list so they can join this discussion, here)

-Original Message-
From: Sean Dague [mailto:s...@dague.net] 
Sent: Monday, February 02, 2015 8:19 AM
To: openstack-...@lists.openstack.org
Subject: Re: [openstack-dev] [Product] [all][log] Openstack HTTP error codes

On 02/01/2015 06:20 PM, Morgan Fainberg wrote:
 Putting on my sorry-but-it-is-my-job-to-get-in-your-way hat (aka security), 
 let's be careful how generous we are with the user and data we hand back. It 
 should give enough information to be useful but no more. I don't want to see 
 us opened to weird attack vectors because we're exposing internal state too 
 generously. 
 
 In short let's aim for a slow roll of extra info in, and evaluate each data 
 point we expose (about a failure) before we do so. Knowing more about a 
 failure is important for our users. Allowing easy access to information that 
 could be used to attack / increase impact of a DOS could be bad. 
 
 I think we can do it but it is important to not swing the pendulum too far 
 the other direction too fast (give too much info all of a sudden). 

Security by cloud obscurity?

I agree we should evaluate information sharing with security in mind.
However, the black boxing level we have today is bad for OpenStack. At a
certain point once you've added so many belts and suspenders, you can no
longer walk normally any more.

Anyway, lets stop having this discussion in abstract and actually just
evaluate the cases in question that come up.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [all][log] Openstack HTTP error codes

[Rochelle Grober]

Hi folks!

Changed the tags a bit because this is a discussion for all projects and 
dovetails with logging rationalization/standards/

At the Paris summit, we had a number of session on logging that kept circling 
back to Error Codes.  But, these codes would not be http codes, rather, as 
others have pointed out, codes related to the calling entities and referring 
entities and the actions that happened or didn’t.  Format suggestions were 
gathered from the Operators and from some senior developers.  The Logging 
Working Group is planning to put forth a spec for discussion on formats and 
standards before the Ops mid-cycle meetup.

Working from a Glance proposal on error codes:  
https://review.openstack.org/#/c/127482/ and discussions with operators and 
devs, we have a strawman to propose.  We also have a number of requirements 
from Ops and some Devs.

Here is the basic idea:

Code for logs would have four segments:
Project Vendor/Component  Error Catalog 
number Criticality
Def [A-Z] [A-Z] [A-Z]   -  [{0-9}|{A-Z}][A-Z] - 
[-]-   [0-9]
Ex.  CIN-   NA- 
   0001- 2
Cinder   NetApp 
   driver error no  Criticality
Ex.  GLA-  0A-  
   0051   3
Glance  Api 
error no   Criticality
Three letters for project,  Either a two letter vendor code or a number and 
letter for 0+letter for internal component of project (like API=0A, Controller 
=0C, etc),  four digit error number which could be subsetted for even finer 
granularity, and a criticality number.

This is for logging purposes and tracking down root cause faster for operators, 
but if an error is generated, why can the same codes be used internally for the 
code as externally for the logs?  This also allows for a unique message to be 
associated with the error code that is more descriptive and that can be pre 
translated.  Again, for logging purposes, the error code would not be part of 
the message payload, but part of the headers.  Referrer IDs and other info 
would still be expected in the payload of the message and could include 
instance ids/names, NICs or VIFs, etc.  The message headers is code in Oslo.log 
and when using the Oslo.log library, will be easy to use.

Since this discussion came up, I thought I needed to get this info out to folks 
and advertise that anyone will be able to comment on the spec to drive it to 
agreement.  I will be  advertising it here and on Ops and Product-WG mailing 
lists.  I’d also like to invite anyone who want to participate in discussions 
to join them.  We’ll be starting a bi-weekly or weekly IRC meeting (also 
announced in the stated MLs) in February.

And please realize that other than Oslo.log, the changes to make the errors 
more useable will be almost entirely community created standards with community 
created tools to help enforce them.  None of which exist yet, FYI.

--RockyG






From: Eugeniya Kudryashova [mailto:ekudryash...@mirantis.com]
Sent: Thursday, January 29, 2015 8:33 AM
To: openstack-...@lists.openstack.org
Subject: [openstack-dev] [api][nova] Openstack HTTP error codes


Hi, all



Openstack APIs interact with each other and external systems partially by 
passing of HTTP errors. The only valuable difference between types of 
exceptions is HTTP-codes, but current codes are generalized, so external system 
can’t distinguish what actually happened.


As an example two different failures below differs only by error message:


request:

POST /v2/790f5693e97a40d38c4d5bfdc45acb09/servers HTTP/1.1

Host: 192.168.122.195:8774http://192.168.122.195:8774

X-Auth-Project-Id: demo

Accept-Encoding: gzip, deflate, compress

Content-Length: 189

Accept: application/json

User-Agent: python-novaclient

X-Auth-Token: 2cfeb9283d784cfba694f3122ef413bf

Content-Type: application/json


{server: {name: demo, imageRef: 171c9d7d-3912-4547-b2a5-ea157eb08622, 
key_name: test, flavorRef: 42, max_count: 1, min_count: 1, 
security_groups: [{name: bar}]}}

response:

HTTP/1.1 400 Bad Request

Content-Length: 118

Content-Type: application/json; charset=UTF-8

X-Compute-Request-Id: req-a995e1fc-7ea4-4305-a7ae-c569169936c0

Date: Fri, 23 Jan 2015 10:43:33 GMT


{badRequest: {message: Security group bar not found for project 
790f5693e97a40d38c4d5bfdc45acb09., code: 400}}


and


request:

POST /v2/790f5693e97a40d38c4d5bfdc45acb09/servers HTTP/1.1

Host: 192.168.122.195:8774http://192.168.122.195:8774

X-Auth-Project-Id: demo

Accept-Encoding: gzip, deflate, compress

[Openstack-operators] [Cross Project][Log][Profiler] Specs are in review and need constructive feedback soonest

[Rochelle Grober]

The Cross Project team meeting discussed the following specs today:



 * Discuss openstack-spec: log guidelines [1]

 * Discuss openstack-spec: OSProfiler [2]

 * Open discussion  announcements



 [1] https://review.openstack.org/#/c/132552/

 [2] https://review.openstack.org/#/c/134839/

For [1], the log guidelines spec is likely to reach approval for merge by the 
next meeting on 1/13.  The spec has not been discussed on any mailing list for 
a while, so might be flying a bit under the radar.  This log guidelines spec is 
high level and states a number of best practices for developers when writing 
the logging chunks of their project code.  It also codifies the set of log 
levels: Error; Warning; Info; etc. and removes the AUDIT level.

Operators and others who hold an interest in this:  *PLEASE REVIEW [1] AND 
COMMENT ON THE REVIEW BY 1/13*  Lack of comments, as always, implies approval.  
If there is only light discussion of the spec, it will be approved.  If there 
is lively discussion, the discussion will move towards consensus before final 
approval.

The spec will then be used to expand the code review checklist so that its 
contents are enforced.

For [2], the OSProfiler spec is not as far along, but if you as an Ops (or 
interested developer) sees an issue or has a suggestion for improvement, you 
should also comment on this review.  The profiler already has code in use and 
is considered useful by a number of developers and some DevOps, which is why 
this spec and its review is being highlighted for this audience.

Ops, if you've never commented on a review before, don't let that hurdle stop 
you.  Check  out this page:  
http://docs.openstack.org/infra/manual/developers.html#code-review

It tells you how to post a review.  It doesn't tell you that you need get an 
account on Launchpad.net, but that info is here:
http://docs.openstack.org/infra/manual/developers.html#account-setup . I'm not 
sure if you have to be a member of OpenStack for commenting, but I'm sure folks 
will chime in if you do;-)

If working through these docs is hard for you, go out to IRC and folks will 
help get you started so you *can* comment on the reviews.

Also, if there is a section of the spec that you feel needs broader attention, 
start a discussion of it on the mailing list.  Please reference the review and 
the part of the spec you want to discuss, but put it out there on the mailing 
list so the issues/suggestions you have get addressed.

--Rocky


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators