[openstack-dev] [oslo] proposing Ken Giusti for oslo-core

[Doug Hellmann]

Ken has been managing oslo.messaging for ages now but his participation
in the team has gone far beyond that single library. He regularly
attends meetings, including the PTG, and has provided input into several
of our team decisions recently.

I think it's time we make him a full member of the oslo-core group.

Please respond here with a +1 or -1 to indicate your opinion.

Thanks,
Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Robert Collins's message of 2018-03-26 10:46:22 +1300:
> On 26 March 2018 at 09:08, Doug Hellmann <d...@doughellmann.com> wrote:
> > Excerpts from Doug Hellmann's message of 2018-03-25 16:04:11 -0400:
> >> A few of the jobs failed because the dependencies were wrong. In a few
> >> cases I was able to figure out what was wrong, but I can use some help
> >> from project teams more familiar with the code bases to debug the
> >> remaining failures.
> >
> > If you need to raise the lower bounds in a requirements file, please
> > update that file as well as lower-constraints.txt in the patch. You may
> > need to add a Depends-On for https://review.openstack.org/555402 in
> > order to have a version specifier that is different from the value in
> > the global requirements list.
> 
> Nice stuff; I'm so glad to see this evolution happening.
> 
> -Rob
> 

Thanks for laying such a firm foundation for us, Robert!

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-03-25 16:04:11 -0400:
> A few of the jobs failed because the dependencies were wrong. In a few
> cases I was able to figure out what was wrong, but I can use some help
> from project teams more familiar with the code bases to debug the
> remaining failures.

If you need to raise the lower bounds in a requirements file, please
update that file as well as lower-constraints.txt in the patch. You may
need to add a Depends-On for https://review.openstack.org/555402 in
order to have a version specifier that is different from the value in
the global requirements list.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-03-22 16:16:06 -0400:
> Excerpts from Doug Hellmann's message of 2018-03-21 16:02:06 -0400:
> > Excerpts from Doug Hellmann's message of 2018-03-15 07:03:11 -0400:
> > > 
> > > TL;DR
> > > -
> > > 
> > > Let's stop copying exact dependency specifications into all our
> > > projects to allow them to reflect the actual versions of things
> > > they depend on. The constraints system in pip makes this change
> > > safe. We still need to maintain some level of compatibility, so the
> > > existing requirements-check job (run for changes to requirements.txt
> > > within each repo) will change a bit rather than going away completely.
> > > We can enable unit test jobs to verify the lower constraint settings
> > > at the same time that we're doing the other work.
> > 
> > The new job definition is in https://review.openstack.org/555034 and I
> > have updated the oslo.config patch I mentioned before to use the new job
> > instead of one defined in the oslo.config repo (see
> > https://review.openstack.org/550603).
> > 
> > I'll wait for that job patch to be reviewed and approved before I start
> > adding the job to a bunch of other repositories.
> > 
> > Doug
> 
> The job definition for openstack-tox-lower-constraints [1] was approved
> today (thanks AJaegar and pabelenger).
> 
> I have started proposing the patches to add that job to the repos listed
> in openstack/requirements/projects.txt using the topic
> "requirements-stop-syncing" [2]. I hope to have the rest of those
> proposed by the end of the day tomorrow, but since they have to run in
> batches I don't know if that will be possible.
> 
> The patch to remove the update proposal job is ready for review [3].
> 
> As is the patch to allow project requirements to diverge by changing the
> rules in the requirements-check job [4].
> 
> We ran into a snag with a few of the jobs for projects that rely on
> having service projects installed. There have been a couple of threads
> about that recently, but Monty has promised to start another one to
> provide all of the necessary context so we can fix the issues and move
> ahead.
> 
> Doug
> 

All of the patches to define the lower-constraints test jobs have been
proposed [1], and many have already been approved and merged (thank you
for your quick reviews).

A few of the jobs are failing because the projects depend on installing
some other service from source. We will work out what to do with those
when we solve that problem in a more general way.

A few of the jobs failed because the dependencies were wrong. In a few
cases I was able to figure out what was wrong, but I can use some help
from project teams more familiar with the code bases to debug the
remaining failures.

In a few cases projects didn't have python 3 unit test jobs, so I
configured the new job to use python 2. Teams should add a step to their
python 3 migration plan to update the version of python used in the new
job, when that is possible.

I believe we are now ready to proceed with updating the
requirements-check job to relax the rules about which changes are
allowed [2].

Doug

[1] https://review.openstack.org/#/q/topic:requirements-stop-syncing+status:open
[2] https://review.openstack.org/555402

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Doug Hellmann]

Excerpts from Stephen Finucane's message of 2018-03-23 17:25:42 +:
> On Fri, 2018-03-23 at 12:23 -0400, Doug Hellmann wrote:
> > Excerpts from Monty Taylor's message of 2018-03-23 08:03:22 -0500:
> > > On 03/22/2018 05:43 AM, Stephen Finucane wrote:
> > > > 
> > > > That's unfortunate. What we really need is a migration path from the
> > > > 'pbr' way of doing things to something else. I see three possible
> > > > avenues at this point in time:
> > > > 
> > > > 1. Start using 'sphinx.ext.autosummary'. Apparently this can do 
> > > > similar
> > > >things to 'sphinx-apidoc' but it takes the form of an extension.
> > > >From my brief experiments, the output generated from this is
> > > >radically different and far less comprehensive than what 'sphinx-
> > > >apidoc' generates. However, it supports templating so we could
> > > >probably configure this somehow and add our own special directive
> > > >somewhere like 'openstackdocstheme'
> > > > 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time 
> > > > back
> > > >against upstream Sphinx [1]. This essentially does what the PBR
> > > >extension does but moves configuration into 'conf.py'. However, 
> > > > this
> > > >is currently held up as I can't adequately explain the 
> > > > differences
> > > >between this and 'sphinx.ext.autosummary' (there's definite 
> > > > overlap
> > > >but I don't understand 'autosummary' well enough to compare 
> > > > them).
> > > > 3. Modify the upstream jobs that detect the pbr integration and have
> > > >them run 'sphinx-apidoc' before 'sphinx-build'. This is the least
> > > >technically appealing approach as it still leaves us unable to 
> > > > build
> > > >stuff locally and adds yet more "magic" to the gate, but it does 
> > > > let
> > > >us progress.
> > > 
> > > I'd suggest a #4:
> > > 
> > > Take the sphinx.ext.apidoc extension and make it a standalone extension 
> > > people can add to doc/requirements.txt and conf.py. That way we don't 
> > > have to convince the sphinx folks to land it.
> > > 
> > > I'd been thinking for a while "we should just write a sphinx extension 
> > > with the pbr logic in it" - but hadn't gotten around to doing anything 
> > > about it. If you've already written that extension - I think we're in 
> > > potentially great shape!
> > 
> > That also has the benefit that we don't have to wait for a new sphinx
> > release to start using it.
> 
> I can do this. Where will it live? pbr? openstackdocstheme? Somewhere
> else?
> 
> Stephen
> 

I think the idea is to make a new thing. If you put it in the
sphinx-contrib org on github it will be easy for other people to
contribute and use it.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Doug Hellmann]

Excerpts from Monty Taylor's message of 2018-03-23 08:03:22 -0500:
> On 03/22/2018 05:43 AM, Stephen Finucane wrote:
> > 
> > That's unfortunate. What we really need is a migration path from the
> > 'pbr' way of doing things to something else. I see three possible
> > avenues at this point in time:
> > 
> > 1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar
> >things to 'sphinx-apidoc' but it takes the form of an extension.
> >From my brief experiments, the output generated from this is
> >radically different and far less comprehensive than what 'sphinx-
> >apidoc' generates. However, it supports templating so we could
> >probably configure this somehow and add our own special directive
> >somewhere like 'openstackdocstheme'
> > 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back
> >against upstream Sphinx [1]. This essentially does what the PBR
> >extension does but moves configuration into 'conf.py'. However, this
> >is currently held up as I can't adequately explain the differences
> >between this and 'sphinx.ext.autosummary' (there's definite overlap
> >but I don't understand 'autosummary' well enough to compare them).
> > 3. Modify the upstream jobs that detect the pbr integration and have
> >them run 'sphinx-apidoc' before 'sphinx-build'. This is the least
> >technically appealing approach as it still leaves us unable to build
> >stuff locally and adds yet more "magic" to the gate, but it does let
> >us progress.
> 
> I'd suggest a #4:
> 
> Take the sphinx.ext.apidoc extension and make it a standalone extension 
> people can add to doc/requirements.txt and conf.py. That way we don't 
> have to convince the sphinx folks to land it.
> 
> I'd been thinking for a while "we should just write a sphinx extension 
> with the pbr logic in it" - but hadn't gotten around to doing anything 
> about it. If you've already written that extension - I think we're in 
> potentially great shape!

That also has the benefit that we don't have to wait for a new sphinx
release to start using it.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-03-21 16:02:06 -0400:
> Excerpts from Doug Hellmann's message of 2018-03-15 07:03:11 -0400:
> > 
> > TL;DR
> > -
> > 
> > Let's stop copying exact dependency specifications into all our
> > projects to allow them to reflect the actual versions of things
> > they depend on. The constraints system in pip makes this change
> > safe. We still need to maintain some level of compatibility, so the
> > existing requirements-check job (run for changes to requirements.txt
> > within each repo) will change a bit rather than going away completely.
> > We can enable unit test jobs to verify the lower constraint settings
> > at the same time that we're doing the other work.
> 
> The new job definition is in https://review.openstack.org/555034 and I
> have updated the oslo.config patch I mentioned before to use the new job
> instead of one defined in the oslo.config repo (see
> https://review.openstack.org/550603).
> 
> I'll wait for that job patch to be reviewed and approved before I start
> adding the job to a bunch of other repositories.
> 
> Doug

The job definition for openstack-tox-lower-constraints [1] was approved
today (thanks AJaegar and pabelenger).

I have started proposing the patches to add that job to the repos listed
in openstack/requirements/projects.txt using the topic
"requirements-stop-syncing" [2]. I hope to have the rest of those
proposed by the end of the day tomorrow, but since they have to run in
batches I don't know if that will be possible.

The patch to remove the update proposal job is ready for review [3].

As is the patch to allow project requirements to diverge by changing the
rules in the requirements-check job [4].

We ran into a snag with a few of the jobs for projects that rely on
having service projects installed. There have been a couple of threads
about that recently, but Monty has promised to start another one to
provide all of the necessary context so we can fix the issues and move
ahead.

Doug

[1] https://review.openstack.org/555034
[2] 
https://review.openstack.org/#/q/topic:requirements-stop-syncing+(status:open+OR+status:merged)
[3] https://review.openstack.org/555426
[4] https://review.openstack.org/555402

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Adding "not docs" banner to specs website?

[Doug Hellmann]

Excerpts from Rochelle Grober's message of 2018-03-22 01:05:42 +:

> It could be *really* useful if you could include the date (month/year
> would be good enough)of the last significant patch (not including
> the reformat to Openstackdocstheme).  That could give folks a great
> stick in the mud for what "past" is for the spec.  It might even
> incent some to see if there are newer, conflicting or enhancing
> specs or docs to reference.
> 
> --Eoxky

The docs theme includes this information just below the title on
each page. See https://docs.openstack.org/reno/latest/ and look for
"UPDATED" for an example.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-03-15 07:03:11 -0400:
> 
> TL;DR
> -
> 
> Let's stop copying exact dependency specifications into all our
> projects to allow them to reflect the actual versions of things
> they depend on. The constraints system in pip makes this change
> safe. We still need to maintain some level of compatibility, so the
> existing requirements-check job (run for changes to requirements.txt
> within each repo) will change a bit rather than going away completely.
> We can enable unit test jobs to verify the lower constraint settings
> at the same time that we're doing the other work.

The new job definition is in https://review.openstack.org/555034 and I
have updated the oslo.config patch I mentioned before to use the new job
instead of one defined in the oslo.config repo (see
https://review.openstack.org/550603).

I'll wait for that job patch to be reviewed and approved before I start
adding the job to a bunch of other repositories.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Adding "not docs" banner to specs website?

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2018-03-19 20:17:37 +:
> On 2018-03-19 16:09:14 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > We want them all to use the openstackdocstheme so you could look
> > into creating a "subclass" of that one with the extra content in
> > the header, then ensure all of the specs repos use it.  We would
> > have to land a small patch to trigger a rebuild, but the patch
> > switching them from oslosphinx to openstackdocstheme would serve
> > for that and a small change to the readme or another file would do it
> > for any that are already using the theme.
> 
> Seems like a reasonable incentive for some needed cleanup.

And if I wasn't clear, we would want to put that subclass in the
openstackdocstheme repo so we can easily keep the styles up to date
over time.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Adding "not docs" banner to specs website?

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2018-03-19 19:06:38 +:
> On Mon, Mar 19, 2018 at 3:46 PM, Jeremy Stanley  wrote:
> 
> > On 2018-03-19 14:57:58 + (+), Jim Rollenhagen wrote:
> > [...]
> > > What do folks think about a banner at the top of the specs website
> > > (or each individual spec) that points this out? I'm happy to do
> > > the work if we agree it's a good thing to do.
> > [...]
> >
> > Sounds good in principle, but the execution may take a bit of work.
> > Specs sites are independently generated Sphinx documents stored in
> > different repositories managed by different teams, and don't
> > necessarily share a common theme or configuration.
> 
> 
> Huh, I had totally thought there was a theme for the specs site that
> most/all projects use. I may try to accomplish this anyway, but will likely
> be more work that I thought. I'll poke around at options (small sphinx
> plugin, etc).

We want them all to use the openstackdocstheme so you could look
into creating a "subclass" of that one with the extra content in
the header, then ensure all of the specs repos use it.  We would
have to land a small patch to trigger a rebuild, but the patch
switching them from oslosphinx to openstackdocstheme would serve
for that and a small change to the readme or another file would do it
for any that are already using the theme.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ALL][PTLs] Community Goals for Rocky: Toggle the debug option at runtime

[Doug Hellmann]

> On Mar 16, 2018, at 6:08 PM, Mohammed Naser  wrote:
> 
>> On Fri, Mar 16, 2018 at 5:34 PM, Jeremy Stanley  wrote:
>>> On 2018-03-16 21:22:51 + (+), Jim Rollenhagen wrote:
>>> [...]
>>> It seems mod_wsgi doesn't want python applications catching SIGHUP,
>>> as Apache expects to be able to catch that. By default, it even ensures
>>> signal handlers do not get registered.[0]
>> [...]
>>> Given we just had a goal to make all API services runnable as a WSGI
>>> application, it seems wrong to enable mutable config for API services.
>>> It's a super useful thing though, so I'd love to figure out a way we can do
>>> it.
>> [...]
>> 
>> Given these are API services, can the APIs grow a (hopefully
>> standardized) method to trigger this in lieu of signal handling? Or
>> if the authentication requirements are too much, Zuul and friends
>> have grown RPC sockets which can be used to inject these sorts of
>> low-level commands over localhost to their service daemons (or could
>> probably also do similar things over UNIX sockets if you don't want
>> listeners on the loopback interface).
> 
> Throwing an idea out there, but maybe listening to file modification
> events using something like inotify could be a possibility?

Both of those are good ideas. I believe adding those things to oslo.service 
would make them available to all applications. 

> 
>> --
>> Jeremy Stanley
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][neutron][arista] Release of openstack/networking-arista failed

[Doug Hellmann]

This Arista release is failing because the packaging job can't run "tox
-e venv" because neutron is listed in the requirements.txt for the
Arista code and in the constraints file.

Excerpts from zuul's message of 2018-03-16 19:50:48 +:
> Build failed.
> 
> - release-openstack-python 
> http://logs.openstack.org/25/25ac528d6771d3440fac428294194e08939fb5aa/release/release-openstack-python/e550904/
>  : FAILURE in 3m 30s
> - announce-release announce-release : SKIPPED
> - propose-update-constraints propose-update-constraints : SKIPPED
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ALL][PTLs] Community Goals for Rocky: Toggle the debug option at runtime

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2018-03-16 13:43:00 +:
> On 2018-03-16 08:34:28 -0500 (-0500), Sean McGinnis wrote:
> > On Mar 16, 2018, at 04:02, Jean-Philippe Evrard  
> > wrote:
> > 
> > > For OpenStack-Ansible, we don't need to do anything for that
> > > community goal.  I am not sure how we can remove our name from
> > > the storyboard, so I just inform you here.
> > 
> > I believe you can just mark the task as done if there is no
> > additional work required.
> 
> Yeah, either "merged" or "invalid" states should work. I'd lean
> toward suggesting "invalid" in this case since the task did not
> require any changes merged to your source code.

Yes, we've been using "invalid" to indicate that no work was needed.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Matthew Thode's message of 2018-03-15 18:36:50 -0500:
> On 18-03-15 19:29:37, Doug Hellmann wrote:
> > Excerpts from Matthew Thode's message of 2018-03-15 10:24:10 -0500:
> > > On 18-03-15 07:03:11, Doug Hellmann wrote:
> > > > What I Want to Do
> > > > -
> > > > 
> > > > 1. Update the requirements-check test job to change the check for
> > > >an exact match to be a check for compatibility with the
> > > >upper-constraints.txt value.
> > > > 
> > > >We would check the value for the dependency from 
> > > > upper-constraints.txt
> > > >against the range of allowed values in the project. If the
> > > >constraint version is compatible, the dependency range is OK.
> > > > 
> > > >This rule means that in order to change the dependency settings
> > > >for a project in a way that are incompatible with the constraint,
> > > >the constraint (and probably the global requirements list) would
> > > >have to be changed first in openstack/requirements. However, if
> > > >the change to the dependency is still compatible with the
> > > >constraint, no change would be needed in openstack/requirements.
> > > >For example, if the global list constraints a library to X.Y.Z
> > > >and a project lists X.Y.Z-2 as the minimum version but then needs
> > > >to raise that because it needs a feature in X.Y.Z-1, it can do
> > > >that with a single patch in-tree.
> > > > 
> > > 
> > > I think what may be better is for global-requirements to become a
> > > gathering place for projects that requirements watches to have their
> > > smallest constrainted installable set defined in.
> > > 
> > > Upper-constraints has a req of foo===2.0.3
> > > Project A has a req of foo>=1.0.0,!=1.6.0
> > > Project B has a req of foo>=1.4.0
> > > Global reqs would be updated with foo>=1.4.0,!=1.6.0
> > > Project C comes along and sets foo>=2.0.0
> > > Global reqs would be updated with foo>=2.0.0
> > > 
> > > This would make global-reqs descriptive rather than prescriptive for
> > > versioning and would represent the 'true' version constraints of
> > > openstack.
> > 
> > It sounds like you're suggesting syncing in the other direction, which
> > could be useful. I think we can proceed with what I've described and
> > consider the work to build what you describe as a separate project.
> > 
> 
> Yes, this would be a follow-on thing.
> 
> > > 
> > > >We also need to change requirements-check to look at the exclusions
> > > >to ensure they all appear in the global-requirements.txt list
> > > >(the local list needs to be a subset of the global list, but
> > > >does not have to match it exactly). We can't have one project
> > > >excluding a version that others do not, because we could then
> > > >end up with a conflict with the upper constraints list that could
> > > >wedge the gate as we had happen in the past.
> > > > 
> > > 
> > > How would this happen when using constraints?  A project is not allowed
> > > to have a requirement that masks a constriant (and would be verified via
> > > the requirements-check job).
> > 
> > If project A excludes version X before the constraint list is updated to
> > use it, and then project B starts trying to depend on version X, they
> > become incompatible.
> > 
> > We need to continue to manage our declarations of incompatible versions
> > to ensure that the constraints list is a good list of versions to test
> > everything under.
> > 
> > > There's a failure mode not covered, a project could add a mask (!=) to
> > > their requirements before we update constraints.  The project that was
> > > passing the requirements-check job would then become incompatable.  This
> > > means that the requirements-check would need to be run for each
> > > changeset to catch this as soon as it happens, instead of running only
> > > on requirements changes.
> > 
> > I'm not clear on what you're describing here, but it sounds like a
> > variation of the failure modes that would be prevented if we require
> > exclusions to exist in the global list before they could be added to the
> > local list.
> > 
> 
> Yes, that'd work (require exclusions to be global before lo

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Matthew Thode's message of 2018-03-15 10:24:10 -0500:
> On 18-03-15 07:03:11, Doug Hellmann wrote:
> > What I Want to Do
> > -
> > 
> > 1. Update the requirements-check test job to change the check for
> >an exact match to be a check for compatibility with the
> >upper-constraints.txt value.
> > 
> >We would check the value for the dependency from upper-constraints.txt
> >against the range of allowed values in the project. If the
> >constraint version is compatible, the dependency range is OK.
> > 
> >This rule means that in order to change the dependency settings
> >for a project in a way that are incompatible with the constraint,
> >the constraint (and probably the global requirements list) would
> >have to be changed first in openstack/requirements. However, if
> >the change to the dependency is still compatible with the
> >constraint, no change would be needed in openstack/requirements.
> >For example, if the global list constraints a library to X.Y.Z
> >and a project lists X.Y.Z-2 as the minimum version but then needs
> >to raise that because it needs a feature in X.Y.Z-1, it can do
> >that with a single patch in-tree.
> > 
> 
> I think what may be better is for global-requirements to become a
> gathering place for projects that requirements watches to have their
> smallest constrainted installable set defined in.
> 
> Upper-constraints has a req of foo===2.0.3
> Project A has a req of foo>=1.0.0,!=1.6.0
> Project B has a req of foo>=1.4.0
> Global reqs would be updated with foo>=1.4.0,!=1.6.0
> Project C comes along and sets foo>=2.0.0
> Global reqs would be updated with foo>=2.0.0
> 
> This would make global-reqs descriptive rather than prescriptive for
> versioning and would represent the 'true' version constraints of
> openstack.

It sounds like you're suggesting syncing in the other direction, which
could be useful. I think we can proceed with what I've described and
consider the work to build what you describe as a separate project.

> 
> >We also need to change requirements-check to look at the exclusions
> >to ensure they all appear in the global-requirements.txt list
> >(the local list needs to be a subset of the global list, but
> >does not have to match it exactly). We can't have one project
> >excluding a version that others do not, because we could then
> >end up with a conflict with the upper constraints list that could
> >wedge the gate as we had happen in the past.
> > 
> 
> How would this happen when using constraints?  A project is not allowed
> to have a requirement that masks a constriant (and would be verified via
> the requirements-check job).

If project A excludes version X before the constraint list is updated to
use it, and then project B starts trying to depend on version X, they
become incompatible.

We need to continue to manage our declarations of incompatible versions
to ensure that the constraints list is a good list of versions to test
everything under.

> There's a failure mode not covered, a project could add a mask (!=) to
> their requirements before we update constraints.  The project that was
> passing the requirements-check job would then become incompatable.  This
> means that the requirements-check would need to be run for each
> changeset to catch this as soon as it happens, instead of running only
> on requirements changes.

I'm not clear on what you're describing here, but it sounds like a
variation of the failure modes that would be prevented if we require
exclusions to exist in the global list before they could be added to the
local list.

> 
> >We also need to verify that projects do not cap dependencies for
> >the same reason. Caps prevent us from advancing to versions of
> >dependencies that are "too new" and possibly incompatible. We
> >can manage caps in the global requirements list, which would
> >cause that list to calculate the constraints correctly.
> > 
> >This change would immediately allow all projects currently
> >following the global requirements lists to specify different
> >lower bounds from that global list, as long as those lower bounds
> >still allow the dependencies to be co-installable. (The upper
> >bounds, managed through the upper-constraints.txt list, would
> >still be built by selecting the newest compatible version because
> >that is how pip's dependency resolver works.)
> > 
> > 2. We should stop syncing dependencies by turning off the
> >propose-update-requirements job enti

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2018-03-15 14:28:49 +:
> On 2018-03-15 07:03:11 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > 1. Update the requirements-check test job to change the check for
> >an exact match to be a check for compatibility with the
> >upper-constraints.txt value.
> [...]
> 
> I thought it might be possible to even just do away with this job
> entirely, but some cursory testing shows that if you supply a
> required versionspec which excludes your constrained version of the
> same package, you'll still get the constrained version installed
> even though you indicated it wasn't in your "supported" range. Might
> be a nice patch to work on upstream in pip, making it explicitly
> error on such a mismatch (and _then_ we might be able to stop
> bothering with this job).
> 
> >We also need to change requirements-check to look at the exclusions
> >to ensure they all appear in the global-requirements.txt list
> >(the local list needs to be a subset of the global list, but
> >does not have to match it exactly). We can't have one project
> >excluding a version that others do not, because we could then
> >end up with a conflict with the upper constraints list that could
> >wedge the gate as we had happen in the past.
> [...]
> 
> At first it seems like this wouldn't end up being necessary; as long
> as you're not setting an upper bound or excluding the constrained
> version, there shouldn't be a coinstallability problem right? Though

That second case is what this prevents. There's a race condition between
updating the requirements range (and exclusions) in a project tree and
updating the upper-constraints.txt list. The check forces those lists to
be updated in an order that avoids a case where the version in
constraints is not compatible with an app installed in an integration
test job.

> I suppose there are still a couple of potential pitfalls if we don't
> check exclusions: setting an exclusion for a future version which
> hasn't been released yet or is otherwise higher than the global
> upper constraint; situations where we need to roll back a constraint
> to an earlier version (e.g., we discover a bug in it) and some
> project has that earlier version excluded. So I suppose there is
> some merit to centrally coordinating these, making sure we can still
> pick sane constraints which work for all projects (mental
> exercise: do we also need to build a tool which can make sure that
> proposed exclusions don't eliminate all possible version numbers?).

Yes, those are all good failure cases that this prevents, too.

> >As the minimum
> >versions of dependencies diverge within projects, there will no
> >longer *be* a real global set of minimum values. Tracking a list of
> >"highest minimums", would either require rebuilding the list from the
> >settings in all projects, or requiring two patches to change the
> >minimum version of a dependency within a project.
> [...]
> 
> It's also been suggested in the past that package maintainers for
> some distributions relied on the ranges in our global requirements
> list to determine what the minimum acceptable version of a
> dependency is so they know whether/when it needs updating (fairly
> critical when you consider that within a given distro some
> dependencies may be shared by entirely unrelated software outside
> our ecosystem and may not be compatible with new versions as soon as
> we are). On the other hand, we never actually _test_ our lower
> bounds, so this was to some extent a convenient fiction anyway.

The lack of testing is an issue, but the tight coupling of those
lower bounds is a bigger problem to me. I know that distros don't
necessarily package exactly what we have in the upper-constraints.txt
list, but they're doing their own testing with those alternatives.

> 
> > 1. Set up a new tox environment called "lower-constraints" with
> >base-python set to "python3" and with the deps setting configured
> >to include a copy of the existing global lower constraints file
> >from the openstack/requirements repo.
> [...]
> 
> I didn't realize lower-constraints.txt already existed (looks like
> it got added a little over a week ago). Reviewing the log it seems

Yes, Dirk did that work.

> to have been updated based on individual projects' declared minimums
> so far which seems to make it a questionable starting point for a
> baseline. I suppose the assumption is that projects have been
> merging requirements proposals which bump their declared
> lower-bounds, though experience suggests that this doesn't happen
> consistently in projects receiving g-r update

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Matthew Thode's message of 2018-03-15 10:05:50 -0500:
> On 18-03-15 09:45:38, Doug Hellmann wrote:
> > Excerpts from Thierry Carrez's message of 2018-03-15 14:34:50 +0100:
> > > Doug Hellmann wrote:
> > > > [...]
> > > > TL;DR
> > > > -
> > > > 
> > > > Let's stop copying exact dependency specifications into all our
> > > > projects to allow them to reflect the actual versions of things
> > > > they depend on. The constraints system in pip makes this change
> > > > safe. We still need to maintain some level of compatibility, so the
> > > > existing requirements-check job (run for changes to requirements.txt
> > > > within each repo) will change a bit rather than going away completely.
> > > > We can enable unit test jobs to verify the lower constraint settings
> > > > at the same time that we're doing the other work.
> > > 
> > > Thanks for the very detailed plan, Doug. It all makes sense to me,
> > > although I have a precision question (see below).
> > > 
> > > > [...]
> > > >We also need to change requirements-check to look at the exclusions
> > > >to ensure they all appear in the global-requirements.txt list
> > > >(the local list needs to be a subset of the global list, but
> > > >does not have to match it exactly). We can't have one project
> > > >excluding a version that others do not, because we could then
> > > >end up with a conflict with the upper constraints list that could
> > > >wedge the gate as we had happen in the past.
> > > > [...]
> > > > 2. We should stop syncing dependencies by turning off the
> > > >propose-update-requirements job entirely.
> > > > 
> > > >Turning off the job will stop the bot from proposing more
> > > >dependency updates to projects.
> > > > [...]
> > > > After these 3 steps are done, the requirements team will continue
> > > > to maintain the global-requirements.txt and upper-constraints.txt
> > > > files, as before. Adding a new dependency to a project will still
> > > > involve a review step to add it to the global list so we can monitor
> > > > licensing, duplication, python 3 support, etc. But adjusting the
> > > > version numbers once that dependency is in the global list will be
> > > > easier.
> > > 
> > > How would you set up an exclusion in that new world order ? We used to
> > > add it to the global-requirements file and the bot would automatically
> > > sync it to various consuming projects.
> > > 
> > > Now since any exclusion needs to also appear on the global file, you
> > > would push it first in the global-requirements, then to the project
> > > itself, is that correct ? In the end the global-requirements file would
> > > only contain those exclusions, right ?
> > > 
> > 
> > The first step would need to be adding it to the global-requirements.txt
> > list. After that, it would depend on how picky we want to be. If the
> > upper-constraints.txt list is successfully updated to avoid the release,
> > we might not need anything in the project. If the project wants to
> > provide detailed guidance about compatibility, then they could add the
> > exclusion. For example, if a version of oslo.config breaks cinder but
> > not nova, we might only put the exclusion in global-requirements.txt and
> > the requirements.txt for cinder.
> > 
> 
> I wonder if we'd be able to have projects decide via a flag in their tox
> or zuul config if they'd like to opt into auto-updating exclusions only.
> 

We could just change the job that does the sync and use the existing
projects.txt file, couldn't we?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2018-03-15 14:34:50 +0100:
> Doug Hellmann wrote:
> > [...]
> > TL;DR
> > -
> > 
> > Let's stop copying exact dependency specifications into all our
> > projects to allow them to reflect the actual versions of things
> > they depend on. The constraints system in pip makes this change
> > safe. We still need to maintain some level of compatibility, so the
> > existing requirements-check job (run for changes to requirements.txt
> > within each repo) will change a bit rather than going away completely.
> > We can enable unit test jobs to verify the lower constraint settings
> > at the same time that we're doing the other work.
> 
> Thanks for the very detailed plan, Doug. It all makes sense to me,
> although I have a precision question (see below).
> 
> > [...]
> >We also need to change requirements-check to look at the exclusions
> >to ensure they all appear in the global-requirements.txt list
> >(the local list needs to be a subset of the global list, but
> >does not have to match it exactly). We can't have one project
> >excluding a version that others do not, because we could then
> >end up with a conflict with the upper constraints list that could
> >wedge the gate as we had happen in the past.
> > [...]
> > 2. We should stop syncing dependencies by turning off the
> >propose-update-requirements job entirely.
> > 
> >Turning off the job will stop the bot from proposing more
> >dependency updates to projects.
> > [...]
> > After these 3 steps are done, the requirements team will continue
> > to maintain the global-requirements.txt and upper-constraints.txt
> > files, as before. Adding a new dependency to a project will still
> > involve a review step to add it to the global list so we can monitor
> > licensing, duplication, python 3 support, etc. But adjusting the
> > version numbers once that dependency is in the global list will be
> > easier.
> 
> How would you set up an exclusion in that new world order ? We used to
> add it to the global-requirements file and the bot would automatically
> sync it to various consuming projects.
> 
> Now since any exclusion needs to also appear on the global file, you
> would push it first in the global-requirements, then to the project
> itself, is that correct ? In the end the global-requirements file would
> only contain those exclusions, right ?
> 

The first step would need to be adding it to the global-requirements.txt
list. After that, it would depend on how picky we want to be. If the
upper-constraints.txt list is successfully updated to avoid the release,
we might not need anything in the project. If the project wants to
provide detailed guidance about compatibility, then they could add the
exclusion. For example, if a version of oslo.config breaks cinder but
not nova, we might only put the exclusion in global-requirements.txt and
the requirements.txt for cinder.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Doug Hellmann]

Excerpts from Thomas Morin's message of 2018-03-15 10:15:38 +0100:
> Hi Doug,
> 
> Doug Hellmann, 2018-03-14 23:42:
> > We keep doing lots of infra-related work to make it "easy" to do
> >  when it comes to
> > managing dependencies.  There are three ways to address the issue
> > with horizon and neutron, and none of them involve adding features
> > to pbr.
> > 
> > 1. Things that are being used like libraries need to release like
> >libraries. Real releases. With appropriate version numbers. So
> >that other things that depend on them can express valid
> > dependencies.
> > 
> > 2. Extract the relevant code into libraries and release *those*.
> > 
> > 3. Things that are not stable enough to be treated as a library
> >shouldn't be used that way. Move the things that use the
> > application
> >code as library code back into the repo with the thing that they
> >are tied to but that we don't want to (or can't) treat like a
> >library.
> 
> What about the case where there is co-development of features across
> repos ? One specific case I have in mind is the Neutron stadium where

We do that all the time with the Oslo libraries. It's not as easy as
having everything in one repo, but we manage.

> we sometimes have features in neutron repo that are worked on as a pre-
> requisite for things that will be done in a neutron-* or networking-*
> project. Another is a case for instance where we need to add in project
> X a tempest test to validate the resolution of a bug for which the fix
> actually happened in project B (and where B is not a library).

If the tempest test can't live in B because it uses part of X, then I
think X and B are really one thing and you're doing more work than you
need to be doing to keep them in separate libraries.

> My intuition is that it is not illegitimate to expect this kind of
> development workflow to be feasible; but at the same time I read your
> suggestion above as meaning that it belongs to the real of "things we
> shouldn't be doing in the first place".  The only way I can reconcile

You read me correctly.

We install a bunch of components from source for integration tests
in devstack-gate because we want the final releases to work together.
But those things only interact via REST APIs, and don't import each
other.  The cases with neutron and horizon are different. Even the
*unit* tests of the add-ons require code from the "parent" app. That
indicates a level of coupling that is not being properly addressed by
the release model and code management practices for the parent apps.

> the two would be to conclude we should collapse all the module in
> neutron-*/networking-* into neutron, but doing that would have quite a
> lot of side effects (yes, this is an understatement).

That's not the only way to do it. The other way would be to properly
decompose the shared code into a library and then provide *stable
APIs* so code can be consumed by the add-on modules. That will make
evolving things a little more difficult because of the stability
requirement. So it's a trade off. I think the teams involved should
make that trade off (in one direction or another), instead of
building tools to continue to avoid dealing with it.

So let's start by examining the root of the problem: Why are the things
that need to import neutron/horizon not part of the neutron/horizon
repositories in the first place?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][requirements] a plan to stop syncing requirements into projects

[Doug Hellmann]

Back in Barcelona for the Ocata summit I presented a rough outline
of a plan for us to change the way we manage dependencies across
projects so that we can stop syncing them [1]. We've made some
progress, and I think it's time to finish the work so I'm volunteering
to take some of it up during Rocky. This email is meant to rehash
and update the proposal, and fill in some of the missing details.

[1] https://etherpad.openstack.org/p/ocata-requirements-notes

TL;DR
-

Let's stop copying exact dependency specifications into all our
projects to allow them to reflect the actual versions of things
they depend on. The constraints system in pip makes this change
safe. We still need to maintain some level of compatibility, so the
existing requirements-check job (run for changes to requirements.txt
within each repo) will change a bit rather than going away completely.
We can enable unit test jobs to verify the lower constraint settings
at the same time that we're doing the other work.

Some History


Back in the dark ages of OpenStack development we had a lot of
trouble keeping the dependencies of all of our various projects
configured so they were co-installable. Usually, but not always,
the problems were caused by caps or "exclusions" (version != X) on
dependencies in one project but not in another. Because pip's
dependency resolver does not take into account the versions of
dependencies needed by existing packages, it was quite easy to
install things in the "wrong" order and end up with incompatible
libraries so services wouldn't start or couldn't import plugins.

The first (working) solution to the problem was to develop a
dependency management system based the openstack/requirements
repository. This system and our policies required projects to copy
exactly the settings for all of their dependencies from a global
list managed by a team of reviewers (first the release team, and
later the requirements team). By copying exactly the same settings
into all projects we ensured that they were "co-installable" without
any dependency conflicts. Having a centralized list of dependencies
with a review team also gave us an opportunity to look for duplicates,
packages using incompatible licenses, and otherwise curate the list
of dependencies. More on that later.

Some time after we had the centralized dependency management system
in place, Robert Collins worked with the PyPA folks to add a feature
to pip to constrain installed versions of packages that are actually
installed, while still allowing a range of versions to be specified
in the dependency list. We were then able to to create a list of
"upper constraints" -- the highest, or newest, versions -- of all
of the packages we depend on and set up our test jobs to use that
list to control what is actually installed. This gives us the ability
to say that we need at least version X.Y.Z of a package and to force
the selection of X.Y+1.0 because we want to test with that version.

The constraint feature means that we no longer need to have all of
the dependency specifications match exactly, since we basically
force the installation of a specific version anyway. We've been
running with both constraints and requirements syncing enabled for
a while now, and I think we should stop syncing the settings to
allow projects to let their lower bounds (the minimum versions of
their dependencies) diverge.

That divergence is useful to folks creating packages for just some
of the services, especially when they are going to be deployed in
isolation where co-installability is not required. Skipping the
syncs will also mean we end up releasing fewer versions of stable
libraries, because we won't be raising the minimum supported versions
of their dependencies automatically. That second benefit is my
motivation for focusing on this right now.

Our Requirements


We have three primary requirements for managing the dependency list:

1. Maintain a list of co-installable versions of all of our
   dependencies.

2. Avoid breaking or deadlocking any of our gate jobs due to
   dependency conflicts.

3. Continue to review new dependencies for licensing, redundancy,
   etc.

I believe the upper-constraints.txt file in openstack/releases
satisfies the first two of these requirements. The third means we
need to continue to *have* a global requirements list, but we can
change how we manage it.

In addition to these hard requirements, it would be nice if we could
test the lower bounds of dependencies in projects to detect when a
project is using a feature of a newer version of a library than
their dependencies indicate. Although that is a bit orthogonal to
the syncing issue, I'm going to describe one way we could do that
because the original plan of keeping a global list of "lower
constraints" breaks our ability to stop syncing the same lower
bounds into all of the projects somewhat.

What I Want to Do
-

1. Update the requirements-check test job to change 

Re: [openstack-dev] [reno] moved to storyboard

[Doug Hellmann]

I was remiss in not thanking fungi for his help with the move, and
diablo_rojo for preparing the docs explaining the rest of the steps I
needed to take afterwards. Thank you both!

Excerpts from Kendall Nelson's message of 2018-03-14 22:55:15 +:
> Woot woot!
> 
> -Kendall (diablo_rojo)
> 
> On Wed, Mar 14, 2018 at 7:51 AM Doug Hellmann <d...@doughellmann.com> wrote:
> 
> > The bug tracker for reno has moved to storyboard:
> > https://storyboard.openstack.org/#!/project/933
> >
> > Doug
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Doug Hellmann]

Excerpts from Tony Breeds's message of 2018-03-15 11:59:00 +1100:
> On Thu, Mar 15, 2018 at 07:16:11AM +0900, Akihiro Motoki wrote:
> > The current version of proposed patches which drops tox_install.sh
> > works in our CI. Even if we have neutron>=12.0.0 (queens) or
> > horizon>=13.0.0 (queens), if we have "required-projects" in zuul v3
> > config, tox-sibling role ensures to install the latest master of
> > neutron/horizon. It is okay in our CI.
> > 
> > On the other hand, this change brings a couple of problems. I think it
> > is worth discussed broadly here.
> > 
> > (1) it makes difficult to run tests in local environment
> > We have only released version of neutron/horizon on PyPI. It means
> > PyPI version (i.e. queens) is installed when we run tox in our local
> > development. Most neutron stadium projects and horizon plugins depends
> > on the latest master. Test run in local environment will be broken. We
> > need to install the latest neutron/horizon manually. This confuses
> > most developers. We need to ensure that tox can run successfully in a
> > same manner in our CI and local environments.
> 
> This is an issue I agree and one we need to think about but it will be
> somewhat mitigated for local development by pbr siblings[1]
> 
> In the short term, developers can do something like:
> 
> for env in pep8,py35,py27 ; do
>tox -e $env --notest
>.tox/$env/bin/pip install -e /path/to/{horizon,neutron}
>tox -e $env
> done
> 
> Which is far from ideal but gives as a little breathing room to decide
> if we need to revert and try again in a while or persist with the plan
> as it stands.
> 
> pbr siblings wont fix all the issues we have and still makes consumption of
> neutron and horizon (and plugins / stadium projects) difficult outside
> of test.
>  
> > (2) neutron/horizon version in requirements.txt is confusing
> > In the cycle-with-milestone model, a new version of neutron/horizon
> > will be released only when a release is shipped.
> > The code depends on the latest master, but requirements.txt says it
> > depends on queens or later. It sounds confusing.
> 
> Yes we either need to create a new release-model or switch
> neutron/horizon to the cycle-with-intermediary model and encourage
> appropriate releases.  I'd really like to avoid publishing daily to pypi.

We keep doing lots of infra-related work to make it "easy" to do
things we shouldn't be doing in the first place when it comes to
managing dependencies.  There are three ways to address the issue
with horizon and neutron, and none of them involve adding features
to pbr.

1. Things that are being used like libraries need to release like
   libraries. Real releases. With appropriate version numbers. So
   that other things that depend on them can express valid dependencies.

2. Extract the relevant code into libraries and release *those*.

3. Things that are not stable enough to be treated as a library
   shouldn't be used that way. Move the things that use the application
   code as library code back into the repo with the thing that they
   are tied to but that we don't want to (or can't) treat like a
   library.

Let's stop making things hard on ourselves and start managing this
code properly.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [First Contact][SIG] [PTG] Summary of Discussions

[Doug Hellmann]

Excerpts from Jay S Bryant's message of 2018-03-14 10:38:37 -0500:
> 
> On 3/14/2018 10:04 AM, Petr Kovar wrote:
> > On Tue, 13 Mar 2018 18:57:24 -0500
> > Jay S Bryant  wrote:
> >
> >> Amy,
> >>
> >> The top level page for projects is referenced under documentation from
> >> here:  https://docs.openstack.org/queens/projects.html
> >>
> >> So, I think we have that one covered for people who are just looking for
> >> the top level documentation.
> > Yes, we have that covered. Just to clarify this a bit further, we also have
> > project lists like https://docs.openstack.org/queens/install/,
> > https://docs.openstack.org/queens/admin/ and
> > https://docs.openstack.org/queens/configuration/, what's missing is
> > https://docs.openstack.org/queens/contributor/.
> >
> > Cheers,
> > pk
> >
> Petr,
> 
> Do we need a contributor link per-release?  I thought in past 
> discussions that the contributor info should always go to latest and 
> that was why this is slightly different.
> 
> Jay

We can have a per-series page that lists all projects and links to their
"latest" contributor doc page.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [reno] moved to storyboard

[Doug Hellmann]

The bug tracker for reno has moved to storyboard:
https://storyboard.openstack.org/#!/project/933

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] [oslo] new unified limit library

[Doug Hellmann]

Excerpts from Lance Bragstad's message of 2018-03-12 11:45:28 -0500:
> I missed the document describing the process for this sort of thing [0].
> So I'm back tracking a bit to go through a more formal process.
> 
> [0]
> http://specs.openstack.org/openstack/oslo-specs/specs/policy/new-libraries.html
> 
> # Proposed new library oslo.limit
> 
> This is a proposal to create a new library dedicated to enabling more
> consistent quota and limit enforcement across OpenStack.
> 
> ## Proposed library mission
> 
> Enforcing quotas and limits across OpenStack has traditionally been a
> tough problem to solve. Determining enforcement requires quota knowledge
> from the service along with information about the project owning the
> resource. Up until the Queens release, quota calculation and enforcement
> has been left to the services to implement, forcing them to understand
> complexities of keystone project structure. During the Pike and Queens
> PTG, there were several productive discussions towards redesigning the
> current approach to quota enforcement. Because keystone is the authority
> of project structure, it makes sense to allow keystone to hold the
> association between a resource limit and a project. This means services
> still need to calculate quota and usage, but the problem should be
> easier for services to implement since developers shouldn't need to
> re-implement possible hierarchies of projects and their associated
> limits. Instead, we can offload some of that work to a common library
> for services to consume that handles enforcing quota calculation based
> on limits associated to projects in keystone. This proposal is to have a
> new library called oslo.limit that fills that need.
> 
> ## Consuming projects
> 
> The services consuming this work will be any service that currently
> implements a quota system, or plans to implement one. Since keystone
> already supports unified limits and association of limits to projects,
> the implementation for consuming projects is easier. instead of having
> to re-write that implementation, developers need to ensure quota
> calculation to passed to the oslo.limit library somewhere in the API's
> validation layer. The pattern described here is very similar to the
> pattern currently used by services that leverage oslo.policy for
> authorization decisions.
> 
> ## Alternative libraries
> 
> It looks like there was an existing library that attempted to solve some
> of these problems, called delimiter [1]. It looks like delimiter could
> be used to talk to keystone about quota enforcement, where as the
> existing approach with oslo.limit would be to use keystone directly.
> Someone more familiar with the library (harlowja?) can probably shed
> more light on it's intended uses (I couldn't find much documentation),
> but the presentation linked in a previous note was helpful.
> 
> [1] https://github.com/openstack/delimiter
> 
> ## Proposed adoption model/plan
> 
> The unified limit API [2] in keystone is currently marked as
> experimental, but the keystone team is actively collecting and
> addressing feedback that will result in stabilizing the API.
> Stabilization changes that effect the oslo.limit library will also be
> addressed before version 1.0.0 is released. From there, we can look to
> incorporate the library into various services that either have an
> existing quota implementation, or services that have a quota requirement
> but no implementation.
> 
> This should help us refine the interfaces between services and
> oslo.limit, while providing a facade to handle complexities of project
> hierarchies. This should enable adoption by simplifying the process and
> making it easier for quota to be implemented in a consistent way across
> services.
> 
> [2]
> https://docs.openstack.org/keystone/latest/admin/identity-unified-limits.html
> 
> ## Reviewer activity
> 
> At first thought, it makes sense to model the reviewer structure after
> the oslo.policy library, where the core team consists of people not only
> interested in limits and quota, but also people familiar with the
> keystone implementation of the unified limits API.
> 
> ## Implementation
> 
> ### Primary Authors:
> 
>   Lance Bragstad (lbrags...@gmail.com) lbragstad
>   You?
> 
> ### Other contributors:
> 
>   You?
> 
> ## Work Items
> 
> * Create a new library called oslo.limit
> * Create a core group for the project
> * Define the minimum we need to enforce quota calculations in oslo.limit
> * Propose an implementation that allows services to test out quota
> enforcement via unified limits
> 
> ## References
> 
> Rocky PTG Etherpad for unified limits:
> https://etherpad.openstack.org/p/unified-limits-rocky-ptg
> 
> ## Revision History
> 
> Introduced in Rocky
> 
> On 03/07/2018 11:55 PM, Joshua Harlow wrote:
> > So the following was a prior effort:
> >
> > https://github.com/openstack/delimiter
> >
> > Maybe just continue down the path of that and/or take that whole repo
> > over and iterate (or 

Re: [openstack-dev] [oslo] Oslo PTG Summary

[Doug Hellmann]

I can’t see 

https://docs.google.com/presentation/d/e/2PACX-1vQpaSSm7Amk9q4sBEAUi_IpyJ4l07qd3t5T_BPZkdLWfYbtSpSmF7obSB1qRGA65wjiiq2Sb7H2ylJo/pub?start=false=false=3000=id.p
 




> On Mar 12, 2018, at 11:39 AM, Ken Giusti  wrote:
> 
> Hi Josh - I'm able to view all of them, but I probably have special
> google powers ;)
> 
> Which links are broken for you?
> 
> thanks,
> 
> On Thu, Mar 8, 2018 at 3:53 PM, Joshua Harlow  wrote:
>> 
>> Can we get some of those doc links opened.
>> 
>> 'You need permission to access this published document.' I am getting for a
>> few of them :(
>> 
>> 
>> Ben Nemec wrote:
>>> 
>>> Hi,
>>> 
>>> Here's my summary of the discussions we had in the Oslo room at the PTG.
>>> Please feel free to reply with any additions if I missed something or
>>> correct anything I've misrepresented.
>>> 
>>> oslo.config drivers for secret management
>>> -
>>> 
>>> The oslo.config implementation is in progress, while the Castellan
>>> driver still needs to be written. We want to land this early in Rocky as
>>> it is a significant change in architecture for oslo.config and we want
>>> it to be well-exercised before release.
>>> 
>>> There are discussions with the TripleO team around adding support for
>>> this feature to its deployment tooling and there will be a functional
>>> test job for the Castellan driver with Custodia.
>>> 
>>> There is a weekly meeting in #openstack-meeting-3 on Tuesdays at 1600
>>> UTC for discussion of this feature.
>>> 
>>> oslo.config driver implementation: https://review.openstack.org/#/c/513844
>>> spec:
>>> 
>>> https://specs.openstack.org/openstack/oslo-specs/specs/queens/oslo-config-drivers.html
>>> 
>>> Custodia key management support for Castellan:
>>> https://review.openstack.org/#/c/515190/
>>> 
>>> "stable" libraries
>>> --
>>> 
>>> Some of the Oslo libraries are in a mature state where there are very
>>> few, if any, meaningful changes to them. With the removal of the
>>> requirements sync process in Rocky, we may need to change the release
>>> process for these libraries. My understanding was that there were no
>>> immediate action items for this, but it was something we need to be
>>> aware of.
>>> 
>>> dropping support for mox3
>>> -
>>> 
>>> There was some concern that no one from the Oslo team is actually in a
>>> position to support mox3 if something were to break (such as happened in
>>> some libraries with Python 3.6). Since there is a community goal to
>>> remove mox from all OpenStack projects in Rocky this will hopefully not
>>> be a long-term problem, but there was some discussion that if projects
>>> needed to keep mox for some reason that they would be asked to provide a
>>> maintainer for mox3. This topic is kind of on hold pending the outcome
>>> of the community goal this cycle.
>>> 
>>> automatic configuration migration on upgrade
>>> 
>>> 
>>> There is a desire for oslo.config to provide a mechanism to
>>> automatically migrate deprecated options to their new location on
>>> version upgrades. This is a fairly complex topic that I can't cover
>>> adequately in a summary email, but there is a spec proposed at
>>> https://review.openstack.org/#/c/520043/ and POC changes at
>>> https://review.openstack.org/#/c/526314/ and
>>> https://review.openstack.org/#/c/526261/
>>> 
>>> One outcome of the discussion was that in the initial version we would
>>> not try to handle complex migrations, such as the one that happened when
>>> we combined all of the separate rabbit connection opts into a single
>>> connection string. To start with we will just raise a warning to the
>>> user that they need to handle those manually, but a templated or
>>> hook-based method of automating those migrations could be added as a
>>> follow-up if there is sufficient demand.
>>> 
>>> oslo.messaging plans
>>> 
>>> 
>>> There was quite a bit discussed under this topic. I'm going to break it
>>> down into sub-topics for clarity.
>>> 
>>> oslo.messaging heartbeats
>>> =
>>> 
>>> Everyone seemed to be in favor of this feature, so we anticipate
>>> development moving forward in Rocky. There is an initial patch proposed
>>> at https://review.openstack.org/546763
>>> 
>>> We felt that it should be possible to opt in and out of the feature, and
>>> that the configuration should be done at the application level. This
>>> should _not_ be an operator decision as they do not have the knowledge
>>> to make it sanely.
>>> 
>>> There was also a desire to have a TTL for messages.
>>> 
>>> bug cleanup
>>> ===
>>> 
>>> There are quite a few launchpad bugs open against oslo.messaging that
>>> were reported against old, 

Re: [openstack-dev] [cinder] [oslo] cinder.conf generation is broken for my_ip, building non-reproducibly

[Doug Hellmann]

Excerpts from Thomas Goirand's message of 2018-03-12 09:17:19 +0100:
> Hi,
> 
> When inspecting Cinder's (Queens release) cinder.conf, I can see:
> 
> # Warning: Failed to format sample for my_ip
> # unhashable type: 'HostAddress'

This part sounds like it might be a bug in oslo.config, which
does define a HostAddress class, but that class appears to have the
needed method. Please file a bug.

> 
> So it seems there's an issue in either Cinder or Oslo. How can I
> investigate and fix this?
> 
> It's very likely that I'm once more the only person in the OpenStack
> community that is really checking config file generation (it used to be
> like that for past releases), and therefore the only one who noticed it.
> 
> Also, looking at the code, this seems to be yet-another-instance of
> "package cannot be built reproducible" [1] with the build host config
> leaking in the configuration (well, once that's fixed...). Indeed, in
> the code I can read:
> 
> cfg.HostAddressOpt('my_ip',
>default=netutils.get_my_ipv4(),
>help='IP address of this host'),
> 
> This means that, when that's repaired, build Cinder will write something
> like this:
> 
> #my_ip = 1.2.3.4
> 
> With 1.2.3.4 being the value of netutils.get_my_ipv4(). This is easily
> fixed by adding something like this:
> 
> sample-default=''
> 
> I'm writing this here for Cinder, but there's been numerous cases like
> this already. The most common mistake being the hostname of the build
> host leaking in the configuration. While this is easily fixed at the
> packaging level fixing the config file after generating it with
> oslo.config, often that config file is also built with the sphinx doc,
> and then that file isn't built reproducibly. That's harder to detect,
> and easier fixed upstream.

These sorts of issues are bugs in the consumers of oslo.config, and
should be filed there, since the library can't choose reasonable default
values.

Doug

> 
> Cheers,
> 
> Thomas Goirand (zigo)
> 
> [1] https://reproducible-builds.org/
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] Technical Committee Status update, March 9th

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2018-03-09 11:28:25 +0100:

[...]

> We started with a retrospective of how we got things done so far with
> this membership. Async collaborations worked well, and the weekly
> reports were seen as helpful. We should probably try to steer the topic
> of discussions in office hours a bit more. We also need to more

I created
https://etherpad.openstack.org/p/tc-office-hour-conversation-starters
and seeded it with one topic. Please feel free to add others.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Interop-wg] [QA] [PTG] [Interop] [Designate] [Heat] [TC]: QA PTG Summary- Interop test for adds-on project

[Doug Hellmann]

Excerpts from Zane Bitter's message of 2018-03-08 15:51:05 -0500:
> On 08/03/18 12:57, Doug Hellmann wrote:
> > Why would the repos be owned by anyone other than the original project
> > team?
> 
> A few reasons I think it makes sense in this instance:
> 
> * Not every set of trademark tests will necessarily belong to a single 
> project. Tempest itself is an example of this - in fact that's basically 
> how the QA program came to exist. Vertical-specific trademark programs 
> are another example that we anticipate in the future.
> * Allowing projects to create their own repos means that there's no 
> co-ordination point to ensure e.g. a consistent naming scheme. Amongst 
> other things, this could potentially cause confusion about which plugins 
> are trademark-candidates-only and which are just regular tempest plugins.

If these new plugins might contain "candidate" tests and all tests
are potentially candidates, how are these new repos different from
the existing repos that already contain all of the tests? It seems
like at least part of the problem with the current system was
triggered by confusion about when to move tests around to satisfy
the policy. Can we avoid that problem with the new system? If we're
not going to move the tests into Tempest itself and have the QA
team manage them, why not simply take the tests from the repos where
they already live?

> * By registering trademark plugins all in one place it makes it easy to 
> determine how many there are, which plugins exist (e.g. are there any 
> extant plugins that are not referenced by refstack? This is a question 
> you can answer in 20s if they're all registered in the same place.)
> * The goal is for maintenance of these plugins to be a collaborative 
> effort by the project team, the QA team, and RefStack. If the first step 
> for a project establishing a trademark test plugin involves the project 
> team reaching out to the QA team then that's a good foot to start on. If 
> teams create the repos in their own projects and fly under QA's radar 
> then QA folks might not even be aware that they've become core reviewers 
> on the repo.

I thought the QA team no longer wanted to be responsible for these
extra tests. Has that changed again? I've lost track of everyone's
positions, I'm afraid. Maybe we could get people to start voting
on the actual resolutions so it's easier to keep track of that?

As you pointed out earlier, when contributors to a repo are allowed
to vote in the election for the team lead that owns the repo. We
should think through the implications of that fact when we consider
who will own these new repos (if we actually need anything new and
we can't just use the existing repos).

> I guess we have examples of both models in the community... e.g. 
> puppet-openstack vs. Horizon plugins. I wonder if there are any lessons 
> we can draw on to see which works better, and when.
> 
> cheers,
> Zane.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Pros and Cons of face-to-face meetings

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2018-03-08 18:34:51 +:
> On 2018-03-08 12:16:18 -0600 (-0600), Jay S Bryant wrote:
> [...]
> > Cinder has been doing this for many years and it has worked
> > relatively well. It requires a good remote speaker and it also
> > requires the people in the room to be sensitive to the needs of
> > those who are remote. I.E. planning topics at a time appropriate
> > for the remote attendees, ensuring everyone speaks up, etc. If
> > everyone, however, works to be inclusive with remote participants
> > it works well.
> > 
> > We have even managed to make this work between separate mid-cycles
> > (Cinder and Nova) in the past before we did PTGs.
> [...]
> 
> I've seen it work okay when the number of remote participants is
> small and all are relatively known to the in-person participants.
> Even so, bridging Doug into the TC discussion at the PTG was
> challenging for all participants.

I agree, and I'll point out I was just across town (snowed in at a
different hotel).

The conversation the previous day with just the 5-6 people on the
release team worked a little bit better, but was still challenging
at times because of audio quality issues.

So, yes, this can be made to work. It's not trivial, though, and
the degree to which it works depends a lot on the participants on
both sides of the connection. I would not expect us to be very
productive with a large number of people trying to be active in the
conversation remotely.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Interop-wg] [QA] [PTG] [Interop] [Designate] [Heat] [TC]: QA PTG Summary- Interop test for adds-on project

[Doug Hellmann]

Excerpts from Zane Bitter's message of 2018-03-08 12:45:11 -0500:
> On 07/03/18 08:44, Ghanshyam Mann wrote:
> > I mean i am all ok with separate plugin which is more easy for QA team 
> > but ownership to QA is kind of going to same direction(QA team 
> > maintaining interop ads-on tests) in more difficult way.
> 
> After reading this and the logs from the QA meeting,[1] I feel like 
> there is some confusion/miscommunication over what the proposed 
> resolution means by 'ownership'. Basically every Git repo has to be 
> registered to *some* project in 
> http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml
> 
> The proposal was to register the trademark test plugins to the QA 
> project. The implications of this are fairly minimal in my view:
> 
> * The project gets a say on any new repo creation requests (this will 
> help maintain e.g. a consistent naming scheme IMO)
> * Contributors to the repos are considered contributors to the project, 
> get to vote in the PTL elections, and are allowed to put the logo 
> sticker on their laptop.[2] (This seems appropriate to me, and in the 
> best case might even help convert some people into becoming core 
> reviewers for QA in the long term.)
> * The project would have to meet any other obligations in regards to 
> those repos that the TC delegates to project teams and PTLs - though 
> none of the ones I can think of (releases, tracking project-wide goals) 
> would really apply in practice to the repos we're talking about.
> 
> Perhaps I am missing something that you have a specific concern with?
> 
> It is *not* meant to imply that the project has an obligation to write 
> tests (nobody expects this, in fact), nor that the core reviewers it 
> contributes to the core review team for the repo have any stronger 
> obligation to do reviews than any of the other core reviewers (we really 
> want all 3 teams to contribute to reviews, since they each bring 
> different expertise).
> 
> I think we have two options that could resolve this:
> * Change the wording to ensure that future readers cannot interpret the 
> resolution as placing obligations on the QA team that we didn't intend 
> and they do not want; or
> * Register the Git repos to the refstack project instead.
> 
> cheers,
> Zane.
> 
> [1] 
> http://eavesdrop.openstack.org/meetings/qa/2018/qa.2018-03-08-07.59.log.html#l-34
> [2] kidding! Everyone knows you can't have the sticker until after the 
> initiation ;)
> 

Why would the repos be owned by anyone other than the original project
team?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ptl][release] missing queens release notes pages

[Doug Hellmann]

We have quite a few projects that publish release notes but have
not approved the patch to ensure their queens release notes are
published properly. Please look for a patch from the proposal bot
with the subject "Update reno for stable/queens", fix it if needed,
and approve it quickly.

Doug

barbican no release notes page at 
https://docs.openstack.org/releasenotes/barbican/queens.html
cloudkitty-dashboard no release notes page at 
https://docs.openstack.org/releasenotes/cloudkitty-dashboard/queens.html
cloudkitty no release notes page at 
https://docs.openstack.org/releasenotes/cloudkitty/queens.html
congress-dashboard no release notes page at 
https://docs.openstack.org/releasenotes/congress-dashboard/queens.html
instack-undercloud no release notes page at 
https://docs.openstack.org/releasenotes/instack-undercloud/queens.html
kolla-ansible no release notes page at 
https://docs.openstack.org/releasenotes/kolla-ansible/queens.html
kolla no release notes page at 
https://docs.openstack.org/releasenotes/kolla/queens.html
kuryr no release notes page at 
https://docs.openstack.org/releasenotes/kuryr/queens.html
manila no release notes page at 
https://docs.openstack.org/releasenotes/manila/queens.html
networking-ovn no release notes page at 
https://docs.openstack.org/releasenotes/networking-ovn/queens.html
os-net-config no release notes page at 
https://docs.openstack.org/releasenotes/os-net-config/queens.html
puppet-tripleo no release notes page at 
https://docs.openstack.org/releasenotes/puppet-tripleo/queens.html
python-heatclient no release notes page at 
https://docs.openstack.org/releasenotes/python-heatclient/queens.html
python-manilaclient no release notes page at 
https://docs.openstack.org/releasenotes/python-manilaclient/queens.html
tempest no release notes page at 
https://docs.openstack.org/releasenotes/tempest/queens.html
tripleo-common no release notes page at 
https://docs.openstack.org/releasenotes/tripleo-common/queens.html
tripleo-heat-templates no release notes page at 
https://docs.openstack.org/releasenotes/tripleo-heat-templates/queens.html
tripleo-image-elements no release notes page at 
https://docs.openstack.org/releasenotes/tripleo-image-elements/queens.html
tripleo-puppet-elements no release notes page at 
https://docs.openstack.org/releasenotes/tripleo-puppet-elements/queens.html
tripleo-ui no release notes page at 
https://docs.openstack.org/releasenotes/tripleo-ui/queens.html
tripleo-validations no release notes page at 
https://docs.openstack.org/releasenotes/tripleo-validations/queens.html
watcher-dashboard no release notes page at 
https://docs.openstack.org/releasenotes/watcher-dashboard/queens.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [sdk] Cleaning up openstacksdk core team

[Doug Hellmann]

Excerpts from Monty Taylor's message of 2018-02-26 11:41:18 +:
> Hey all,
> 
> A bunch of stuff has changed in SDK recently, and a few of the 
> historical sdk core folks have also not been around. I'd like to propose 
> removing the following people from the core team:
> 
>Everett Towes
>Jesse Noller
>Richard Theis
>Terry Howe
> 
> They're all fantastic humans but they haven't had any activity in quite 
> some time - and not since all the changes of the sdk/shade merge. As is 
> normal in OpenStack land, they'd all be welcome back if they found 
> themselves in a position to dive in again.
> 
> Any objections?
> 
> Monty
> 

+1 for cleanup. As you say, we can add them back easily if we need to.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptl][all][python3] collecting current status of python 3 support in projects

[Doug Hellmann]

Excerpts from Thomas Goirand's message of 2018-02-24 21:56:43 +0100:
> On 02/23/2018 12:29 AM, Doug Hellmann wrote:
> > I am trying to update the wiki document with the current state of
> > support for Python 3 projects as part of preparing for a discussion
> > about moving from "Python 2 first, then 3" to "Python 3 first, then
> > 2" development.
> > 
> > I have added the missing libraries and services (at least those
> > managed by the release team) and done my best to figure out if there
> > are unit and functional/integration test jobs for each project.
> > 
> > I need your help to verify the information I have collected and fill in
> > any gaps.
> > 
> > Please look through the tables in [1] and if your projects' status
> > is out of date either update the page directly or email me (off
> > list) with the updates.
> > 
> > Thanks!
> > Doug
> > 
> > [1] 
> > https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects
> 
> Hi Doug!
> 
> As I've been working over the course of this week on switching all of
> Debian OpenStack to Py3, I have a bit experience with it. Unfortunately,
> I can only tell about unit tests, as I haven't run functional tests yet.
> 
> Mostly, it's working well, and even in Python 3.6 in Sid.
> 
> Though what I've seen often, is the tooling, and especially the sphinx
> docs, expecting Python 2 to be there. For example (and that's just an
> example, I'm not pointing finger at any project here...) generating the
> sphinx doc of Cinder calls binaries in the "tools" folder (ie:
> tools/generate_driver_list.py) which has "#! /usr/bin/env python" as
> first line. Of course, under my Python 3 only environment, it just fails
> miserably, and I had to patch the files.
> 
> Another example would be Congress generating its lexer with some Python
> 2 type of exception (those with coma instead of "as"). I fixed that at
> build time with Victor's sixer tool (which really is awesome, thanks for
> it Victor!).
> 
> Then there's Nova which annoyed me when generating the doc because of
> seemingly a bug in the Python 3 version of blockdiag (I may be wrong,
> but I don't think Nova itself is at fault here).
> 
> I would have more details like this, but I guess you understand the
> general issue I'm raising: mostly we need to get rid of Python 2
> completely, because otherwise, it's expected to be the default. So I'm
> really looking forward it happens upstream.
> 
> LET'S KILL PYTHON 2 SUPPORT !!! :)
> 
> More seriously, it'd be nice if all the docs tooling were effectively
> switching to Python 3, otherwise other issues will be reported.
> 
> Also, it is annoying to see that manila-ui isn't Python 3 ready at all.
> I guess I'll simply skip manila-ui for this release (since all of
> Horizon is already switched to Python 3 on my side). I'm expecting to
> see more of these Horizon plugins to not be ready (I haven't completely
> finished that part...).
> 
> I hope this helps,
> Cheers,
> 
> Thomas Goirand (zigo)
> 

Thanks for bringing this up, Thomas. You make a good point about
addressing python 2 use outside of just our test jobs, and the issue
is easily actionable.

We do have the ability to specify a python 3 version of the doc
build job, so maybe we can take some time this cycle to move all
projects over to using it and resolve the issues you've spotted.

Does anyone want to volunteer to help with that migration? I'll bet a
lot of projects will just work, and the ones that don't shouldn't be
difficult if the unit tests run under the python 3.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ptl][all][python3] collecting current status of python 3 support in projects

[Doug Hellmann]

I am trying to update the wiki document with the current state of
support for Python 3 projects as part of preparing for a discussion
about moving from "Python 2 first, then 3" to "Python 3 first, then
2" development.

I have added the missing libraries and services (at least those
managed by the release team) and done my best to figure out if there
are unit and functional/integration test jobs for each project.

I need your help to verify the information I have collected and fill in
any gaps.

Please look through the tables in [1] and if your projects' status
is out of date either update the page directly or email me (off
list) with the updates.

Thanks!
Doug

[1] 
https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][release] "why do I have new tags for old versions?"

[Doug Hellmann]

The release team is doing some housekeeping work in openstack/releases
and in the course of that work we modified the deliverable files
for some very old series. Including all the way back to Austin.

Because those files changed, the tag-releases job ran. And because,
apparently, some of the very very old tags had never been imported
into git, they were added today based on the SHAs we had established
when we first imported the history into the repository. So, if you
notice very old tags like "2010.1" showing up in places, that's why.

Our shiny and modern build machinery doesn't work with the state
of those repos from that long ago, so aside from the tags we don't
think anything else was rebuilt (no build artifacts on tarballs.o.o
were changed, for example).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ptl][all][release] published release notes may be out of date

[Doug Hellmann]

Thomas Bechtold reported some issues with the oslo.config release
notes not showing some information. As part of investigating the
problem, I discovered that reno's own release notes were not up to
date, either. I think the problem is a combination of a (now fixed)
reno bug [1] and some unrelated issues with the doc publishing jobs.

The issue is most likely to affect libraries, since those have been
frozen and may not have landed patches to retrigger the publishing
jobs, but server projects without a lot of activity over the last
few weeks may also have out of date release notes.

Please take a look at your projects' release notes and documentation.
If they were last updated before 1 Feb 2018, land a trivial (or
real) patch to the project to cause the documentation jobs to run.
If you have a current global requirements sync patch or translation
patch those are good candidates. The change in [2] is an example
of a trivial patch that might be good if you need one.

Doug

[1] https://bugs.launchpad.net/reno/+bug/1746076
[2] https://review.openstack.org/545846

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

[Doug Hellmann]

Excerpts from Robert Collins's message of 2018-02-20 22:58:59 +1300:
> On 20 February 2018 at 04:39, Andrey Kurilin  wrote:
> > Can someone explain me the reason for including "tests" module into
> > packages?
> 
> Namespacing the tests makes the test ids unique which is very helpful
> for aggregating test data as we do. Including that in the tar.gz that
> is uploaded to PyPI is pretty standard - a) its how you can verify
> that what you downloaded works in your context (and no, going to git
> is not a good answer there because that means you now need the entire
> ecosystem of tools to build man pages etc etc etc). and b) its
> providing the full source of the thing we're releasing. Thats you
> know, how F/LOSS works.

Thanks, Robert, that's the argument I was trying to remember earlier in
the thread.

> It should be possible, if we care to, to exclude the tests from wheels
> made from those distributions, which would make the footprint for
> binary usage smaller - I have no strong opinion on whether thats wise
> or not, but I will say I can't see a use case for needing the tests in
> that scenario.
> 
> Similarly whether those tests should be included in Linux distribution
> packages or not is a debate I don't care to enter - but again I don't
> see a use case: binary distributions are presumed to be integration
> tested by the distributor, not the consumers.
> 
> I don't think importing code from another packages 'tests' module is
> wrong or right - python is very much a consenting-adults language -
> but I do think that in OpenStack with the sheer number of people
> involved we should set very clear guidance; and I'd suggest that
> saying its not supported is a good default: if folk want to offer a
> contract where something can be imported they can always put it in a
> different package.

Exactly. The Oslo team struggles sometimes to support project teams
using the libraries in unexpected ways. This is one of those
unexpected uses, and I think we don't want to support it because
we have not designed the test suite with backwards compatibility
in mind. We just need to be clear about that.

> 
> In summary:
>  - moving 'tests' to the root is a poor idea, please don't do it - we
> had this debate back in 2011 or so and nothing has changed that I can
> see.
>  - we can, and perhaps should, exclude $package.tests from wheels to
> save bandwidth (but *not* from .tar.gz).
>  - linux distributions should IMO follow what we put in wheels.
> 
> -Rob
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

[Doug Hellmann]

Excerpts from Michael Bayer's message of 2018-02-19 10:55:52 -0500:
> wow that's heavy-handed.  should that be in an oslo utility package of
> some kind ?

I thought about that, but figured we should wait and see whether we
actually want to take the approach before polishing it. If we do we can
add a function in oslo.utils.

> 
> On Mon, Feb 19, 2018 at 10:52 AM, Doug Hellmann <d...@doughellmann.com> wrote:
> > Excerpts from Doug Hellmann's message of 2018-02-19 10:15:34 -0500:
> >> Excerpts from Michael Bayer's message of 2018-02-19 10:00:59 -0500:
> >> > Hi list -
> >> >
> >> > Apparently Cinder was misled by my deprecations within the
> >> > oslo_db.sqlalchemy.test_base package of DbFixture and DbTestCase, and
> >> > in https://review.openstack.org/#/c/522290/ the assumption was made
> >> > that these should be imported from oslo_db.tests.sqlalchemy.This
> >> > is an immense mistake on my part that I did not expect people to go
> >> > looking for the same names elsewhere in private packages and now we
> >> > have a serious downstream issue as these modules are not packaged, as
> >> > well as the possibility that the oslo_db.tests. package is now locked
> >> > in time and I have to add deprecations there also.
> >> >
> >> > If anyone knows of projects (or feels like helping me search) that are
> >> > importing *anything* from oslo_db.tests these must be reverted ASAP.
> >> >
> >>
> >> If we have modules or classes we don't expect people to be importing
> >> directly, we need to prefix the names with _ to comply with the naming
> >> conventions we have previously told everyone to look for to recognize
> >> private code.
> >>
> >> I think it's safe to treat "tests" as an exception (after resolving
> >> this case) but we should probably document that.
> >>
> >> Doug
> >
> > Once we resolve the current set of imports, we can land a patch like
> > https://review.openstack.org/545859 to prevent this from happening in
> > the future.
> >
> > Doug
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

[Doug Hellmann]

IIRC we started doing that so that consumers building their own packages
can run the tests for the packages easily. I don't know how many people
are doing that, and apparently at least some downstream consumers aren't
packaging everything anyway so they couldn't run those tests.

Excerpts from Andrey Kurilin's message of 2018-02-19 17:39:11 +0200:
> Can someone explain me the reason for including "tests" module into
> packages?
> 
> 2018-02-19 17:00 GMT+02:00 Michael Bayer :
> 
> > Hi list -
> >
> > Apparently Cinder was misled by my deprecations within the
> > oslo_db.sqlalchemy.test_base package of DbFixture and DbTestCase, and
> > in https://review.openstack.org/#/c/522290/ the assumption was made
> > that these should be imported from oslo_db.tests.sqlalchemy.This
> > is an immense mistake on my part that I did not expect people to go
> > looking for the same names elsewhere in private packages and now we
> > have a serious downstream issue as these modules are not packaged, as
> > well as the possibility that the oslo_db.tests. package is now locked
> > in time and I have to add deprecations there also.
> >
> > If anyone knows of projects (or feels like helping me search) that are
> > importing *anything* from oslo_db.tests these must be reverted ASAP.
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-02-19 10:15:34 -0500:
> Excerpts from Michael Bayer's message of 2018-02-19 10:00:59 -0500:
> > Hi list -
> > 
> > Apparently Cinder was misled by my deprecations within the
> > oslo_db.sqlalchemy.test_base package of DbFixture and DbTestCase, and
> > in https://review.openstack.org/#/c/522290/ the assumption was made
> > that these should be imported from oslo_db.tests.sqlalchemy.This
> > is an immense mistake on my part that I did not expect people to go
> > looking for the same names elsewhere in private packages and now we
> > have a serious downstream issue as these modules are not packaged, as
> > well as the possibility that the oslo_db.tests. package is now locked
> > in time and I have to add deprecations there also.
> > 
> > If anyone knows of projects (or feels like helping me search) that are
> > importing *anything* from oslo_db.tests these must be reverted ASAP.
> > 
> 
> If we have modules or classes we don't expect people to be importing
> directly, we need to prefix the names with _ to comply with the naming
> conventions we have previously told everyone to look for to recognize
> private code.
> 
> I think it's safe to treat "tests" as an exception (after resolving
> this case) but we should probably document that.
> 
> Doug

Once we resolve the current set of imports, we can land a patch like
https://review.openstack.org/545859 to prevent this from happening in
the future.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

[Doug Hellmann]

Excerpts from Michael Bayer's message of 2018-02-19 10:00:59 -0500:
> Hi list -
> 
> Apparently Cinder was misled by my deprecations within the
> oslo_db.sqlalchemy.test_base package of DbFixture and DbTestCase, and
> in https://review.openstack.org/#/c/522290/ the assumption was made
> that these should be imported from oslo_db.tests.sqlalchemy.This
> is an immense mistake on my part that I did not expect people to go
> looking for the same names elsewhere in private packages and now we
> have a serious downstream issue as these modules are not packaged, as
> well as the possibility that the oslo_db.tests. package is now locked
> in time and I have to add deprecations there also.
> 
> If anyone knows of projects (or feels like helping me search) that are
> importing *anything* from oslo_db.tests these must be reverted ASAP.
> 

If we have modules or classes we don't expect people to be importing
directly, we need to prefix the names with _ to comply with the naming
conventions we have previously told everyone to look for to recognize
private code.

I think it's safe to treat "tests" as an exception (after resolving
this case) but we should probably document that.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs] About the convention to use '.' instead of 'source'.

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2018-02-18 16:01:52 +:
> On 2018-02-18 03:55:51 -0600 (-0600), Monty Taylor wrote:
> [...]
> > I'd honestly argue in favor of assuming bash and using 'source'
> > because it's more readable. We don't make allowances for alternate
> > shells in our examples anyway.
> > 
> > I personally try to use 'source' vs . and $() vs. `` as
> > aggressively as I can.
> > 
> > That said - I completely agree with fungi on the description of
> > the tradeoffs of each direction, and I do think it's valuable to
> > pick one for the docs.
> 
> Yes, it's not my call but I too would prefer more readable examples
> over a strict adherence to POSIX. As long as we say somewhere that
> our examples assume the user is in a GNU bash(1) environment and
> that the examples may require minor adjustment for other shells, I
> think that's a perfectly reasonable approach. If there's a
> documentation style guide, that too would be a great place to
> encourage examples following certain conventions such as source
> instead of ., $() instead of ``, [] instead of test, an so on... and
> provide a place to explain the rationale so that reviewers have a
> convenient response they can link for bulk "improvements" which seem
> to indicate ignorance of our reasons for these choices.

I've proposed reverting the style-guide change that seems to have led to
this discussion in https://review.openstack.org/#/c/545718/2

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Migration of Tempest / Grenade jobs to Zuul v3 native

[Doug Hellmann]

Excerpts from Andrea Frittoli's message of 2018-02-15 23:31:02 +:
> Dear all,
> 
> this is the first or a series of ~regular updates on the migration of
> Tempest / Grenade jobs to  Zuul v3 native.
> 
> The QA team together with the infra team are working on providing the
> OpenStack community with a set of base Tempest / Grenade jobs that can be
> used as a basis to write new CI jobs / migrate existing legacy ones with a
> minimal effort and very little or no Ansible knowledge as a precondition.
> 
> The effort is tracked in an etherpad [0]; I'm trying to keep the
> etherpad up to date but it may not always be a source of truth.
> 
> Useful jobs available so far:
> - devstack-tempest [0] is a simple tempest/devstack job that runs keystone
> glance nova cinder neutron swift and tempest *smoke* filter
> - tempest-full [1] is similar but runs a full test run - it replaces the
> legacy tempest-dsvm-neutron-full from the integrated gate
> - tempest-full-py3 [2] runs a full test run on python3 - it replaces the
> legacy tempest-dsvm-py35
> 
> Both tempest-full and tempest-full-py3 are part of integrated-gate
> templates, starting from stable/queens on.
> The other stable branches still run the legacy jobs, since devstack ansible
> changes have not been backported (yet). If we do backport it will be up to
> pike maximum.
> 
> Those jobs work in single node mode only at the moment. Enabling multinode
> via job configuration only require a new Zuul feature [4][5] that should be
> available soon; the new feature allows defining host/group variables in the
> job definition, which means setting variables which are specific to one
> host or a group of hosts.
> Multinode DVR and Ironic jobs will require migration of the ovs-* roles
> form devstack-gate to devstack as well.
> 
> Grenade jobs (single and multinode) are still legacy, even if the *legacy*
> word has been removed from the name.
> They are currently temporarily hosted in the neutron repository. They are
> going to be implemented as Zuul v3 native in the grenade repository.
> 
> Roles are documented, and a couple of migration tips for DEVSTACK_GATE
> flags is available in the etherpad [0]; more comprehensive examples /
> docs will be available as soon as possible.
> 
> Please let me know if you find this update useful and / or if you would
> like to see different information in it.
> I will send further updates as soon as significant changes / new features
> become available.
> 
> Andrea Frittoli (andreaf)
> 
> [0] https://etherpad.openstack.org/p/zuulv3-native-devstack-tempest-jobs
> [1] http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n1
> [2] http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n29
> [3] http://git.openstack.org/cgit/openstack/tempest/tree/.zuul.yaml#n47
> [4] https://etherpad.openstack.org/p/zuulv3-group-variables
> [5] https://review.openstack.org/#/c/544562/

Thanks for this post, Andrea. I know the QA & Infra teams have been
doing a lot of work to complete the migration and improve our CI
systems and I look forward to being able to track the work via
future update emails.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [reno] an alternative approach to known issues

[Doug Hellmann]

Excerpts from Gabriele Cerami's message of 2018-02-08 22:43:56 +:
> Hi,
> 
> sometimes it happens, while reviewing a patch, to find an issue that
> is not quite a bug, because it doesn't limit functionality, but
> may represent a problem in some corner case, or with some possible
> future modification in some component involved in the patch; it may
> best be described as a weakness in the code, which may happen only under
> certain circumstances.
> The author, for some time or complexity constraint is creating a
> technical debt, or making a micro design choice.
> 
> How to keep track of the issue ? How, after 6 month when there's time
> and bandwidth to look at the problem again, can this note be found and
> issue dealt in the way it deserves ?
> How to help prioritize then the list of issues left behind during the
> duration of a release ?
> Nobody is going to read all the comments on all the merged patches in
> the past months, to find all the objections.
> Also technical debts cannot be treated like bugs, because they have a
> different life span. A bug is opened and closed for good after a while.
> A technical debt may be carried for long time, and it would be perfectly
> natural to mark it as something to just live with, and pay the interest
> for, because the time required to solve it it's not worth it. And
> despite that, it's important to keep track of them because an eventual
> reevaluation of the interests cost or a change in the surroundings (a
> new requirement that breaks an assumption) may lead to a different
> decision after some time.
> 
> The way technical debts are treated right now officially is by adding a
> TODO note inside the code, or maybe adding a "issue" field in release
> notes.
> I would like to expand this TODO note, and the known issue field,
> make it become something more structured.
> I thought about reno, to create a technical debt register/micro design
> document.
> A developer would generate a UUID, put on the code a comment
> 
> # TD: 
> 
> and then add the description in reno. A simple yaml associative array
> with three or four keys: UUID, description, consequences, options, which
> may describe either the problem or the micro design choice and
> assumption without which the code may show these weaknesses.
> The description would stay with the code, submitted with the same
> patch with which it was introduced. Then when it's time, a report on all
> these description could be created to evaluate, prioritize and
> eventually close the gap that was created, or just mark that as "prefer
> to just deal with the consequences"
> 
> One may later incur in a problem a number of times, find the piece of
> code responsible, and see that the problem is know, and immediately
> raise its impact to request a reevaluation.
> Or we may realize that the code that creates a certain amount of
> weaknesses is going to be deleted, and we can close all the items
> related to it.
> 
> The creation and handling of such items could add too much of a burden
> to the developer, for these reasons, I would prefer to automate some
> part of the creation, for example the UUID generation, date expansion,
> status change on the item.
> 
> I used this, to try out how this automation could work
> 
> https://review.openstack.org/538233
> 
> which could add basic logic to the templates, to automate some of the
> tasks.
> 
> This idea certainly requires refinement (for example what happens when
> the weakness is discovered at a later time), but I would like to
> understand if it's possible to use reno for this approach. Any feedback
> would be highly appreciated.
> 
> Thanks
> 

What makes reno a good fit for this task? It seems like updating a
regular documentation page in the source tree would work just as well,
since presumably these technical debt descriptions don't need to be
backported to stable branches.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum][release] release-post job for openstack/releases failed

[Doug Hellmann]

Excerpts from Sean McGinnis's message of 2018-02-08 13:00:52 -0600:
> The release job for magnum failed, but luckily it was after tagging and
> branching the release. It was not able to get to the point of uploading a
> tarball to http://tarballs.openstack.org/magnum/ though.
> 
> The problem the job encountered is that magnum is now configured to publish to
> Pypi. The tricky part ends up being that the "magnum" package on Pypi is not
> this magnum project. It appears to be an older abandoned project by someone,
> not related to OpenStack.
> 
> There is an openstack-magnum registered. But since the setup.cfg file in
> openstack/magnum has "name = magnum", it attempts to publish to the one that 
> is
> not ours.
> 
> I have put up a patch to openstack/magnum to change the name to
> openstack-magnum here:
> 
> https://review.openstack.org/#/c/542371/
> 
> That, or something like it, will need to merge and be backported to
> stable/queens before we can get this project published.
> 
> If there are any questions, please feel free to drop in to the
> #openstack-release channel.
> 
> Thanks,
> Sean

Another alternative is to change the job configuration for magnum to use
release-openstack-server instead of publish-to-pypi, at least for the
near term. That would give the magnum team more time to make the changes
need to modify the sdist name for the package.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][ironic][release] release-post job for openstack/releases failed

[Doug Hellmann]

Excerpts from zuul's message of 2018-02-06 00:17:48 +:
> Build failed.
> 
> - tag-releases 
> http://logs.openstack.org/c3/c3ae8a1d87084bcee73e96e2f9e2d174ee610ed4/release-post/tag-releases/48a1f5a/
>  : TIMED_OUT in 32m 09s
> - publish-static publish-static : SKIPPED
> 

After looking at the logs from this failed job, it looks like the
release itself was successful but the branch was not created properly
(the job timed out before that step). I have run the script to create
the branch by hand, so the patches submitted as part of the process came
from me instead of the bot. Please treat them as automatically
bot-generated patches anyway and take them over and fix them if they
have problems (that usually only applies to the sphinx update for reno,
but keep an eye on all of them just in case).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures] [i18n][senlin][release] Tag of openstack/python-senlinclient failed

[Doug Hellmann]

Excerpts from zuul's message of 2018-02-01 17:03:06 +:
> Build failed.
> 
> - publish-openstack-releasenotes 
> http://logs.openstack.org/f8/f84d8220a3df4421c1cfa7ee7b1e551b57c3505d/tag/publish-openstack-releasenotes/49c0e16/
>  : POST_FAILURE in 5m 48s
> 

This failure to build the senlin client release notes appears to
have something to do with the internationalization setup. It is
looking for a CSS file under the fr translation, for some reason.
Perhaps this is related to the race condition we know that the
publish jobs have?

Doug

rsync: failed to set permissions on 
"/afs/.openstack.org/docs/releasenotes/python-senlinclient/fr/_static/css/.bootstrap.css.nwixts":
 No such file or directory (2)
rsync: rename 
"/afs/.openstack.org/docs/releasenotes/python-senlinclient/fr/_static/css/.bootstrap.css.nwixts"
 -> "fr/_static/css/bootstrap.css": No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1183) [sender=3.1.1]
Traceback (most recent call last):
 File "/tmp/ansible_bq6ijf9y/ansible_module_zuul_afs.py", line 115, in 
   main()
 File "/tmp/ansible_bq6ijf9y/ansible_module_zuul_afs.py", line 110, in main
   output = afs_sync(p['source'], p['target'])
 File "/tmp/ansible_bq6ijf9y/ansible_module_zuul_afs.py", line 95, in afs_sync
   output['output'] = subprocess.check_output(shell_cmd, shell=True)
 File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
   **kwargs).stdout
 File "/usr/lib/python3.5/subprocess.py", line 708, in run
   output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/bin/bash -c "mkdir -p 
/afs/.openstack.org/docs/releasenotes/python-senlinclient/ && /usr/bin/rsync 
-rtp --safe-links --delete-after --out-format='<>%i %n%L' 
--filter='merge /tmp/tmp9i7el2ow' 
/var/lib/zuul/builds/49c0e164949c43b68c05856f6cc6452e/work/artifacts/ 
/afs/.openstack.org/docs/releasenotes/python-senlinclient/"' returned non-zero 
exit status 23


___
Release-job-failures mailing list
release-job-failu...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirement][cyborg]FFE - pyspdk requirement dependency

[Doug Hellmann]

Excerpts from We We's message of 2018-01-31 01:54:04 +0800:
> > Hi,
> 
> > I have modified and resubmitted  pyspdk to the pypi. Please check it.
> 
> > Thx,
> 
> > Helloway

Is there a public source repository for the library somewhere?

> 
> > 在 2018年1月30日，下午12:52，We We > 
> > 写道：
> > 
> > Hi,
> > The pyspdk is a important tool library [1] which  supports Cyborg SPDK 
> > driver [2] to manage the backend SPDK-base app, so we need to upload pyspdk 
> > into the pypi [3]  and then append 'pyspdk>=0.0.1’ item into 
> > ‘OpenStack/Cyborg/requirements.txt’ , so that  SPDK driver can be built 
> > correctly when zuul runs. However, It's not what we thought it would be, if 
> > we want to  add the new requirements, we should get support from upstream 
> > OpenStack/requirements [4] to append 'pyspdk>=0.0.1’ item.
> > 
> > I'm sorry for propose the request so late. Please Please help.
> > 
> > 
> > [1] https://review.gerrithub.io/#/c/379741/ 
> > 
> > [2] https://review.openstack.org/#/c/538164/11 
> > 
> > [3] https://pypi.python.org/pypi/pyspdk/0.0.1 
> > 
> > [4] https://github.com/openstack/requirements 
> > 
> > 
> > 
> > Regards,
> > Helloway
> > 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][searchlight] problem with release job configurations

[Doug Hellmann]

Excerpts from McLellan, Steven's message of 2018-01-30 15:03:59 +:
> Hi Doug,
> 
> Apologies, I was travelling all day yesterday. I've put up  
> https://review.openstack.org/539231 to change the project config and made the 
> release review (https://review.openstack.org/538321) depend on it.
> 
> Thanks for the detailed information!
> 
> Steve

Thanks for hopping on the fix so quickly, Steve.

> 
> On 1/29/18, 6:21 PM, "Doug Hellmann" <d...@doughellmann.com> wrote:
> 
> Both searchlight-ui has a configuration issue that the release team
> cannot fix by ourselves. We need input from the searchlight team about
> how to resolve it.
> 
> As you'll see from [2] the release validation logic is categorizing
> searchlight-ui as a horizon-plugin. It is then rejecting the release
> request [1] because, according to the settings in project-config,
> the repository is configured to use publish-to-pypi instead of the
> expected publish-to-pypi-horizon.
> 
> The difference between the two jobs is the latter installs horizon
> before trying to build the package. Many horizon plugins apparently
> needed this. We don't know if searchlight does.
> 
> There are 2 possible ways to fix the issue:
> 
> 1. Set release-type to "python-pypi" in [1] to tell the validation code
>that publish-to-pypi is the expected job.
> 2. Change the release job for the repository in project-config.
> 
> Please let us know which fix is correct by either updating [1] with the
> release-type or a Depends-On link to the change in project-config to use
> the correct release job.
> 
> Doug
> 
> 
> [1] https://review.openstack.org/#/c/538321/
> [2] 
> http://logs.openstack.org/21/538321/1/check/openstack-tox-validate/3afbe28/tox/validate-request-results.log
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][mistral][release][requirements] Pre-release of openstack/mistral-extra failed

[Doug Hellmann]

Excerpts from Brad P. Crochet's message of 2018-01-30 08:16:52 -0500:
> On Mon, Jan 29, 2018 at 8:02 PM, Doug Hellmann <d...@doughellmann.com> wrote:
> > Excerpts from zuul's message of 2018-01-30 00:40:13 +:
> >> Build failed.
> >>
> >> - release-openstack-python 
> >> http://logs.openstack.org/53/533a5ee424ebf6937f03d3b1d9d5b52e8ecb/pre-release/release-openstack-python/44f2fd4/
> >>  : FAILURE in 7m 58s
> >> - announce-release announce-release : SKIPPED
> >> - propose-update-constraints propose-update-constraints : SKIPPED
> >>
> >
> > This release appears to have failed because tox.ini is set up to use the
> > old style of constraints list management and mistral-extra appears in
> > the constraints list.
> >
> > I don't know why the tox environment is being used to build the package;
> > I thought we stopped doing that.
> >
> > One solution is to fix the tox.ini to put the constraints specification
> > in the "deps" field. The patch [1] to oslo.config making a similar
> > change should show you what is needed.
> >
> > Doug
> >
> > [1] https://review.openstack.org/#/c/524496/1/tox.ini
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> Hopefully https://review.openstack.org/539204 fixes it.
> 
> Brad
> 

Thanks!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][mistral][release][requirements] Pre-release of openstack/mistral-extra failed

[Doug Hellmann]

Excerpts from zuul's message of 2018-01-30 00:40:13 +:
> Build failed.
> 
> - release-openstack-python 
> http://logs.openstack.org/53/533a5ee424ebf6937f03d3b1d9d5b52e8ecb/pre-release/release-openstack-python/44f2fd4/
>  : FAILURE in 7m 58s
> - announce-release announce-release : SKIPPED
> - propose-update-constraints propose-update-constraints : SKIPPED
> 

This release appears to have failed because tox.ini is set up to use the
old style of constraints list management and mistral-extra appears in
the constraints list.

I don't know why the tox environment is being used to build the package;
I thought we stopped doing that.

One solution is to fix the tox.ini to put the constraints specification
in the "deps" field. The patch [1] to oslo.config making a similar
change should show you what is needed.

Doug

[1] https://review.openstack.org/#/c/524496/1/tox.ini

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [blazar][release] release job configuration issues

[Doug Hellmann]

Both blazar-dashboard and blazar-nova have configuration issues blocking
their release and the release team needs input from the blazar team to
resolve the problems.

The validation output for blazar-dashboard [2] shows that the repo is
being treated as a horizon plugin but it is configured to use the
release-openstack-server jobs. We think the correct way to resolve this
is to update project-config to use publish-to-pypi-horizon. However, if
horizon is not needed then project-config should be updated to use
publish-to-pypi and the release-type in [1] should be updated to
"python-pypi".

The validation output for blazar-nova shows a similar problem [4]. In
this case, we think the correct solution is to change project-config so
that the repo uses publish-to-pypi instead of release-openstack-server.

Please update those settings and update the release requests with
Depends-On links to the project-config patches so we can process the
releases.

Doug

[1] https://review.openstack.org/#/c/538175/
[2] 
http://logs.openstack.org/75/538175/3/check/openstack-tox-validate/7ed5005/tox/validate-request-results.log
[3] https://review.openstack.org/#/c/538139/
[4] 
http://logs.openstack.org/39/538139/5/check/openstack-tox-validate/05a7503/tox/validate-request-results.log

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release][searchlight] problem with release job configurations

[Doug Hellmann]

Both searchlight-ui has a configuration issue that the release team
cannot fix by ourselves. We need input from the searchlight team about
how to resolve it.

As you'll see from [2] the release validation logic is categorizing
searchlight-ui as a horizon-plugin. It is then rejecting the release
request [1] because, according to the settings in project-config,
the repository is configured to use publish-to-pypi instead of the
expected publish-to-pypi-horizon.

The difference between the two jobs is the latter installs horizon
before trying to build the package. Many horizon plugins apparently
needed this. We don't know if searchlight does.

There are 2 possible ways to fix the issue:

1. Set release-type to "python-pypi" in [1] to tell the validation code
   that publish-to-pypi is the expected job.
2. Change the release job for the repository in project-config.

Please let us know which fix is correct by either updating [1] with the
release-type or a Depends-On link to the change in project-config to use
the correct release job.

Doug


[1] https://review.openstack.org/#/c/538321/
[2] 
http://logs.openstack.org/21/538321/1/check/openstack-tox-validate/3afbe28/tox/validate-request-results.log

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] PTL non-candidacy

[Doug Hellmann]

Excerpts from Brian Rosmaita's message of 2018-01-29 14:18:18 -0500:
> I've been PTL of Glance through some rocky times, but have decided not
> to stand for election for the Rocky cycle.  My plan is to stick
> around, attend to my duties as a glance core contributor, and support
> my successor in whatever way I can to make for a smooth transition.
> After three consecutive cycles of me, it's time for some new ideas and
> new approaches.
> 
> For anyone out there who hasn't contributed to Glance yet, the Glance
> community is friendly and welcoming, and we've got a backlog of
> "untargeted" specs ready for you to pick up.  Weekly meetings are
> 14:00 UTC on Thursdays in #openstack-meeting-4.
> 
> cheers,
> brian
> 

Thank you for carrying the mantle for so long, Brian. I know it
hasn't necessarily been easy but you've dealt with the challenges
well and helped the team move to a healthier state than it was in
when you started in the role.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][requirements][horizon] django-openstack-auth retirement

[Doug Hellmann]

Excerpts from Akihiro Motoki's message of 2018-01-30 00:36:30 +0900:
> Hi the release team and the requirements team,
> 
> I would like to have on django-openstack-auth (DOA) retirement.
> In the thread of the announce of DOA retirement last week, I was
> advised to release a transition package which provides no python
> module and make horizon depend on it so that the transition can be
> smooth.
> http://lists.openstack.org/pipermail/openstack-dev/2018-January/thread.html#126428
> 
> To achieve this, the horizon team needs:
> * to release django-openstack-auth 4.0.0 (the current version is 3.5.0
> so 4.0.0 makes sense) https://review.openstack.org/#/c/538709/
> * to add django-openstack-auth 4.0.0 to g-r and u-c (for queens)
> * to add django-openstack-auth 4.0.0 to horizon queens RC1

I think what Jeremy was proposing in the thread you linked to was
that the new version of django-openstack-auth should depend on
Horizon, so that any projects that depend on django-openstack-auth
but that do not depend on Horizon will still have the relevant
packages installed when they install django_openstack_auth.

We would not need to update the global requirements or constraints lists
to do that.

Doug

> 
> I think there are two options in horizon queens:
> - to release the transition package of django-openstack-auth 4.0.0 as
> described above, or
> - to just document the retirement of django-openstack-auth
> 
> The requirement release is in 9 hours.
> I would like to ask advices from the release and requirements team.
> 
> Thanks,
> Akihiro
> 
> 2018-01-27 2:45 GMT+09:00 Jeremy Stanley :
> > On 2018-01-24 08:47:30 -0600 (-0600), Monty Taylor wrote:
> > [...]
> >> Horizon and neutron were updated to start publishing to PyPI
> >> already.
> >>
> >> https://review.openstack.org/#/c/531822/
> >>
> >> This is so that we can start working on unwinding the neutron and
> >> horizon specific versions of jobs for neutron and horizon plugins.
> >
> > Nice! I somehow missed that merging a couple of weeks back. In that
> > case, I suppose we could in theory do one final transitional package
> > upload of DOA depending on the conflicting Horizon release if others
> > think that's a good idea.
> > --
> > Jeremy Stanley
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ALL][requirements] A freeze is coming and you should be prepared

[Doug Hellmann]

Excerpts from Matthew Thode's message of 2018-01-28 20:47:42 -0600:
> On 18-01-27 21:37:53, Matthew Thode wrote:
> > On 18-01-26 23:05:11, Matthew Thode wrote:
> > > On 18-01-26 00:12:38, Matthew Thode wrote:
> > > > On 18-01-24 22:32:27, Matthew Thode wrote:
> > > > > On 18-01-24 01:29:47, Matthew Thode wrote:
> > > > > > On 18-01-23 01:23:50, Matthew Thode wrote:
> > > > > > > Requirements is freezing Friday at 23:59:59 UTC so any last
> > > > > > > global-requrements updates that need to get in need to get in now.
> > > > > > > 
> > > > > > > I'm afraid that my condition has left me cold to your pleas of 
> > > > > > > mercy.
> > > > > > > 
> > > > > > 
> > > > > > Just your daily reminder that the freeze will happen in about 3 days
> > > > > > time.  Reviews seem to be winding down for requirements now (which 
> > > > > > is
> > > > > > a good sign this release will be chilled to perfection).
> > > > > > 
> > > > > 
> > > > > There's still a couple of things that may cause bumps for iso8601 and
> > > > > oslo.versionedobjects but those are the main things.  The msgpack 
> > > > > change
> > > > > is also rolling out (thanks dirk :D).  Even with all these changes
> > > > > though, in this universe, there's only one absolute. Everything 
> > > > > freezes!
> > > > > 
> > > > > https://review.openstack.org/535520 (oslo.serialization)
> > > > > 
> > > > 
> > > > Last day, gate is sad and behind, but not my fault you waited til the
> > > > last minute :P  (see my first comment).  The Iceman Cometh!
> > > > 
> > > 
> > > All right everyone, Chill.  Looks like we have another couple days to
> > > get stuff in for gate's slowness.  The new deadline is 23:59:59 UTC
> > > 29-01-2018.
> > > 
> > 
> > It's a cold town.  The current status is as follows.  It looks like the
> > gate is clearing up.  oslo.versionedobjects-1.31.2 and iso8601 will be
> > in a gr bump but that's it.  monasca-tempest-plugin is not going to get
> > in by freeze at this rate (has fixes needed in the review).  There was
> > some stuff needed to get nova-client/osc to work together again, but
> > mriedem seems to have it in hand (and no gr updates it looks like).
> > 
> 
> Allow me to break the Ice. My name is Freeze. Learn it well for it's
> the chilling sound of your doom!  Can you feel it coming? The icy cold
> of space!  It's less than 24 hours til the freeze fomrally happens, the
> only outstanding item is that oslo.versionedobjects seems to need
> another fix for the iso8601 bump.  osc-placement won't be added to
> requirements at this point as there has been no responce on their
> review.
> 
> https://review.openstack.org/538515
> 
> python-vitrageclient looks like it'll make it in if gate doesn't break.
> msgpack may also be late, but we'll see (just workflow'd).
> openstacksdk may need a gr bump, I'm waiting on a response from mordred
> 
> https://review.openstack.org/538695
> 

We also have pending releases for cloudkittyclient, blazarclient,
django-openstack-auth, swiftclient, and zaqarclient. Those are blocked
by the current infra issues, and we really should not freeze the
requirements list until we those libraries are released and we have a
chance to try to update the constraints list to include them.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [blazar][release] we need a new blazar client release

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-01-26 16:41:23 -0500:
> The PyPI service is now validating package metadata more strictly, and
> the classifier values for python-blazarclient do not pass the validation
> checks. This means the 1.0.0 package we built cannot be uploaded to
> PyPI [1].
> 
> The fix will take several steps.
> 
> 1. dmsimard has proposed [2] to master to fix the classifiers.
> 
> 2. However, since the repository has
>already been branched for queens we will also need to backport
>that fix to stable/queens.  David has proposed that backport in
>[3].
> 
> 3. There are 2 other patches in stable/queens that need to be
>approved as well [4].
> 
> 4. After they are all merged we can release 1.0.1 from the stable/queens
>branch using the SHA for the merge commit created when [3] lands.
> 
> So, blazar team, please approve all of those patches and then propose a
> new 1.0.1 release quickly.
> 
> Doug
> 
> [1] 
> http://logs.openstack.org/1d/1d46185bf1e0c18f69038adedd37cf6f6eaf06ab/release/release-openstack-python/13aa058/ara/result/26cee65c-b3cd-4267-9a03-1fe45be043d4/
> [2] https://review.openstack.org/538340 Remove commas in setup.cfg package 
> classifiers
> [3] https://review.openstack.org/538343
> [4] 
> https://review.openstack.org/#/q/status:open+project:openstack/python-blazarclient+branch:stable/queens

In order to speed things along, I'm going to go ahead and use my release
manager ACLs to approve those stable branch changes. So please approve
the one on master so your next release there won't have the same issue.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [blazar][release] we need a new blazar client release

[Doug Hellmann]

The PyPI service is now validating package metadata more strictly, and
the classifier values for python-blazarclient do not pass the validation
checks. This means the 1.0.0 package we built cannot be uploaded to
PyPI [1].

The fix will take several steps.

1. dmsimard has proposed [2] to master to fix the classifiers.

2. However, since the repository has
   already been branched for queens we will also need to backport
   that fix to stable/queens.  David has proposed that backport in
   [3].

3. There are 2 other patches in stable/queens that need to be
   approved as well [4].

4. After they are all merged we can release 1.0.1 from the stable/queens
   branch using the SHA for the merge commit created when [3] lands.

So, blazar team, please approve all of those patches and then propose a
new 1.0.1 release quickly.

Doug

[1] 
http://logs.openstack.org/1d/1d46185bf1e0c18f69038adedd37cf6f6eaf06ab/release/release-openstack-python/13aa058/ara/result/26cee65c-b3cd-4267-9a03-1fe45be043d4/
[2] https://review.openstack.org/538340 Remove commas in setup.cfg package 
classifiers
[3] https://review.openstack.org/538343
[4] 
https://review.openstack.org/#/q/status:open+project:openstack/python-blazarclient+branch:stable/queens

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cliff][osc][barbican][oslo][sdk][all] avoiding option name conflicts with cliff and OSC plugins

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-01-18 10:15:16 -0500:
> We've been working this week to resolve an issue between cliff and
> barbicanclient due to a collision in the option namespace [0].
> Barbicanclient was using the -s option, and then cliff's lister
> command base class added that option as an alias for sort-columns.
> 
> The first attempt at resolving the conflict was to set the conflict
> handler in argparse to 'resolve' [1]. Several reviewers pointed out
> that this would have the unwanted side-effect of making some OSC
> commands support the -s as an alias for --sort-columns while the
> barbicanclient commands would use it for a different purpose.
> 
> For now we have removed the -s alias from within cliff. However,
> we want to avoid this problem coming up in the future so we want a
> better solution.
> 
> The OSC project has a policy that its command plugins do not use
> short options (single letter). There are relatively few of them,
> and it's easy to introduce collisions.  Therefore, they are seen
> as reserved for more common "global" options such as provided by
> the base classes in OSC and cliff.
> 
> I propose that for Rocky we update cliff to change the way options
> are registered so that conflicting options added by command subclasses
> are ignored. This would effectively let cliff "own" the short option
> namespace, and require command classes to use long option names.
> 
> The implementation details need to be worked out a bit, but I think
> we can do that by subclassing ArgumentParser and adding a new
> conflict handler method similar to the existing _handle_conflict_error()
> and _handle_conflict_resolve().
> 
> This is going to introduce backwards-incompatible changes in the
> commands derived from cliff, so before we take any action I wanted
> to solicit input in the plan.
> 
> Please let me know what you think,
> Doug
> 
> [0] https://bugs.launchpad.net/python-barbicanclient/+bug/1743578
> [1] https://docs.python.org/3.5/library/argparse.html#conflict-handler

I have a patch up to implement this in https://review.openstack.org/538335

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] [all] TC Report 18-04

[Doug Hellmann]

Excerpts from Matt Riedemann's message of 2018-01-26 10:08:46 -0600:
> On 1/26/2018 9:57 AM, Doug Hellmann wrote:
> > Ideally we would use the time at the PTG to discuss implementation
> > details.
> 
> For something like the mox one, there really are no implementation 
> details, are there?
> 

That one is unusual in that regard, yes.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] [all] TC Report 18-04

[Doug Hellmann]

Excerpts from Matt Riedemann's message of 2018-01-26 09:38:18 -0600:
> On 1/23/2018 12:40 PM, Chris Dent wrote:
> > ## OpenStack-wide Goals
> > 
> > There are four proposed [OpenStack-wide
> > goals](https://governance.openstack.org/tc/goals/index.html):
> > 
> > * [Add Cold upgrades
> >    capabilities](https://review.openstack.org/#/c/533544/)
> > * [Add Rocky goal to remove
> >    mox](https://review.openstack.org/#/c/532361/)
> > * [
> > Add Rocky goal to enable mutable
> > configuration](https://review.openstack.org/#/c/534605/)
> > * [
> > Add Rocky goal to ensure pagination
> > links](https://review.openstack.org/#/c/532627/)
> > 
> > These need to be validated by the community, but they are not [getting
> > as much
> > feedback](http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-01-23.log.html#t2018-01-23T09:16:51)
> >  
> > 
> > as hoped. There are different theories as to why, from "people are
> > busy", to "people don't feel empowered to comment", to "people don't
> > care". Whatever it is, without input the onus falls on the TC to make
> > choices, increasing the risk that the goals will be perceived as a
> > diktat. As always, we need to work harder to have high fidelity
> > feedback loops. This is especially true in our "mature" phase.
> 
> What's the due date on approving the community wide goals for Rocky? 
> Given the PTG is around the corner (by a month I guess), why not just 
> have the discussion in person at the PTG (for those attending in person 
> anyway)?
> 

Ideally we would use the time at the PTG to discuss implementation
details.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release][ptl][all] extending queens-3 milestone deadlines

[Doug Hellmann]

This morning during the release team meeting [1] the team agreed that
given the issues with CI this week it makes sense to extend the
deadlines for the milestone.

The queens-3 deadline for projects following the cycle-with-milestones
release model is extended to Monday 29 January.

The client-library release deadline is extended to Tuesday 30 January.

Milestone-projects that have already tagged their 0b3 release for Queens
should carry on with preparing their release candidates as planned.

Doug

[1] 
http://eavesdrop.openstack.org/meetings/releaseteam/2018/releaseteam.2018-01-26-15.03.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release][ptl][all] please approve the reno updates for queens

[Doug Hellmann]

As part of creating branches for projects, the job proposes updates to
add a "queens" page to the release notes build. The script prepares a
best-effort version of the update, but local variances in the
repositories means that doesn't always work.

Please take the patches over and fix them, then land them as quickly as
is reasonable to ensure that the release notes for your project are
published correctly.

https://review.openstack.org/#/q/topic:reno-queens

Thanks!
Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Saverio Proto's message of 2018-01-24 22:18:39 +0100:
> > 3.34.0 is a queens series release, which makes it more likely that more
> > other dependencies would need to be updated. Even backporting the
> > changes to the Ocata branch and releasing it from there would require
> > updating several other libraries.
> > 
> 
> That is what I was fearing. Consider that our upgrade schedule is now to
> have Pike by the end of 2018. Unless we try to skip a release.

You should seriously consider upgrading to at least Queens. Pike will be out
of community support by Sept 2018 (https://releases.openstack.org). Some
of the community deployment tools have support for "fast-forward"
upgrades (allowing you to install several versions in series and only
launch the cloud again when the final version is installed). You can
check with the team that manages your deployment tool to see if it
supports this capability.

> > Are you using packages from Canonical, or are you building them
> > yourself?
> 
> I am using the packages from Canonical, but I am familiar patching those
> packages and merge my changes upstream back to Canonical.
> If the problem is just dependencies with the ".deb" packages, I can
> handle that. But if the problem is really python code not working
> together across multiple components, then I have little hope to fix this
> in Newton.

I don't know Canonical's support policies, but it may be possible to
backport a patch to oslo.log to provide the missing information.

> Thanks for the support, if I manage to make some progress on this I will
> send an update on this thread on the mailing list.

Good luck, and please do let me know how it goes.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-ansible] Limiting pip wheel builds for OpenStack clients

[Doug Hellmann]

Excerpts from Mooney, Sean K's message of 2018-01-24 20:30:19 +:
> 
> > -Original Message-
> > From: Major Hayden [mailto:ma...@mhtx.net]
> > Sent: Wednesday, January 24, 2018 8:03 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > 
> > Subject: [openstack-dev] [openstack-ansible] Limiting pip wheel builds
> > for OpenStack clients
> > 
> > Hey there,
> > 
> > I was spelunking into the slow wheel build problems we've been seeing
> > in CentOS and I found that our wheel build process was spending 4-6
> > minutes building cassandra-driver. The wheel build process usually
> > takes 8-12 minutes, so half the time is being spent there.
> > 
> > More digging revealed that cassandra-driver is a dependency of python-
> > monascaclient, which is a dependency of heat. The requirements.txt for
> > heat drags in all of the clients:
> > 
> >   https://github.com/openstack/heat/blob/master/requirements.txt
> [Mooney, Sean K] the python-monascaclient package is presumably an optional
> Dependency of heat as are the other client.
> E.g. I would hope that if you are using a heat with a cloud that does not have
> Monasca it could still run without have python-monascaclient installed so
> All of the clients should proably be move form the requirements.txt to the
> test-requiremetns.txt and only the minimal required packages for heat to work
> should be in requirements.txt.

This is what the "extras" section of setup.cfg is for. It should be
possible to say something like:

[extras]
monasca =
   python-monascaclient>=1.0.0 (or whatever version)


Then users of pip would install a heat that uses monasca with:

   pip install heat[monasca]

and distro packagers would know why any extra packages are needed and
could take appropriate action in their package specifications, too.

> > 
> > We're already doing selective wheel builds and building only the wheels
> > and venvs we need for the OpenStack services which are selected for
> > deployment. Would it make sense to reduce the OpenStack client list for
> > heat during the wheel/venv build? For example, if we're not deploying
> > monasca, should we build/venv the python-monascaclient package (and its
> > dependencies)?
> > 
> > I've opened a bug:
> > 
> >   https://bugs.launchpad.net/openstack-ansible/+bug/1745215
> > 
> > --
> > Major Hayden
> > 
> > ___
> > ___
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> > requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Saverio Proto's message of 2018-01-23 10:21:37 +0100:
> Hello Doug,
> 
> I have run the script:
> here is my output:
> 
> http://paste.openstack.org/show/650913/

It looks like the logger returned by oslo.log does include the
values in the "extra" section but the others do not because the
code to introduce the values isn't invoked, just as when I run the
script.

> At this point I have some questions. Can I upgrade just oslo.log library
> keeping the rest of the stuff in Newton ?

Maybe. oslo.log has several dependencies that may also need to be
updated. Those should be backwards-compatible, but the configuration
you would end up with is not something we have ever tested. I'm also not
sure how you would get the right Ubuntu packages in place. Someone on
the openstack-operators mailing list might be able to help with that.

> The versions of oslo.log have a different numbering scheme than other
> openstack projects, so I cannot understand the versions compatibility.

The releases web site (https://releases.openstack.org) documents the
versions of all of the components released for each series.

> 
> As far as I understand 3.34.0 should be enough for me ? :
> 
>  git tag --contains 1b012d0fc6811f00e032e52ed586fe37e157584d
> 3.34.0
> 3.35.0
> 3.36.0

3.34.0 is a queens series release, which makes it more likely that more
other dependencies would need to be updated. Even backporting the
changes to the Ocata branch and releasing it from there would require
updating several other libraries.

Are you using packages from Canonical, or are you building them
yourself?

Doug

> 
> thank you
> 
> Saverio
> 
> On 22.01.18 23:20, Doug Hellmann wrote:
> > Excerpts from Saverio Proto's message of 2018-01-22 18:45:15 +0100:
> >> Hello Doug,
> >>
> >> in the extra session I see just {"project": "unknown", "version": 
> >> "unknown"}
> >>
> >> here a full line from nova-api:
> >>
> >> {"thread_name": "MainThread", "extra": {"project": "unknown", "version":
> >> "unknown"}, "process": 31142, "relative_created": 3459415335.4091644,
> >> "module": "wsgi", "message":
> >> "2001:xxx::8100::80,2001:xxx::81ff::b0 \"GET
> >> /v2/64b5b50eb21d4efe9783eb1d81a9ec65/os-services HTTP/1.1\" status: 200
> >> len: 1812 time: 0.1813300", "hostname": "nova-0", "filename": "wsgi.py",
> >> "levelno": 20, "lineno": 555, "asctime": "2018-01-22 18:37:02,312",
> >> "msg": "2001:xxx::8100::80,2001:xxx::81ff::b0 \"GET
> >> /v2/64b5b50eb21d4efe9783eb1d81a9ec65/os-services HTTP/1.1\" status: 200
> >> len: 1812 time: 0.1813300", "args": [], "process_name": "MainProcess",
> >> "name": "nova.osapi_compute.wsgi.server", "thread": 140414249163824,
> >> "created": 1516642622.312235, "traceback": null, "msecs":
> >> 312.23511695861816, "funcname": "handle_one_response", "pathname":
> >> "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", "levelname": "INFO"}
> > 
> > It looks like you're running into a limitation of the older version of
> > the library where the context was only logged from openstack source
> > code. This particular log message is coming from the eventlet library.
> > 
> > Try running the script below and saving the output to a pastebin.
> > 
> > Under the newton version of oslo.log, I get
> > http://paste.openstack.org/show/650566/ and under the queens version I
> > get http://paste.openstack.org/show/650569/ which shows me that the
> > "extra" handling is working more or less the same way but the "context"
> > handling is improved in the newer version (lots of the values are null
> > because I don't fully set up the context, but the request_id field has a
> > valid value).
> > 
> > Doug
> > 
> > 
> > #!/usr/bin/env python
> > 
> > from __future__ import print_function
> > 
> > import logging
> > 
> > from oslo_context import context
> > from oslo_log import formatters, log
> > 
> > 
> > ch = logging.StreamHandler()
> > ch.setLevel(logging.DEBUG)
> > 
> > formatter = formatters.JSONFormatter()
> > ch.setFormatter(formatter)
> > 
> > LOG = logging.getLogger()
> > LOG.setLevel(logging.DEBUG)
> > LOG.addHandler(ch)
> > 
> > ctx = context.RequestContext(request_id='the-request-id')
> > 
> > LOG.debug('without extra')
> > print()
> > 
> > LOG.debug('with extra', extra={'context': ctx})
> > print()
> > 
> > log.getLogger().debug('via KeywordArgumentAdapter', context=ctx)
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Saverio Proto's message of 2018-01-22 18:45:15 +0100:
> Hello Doug,
> 
> in the extra session I see just {"project": "unknown", "version": "unknown"}
> 
> here a full line from nova-api:
> 
> {"thread_name": "MainThread", "extra": {"project": "unknown", "version":
> "unknown"}, "process": 31142, "relative_created": 3459415335.4091644,
> "module": "wsgi", "message":
> "2001:xxx::8100::80,2001:xxx::81ff::b0 \"GET
> /v2/64b5b50eb21d4efe9783eb1d81a9ec65/os-services HTTP/1.1\" status: 200
> len: 1812 time: 0.1813300", "hostname": "nova-0", "filename": "wsgi.py",
> "levelno": 20, "lineno": 555, "asctime": "2018-01-22 18:37:02,312",
> "msg": "2001:xxx::8100::80,2001:xxx::81ff::b0 \"GET
> /v2/64b5b50eb21d4efe9783eb1d81a9ec65/os-services HTTP/1.1\" status: 200
> len: 1812 time: 0.1813300", "args": [], "process_name": "MainProcess",
> "name": "nova.osapi_compute.wsgi.server", "thread": 140414249163824,
> "created": 1516642622.312235, "traceback": null, "msecs":
> 312.23511695861816, "funcname": "handle_one_response", "pathname":
> "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", "levelname": "INFO"}

It looks like you're running into a limitation of the older version of
the library where the context was only logged from openstack source
code. This particular log message is coming from the eventlet library.

Try running the script below and saving the output to a pastebin.

Under the newton version of oslo.log, I get
http://paste.openstack.org/show/650566/ and under the queens version I
get http://paste.openstack.org/show/650569/ which shows me that the
"extra" handling is working more or less the same way but the "context"
handling is improved in the newer version (lots of the values are null
because I don't fully set up the context, but the request_id field has a
valid value).

Doug


#!/usr/bin/env python

from __future__ import print_function

import logging

from oslo_context import context
from oslo_log import formatters, log


ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

formatter = formatters.JSONFormatter()
ch.setFormatter(formatter)

LOG = logging.getLogger()
LOG.setLevel(logging.DEBUG)
LOG.addHandler(ch)

ctx = context.RequestContext(request_id='the-request-id')

LOG.debug('without extra')
print()

LOG.debug('with extra', extra={'context': ctx})
print()

log.getLogger().debug('via KeywordArgumentAdapter', context=ctx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Saverio Proto's message of 2018-01-21 18:09:03 +0100:
> Hello,
> 
> I figured out a bug is already open since a long time :(
> https://bugs.launchpad.net/oslo.log/+bug/1564931
> 
> And there is already a review:
> https://review.openstack.org/#/c/367514/
> 
> it looks like the review was not merged, and it went to abandoned
> because of no progress on it for long time.
> 
> I rebased that code on the current master:
> https://review.openstack.org/536149

That patch is not needed on master.

We added the full context as a nested value under the "context" key when
http://git.openstack.org/cgit/openstack/oslo.log/commit/oslo_log/formatters.py?id=1b012d0fc6811f00e032e52ed586fe37e157584d
landed. That change was released as part of 3.35.0 and as the test in
https://review.openstack.org/#/c/536164/ shows the request_id and
global_request_id values are present in the output.

Before that change (and after, as part of our effort to maintain
backwards compatibility) the values from the context were added to the
"extra" section of the output message. That behavior is present in

master: 
http://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/formatters.py?h=master#n246
pike: 
http://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/formatters.py?h=stable%2Fpike#n300
ocata: 
http://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/formatters.py?h=stable%2Focata#n156

It also appears to be present for newton, the version you said you are
using:

http://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/formatters.py?h=newton-eol#n153

Have you looked at the "extras" section of the log output?

Could you provide some sample log output?

Doug

> 
> Saverio
> 
> On 18.01.18 18:14, Doug Hellmann wrote:
> > Excerpts from Doug Hellmann's message of 2018-01-18 11:45:28 -0500:
> >> Excerpts from Saverio Proto's message of 2018-01-18 14:49:21 +0100:
> >>> Hello all,
> >>>
> >>> well this oslo.log library looks like a core thing that is used by
> >>> multiple projects. I feel scared hearing that bugs opened on that
> >>> project are probably just ignored.
> >>>
> >>> should I reach out to the current PTL of OSLO ?
> >>> https://github.com/openstack/governance/blob/master/reference/projects.yaml#L2580
> >>>
> >>> ChangBo Guo are you reading this thread ? Do you think this is a bug or
> >>> a missing feature ? And moreover is really nobody looking at these
> >>> oslo.log bugs ?
> >>
> >> The Oslo team is small, but we do pay attention to bug reports. I
> >> don't think this issue is going to rise to the level of "drop what
> >> you're doing and help because the world is on fire", so I think
> >> Sean is just encouraging you to have a little patience.
> >>
> >> Please do go ahead and open a bug and attach (or paste into the
> >> description) an example of what the log output for your service looks
> >> like.
> >>
> >> Doug
> > 
> > Earlier in the thread you mentioned running the newton versions of
> > neutron and oslo.log. The newton release has been marked end-of-life
> > and is not supported by the community any longer. You may find
> > support from your vendor, but if you're deploying on your own we'll
> > have to work something else out. If we determine that this is a bug
> > in the newton version of the library I won't have any way to give
> > you a new release because the branch is closed.
> > 
> > It should be possible for you to update just oslo.log to a more
> > recent (and supported), although to do so you would have to get the
> > package separately or build your own and that may complicate your
> > deployment.
> > 
> > More recent versions of the JSON formatter change the structure of
> > the data to include the entire context (including the request id)
> > in a separate key.  Are you updating to newton as part of upgrading
> > further than that?  If so, we probably want to wait to debug this
> > until you hit the latest supported version you're planning to deploy,
> > in case the problem is already fixed there.
> > 
> > Doug
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-01-18 15:21:12 -0500:
> Excerpts from Graham Hayes's message of 2018-01-18 19:25:02 +:
> > 
> > On 18/01/18 18:52, Doug Hellmann wrote:
> > > Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +:
> > >> On 18/01/18 16:25, Doug Hellmann wrote:
> > >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +:
> > >>
> > >> 
> > >>
> > >>>
> > >>> In the past the QA team agreed to accept trademark-related tests from
> > >>> all projects in the tempest repo. Has that changed?
> > >>>
> > >>
> > >> There has not been an explict rejection but in all conversations the
> > >> response has been "non core projects are outside the scope of tempest".
> > >>
> > >> Honestly, everytime we have tried to do something to core tempest
> > >> we have had major pushback, and I want to clarify this before I or
> > >> someone else put in the work of porting the base clients, getting CI
> > >> configured*, and proposing the tests to tempest.
> > > 
> > > OK.
> > > 
> > > The current policy doesn't say anything about "core" or different
> > > trademark programs or any other criteria.
> > > 
> > >   The TC therefore encourages the DefCore committee to consider it an
> > >   indication of future technical direction that we do not want tests
> > >   outside of the Tempest repository used for trademark enforcement, and
> > >   that any new or existing tests that cover capabilities they want to
> > >   consider for trademark enforcement should be placed in Tempest.
> > > 
> > > That all seems very clear to me (setting aside some specific word
> > > choices like "future technical direction" that tie the resolution
> > > to language in the bylaws).  Regardless of technical reasons why
> > > it may not be necessary, we still have many social justifications
> > > for doing it the way we originally set out to do it.  Tests related
> > > to trademark enforcement need to go into the tempest repository.
> > > 
> > > The way I think this should work (and the way I remember us describing
> > > it at the time the policy was established) is the Interop WG
> > > (previously DefCore) should identify capabilities and tests, then
> > > ask project teams to reproduce those tests in the tempest repo.
> > > When the tests land, they can be used by the trademark program.
> > > Teams can also, at their leisure, decide whether to remove the
> > > original versions of the tests from whatever repo they existed in
> > > to begin with.
> > > 
> > > Graham, you've proposed a new resolution with several options for
> > > where to put tests for "add-on programs." I don't think we need
> > > that resolution if we want the tests to continue to live in tempest.
> > > The existing resolution doesn't qualify which tests, beyond "for
> > > trademark enforcement" and more words won't make that more clear,
> > > IMO.
> > > 
> > > Now if you *do* want to change the policy, we should talk about
> > > that.  But I can't tell whether you want to change it, you're worried
> > > the policy is unclear, or it is not being followed.  Can you clarify
> > > which it is?
> > 
> > It is not being followed.
> > 
> > I have brought this up at every forum session on these programs, and the
> > people in the room from QA have *always* pushed back on it.
> 
> OK, so that's a problem. I need to hear from the QA team why they've
> reversed that decision.
> 
> > 
> > And, for clarity (I saw this in a few logs) QA have *never* said that
> > they will take the interop designated tests for the DNS project into
> > openstack/tempest.
> 
> When we approved the resolution that describes the current policy, the
> QA team agreed that they would take tests for trademark. There was no
> stipulation about which projects those apply to.

I feel pretty sure that was discussed in a TC meeting, but I can't
find that. I do find Matt and Ken'ichi voting +1 on the resolution
itself.  https://review.openstack.org/#/c/312718/. If I remember
correctly, Ken'ichi was the PTL at the time.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs

[Doug Hellmann]

Excerpts from Graham Hayes's message of 2018-01-18 19:25:02 +:
> 
> On 18/01/18 18:52, Doug Hellmann wrote:
> > Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +:
> >> On 18/01/18 16:25, Doug Hellmann wrote:
> >>> Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +:
> >>
> >> 
> >>
> >>>
> >>> In the past the QA team agreed to accept trademark-related tests from
> >>> all projects in the tempest repo. Has that changed?
> >>>
> >>
> >> There has not been an explict rejection but in all conversations the
> >> response has been "non core projects are outside the scope of tempest".
> >>
> >> Honestly, everytime we have tried to do something to core tempest
> >> we have had major pushback, and I want to clarify this before I or
> >> someone else put in the work of porting the base clients, getting CI
> >> configured*, and proposing the tests to tempest.
> > 
> > OK.
> > 
> > The current policy doesn't say anything about "core" or different
> > trademark programs or any other criteria.
> > 
> >   The TC therefore encourages the DefCore committee to consider it an
> >   indication of future technical direction that we do not want tests
> >   outside of the Tempest repository used for trademark enforcement, and
> >   that any new or existing tests that cover capabilities they want to
> >   consider for trademark enforcement should be placed in Tempest.
> > 
> > That all seems very clear to me (setting aside some specific word
> > choices like "future technical direction" that tie the resolution
> > to language in the bylaws).  Regardless of technical reasons why
> > it may not be necessary, we still have many social justifications
> > for doing it the way we originally set out to do it.  Tests related
> > to trademark enforcement need to go into the tempest repository.
> > 
> > The way I think this should work (and the way I remember us describing
> > it at the time the policy was established) is the Interop WG
> > (previously DefCore) should identify capabilities and tests, then
> > ask project teams to reproduce those tests in the tempest repo.
> > When the tests land, they can be used by the trademark program.
> > Teams can also, at their leisure, decide whether to remove the
> > original versions of the tests from whatever repo they existed in
> > to begin with.
> > 
> > Graham, you've proposed a new resolution with several options for
> > where to put tests for "add-on programs." I don't think we need
> > that resolution if we want the tests to continue to live in tempest.
> > The existing resolution doesn't qualify which tests, beyond "for
> > trademark enforcement" and more words won't make that more clear,
> > IMO.
> > 
> > Now if you *do* want to change the policy, we should talk about
> > that.  But I can't tell whether you want to change it, you're worried
> > the policy is unclear, or it is not being followed.  Can you clarify
> > which it is?
> 
> It is not being followed.
> 
> I have brought this up at every forum session on these programs, and the
> people in the room from QA have *always* pushed back on it.

OK, so that's a problem. I need to hear from the QA team why they've
reversed that decision.

> 
> And, for clarity (I saw this in a few logs) QA have *never* said that
> they will take the interop designated tests for the DNS project into
> openstack/tempest.

When we approved the resolution that describes the current policy, the
QA team agreed that they would take tests for trademark. There was no
stipulation about which projects those apply to.

> 
> To the point that the interop tooling was developed to support plugins
> (which would seem to be in breach of this resolution, but I am sure
> there is reasons for this.)

I can see it being useful to support plugins for evaluating tests before
they are accepted.

> 
> I do want to have option 3 (interop-tempest-plugin), but right now I
> will settle for us either:
> 
> A: Doing what we planned on before (Option 1) (Prefered)
> B: Documenting the fact that things have changed (Option 2), and 
>articulate and record the reasoning for the change.
> 
> I think Add Ons are going to the Board in Dublin for the change from
> Advisory, in the 2018.02 standard so we need to get clarity on this.

I agree.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs

[Doug Hellmann]

Excerpts from Graham Hayes's message of 2018-01-18 17:52:39 +:
> On 18/01/18 16:25, Doug Hellmann wrote:
> > Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +:
> 
> 
> 
> > 
> > In the past the QA team agreed to accept trademark-related tests from
> > all projects in the tempest repo. Has that changed?
> > 
> 
> There has not been an explict rejection but in all conversations the
> response has been "non core projects are outside the scope of tempest".
> 
> Honestly, everytime we have tried to do something to core tempest
> we have had major pushback, and I want to clarify this before I or
> someone else put in the work of porting the base clients, getting CI
> configured*, and proposing the tests to tempest.

OK.

The current policy doesn't say anything about "core" or different
trademark programs or any other criteria.

  The TC therefore encourages the DefCore committee to consider it an
  indication of future technical direction that we do not want tests
  outside of the Tempest repository used for trademark enforcement, and
  that any new or existing tests that cover capabilities they want to
  consider for trademark enforcement should be placed in Tempest.

That all seems very clear to me (setting aside some specific word
choices like "future technical direction" that tie the resolution
to language in the bylaws).  Regardless of technical reasons why
it may not be necessary, we still have many social justifications
for doing it the way we originally set out to do it.  Tests related
to trademark enforcement need to go into the tempest repository.

The way I think this should work (and the way I remember us describing
it at the time the policy was established) is the Interop WG
(previously DefCore) should identify capabilities and tests, then
ask project teams to reproduce those tests in the tempest repo.
When the tests land, they can be used by the trademark program.
Teams can also, at their leisure, decide whether to remove the
original versions of the tests from whatever repo they existed in
to begin with.

Graham, you've proposed a new resolution with several options for
where to put tests for "add-on programs." I don't think we need
that resolution if we want the tests to continue to live in tempest.
The existing resolution doesn't qualify which tests, beyond "for
trademark enforcement" and more words won't make that more clear,
IMO.

Now if you *do* want to change the policy, we should talk about
that.  But I can't tell whether you want to change it, you're worried
the policy is unclear, or it is not being followed.  Can you clarify
which it is?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-01-18 11:45:28 -0500:
> Excerpts from Saverio Proto's message of 2018-01-18 14:49:21 +0100:
> > Hello all,
> > 
> > well this oslo.log library looks like a core thing that is used by
> > multiple projects. I feel scared hearing that bugs opened on that
> > project are probably just ignored.
> > 
> > should I reach out to the current PTL of OSLO ?
> > https://github.com/openstack/governance/blob/master/reference/projects.yaml#L2580
> > 
> > ChangBo Guo are you reading this thread ? Do you think this is a bug or
> > a missing feature ? And moreover is really nobody looking at these
> > oslo.log bugs ?
> 
> The Oslo team is small, but we do pay attention to bug reports. I
> don't think this issue is going to rise to the level of "drop what
> you're doing and help because the world is on fire", so I think
> Sean is just encouraging you to have a little patience.
> 
> Please do go ahead and open a bug and attach (or paste into the
> description) an example of what the log output for your service looks
> like.
> 
> Doug

Earlier in the thread you mentioned running the newton versions of
neutron and oslo.log. The newton release has been marked end-of-life
and is not supported by the community any longer. You may find
support from your vendor, but if you're deploying on your own we'll
have to work something else out. If we determine that this is a bug
in the newton version of the library I won't have any way to give
you a new release because the branch is closed.

It should be possible for you to update just oslo.log to a more
recent (and supported), although to do so you would have to get the
package separately or build your own and that may complicate your
deployment.

More recent versions of the JSON formatter change the structure of
the data to include the entire context (including the request id)
in a separate key.  Are you updating to newton as part of upgrading
further than that?  If so, we probably want to wait to debug this
until you hit the latest supported version you're planning to deploy,
in case the problem is already fixed there.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Saverio Proto's message of 2018-01-18 14:49:21 +0100:
> Hello all,
> 
> well this oslo.log library looks like a core thing that is used by
> multiple projects. I feel scared hearing that bugs opened on that
> project are probably just ignored.
> 
> should I reach out to the current PTL of OSLO ?
> https://github.com/openstack/governance/blob/master/reference/projects.yaml#L2580
> 
> ChangBo Guo are you reading this thread ? Do you think this is a bug or
> a missing feature ? And moreover is really nobody looking at these
> oslo.log bugs ?

The Oslo team is small, but we do pay attention to bug reports. I
don't think this issue is going to rise to the level of "drop what
you're doing and help because the world is on fire", so I think
Sean is just encouraging you to have a little patience.

Please do go ahead and open a bug and attach (or paste into the
description) an example of what the log output for your service looks
like.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs

[Doug Hellmann]

Excerpts from Graham Hayes's message of 2018-01-18 15:33:12 +:

> I had hoped for more of a discussion on this before I jumped back into
> this debate  - but it seams to be stalled still, so here it goes.
> 
> I proposed this initially as we were unclear on where the tests should
> go - we had a resolution that said all tests go into openstack/tempest
> (with a list of reasons why), and the guidance and discussion that been
> had in various summits was that "add-ons" should stay in plugins.
> 
> So right now, we (by the governance rules) should be pushing tests to
> tempest for the new programs.
> 
> In the resolution that placed the tests in tempest there was a few
> reasons proposed:
> 
>   For example, API and behavioral changes must be carefully managed, as
>   must mundane aspects such as test and module naming and location
>   within the test suite. Even changes that leave tests functionally
>   equivalent may cause unexpected consequences for their use in DefCore
>   processes and validation. Placing the tests in a central repository
>   will make it easier to maintain consistency and avoid breaking the
>   trademark enforcement tool.
> 
> This still applies, and even more so for teams that traditionally do not
> have a strong QE contributor / reviewer base (aka projects not in
> "core").
> 
>   Centralizing the tests also makes it easier for anyone running the
>   validation tool against their cloud or cloud distribution to use the
>   tests. It is easier to install the test suite and its dependencies,
>   and it is easier to read and understand a set of tests following a
>   consistent implementation pattern.
> 
> Apparently users do not need central tests anymore, feedback from
> RefStack is that people who run these tests are comfortable dealing
> with extra python packages.
> 
> The point about a single set of tests, in a single location and style
> still stands.
> 
>   Finally, having the tests in a central location makes it easier to
>   ensure that all members of the community have equal input into what
>   the tests do and how they are implemented and maintained.
> 
> Seems like a good value to strive for.
> 
> One of the items that has been used to push back against adding
> "add-ons" to tempest has been that tempest has a defined scope, and
> neither of the current add-ons fit in that scope.
> 
> Can someone clarify what the set of criteria is? I think it will help
> this discussion.
> 
> Another push back is the "scaling" issue - adding more tests will
> overload the QA team.

In the past the QA team agreed to accept trademark-related tests from
all projects in the tempest repo. Has that changed?

> 
> Right now, DNS adds 10 tests, Orchestration adds 22, to a current suite
> of 353.
> 
> I do not think there is many other add-ons proposed yet, and the new
> Vertical programs will probably mainly be re-using tests in the
> openstack/tempest repos as is.
> 
> This is not a big tent-esque influx of programs - the only projects
> that can be added to the trademarks are programs in tc-approved-release
> [4], so I do not see scaling as a big issue, especially as these tests
> are such base concepts that if they need to be changed there is a
> completely new API, so the only overhead will be ensuring that nothing
> in tempest breaks the new tests (which is a good thing for trademark
> tests).
> 
> Personally, for me, I like option 3. I did not initially add it, as I
> knew it would cause endless bikesheding, but I do think it fits both
> a technical and social model.
> 
> I see 2 immediate routes forward:
> 
>  - Option 1, and we start adding these tests asap
>  - Pseudo Option 2, were we delete the resolution at [2] as it clearly
>does not apply anymore, and abandon the review at [1].
> 
> Finally - do not conflate my actions with those of the Designate team.
> I have seen people talking about how this resolution was leverage the
> team needed to move our tests in tree. This is definitely *not* true.
> Having our tests in a plugin is useful to us, and if the above
> resolution passed, I cannot see a situation where we would try to
> move any tests that were not listed in the interop standards.
> 
> This is something I have done as an individual in the community, not
> something the designate team have pushed for.

Thanks for pushing for a clear resolution to this, Graham.

> 
> 
> [4] -
> https://governance.openstack.org/tc/reference/tags/tc_approved-release.html
> 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

[openstack-dev] [cliff][osc][barbican][oslo][sdk][all] avoiding option name conflicts with cliff and OSC plugins

[Doug Hellmann]

We've been working this week to resolve an issue between cliff and
barbicanclient due to a collision in the option namespace [0].
Barbicanclient was using the -s option, and then cliff's lister
command base class added that option as an alias for sort-columns.

The first attempt at resolving the conflict was to set the conflict
handler in argparse to 'resolve' [1]. Several reviewers pointed out
that this would have the unwanted side-effect of making some OSC
commands support the -s as an alias for --sort-columns while the
barbicanclient commands would use it for a different purpose.

For now we have removed the -s alias from within cliff. However,
we want to avoid this problem coming up in the future so we want a
better solution.

The OSC project has a policy that its command plugins do not use
short options (single letter). There are relatively few of them,
and it's easy to introduce collisions.  Therefore, they are seen
as reserved for more common "global" options such as provided by
the base classes in OSC and cliff.

I propose that for Rocky we update cliff to change the way options
are registered so that conflicting options added by command subclasses
are ignored. This would effectively let cliff "own" the short option
namespace, and require command classes to use long option names.

The implementation details need to be worked out a bit, but I think
we can do that by subclassing ArgumentParser and adding a new
conflict handler method similar to the existing _handle_conflict_error()
and _handle_conflict_resolve().

This is going to introduce backwards-incompatible changes in the
commands derived from cliff, so before we take any action I wanted
to solicit input in the plan.

Please let me know what you think,
Doug

[0] https://bugs.launchpad.net/python-barbicanclient/+bug/1743578
[1] https://docs.python.org/3.5/library/argparse.html#conflict-handler

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zuul] requirements-check FAILURE

[Doug Hellmann]

Excerpts from Matthew Treinish's message of 2018-01-17 17:16:19 -0500:
> On Wed, Jan 17, 2018 at 10:01:13PM +, Kwan, Louie wrote:
> > Would like to add the following module to openstack.masakari project
> > 
> > https://github.com/pytransitions/transitions
> > 
> > Got the following error with zuul requirements-check
> > 
> > Requirement set([Requirement(package=u'transitions', location='', 
> > specifiers='>=0.6.4', markers=u'', comment='', extras=frozenset([]))]) not 
> > in openstack/requirements
> > 
> > http://logs.openstack.org/88/534888/3/check/requirements-check/edec7bf/ara/
> > 
> > Any tip or insight to fix it?
> 
> That error is caused by the dependency you're adding not being tracked in
> global requirements. To add it to the masakari project you first have to 
> add it to the openstack/requirements project.
> 
> The process for doing that is documented in:
> 
> https://docs.openstack.org/requirements/latest/
> 
> That link also explains the reasoning behind why we handle adding dependencies
> centrally like this.
> 
> -Matt Treinish

Please take a little time to look through the list of dependencies
in that repository to see if we have a finite state machine library
in the list already.  If so, see if you can use that one instead
of adding a new dependency to the system.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][ptl][goals][storyboard] tracking the rocky goals with storyboard

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-01-12 15:37:42 -0500:
> Since we are discussing goals for the Rocky cycle, I would like to
> propose a change to the way we track progress on the goals.
> 
> We've started to see lots and lots of changes to the goal documents,
> more than anticipated when we designed the system originally. That
> leads to code review churn within the governance repo, and it means
> the goal champions have to wait for the TC to review changes before
> they have complete tracking information published somewhere. We've
> talked about moving the tracking out of git and using an etherpad
> or a wiki page, but I propose that we use storyboard.
> 
> Specifically, I think we should create 1 story for each goal, and
> one task for each project within the goal. We can then use a board
> to track progress, with lanes like "New", "Acknowledged", "In
> Progress", "Completed", and "Not Applicable". It would be the
> responsibility of the goal champion to create the board, story, and
> tasks and provide links to the board and story in the goal document
> (so we only need 1 edit after the goal is approved). From that point
> on, teams and goal champions could collaborate on keeping the board
> up to date.
> 
> Not all projects are registered in storyboard, yet. Since that
> migration is itself a goal under discussion, I think for now we can
> just associate all tasks with the governance repository.
> 
> It doesn't look like changes to a board trigger any sort of
> notifications for the tasks or stories involved, but that's probably
> OK. If we really want notifications we can look at adding them as
> a feature of Storyboard at the board level.
> 
> How does this sound as an approach? Does anyone have any reservations
> about using storyboard this way?
> 
> Doug

Since the feedback has been positive, I wrote up the policy changes to
go along with this. Please continue any discussion of the idea over
there.

https://review.openstack.org/534443

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] storyboard evaluation

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2018-01-16 16:29:32 +:
> On 2018-01-16 06:55:03 -0800 (-0800), Emilien Macchi wrote:
> [...]
> > I created a dev instance of storyboard and imported all bugs from
> > TripleO so we could have a look at how it would be if we were using
> > the tool:
> [...]
> 
> Awesome! We do also have a https://storyboard-dev.openstack.org/
> deployment we can do test migrations into if you'd prefer something
> more central with which to play around.
> 
> > - how do we deal milestones in stories and also how can we have a
> > dashboard with an overview per milestone (useful for PTL + TripleO
> > release managers).
> 
> So far, the general suggestion for stuff like this is to settle on a
> consistent set of story tags to apply. It really depends on whether
> you're trying to track this at a story or task level (there is no
> per-task tagging implemented yet at any rate). I could imagine, for
> example, setting something like tripleo-r2 as a tag on stories whose
> TripleO deliverable tasks are targeting Rocky milestone #2, and then
> you could have an automatic board with stories matching that tag and
> lanes based on the story status.

That sounds like it might also be a useful way to approach the goal
tracking. Can someone point me to an example of how to set up an
automatic board like that?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] not run for PTL

[Doug Hellmann]

Excerpts from ChangBo Guo's message of 2018-01-16 18:55:26 +0800:
> Hi Oslo folks,
> 
> I taken the role of PTL in last 2 cycles, and would like to focus on coding
> this cycle.
>  it's time to let new leader to make oslo better.  So  I won't be running
> for PTL reelection for Rocky cycle.   Thanks all of your support and trust
> in last 2 cycles.
> 

Thank you for serving as PTL, gcb! The libraries have been quite stable
lately under your leadership.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs

[Doug Hellmann]

Excerpts from Erno Kuvaja's message of 2018-01-15 12:59:44 +:

> I think the worst case scenario is that we scrape together what ever
> we can just to have something to say that we test it and not have
> consistency nor clear responsibility of who, what and how.
> (Unfortunately I think this is the current situation and I'm super
> happy to hear that this is being discussed and the decision is not
> made lightly.)

That seems very far from the current situation to me.

We have a large integration test suite written primarily by
contributors to the projects it tests. A subset of that is used for
the trademark tests. That same subset is in 1 repo, managed by the
QA team, who apply the extra review criteria needed for the trademark
program to be stable.

The fact that so many people seem uninformed about how all of this
works is exactly why I think it's a mistake to spread the tests out
and have a bunch of different teams applying different review
criteria to them.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [pbr] support v_version

[Doug Hellmann]

Excerpts from Gaetan's message of 2018-01-15 19:29:01 +0100:
> >
> >
> > > I guess it somehow didn't used my build of the pbr package when
> > > running in gitlab pipeline.
> >
> > It sounds like the CE pipeline is not building packages in the same way?
> > Or is using an old version of pbr?
> >
> > I guess it was the pbr version from pypi.python.org, not my customized
> build
> published on our internal pypi
> https://books.sonatype.com/nexus-book/3.0/reference/pypi.html
> 
> > > Do you plan on releasing PBR soon on pypi?
> > > I have to build myself and push it on our internal nexus pypi, but I
> > think
> > > the
> > > safest way is to wait for an official pbr release on pypi.python.org :)
> >
> > Unfortunately we're approaching a significant set of feature freeze
> > deadlines for this OpenStack development cycle. Because it is very
> > difficult to "pin" a library like pbr that appears in the setup_requires
> > list for projects in our CI pipelines, and this next release of pbr
> > would have quite a few changes, we have decided in our meeting today
> > to be cautious and delay the next release for a few weeks.
> >
> Have you ever considered using pipenv/pipfile ? I started using it a few
> months ago it helps a lot with dependency freezing

We have a system in place to constrain most of the libraries we use
(using pip's -c option) but pip does not pay attention to that list in
all code paths when installing things listed in setup_requires.

> > I have scheduled a reminder to tag the pbr release during the first
> > week of March, after the OpenStack release is safely completed.
> >
> > I'm sorry if this causes inconvenience. We're going to work on
> > ensuring we have more frequent releases so we reduce our chance of
> > ending up in a similar situation again in the future.
> >
> 
> That is not a problem by itself, I still have this self-hosted Pypi
> repository
> in the meantime.

OK, good.

> To react on the IRL log, indeed, the proposal I make in this thread is
> purely optional,
> for my need, gitlab handle correctly the protection against unwanted
> publication.
> Just hope it will be useful for other project, probably not for OpenStack
> itself.
> 
> Thanks a lot for supporting external projects :)
> 
> Gaetan

Thanks for working with us to improve pbr! :-)

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [pbr] support v_version

[Doug Hellmann]

Excerpts from Gaetan's message of 2018-01-15 17:29:01 +0100:
> First, thanks a lot for your support and your kindness ! Really appreciate
> that :)
> 
> > > Do you know where I need to hack PBR to fix it?
> >
> > So 'pbr' correctly parses the prefixed tags, but it's just the output
> > packages (sdists, wheels) that always unversioned? If so, this sounds
> > correct. Python packaging, as defined in PEP-440 [1], doesn't use the
> > 'v' prefixes, so it doesn't really make sense to stick them in here.
> > Out of curiosity, what's your rationale for modifying the package name?
> >
> 
> The package name is not changed actually. With the patch in PBR that has
> been merged, one could add a tag named "v1.0.0" on mypackage package,
> and the sdist will generate a distribution package
> 
> mypackage-0.0.4.tar.gz
> 
> 
> So I think (hope?) this is still PEP440 compliant.
> 
> I tried this feature on another software that also uses pbr and there is no
> problem: v version works great with sdist/bdist/wheel packages.
> 
> I use it inside a Gitlab CE pipeline on a tag pipeline (CI is triggered when
> a git tag that follows the "v*" regular expression), and instead of
> creating
> a package
> 
> mypackage-0.0.4-py2.py3-none-any.whl
> 
> it created
> 
> mypackage-0.0.3.dev3-py2.py3-none-any.whl.
> 
> 
> When I retried manually on my development environment, pbr works
> perfectly again on the same code.
> I guess it somehow didn't used my build of the pbr package when
> running in gitlab pipeline.

It sounds like the CE pipeline is not building packages in the same way?
Or is using an old version of pbr?

> 
> Do you plan on releasing PBR soon on pypi?
> I have to build myself and push it on our internal nexus pypi, but I think
> the
> safest way is to wait for an official pbr release on pypi.python.org :)

Unfortunately we're approaching a significant set of feature freeze
deadlines for this OpenStack development cycle. Because it is very
difficult to "pin" a library like pbr that appears in the setup_requires
list for projects in our CI pipelines, and this next release of pbr
would have quite a few changes, we have decided in our meeting today
to be cautious and delay the next release for a few weeks.

I have scheduled a reminder to tag the pbr release during the first
week of March, after the OpenStack release is safely completed.

I'm sorry if this causes inconvenience. We're going to work on
ensuring we have more frequent releases so we reduce our chance of
ending up in a similar situation again in the future.

Doug

[1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2018-01-15.log.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [pbr] support v_version

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2018-01-15 14:40:14 +:
> On 2018-01-09 10:25:56 +0100 (+0100), Gaetan wrote:
> > I have submitted this patch ([1]) that add support for v_version
> > in PBR. Basically I can tag v1.0.0 instead of 1.0.0 to release
> > 1.0.0.
> [...]
> 
> Looks like the patch you linked has since merged. Any issues with it
> so far?
> 
> > Second point, to go to the end of the logic of my change, I would
> > like to propose an optional way (in setup.cfg?) to **prevent** any
> > tag without the 'v' prefix, ie, where a bare version tag like
> > `1.0.0` is not to be considered as a valid version.
> [...]
> 
> I'm not heavily involved in PBR maintenance so my ideas may be
> terrible, but have you considered just making it possible to set a
> version pattern option in setup.cfg so that this sort of filtering
> is more easily generalized?

That seems reasonable. It may even help solve the package filename
problem.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [storyboard] need help figuring out how to use auth with storyboard client

[Doug Hellmann]

Excerpts from Adam Coldrick's message of 2018-01-15 11:26:14 +:
> On Fri, 2018-01-12 at 21:30 +, Jeremy Stanley wrote:
> > On 2018-01-12 15:57:44 -0500 (-0500), Doug Hellmann wrote:
> > > The storyboard client docs mention an "access token" [1] as
> > > something
> > > a client needs in order to create stories and make other sorts of
> > > changes.  They don't explain what that token is or how to get one,
> > > though.
> > > 
> > > Where do I get a token? How long does the token work? Can I safely
> > > put a token in a configuration file, or do I need to get a new one
> > > each time I want to do something with the client?
> > 
> > https://docs.openstack.org/infra/storyboard/webapi/v1.html#api
> > suggests that logging in and going to
> > https://storyboard.openstack.org/#!/profile/tokens will allow you to
> > issue one (with up to a 10-year expiration based on my modest
> > experimentation). I believe this to be the same solution we're using
> > to grant teh storyboard-its Gerrit plugin to update tasks/stories
> > from review.openstack.org.
> 
> This is likely the easiest solution. Some other options:
> 
> - Admin users can issue tokens for any users, so an automation user
>   could have a token granted by infra-root using the API directly (see
>   the API docs[0] for detail).

The script I'm thinking of would create the story, tasks, and board
associated with a community goal. So it could be run by a goal champion
on their local system, and wouldn't need a special user to own the
results.

> - Add some functionality in python-storyboardclient to handle
>   authenticating with the OpenID provider that the API sends a
>   redirect link for.

That would be useful, although having directions for getting a token
manually is probably fine, for this case.

> 
> [0]: https://docs.openstack.org/infra/storyboard/webapi/v1.html#post--v
> 1-users--user_id--tokens
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] proposing Stephen Finucan for oslo-core

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-01-08 09:55:26 -0500:
> Stephen (sfinucan) has been working on pbr, oslo.config, and
> oslo.policy and reviewing several of the other Oslo libraries for
> a while now. His reviews are always helpful and I think he would
> make a good addition to the oslo-core team.
> 
> As per our usual practice, please reply here with a +1 or -1 and
> any reservations.
> 
> Doug

As we have gone a week with no objections, I added Stephen to the
oslo-core group this morning.

Stephen, welcome to the team!

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [storyboard] need help figuring out how to use auth with storyboard client

[Doug Hellmann]

The storyboard client docs mention an "access token" [1] as something
a client needs in order to create stories and make other sorts of
changes.  They don't explain what that token is or how to get one,
though.

Where do I get a token? How long does the token work? Can I safely
put a token in a configuration file, or do I need to get a new one
each time I want to do something with the client?

Doug

[1] https://docs.openstack.org/infra/python-storyboardclient/usage.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tc][ptl][goals][storyboard] tracking the rocky goals with storyboard

[Doug Hellmann]

Since we are discussing goals for the Rocky cycle, I would like to
propose a change to the way we track progress on the goals.

We've started to see lots and lots of changes to the goal documents,
more than anticipated when we designed the system originally. That
leads to code review churn within the governance repo, and it means
the goal champions have to wait for the TC to review changes before
they have complete tracking information published somewhere. We've
talked about moving the tracking out of git and using an etherpad
or a wiki page, but I propose that we use storyboard.

Specifically, I think we should create 1 story for each goal, and
one task for each project within the goal. We can then use a board
to track progress, with lanes like "New", "Acknowledged", "In
Progress", "Completed", and "Not Applicable". It would be the
responsibility of the goal champion to create the board, story, and
tasks and provide links to the board and story in the goal document
(so we only need 1 edit after the goal is approved). From that point
on, teams and goal champions could collaborate on keeping the board
up to date.

Not all projects are registered in storyboard, yet. Since that
migration is itself a goal under discussion, I think for now we can
just associate all tasks with the governance repository.

It doesn't look like changes to a board trigger any sort of
notifications for the tasks or stories involved, but that's probably
OK. If we really want notifications we can look at adding them as
a feature of Storyboard at the board level.

How does this sound as an approach? Does anyone have any reservations
about using storyboard this way?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Saverio Proto's message of 2018-01-12 09:17:55 +0100:
> > Which service's logs are missing the request_id?
> > 
> If I look at neutron-server logs with my current setup I get two files:
> 
> neutron-server.log
> neutron-server.json
> 
> the standard log file has in all neutron.wsgi lines information like:
> 
>  neutron.wsgi [req-4fda8017-50c7-40eb-9e7b-710e7fba0d01
> 97d349b9499b4bd29c5e167c65ca1fb3 d447c836b6934dfab41a03f1ff96d879 - - -]
> 
> where req-UUID is the request ID, and the other two values are the user
> UUID and the keystone project UUID.
> 
> when I look at the same line in the JSON output this information is missing.
> 
> I am starting neutron-server with the command line option
> --log-config-append=/etc/neutron/logging.neutron-server.conf
> 
> where the conf file looks like
> 
> [loggers]
> keys = root, neutron
> 
> [handlers]
> keys = logfile, jsonfile, null
> 
> [formatters]
> keys = context, json, default
> 
> [logger_root]
> level = WARNING
> handlers = null
> 
> [logger_neutron]
> level = INFO
> handlers = logfile, jsonfile
> qualname = neutron
> 
> [handler_logfile]
> class = handlers.WatchedFileHandler
> args = ('/var/log/neutron/neutron-server.log',)
> formatter = context
> 
> [handler_jsonfile]
> level = INFO
> class = handlers.WatchedFileHandler
> args = ('/var/log/neutron/neutron-server.json',)
> formatter = json
> 
> [handler_null]
> class = logging.NullHandler
> formatter = default
> args = ()
> 
> [formatter_context]
> class = oslo_log.formatters.ContextFormatter
> 
> [formatter_json]
> class = oslo_log.formatters.JSONFormatter
> 
> [formatter_default]
> format = %(message)s
> 
> 
> I had a look at nova-api and I have the same problem. Anyway in my
> Kibana I never so a req-UUID what so ever, so this looks like a problem
> with all the openstack services.
> 
> Is it a problem with my logging configuration ?
> 
> thank you
> 
> Saverio
> 

I don't think this is a configuration problem.

Which version of the oslo.log library do you have installed?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Clarifying testing recommendation for interop programs

[Doug Hellmann]

Excerpts from Luigi Toscano's message of 2018-01-12 10:49:55 +0100:
> On Thursday, 11 January 2018 23:52:00 CET Matt Riedemann wrote:
> > On 1/11/2018 10:36 AM, Colleen Murphy wrote:
> > > 1) All trademark-related tests should go in the tempest repo, in
> > > accordance
> > > 
> > > with the original resolution. This would mean that even projects that
> > > have
> > > never had tests in tempest would now have to add at least some of
> > > their
> > > black-box tests to tempest.
> > > 
> > > The value of this option is that centralizes tests used for the Interop
> > > program in a location where interop-minded folks from the QA team can
> > > control them. The downside is that projects that so far have avoided
> > > having a dependency on tempest will now lose some control over the
> > > black-box tests that they use for functional and integration that would
> > > now also be used for trademark certification.
> > > There's also concern for the review bandwidth of the QA team - we can't
> > > expect the QA team to be continually responsible for an ever-growing list
> > > of projects and their trademark tests.
> > 
> > How many tests are we talking about for designate and heat? Half a
> > dozen? A dozen? More?
> > 
> > If it's just a couple of tests per project it doesn't seem terrible to
> > have them live in Tempest so you get the "interop eye" on reviews, as
> > noted in your email. If it's a considerable amount, then option 2 seems
> > the best for the majority of parties.
> 
> I would argue that it does not scale; what if some test is taken out from the 
> interoperability, and others are added? It would mean moving tests from one 
> repository to another, with change of paths. I think that the solution 2, 
> where the repository where a test belong and the functionality of a test are 
> not linked, is better.
> 
> Ciao

How often do the interop test suites change in that way?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log] JSON logs are missing the request ID

[Doug Hellmann]

Excerpts from Saverio Proto's message of 2018-01-11 15:23:46 +0100:
> Hello,
> 
> we recently enabled the JSON logging to feed a Kibana dashboard and look
> at the logs with modern tooling.
> 
> however it looks like in our Openstack Newton deployment that some
> information in the JSON files is missing.
> 
> most important missing bit is the request-id, that we use to track an
> event across multiple log files on multiple hosts.
> 
> Looking at the code it really looks like the request ID is there for the
> context formatter and not for the json formatter.
> 
> https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L208
> 
> https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L460
> 
> I am an operator and a very bad python developer, so can anyone confirm
> that is really missing in the code, and it is not me configuring stuff
> wrongly ?
> 
> If it is really missing the request-id in the json log formatter, should
> I open a bug about this ?
> 
> thank you
> 
> Saverio
> 

The newton version of the JSONFormatter adds all of the values from the
context to the log record:

http://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/formatters.py?h=newton-eol#n142

That should include the request_id.

Which service's logs are missing the request_id?

Chaining request_id values from one service to the next was a separate
piece of work, and I don't remember off the top of my head when that was
added. Perhaps someone else can. I think Sean Dague drove a lot of that
work.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] Retiring ceilometerclient

[Doug Hellmann]

Excerpts from Monty Taylor's message of 2018-01-10 18:08:52 -0600:
> On 01/10/2018 05:44 PM, Doug Hellmann wrote:
> > Excerpts from Monty Taylor's message of 2018-01-10 17:40:28 -0600:
> >> On 01/10/2018 04:10 PM, Jon Schlueter wrote:
> >>> On Thu, Nov 23, 2017 at 4:12 PM, gordon chung <g...@live.ca> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2017-11-22 04:18 AM, Julien Danjou wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Now that the Ceilometer API is gone, we really don't need
> >>>>> ceilometerclient anymore. I've proposed a set of patches to retire it:
> >>>>>
> >>>>>  https://review.openstack.org/#/c/522183/
> >>>>>
> >>>
> >>>
> >>> So my question here is are we missing a process check for retiring a
> >>> project that is still in
> >>> the requirements of several other OpenStack projects?
> >>>
> >>> I went poking around and found that rally [4], heat [1], aodh [3] and
> >>> mistral [2] still had references to
> >>> ceilometerclient in the RPM packaging in RDO Queens, and on digging a
> >>> bit more they
> >>> were still in the requirements for at least those 4 projects.
> >>>
> >>> I would think that a discussion around retiring a project should also
> >>> include at least enumerating
> >>> which projects are currently consuming it [5].  That way a little bit
> >>> of pressure on those consumers
> >>> can be exerted to evaluate their usage of an about to be retired
> >>> project.  It shouldn't stop the
> >>> discussions around retiring a project just a data point for decision 
> >>> making.
> >>
> >> It's worth pointing out that openstacksdk has ceilometer REST API
> >> support in it, although it is special-cased since ceilometer was retired
> >> before we even made the service-types-authority:
> >>
> >> http://git.openstack.org/cgit/openstack/python-openstacksdk/tree/openstack/connection.py#n234
> 
> Whoops, that's not ceilometer - that's gnocchi I think?
> 
> ceilometer support *does* have a service-types-authority reference so 
> *isn't* special-cased and is here:
> 
> http://git.openstack.org/cgit/openstack/python-openstacksdk/tree/openstack/meter
> 
> >> We can either keep it there indefinitely (there is no cost to keeping
> >> it, other than that one "self._load('metric')" line) - or we could take
> >> this opportunity to purge it from sdk as well.
> >>
> >> BUT - if we're going to remove it from SDK I'd rather we do it in the
> >> very-near-future because we're getting closer to a 1.0 for SDK and once
> >> that happens if ceilometer is still there ceilometer support will remain
> >> until the end of recorded history.
> >>
> >> We could keep it and migrate the heat/mistral/rally/aodh
> >> ceilometerclient uses to be SDK uses (although heaven knows how we test
> >> that without a ceilometer in devstack)
> >>
> >> I honestly do not have a strong opinion in either direction and welcome
> >> input on what people would like to see done.
> >>
> >> Monty
> >>
> > 
> > If ceilometer itself is deprecated, do we need to maintain support
> > in any of our tools?
> 
> We do not - although if we had had ceilometer support in shade I would 
> be very adamant that we continue to support it to the best of our 
> ability for forever, since you never know who out there is running on an 
> old cloud that still has it.
> 
> This is why I could go either way personally from an SDK perspective - 
> we don't have a 1.0 release of SDK yet, so if we do think it's best to 
> just clean house, now's the time.
> 

I favor dropping support in the SDK. I'm not sure what that means
for the service projects that seem to be using it, though. Do they
actually need it?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] Retiring ceilometerclient

[Doug Hellmann]

Excerpts from Monty Taylor's message of 2018-01-10 17:40:28 -0600:
> On 01/10/2018 04:10 PM, Jon Schlueter wrote:
> > On Thu, Nov 23, 2017 at 4:12 PM, gordon chung  wrote:
> >>
> >>
> >>
> >> On 2017-11-22 04:18 AM, Julien Danjou wrote:
> >>> Hi,
> >>>
> >>> Now that the Ceilometer API is gone, we really don't need
> >>> ceilometerclient anymore. I've proposed a set of patches to retire it:
> >>>
> >>> https://review.openstack.org/#/c/522183/
> >>>
> > 
> > 
> > So my question here is are we missing a process check for retiring a
> > project that is still in
> > the requirements of several other OpenStack projects?
> > 
> > I went poking around and found that rally [4], heat [1], aodh [3] and
> > mistral [2] still had references to
> > ceilometerclient in the RPM packaging in RDO Queens, and on digging a
> > bit more they
> > were still in the requirements for at least those 4 projects.
> > 
> > I would think that a discussion around retiring a project should also
> > include at least enumerating
> > which projects are currently consuming it [5].  That way a little bit
> > of pressure on those consumers
> > can be exerted to evaluate their usage of an about to be retired
> > project.  It shouldn't stop the
> > discussions around retiring a project just a data point for decision making.
> 
> It's worth pointing out that openstacksdk has ceilometer REST API 
> support in it, although it is special-cased since ceilometer was retired 
> before we even made the service-types-authority:
> 
> http://git.openstack.org/cgit/openstack/python-openstacksdk/tree/openstack/connection.py#n234
> 
> We can either keep it there indefinitely (there is no cost to keeping 
> it, other than that one "self._load('metric')" line) - or we could take 
> this opportunity to purge it from sdk as well.
> 
> BUT - if we're going to remove it from SDK I'd rather we do it in the 
> very-near-future because we're getting closer to a 1.0 for SDK and once 
> that happens if ceilometer is still there ceilometer support will remain 
> until the end of recorded history.
> 
> We could keep it and migrate the heat/mistral/rally/aodh 
> ceilometerclient uses to be SDK uses (although heaven knows how we test 
> that without a ceilometer in devstack)
> 
> I honestly do not have a strong opinion in either direction and welcome 
> input on what people would like to see done.
> 
> Monty
> 

If ceilometer itself is deprecated, do we need to maintain support
in any of our tools?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] Retiring ceilometerclient

[Doug Hellmann]

Excerpts from gordon chung's message of 2018-01-10 23:28:05 +:
> 
> On 2018-01-10 05:10 PM, Jon Schlueter wrote:
> > I would think that a discussion around retiring a project should also
> > include at least enumerating
> > which projects are currently consuming it [5].  That way a little bit
> > of pressure on those consumers
> > can be exerted to evaluate their usage of an about to be retired
> > project.  It shouldn't stop the
> > discussions around retiring a project just a data point for decision making.
> 
> this is a very valid point. this is something overlooked on my part.
> 
> out of curiosity, but what's the effect have 'retiring' something in 
> openstack and having services still referencing ceilometerclient? is it 
> that it will not get packaged in centos/ubuntu and therefore will be 
> missing requirements when installed? or can you not build the package at 
> all?
> 
> cheers,
> 

The python-ceilometer client is empty now except for a README file
explaining that the project is retired. So if there's a bug in the
library, there's no convenient way for anyone to fix it.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][release][nova][infra] Release of openstack/nova failed

[Doug Hellmann]

Excerpts from zuul's message of 2018-01-10 01:38:45 +:
> Build failed.
> 
> - release-openstack-python-without-pypi 
> http://logs.openstack.org/d6/d6ce901fe280004edbe2f27d8af374ff905161d6/release/release-openstack-python-without-pypi/10069b4/
>  : FAILURE in 4m 23s
> - announce-release announce-release : SKIPPED
> 

The failure from [1] is during the step where the "sibling" packages are
installed and says "pip python module is required".

Do we need to update a bindep file somewhere?

[1] 
http://logs.openstack.org/d6/d6ce901fe280004edbe2f27d8af374ff905161d6/release/release-openstack-python-without-pypi/10069b4/ara/result/eead3240-2a6f-40ba-8f59-9d80bc408603/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][oslo.log]Re: Error will be occurred if watch_log_file option is true

[Doug Hellmann]

Excerpts from Rikimaru Honjo's message of 2018-01-09 18:11:09 +0900:
> Hello,
> 
> On 2018/01/04 23:12, Doug Hellmann wrote:
> > Excerpts from Rikimaru Honjo's message of 2018-01-04 18:22:26 +0900:
> >> Hello,
> >>
> >> The below bug was reported in Masakari's Launchpad.
> >> I think that this bug was caused by oslo.log.
> >> (And, the root cause is a bug of pyinotify using by oslo.log. The detail is
> >> written in the bug report.)
> >>
> >> * masakari-api failed to launch due to setting of watch_log_file and 
> >> log_file
> >> https://bugs.launchpad.net/masakari/+bug/1740111
> >>
> >> There is a possibility that this bug will affects all openstack components 
> >> using oslo.log.
> >> (But, the processes working with uwsgi[1] wasn't affected when I tried to 
> >> reproduce.
> >> I haven't solved the reason of this yet...)
> >>
> >> Could you help us?
> >> And, what should we do...?
> >>
> >> [1]
> >> e.g. nova-api, cinder-api, keystone...
> >>
> >> Best regards,
> > 
> > The bug is in pyinotify. According to the git repo [1] that project
> > was last updated in June of 2015.  I recommend we move off of
> > pyinotify entirely, since it appears to be unmaintained.
> > 
> > If there is another library to do the same thing we should switch
> > to it (there seem to be lots of options [2]). If there is no viable
> > replacement or fork, we should deprecate that log watching feature
> > (and anything else for which we use pyinotify) and remove it ASAP.
> > 
> > We'll need a volunteer to do the evaluation and update oslo.log.
> > 
> > Doug
> > 
> > [1] https://github.com/seb-m/pyinotify
> > [2] https://pypi.python.org/pypi?%3Aaction=search=inotify=search
> Thank you for replying.
> 
> I haven't deeply researched, but inotify looks good.
> Because "weight" of inotify is the largest, and following text is described.
> 
> https://pypi.python.org/pypi/inotify/0.2.9
> > This project is unrelated to the *PyInotify* project that existed prior to 
> > this one (this project began in 2015). That project is defunct and no 
> > longer available.
> PyInotify is defunct and no longer available...
> 

The inotify package seems like a good candidate to replace pyinotify.

Have you looked at how hard it would be to change oslo.log? If so, does
using the newer library eliminate the bug you had?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][requirements][cinder] Handling requirements for driverfixes branches

[Doug Hellmann]

Excerpts from Eric Harney's message of 2018-01-08 14:05:35 -0500:
> Hi all,
> 
> I'm trying to sort out how to run unit tests on Cinder driverfixes branches.
> 
> These branches are similar to stable branches, but live longer (and have
> a different set of rules for what changes are appropriate).
> 
> In order for unit tests to work on these branches, requirements need to
> be pinned in the same way they are for stable branches (i.e.
> driverfixes/ocata matches stable/ocata's requirements).  Currently, unit
> test jobs on these branches end up using requirements from master.
> 
> It is not clear how I can pin requirements on these branches, since they
> aren't recognized as equivalent to stable branches by any of the normal
> tooling used in CI.  I tried manually adding an upper-constraints.txt
> here [1] but this does not result in the correct dependencies being used.
> 
> Where do changes need to be made for us to set the
> requirements/upper-constraints correctly for these branches?
> 
> 
> [1] https://review.openstack.org/#/c/503711/
> 
> Thanks,
> Eric
> 

You'll want to just update the UPPER_CONSTRAINTS_FILE setting in tox.ini
to point to the one for the relevant stable branch. If the branch no
longer exists, you should be able to refer to the version of the file
using the $release-eol tag instead of the branch name.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [reno] questions about reno

[Doug Hellmann]

Excerpts from Gaetan's message of 2018-01-08 17:59:55 +0100:
> Hello
> 
> Thanks for your answers!
> 
> - unreleased notes appear in release note with a title such as "0.1.0-2".
> > > Is it possible to not have any title or use "0.1.0-dev2" pattern like
> > pbr ?
> >
> > I'm not sure why it matters, but if you want to work on that patch I'll
> > help with reviews.
> >
> 
> Ok, I'll see what I can do :)
> 
> >
> > > - I guess that all notes should stay in the same folder version after
> > > versions, and the
> > > release notes of all versions will keep being automatically generated.
> > > Don't you think
> > > it might get difficult to manage all theses files? Is is possible to move
> > > them in different folder (at least a folder "archives?)
> >
> > We've put off doing anything like that until we have a project with
> > enough notes that we can observe the problems and decide how to fix
> > them. Have you already reached that point or are you anticipating
> > problems in the future?
> >
> 
> No, just started using reno and just see that this folder might get messy
> quickly. Maybe I over-think this, I agree with you to observe first
> 
> >
> > > - it is possible to generate the NEWS file using reno ? I started trying
> > > conversion with pandoc but the result are not great.
> >
> > How is the NEWS file different from CHANGES.txt that pbr produces? Is it
> > the format, or the content?
> >
> 
> So, I like PBR that generates ChangeLog from the git history, but it has
> lot of details (maybe too much). So, I was thinking to store in NEWS only
> the release note
> As an example, you can look how I plan to use it for Guake:
> https://github.com/Guake/guake
> I usually write the NEWS at each release manually (
> https://github.com/Guake/guake/blob/master/NEWS), and that's where reno
> shines in my eyes :)

That looks similar to the output of the report command, although maybe
with less detail.

I can see a couple of ways to do this.

1. Use the report format as it is now.

2. Add a section to the note file to include a "highlight" or "news"
   entry that is ignored most of the time but can be used to produce
   summaries like this.

3. Try to somehow derive a summary from the text in the notes
   automatically.

Option 1 might work, although maybe you would end up with more detail
than you really want? If you were able to go this route, you could take
advantage of our plans to include release notes in source distributions
automatically (so you wouldn't even need to check the file into git).

Option 2 is do-able, but I'm a little concerned that having "magic"
sections makes the processing of the input files more complicated so
I'll have to think about whether it's really a good idea or not.

Option 3 sounds relatively hard, given that release notes just need to
be valid restructuredtext (meaning they don't need to be a list or other
structure that would be easy to take the "first" part of.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] proposing Stephen Finucan for oslo-core

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2018-01-08 09:55:26 -0500:
> Stephen (sfinucan) has been working on pbr, oslo.config, and

Oops, that's stephenfin. Sorry for the confusion.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev