[openstack-dev] [release][ptl] new "linter" rules for openstack/releases repository

[Doug Hellmann]

The release team is working on a series of patches to improve the
data validation within openstack/releases. Part of that work is to
apply consistent formatting rules for all of the YAML files, so it
is easier to build tools that automatically update those files. To
enable those consistent rules, we have had to "normalize" the use
of whitespace in a bunch of the existing files.

These changes mean that if you have built your own automation for
adding new releases, you might have to make adjustments. If you do,
please take the time to look at the tools within the repo (in
particular the new-release, interactive-release, and edit-deliverable
commands) to see if they meet your needs, or can be extended to do
so. There's not much point in all of us building our own tools. :-)

If you're curious about the actual changes, you can have a look at the
patch series for "queens-indentation" at
https://review.openstack.org/#/q/project:openstack/releases+topic:queens-indentation

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][ptl][docs] adding redirects for moved pages

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-08-01 12:29:38 -0400:
> When we started the doc-migration projects we knew that moving all
> of the content around would break a lot of external links. We said
> we would deal with it if we had time at the end of the cycle. We
> have put the changes into place to allow us to do that now. I'm
> going to explain it here, and answer questions in this thread, and
> then take the results and put them into the docs contributor guide.
> 
> 1. Add a .htaccess file to your repo.
> 
> The first step is to add the configuration file Apache needs to know
> what the redirect rules are, .htaccess. We have a global file in the
> openstack-manuals repository but in the spirit of not making the docs
> team a bottleneck for doc-related changes we have also configured
> Apache to allow a .htaccess file in each project's documentation.

I should add that we *only* allow Redirect and RedirectMatch rules in
these files, and that we are relying on review teams to avoid adding
rules that break access to their docs or redirect off-site. Reviewers,
please, be careful.

Doug

> 
> Sphinx needs to be told to include the file in the build output by
> adding it to the list of "extra" files. This patch for nova shows
> how that is done by editing doc/source/conf.py to set html_extra_path:
> https://review.openstack.org/#/c/487932/5/doc/source/conf.py
> 
> If that path is set to '_extras' then the patch should also create
> doc/source/_extras/.htaccess containing the redirects needed. The
> contents of that file can be written by hand, or computed with a command
> like:
> 
> git log --follow --name-status \
> --format='%H' origin/stable/ocata.. \
> -- doc/source | \
> grep ^R | \
> grep .rst | \
> cut -f2- | \
> sed -e 's|doc/source/|^/nova/([^/]+)/|' \
> -e 's|doc/source/|/nova/$1/|' \
> -e 's/.rst/.html$/' \
> -e 's/.rst/.html/' \
> -e 's/^/redirectmatch 301 /'
> 
> Obviously you need to replace the 2 references to "nova" with the base
> name of your git repository. The output will look something like:
> 
> redirectmatch 301 ^/nova/([^/]+)/aggregates.html$ 
> /nova/$1/user/aggregates.html
> redirectmatch 301 ^/nova/([^/]+)/architecture.html$ 
> /nova/$1/user/architecture.html
> redirectmatch 301 ^/nova/([^/]+)/block_device_mapping.html$ 
> /nova/$1/user/block-device-mapping.html
> redirectmatch 301 ^/nova/([^/]+)/cells.html$ /nova/$1/user/cells.html
> redirectmatch 301 ^/nova/([^/]+)/conductor.html$ /nova/$1/user/conductor.html
> redirectmatch 301 ^/nova/([^/]+)/feature_classification.html$ 
> /nova/$1/user/feature-classification.html
> redirectmatch 301 ^/nova/([^/]+)/filter_scheduler.html$ 
> /nova/$1/user/filter-scheduler.html
> redirectmatch 301 ^/nova/([^/]+)/placement.html$ /nova/$1/user/placement.html
> 
> Adding the resulting file to your repository will enable rules to
> redirect from paths like /nova/latest/aggregates.html to
> /nova/latest/user/aggregates.html.
> 
> 2. Enable detailed redirects for your project.
> 
> The other major portion of the move that we made was to take
> everything that was under /developer/$project/ and move it to
> /$project/latest/ (with similar moves for other versions). By
> default, any page under /developer/$project/ is now being redirected
> to /$project/latest/ to at least give the user a table of contents
> to find the new page. After a local .htaccess file is added to a
> projects documentation we can allow /developer/$project/(.*) to
> redirect to /$project/latest/$1 which will then redirect *again*
> to the new home of the file.
> 
> To turn that feature on for your repository, set the has_in_tree_htaccess
> flag for the repo by modifying www/project-data/latest.yaml in the
> openstack-manuals repository. See
> https://docs.openstack.org/contributor-guide/doc-tools/template-generator.html
> for details about the other flags you can set to control how your
> project appears on docs.o.o.
> 
> After the has_in_tree_htaccess flag change lands, links to URLs
> like docs.openstack.org/developer/nova/cells.html should (with 2
> redirects) end up at the new home
> docs.openstack.org/nova/latest/user/cells.html.
> 
> Let me know if you have any questions or run into any issues making
> these changes.
> 
> Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][ptl][docs] adding redirects for moved pages

[Doug Hellmann]

When we started the doc-migration projects we knew that moving all
of the content around would break a lot of external links. We said
we would deal with it if we had time at the end of the cycle. We
have put the changes into place to allow us to do that now. I'm
going to explain it here, and answer questions in this thread, and
then take the results and put them into the docs contributor guide.

1. Add a .htaccess file to your repo.

The first step is to add the configuration file Apache needs to know
what the redirect rules are, .htaccess. We have a global file in the
openstack-manuals repository but in the spirit of not making the docs
team a bottleneck for doc-related changes we have also configured
Apache to allow a .htaccess file in each project's documentation.

Sphinx needs to be told to include the file in the build output by
adding it to the list of "extra" files. This patch for nova shows
how that is done by editing doc/source/conf.py to set html_extra_path:
https://review.openstack.org/#/c/487932/5/doc/source/conf.py

If that path is set to '_extras' then the patch should also create
doc/source/_extras/.htaccess containing the redirects needed. The
contents of that file can be written by hand, or computed with a command
like:

git log --follow --name-status \
--format='%H' origin/stable/ocata.. \
-- doc/source | \
grep ^R | \
grep .rst | \
cut -f2- | \
sed -e 's|doc/source/|^/nova/([^/]+)/|' \
-e 's|doc/source/|/nova/$1/|' \
-e 's/.rst/.html$/' \
-e 's/.rst/.html/' \
-e 's/^/redirectmatch 301 /'

Obviously you need to replace the 2 references to "nova" with the base
name of your git repository. The output will look something like:

redirectmatch 301 ^/nova/([^/]+)/aggregates.html$ /nova/$1/user/aggregates.html
redirectmatch 301 ^/nova/([^/]+)/architecture.html$ 
/nova/$1/user/architecture.html
redirectmatch 301 ^/nova/([^/]+)/block_device_mapping.html$ 
/nova/$1/user/block-device-mapping.html
redirectmatch 301 ^/nova/([^/]+)/cells.html$ /nova/$1/user/cells.html
redirectmatch 301 ^/nova/([^/]+)/conductor.html$ /nova/$1/user/conductor.html
redirectmatch 301 ^/nova/([^/]+)/feature_classification.html$ 
/nova/$1/user/feature-classification.html
redirectmatch 301 ^/nova/([^/]+)/filter_scheduler.html$ 
/nova/$1/user/filter-scheduler.html
redirectmatch 301 ^/nova/([^/]+)/placement.html$ /nova/$1/user/placement.html

Adding the resulting file to your repository will enable rules to
redirect from paths like /nova/latest/aggregates.html to
/nova/latest/user/aggregates.html.

2. Enable detailed redirects for your project.

The other major portion of the move that we made was to take
everything that was under /developer/$project/ and move it to
/$project/latest/ (with similar moves for other versions). By
default, any page under /developer/$project/ is now being redirected
to /$project/latest/ to at least give the user a table of contents
to find the new page. After a local .htaccess file is added to a
projects documentation we can allow /developer/$project/(.*) to
redirect to /$project/latest/$1 which will then redirect *again*
to the new home of the file.

To turn that feature on for your repository, set the has_in_tree_htaccess
flag for the repo by modifying www/project-data/latest.yaml in the
openstack-manuals repository. See
https://docs.openstack.org/contributor-guide/doc-tools/template-generator.html
for details about the other flags you can set to control how your
project appears on docs.o.o.

After the has_in_tree_htaccess flag change lands, links to URLs
like docs.openstack.org/developer/nova/cells.html should (with 2
redirects) end up at the new home
docs.openstack.org/nova/latest/user/cells.html.

Let me know if you have any questions or run into any issues making
these changes.

Doug


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-07-31 10:02:19 -0400:
> Excerpts from Doug Hellmann's message of 2017-07-27 19:10:32 -0400:
> > Excerpts from Doug Hellmann's message of 2017-07-27 19:05:16 -0400:
> > > Excerpts from Jeremy Stanley's message of 2017-07-27 22:07:33 +:
> > > > On 2017-07-27 15:40:22 -0400 (-0400), Doug Hellmann wrote:
> > > > [...]
> > > > > Are we over-emphasizing the scale of the discovery problem?
> > > > > 
> > > > > When I search for how to install a package on Ubuntu (or Red Hat
> > > > > or Debian for that matter), I find all sorts of references all over
> > > > > the web (including on the sites for those distros) and I have to
> > > > > sift through it all to find right command or package name to use
> > > > > for my version.  Usually the easiest way to do that is to put the
> > > > > version in the search query.
> > > > > 
> > > > > Are our users incapable of finding the right version of a document
> > > > > if we clearly mark the version to which each document applies? Every
> > > > > URL now has a release series name or version tag in the path. The
> > > > > docs theme inserts the version number into every page as well. Is
> > > > > that sufficient to help a reader understand what version the info
> > > > > they're view is for?
> > > > > 
> > > > > That's not to say we shouldn't do some work to make newer docs easy
> > > > > to find. If we can modify the sitemap to make them appear earlier
> > > > > in search results, that's good. The docs theme allows a project to
> > > > > include links to several older versions to switch between them,
> > > > > too, so users who land on the "wrong" version can switch. We could
> > > > > update that to include branches as well as tags, so that someone
> > > > > can easily move to the series they need without knowing the version
> > > > > number.
> > > > [...]
> > > > 
> > > > The biggest liability is people not realizing there are multiple
> > > > "versions" of OpenStack finding an old install doc and using it
> > > > because they don't know it's out of date, then seeking support
> > > > upstream or just generally contributing to the number of deployments
> > > > unnecessarily stuck on obsolete software. The current solution of
> > > > not publishing installation guides for EOL releases seems like a
> > > > good enough compromise there to me.
> > > 
> > > That content will now live in the project trees. Perhaps part of marking
> > > branches in those repos EOL needs to include deleting the install tree
> > > from the docs? Now that the docs are in a standard location, that could
> > > be part of an EOL script (although it means 2 phases, since we have to
> > > land the patch and let the docs rebuild before we remove the branch).
> > > 
> > > We could also update the openstack-manuals tree to not publish links
> > > to the install guides (either removing the page or replacing it
> > > with a placeholder that explains they should be trying to use a
> > > newer version).
> > > 
> > > Doug
> > > 
> > 
> > Today we're publishing series-specific (e.g., newton) and
> > version-specific (e.g., 10.0.0) docs. I wonder if we should stop
> > publishing docs when we tag repositories, and just stick to the
> > series? There's no way to go back and update those version-specific
> > builds after the fact, so we can't remove the installation guides
> > and we can't rebuild them easily if there are security issues with
> > the content.
> > 
> > Doug
> > 
> 
> I've proposed https://review.openstack.org/489231 to update
> project-config to stop publishing docs when we tag releases.
> 
> Doug
> 

And here's the theme change to show series names instead of version
numbers:

https://review.openstack.org/489252

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-07-27 19:10:32 -0400:
> Excerpts from Doug Hellmann's message of 2017-07-27 19:05:16 -0400:
> > Excerpts from Jeremy Stanley's message of 2017-07-27 22:07:33 +:
> > > On 2017-07-27 15:40:22 -0400 (-0400), Doug Hellmann wrote:
> > > [...]
> > > > Are we over-emphasizing the scale of the discovery problem?
> > > > 
> > > > When I search for how to install a package on Ubuntu (or Red Hat
> > > > or Debian for that matter), I find all sorts of references all over
> > > > the web (including on the sites for those distros) and I have to
> > > > sift through it all to find right command or package name to use
> > > > for my version.  Usually the easiest way to do that is to put the
> > > > version in the search query.
> > > > 
> > > > Are our users incapable of finding the right version of a document
> > > > if we clearly mark the version to which each document applies? Every
> > > > URL now has a release series name or version tag in the path. The
> > > > docs theme inserts the version number into every page as well. Is
> > > > that sufficient to help a reader understand what version the info
> > > > they're view is for?
> > > > 
> > > > That's not to say we shouldn't do some work to make newer docs easy
> > > > to find. If we can modify the sitemap to make them appear earlier
> > > > in search results, that's good. The docs theme allows a project to
> > > > include links to several older versions to switch between them,
> > > > too, so users who land on the "wrong" version can switch. We could
> > > > update that to include branches as well as tags, so that someone
> > > > can easily move to the series they need without knowing the version
> > > > number.
> > > [...]
> > > 
> > > The biggest liability is people not realizing there are multiple
> > > "versions" of OpenStack finding an old install doc and using it
> > > because they don't know it's out of date, then seeking support
> > > upstream or just generally contributing to the number of deployments
> > > unnecessarily stuck on obsolete software. The current solution of
> > > not publishing installation guides for EOL releases seems like a
> > > good enough compromise there to me.
> > 
> > That content will now live in the project trees. Perhaps part of marking
> > branches in those repos EOL needs to include deleting the install tree
> > from the docs? Now that the docs are in a standard location, that could
> > be part of an EOL script (although it means 2 phases, since we have to
> > land the patch and let the docs rebuild before we remove the branch).
> > 
> > We could also update the openstack-manuals tree to not publish links
> > to the install guides (either removing the page or replacing it
> > with a placeholder that explains they should be trying to use a
> > newer version).
> > 
> > Doug
> > 
> 
> Today we're publishing series-specific (e.g., newton) and
> version-specific (e.g., 10.0.0) docs. I wonder if we should stop
> publishing docs when we tag repositories, and just stick to the
> series? There's no way to go back and update those version-specific
> builds after the fact, so we can't remove the installation guides
> and we can't rebuild them easily if there are security issues with
> the content.
> 
> Doug
> 

I've proposed https://review.openstack.org/489231 to update
project-config to stop publishing docs when we tag releases.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][ptl][stable][release] stable/pike branches created for most libraries

[Doug Hellmann]

Excerpts from Dmitry Tantsur's message of 2017-07-29 20:06:18 +0200:
> On 07/28/2017 11:13 PM, Doug Hellmann wrote:
> > As Thierry mentioned in his countdown email today, the release team has
> > now created the stable branches for most deliverables with type
> > "library".
> > 
> > We have 3 exceptions:
> > 
> > 1. python-neutronclient had a late release, so I will be branching it
> > shortly.
> > 2. python-barbicanclient was skipped until they have a chance to resolve
> > the critical bug they are working on.
> > 3. tempest doesn't branch (we should probably reclassify that as
> > something other than a library to avoid issues with the automation)
> > 
> > All libraries should have had updates for the .gitreview file in the new
> > branch, the constraints URL in the tox.ini file, and the reno
> > configuration (in master, if the project uses reno).
> > 
> > Landing any of the patches in the stable/pike branch will trigger a new
> > documentation build publishing to docs.o.o/$project/pike.
> > 
> > Please take over any of the bot-created patches that do not pass your
> > validation jobs and fix them so that they can land as soon as possible.
> 
> Just to clarify: we cannot land the tox.ini change until the requirements 
> repo 
> is actually branched, right?

Good point. The tests for those patches are passing for some projects in
CI, but when the patches are landed it will make it a little harder for
anyone to run the tests for the branch elsewhere because the
requirements repo has not yet been branched.

So, yes, hold off on landing the constraint URL changes.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release] nominating Sean McGinnis for releases-core

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-07-27 11:14:51 -0400:
> Sean has been increasingly active with the release team this cycle, and
> wants to contribute. I think we should go ahead and add him to the
> releases-core group.
> 
> Please respond +1/-1.
> 
> Doug
> 

As all of the feedback has been positive, I went ahead and added Sean
to the releases-core group in gerrit today.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][ptl][stable][release] stable/pike branches created for most libraries

[Doug Hellmann]

As Thierry mentioned in his countdown email today, the release team has
now created the stable branches for most deliverables with type
"library".

We have 3 exceptions:

1. python-neutronclient had a late release, so I will be branching it
   shortly.
2. python-barbicanclient was skipped until they have a chance to resolve
   the critical bug they are working on.
3. tempest doesn't branch (we should probably reclassify that as
   something other than a library to avoid issues with the automation)

All libraries should have had updates for the .gitreview file in the new
branch, the constraints URL in the tox.ini file, and the reno
configuration (in master, if the project uses reno).

Landing any of the patches in the stable/pike branch will trigger a new
documentation build publishing to docs.o.o/$project/pike.

Please take over any of the bot-created patches that do not pass your
validation jobs and fix them so that they can land as soon as possible.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-28 13:23:32 +:
> On 2017-07-28 08:34:18 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > I wasn't able to come up with a way to disable the link without
> > triggering a new build. I didn't want us to have to land a patch in each
> > repo as part of marking it EOL, but if we're going to do that to remove
> > the installation guide anyway then maybe we can disable the bug link at
> > the same time.
> 
> Could you have the buglink in the docs go to a centrally-managed,
> pattern-based redirect per series and then change that redirect to
> point them all to a page describing the EOL process later?

Oh, that's an interesting idea. We could do that in the openstack-manuals
repo and generate new redirects when we change the status of the
series there.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Dmitry Tantsur's message of 2017-07-28 12:29:29 +0200:
> On 07/28/2017 09:12 AM, Andreas Jaeger wrote:
> > On 2017-07-27 21:40, Doug Hellmann wrote:
> >> [...]
> >> Please encourage everyone there to explore options that require the
> >> least amount of effort. An ideal solution is one we can implement
> >> without heroic efforts or having to recruit an army of contributors.
> > 
> > I agree with the points made in general and want to stress we need
> > something that reduces our efforts.
> > 
> > Two more ideas:
> > 
> > 1) What about enhancing the openstackdocstheme to automatically add a
> > watermark if we publish a stable branch document?
> > Like on https://docs.openstack.org/mitaka/config-reference/ - which also
> > includes a warning that the branch is EOL. What about adding at *start*
> > of a branch a message to the index page like "This is the document for
> > the SERIES release. Please see https://releases.openstack.org/ whether
> > the SERIES release is still supported by the OpenStack community."
> 
> Or we can use a similar approach with repository badges, and include an image 
> with URL like https://releases.openstack.org/is-supported/ocata.png, which 
> will 
> turn red when ocata goes EOL.

Ooo, I like *that*. It lets us generate the updated status badge from a
repo that will always be active and isn't likely to get into a state
where we can't land patches, like some of our older stable branches have
a tendency to do.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Andreas Jaeger's message of 2017-07-28 09:12:59 +0200:
> On 2017-07-27 21:40, Doug Hellmann wrote:
> > [...]
> > Please encourage everyone there to explore options that require the
> > least amount of effort. An ideal solution is one we can implement
> > without heroic efforts or having to recruit an army of contributors.
> 
> I agree with the points made in general and want to stress we need
> something that reduces our efforts.
> 
> Two more ideas:
> 
> 1) What about enhancing the openstackdocstheme to automatically add a
> watermark if we publish a stable branch document?
> Like on https://docs.openstack.org/mitaka/config-reference/ - which also
> includes a warning that the branch is EOL. What about adding at *start*
> of a branch a message to the index page like "This is the document for
> the SERIES release. Please see https://releases.openstack.org/ whether
> the SERIES release is still supported by the OpenStack community."

I like that. We could add that information to a sidebar or footer on
every page.

> 2) The openstackdocstheme has the report a bug link feature. We can
> disable this. If we EOL docs (so, push a change before we eol them), we
> could remove the setting so that "report a bug" does not work.

I wasn't able to come up with a way to disable the link without
triggering a new build. I didn't want us to have to land a patch in each
repo as part of marking it EOL, but if we're going to do that to remove
the installation guide anyway then maybe we can disable the bug link at
the same time.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-07-27 19:05:16 -0400:
> Excerpts from Jeremy Stanley's message of 2017-07-27 22:07:33 +:
> > On 2017-07-27 15:40:22 -0400 (-0400), Doug Hellmann wrote:
> > [...]
> > > Are we over-emphasizing the scale of the discovery problem?
> > > 
> > > When I search for how to install a package on Ubuntu (or Red Hat
> > > or Debian for that matter), I find all sorts of references all over
> > > the web (including on the sites for those distros) and I have to
> > > sift through it all to find right command or package name to use
> > > for my version.  Usually the easiest way to do that is to put the
> > > version in the search query.
> > > 
> > > Are our users incapable of finding the right version of a document
> > > if we clearly mark the version to which each document applies? Every
> > > URL now has a release series name or version tag in the path. The
> > > docs theme inserts the version number into every page as well. Is
> > > that sufficient to help a reader understand what version the info
> > > they're view is for?
> > > 
> > > That's not to say we shouldn't do some work to make newer docs easy
> > > to find. If we can modify the sitemap to make them appear earlier
> > > in search results, that's good. The docs theme allows a project to
> > > include links to several older versions to switch between them,
> > > too, so users who land on the "wrong" version can switch. We could
> > > update that to include branches as well as tags, so that someone
> > > can easily move to the series they need without knowing the version
> > > number.
> > [...]
> > 
> > The biggest liability is people not realizing there are multiple
> > "versions" of OpenStack finding an old install doc and using it
> > because they don't know it's out of date, then seeking support
> > upstream or just generally contributing to the number of deployments
> > unnecessarily stuck on obsolete software. The current solution of
> > not publishing installation guides for EOL releases seems like a
> > good enough compromise there to me.
> 
> That content will now live in the project trees. Perhaps part of marking
> branches in those repos EOL needs to include deleting the install tree
> from the docs? Now that the docs are in a standard location, that could
> be part of an EOL script (although it means 2 phases, since we have to
> land the patch and let the docs rebuild before we remove the branch).
> 
> We could also update the openstack-manuals tree to not publish links
> to the install guides (either removing the page or replacing it
> with a placeholder that explains they should be trying to use a
> newer version).
> 
> Doug
> 

Today we're publishing series-specific (e.g., newton) and
version-specific (e.g., 10.0.0) docs. I wonder if we should stop
publishing docs when we tag repositories, and just stick to the
series? There's no way to go back and update those version-specific
builds after the fact, so we can't remove the installation guides
and we can't rebuild them easily if there are security issues with
the content.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-27 21:56:35 +:
> On 2017-07-27 15:40:22 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > Regarding point 2, if we don't have the space to host the content
> > indefinitely, then we need to set a fixed, but longer, retention
> > period before deleting it.  Several years, at least. In the mean
> > time, we could delete builds for intermediate versions (maybe if
> > we have 10.0.0, 10.1.0, and 10.1.1 we only need to keep 10.1.1, for
> > example).  The point is that there are ways to remove some content
> > without removing it all. I know space used to be a real problem
> > when we were hosting on CloudSites, but maybe someone from the infra
> > team can give us some insight into how significant the space issue
> > is today?
> [...]
> 
> The current content for all of docs.openstack.org weighs in at
> roughly 11GiB on disk. I'm not surprised that was a massive Web site
> by CloudSites' standards, but compared to the ~11TiB we host for
> just two months of CI job logs it's a 1ml drop in a liter bucket.

Excellent, so we don't need to worry about space, for now.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-27 22:07:33 +:
> On 2017-07-27 15:40:22 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > Are we over-emphasizing the scale of the discovery problem?
> > 
> > When I search for how to install a package on Ubuntu (or Red Hat
> > or Debian for that matter), I find all sorts of references all over
> > the web (including on the sites for those distros) and I have to
> > sift through it all to find right command or package name to use
> > for my version.  Usually the easiest way to do that is to put the
> > version in the search query.
> > 
> > Are our users incapable of finding the right version of a document
> > if we clearly mark the version to which each document applies? Every
> > URL now has a release series name or version tag in the path. The
> > docs theme inserts the version number into every page as well. Is
> > that sufficient to help a reader understand what version the info
> > they're view is for?
> > 
> > That's not to say we shouldn't do some work to make newer docs easy
> > to find. If we can modify the sitemap to make them appear earlier
> > in search results, that's good. The docs theme allows a project to
> > include links to several older versions to switch between them,
> > too, so users who land on the "wrong" version can switch. We could
> > update that to include branches as well as tags, so that someone
> > can easily move to the series they need without knowing the version
> > number.
> [...]
> 
> The biggest liability is people not realizing there are multiple
> "versions" of OpenStack finding an old install doc and using it
> because they don't know it's out of date, then seeking support
> upstream or just generally contributing to the number of deployments
> unnecessarily stuck on obsolete software. The current solution of
> not publishing installation guides for EOL releases seems like a
> good enough compromise there to me.

That content will now live in the project trees. Perhaps part of marking
branches in those repos EOL needs to include deleting the install tree
from the docs? Now that the docs are in a standard location, that could
be part of an EOL script (although it means 2 phases, since we have to
land the patch and let the docs rebuild before we remove the branch).

We could also update the openstack-manuals tree to not publish links
to the install guides (either removing the page or replacing it
with a placeholder that explains they should be trying to use a
newer version).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][docs] 404s on docs website after the great reorg

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-07-27 14:24:53 -0400:
> Excerpts from Jeremy Stanley's message of 2017-07-27 16:40:08 +:
> > On 2017-07-27 12:23:39 -0400 (-0400), Sean Dague wrote:
> > > In the #openstack-nova channel this morning we were debugging some cells
> > > v2 things, and ran into the fact that the online docs for this -
> > > https://docs.openstack.org/nova/latest/cells.html go to a 404. That's a
> > > previously well known link, people have it in their browser history,
> > > bookmarks, wiki pages, other websites.
> > > 
> > > My understanding of big moves like this is that redirects are important.
> > > Things going blackhole like that not only is an inconvenience to users,
> > > but impacts our search engine rankings, and takes a while for them to
> > > all sift out. I know in sites I run I'm still regularly getting in
> > > bounds to paths on the site that haven't been there for 8 years.
> > > 
> > > It would be really good if we had a way (manual or automated) to have
> > > 301 redirects, that are fixable by the teams that now own the
> > > documentation (the project teams).
> > 
> > We can look at including .htaccess files in the tree I guess? Or
> > some metadata the publish job uses to build them maybe?
> 
> That's exactly what I was thinking.
> 
> 1. Enable .htaccess files by turning on allowoverride for
>docs.openstack.org.
> 
> 2. Add .htaccess files in each tree, as needed (see
>https://review.openstack.org/487932 for an example of how this
>is done with sphinx).
> 
> 3. Update the main .htaccess file in openstack-manuals to redirect
>from the old location of docs in a way that passes the full path.
>Right now we redirect to /project/latest/:
> 
>   redirectmatch 301 "^/developer/([^/]+)/.*$" /$1/latest/
> 
>I think that would change to look something like:
> 
>   redirectmatch 301 "^/developer/([^/]+)/(.*)$" /$1/latest/$2
> 
>We would only want to do that for projects that actually have
>.htaccess files, so we can put a flag in the project-data files in
>openstack-manuals and generate project-specific redirect rules (we're
>already doing that for some other pages).
> 
> Then when someone visits docs.o.o/developer/nova/cells.html it would
> redirect to docs.o.o/nova/latest/cells.html. The nova team then
> need to have a redirect from docs.o.o/nova/latest/cells.html to
> docs.o.o/nova/latest/user/cells.html.
> 
> If folks think that's a good approach, I will start on the patches
> needed in infra and openstack-manuals (1 and 3).
> 
> Doug
> 

The patches are up using the topic in-tree-redirects:

https://review.openstack.org/#/q/topic:in-tree-redirects

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][release][stable][ptl][infra] Strategic discussion regarding the future of documentation for EOL releases

[Doug Hellmann]

[Moving a thread from the openstack-docs list [1] to this list for
broader input.]

[1] http://lists.openstack.org/pipermail/openstack-docs/2017-July/010069.html

Excerpts from 's message of 2017-07-26 07:42:38 -0400:
> Yesterday was a very busy day on IRC, with the discussion about the
> strategy and future maintenance of the documentation for the EOL
> releases coming back to the front.
> 
> I've promised to summarize some of what we discussed, for those who
> weren't there, and sketch out some of the fenceposts along our path forward.
> 
> Summary:
> The main issue, is that when an OpenStack release goes EOL, the branch
> in the main repository goes away, and with it go the docs, which then
> vanish from the public-facing website.
> 
> This has been an open gap for awhile, but only recently became a pain
> point for many operators. I think we can all agree that this is an
> issue, so the "Why should we fix this?" isn't as important as "HOW do we
> fix this?"

Yes, as I said on IRC, I think we have reached a point where enough of
our user base is trailing far enough behind that we need to rethink our
documentation retention policy to make sure we're meeting everyone's
needs.

> This leaves any sites, operators or installations that may be using
> those releases, without any tangible way to research how to install,
> manage or maintain those in-place installations and releases.
> 
> For companies like Canonical, we support OpenStack on Ubuntu LTS
> releases beyond the upstream EOL date of those point releases
> themselves. Other distributions may have a similar gap with the docs
> vanishing before their support of those releases sunsets.
> 
> Ideally, docs should be managed and maintained at the upstream project
> site, not at each disribution's own domain. I'd like to avoid having a
> Canonical version, a Red Hat version, an Oracle version, etc. all with
> the same patches to build local copies staged on our respective domains.
> 
> I've recently put some effort into automating and patching off the older
> versions of the docs based on the github tags, so they build in a local
> tree with mvn/tox, and that tree can be used internally by operators,
> but this should be viewed as a stopgap to a more strategic archiving
> solution for these docs.
> 
> There is an archiving plan[0] that has been discussed, but with the -doc
> team resources being thinned out, there is very little "spare" resources
> to put towards this effort. There's been some discussion[1],[2] to try
> to bring this to the front, but it too suffers from time and resources.
> 
> There are a few key problems/challenges/questions with solving this in a
> repeatable, strategic way:
> 
> * The branches are gone so there's no way to "update" these eol docs,
>   patch them to build, or maintain them going forward for those eol
>   releases.
> 
>   * Should the eol docs be pulled out and put into their own separate
> github repo, where they _can_ be patched and maintained so they
> continue to be buildable and usable by those with the correct
> tools and environment?
>
>   * There's been talk about pulling the docs into a community-
> maintained 'wiki', for ongoing maintenance, but that opens more
> questions too (who should host it? who 'owns' it? who gets write
> vs. read access? etc.)

As Jeremy pointed out on the docs list, this doesn't really help. Moving
the content won't give us more maintainers. I don't think we want to
enable "maintenance" of the docs as such, though. I think we just want
to make them available as they are for users to find. This week we made
some progress there, so that now docs.openstack.org has a landing page
for every series (for example https://docs.openstack.org/austin/).

For the series where docs still existed on the server, we added
links. The earliest release with any docs available was Icehouse.
We have progressively more material as you look at more recent
releases.

> * Where should the docs ultimately "live", until they're truly eol for
>   all parties concerned, and what should that timeline be? 3 years
>   past eol? 5 years? Indefinitely?
> 
>   * We discussed something like: docs.o.o/eol_$release or similar
> indicator to differentiate the 'current' docs from the
> archived/eol docs.
> 
> * Should the docs for eol releases be made available in PDF only
>   format, or indexable/searchable HTML format? There are pros and
>   cons for using either approach, this might need more thought.

I think we want to take a "less is more" approach here. When we had
lots of contributors working on the docs, it made sense to ask them
to do things that I think we can't afford to do any more. So rather
than looking for a new way to do something, let's see if we can
*stop* doing work and achieve more.

My proposal is to put the documentation on docs.openstack.org and
leave it there indefinitely.

Leaving the docs where they are avoids a ton of work to build
archiving systems and 

Re: [openstack-dev] [all][docs] 404s on docs website after the great reorg

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-27 16:40:08 +:
> On 2017-07-27 12:23:39 -0400 (-0400), Sean Dague wrote:
> > In the #openstack-nova channel this morning we were debugging some cells
> > v2 things, and ran into the fact that the online docs for this -
> > https://docs.openstack.org/nova/latest/cells.html go to a 404. That's a
> > previously well known link, people have it in their browser history,
> > bookmarks, wiki pages, other websites.
> > 
> > My understanding of big moves like this is that redirects are important.
> > Things going blackhole like that not only is an inconvenience to users,
> > but impacts our search engine rankings, and takes a while for them to
> > all sift out. I know in sites I run I'm still regularly getting in
> > bounds to paths on the site that haven't been there for 8 years.
> > 
> > It would be really good if we had a way (manual or automated) to have
> > 301 redirects, that are fixable by the teams that now own the
> > documentation (the project teams).
> 
> We can look at including .htaccess files in the tree I guess? Or
> some metadata the publish job uses to build them maybe?

That's exactly what I was thinking.

1. Enable .htaccess files by turning on allowoverride for
   docs.openstack.org.

2. Add .htaccess files in each tree, as needed (see
   https://review.openstack.org/487932 for an example of how this
   is done with sphinx).

3. Update the main .htaccess file in openstack-manuals to redirect
   from the old location of docs in a way that passes the full path.
   Right now we redirect to /project/latest/:

  redirectmatch 301 "^/developer/([^/]+)/.*$" /$1/latest/

   I think that would change to look something like:

  redirectmatch 301 "^/developer/([^/]+)/(.*)$" /$1/latest/$2

   We would only want to do that for projects that actually have
   .htaccess files, so we can put a flag in the project-data files in
   openstack-manuals and generate project-specific redirect rules (we're
   already doing that for some other pages).

Then when someone visits docs.o.o/developer/nova/cells.html it would
redirect to docs.o.o/nova/latest/cells.html. The nova team then
need to have a redirect from docs.o.o/nova/latest/cells.html to
docs.o.o/nova/latest/user/cells.html.

If folks think that's a good approach, I will start on the patches
needed in infra and openstack-manuals (1 and 3).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release] nominating Sean McGinnis for releases-core

[Doug Hellmann]

Sean has been increasingly active with the release team this cycle, and
wants to contribute. I think we should go ahead and add him to the
releases-core group.

Please respond +1/-1.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][oslo.db] nominating Jay Pipes for oslo-db-core

[Doug Hellmann]

I have noticed that Jay has been very deeply involved in several
recent design discussions about oslo.db, and he obviously has a
great deal of experience in the area, so even though he hasn't been
actively reviewing patches recently I think he would be a good
addition to the team. I have asked him, and he is interested and will
try to become more active as well.

Please indicate your opinion with +1/-1.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [relmgt] Non-candidacy for the Queens cycle Release management PTL

[Doug Hellmann]

Excerpts from Sean McGinnis's message of 2017-07-27 08:45:16 -0500:
> > 
> > I have multiple reasons for not running this cycle. First, as part of my
> > expanded role at the Foundation my travel commitments increased a bit.
> > The Release management PTL role requires regular work: I ended up doing
> > a relatively bad job at PTLing this cycle, by being away during a number
> > of critical weeks. Fortunately, I could count on the presence of the
> > other team members (especially dhellmann and dims) to keep things under
> > control!
> 
> Thank you Thierry and team for the great work. Depite what you say, I
> don't think anyone has felt the impact, so - great job!
> 
> Sean
> 

+1 to that -- With your guidance we have solidified the processes
the team uses so well that we can all share the load. That's exactly
the sort of situation we need for all of our cross-project teams.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] Non-candidacy for Queens cycle Infrastructure PTL

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-25 23:37:40 +:
> It has been a pleasure and a privilege to captain the good ship
> Infrastructure for the past four cycles, but it's time to let
> someone else take a turn at the wheel. I am confident in the ability
> of my shipmates to step up to the responsibilities of the position
> (quite likely even better than I have done), and I look forward to
> assisting them as they steer us through the challenging waters
> ahead. I couldn't have accomplished everything I did without the aid
> and support of the rest of our crew, so it's only right I should
> give the same as I received. Thank you all for making it so much
> easier than I ever could have anticipated.
> 
> I'm definitely not jumping ship, though! Much like my Infra PTL
> emeritus predecessors, I'm excited to be able to return to building,
> fixing and improving the systems on which our community relies so
> heavily day in and day out. This should hopefully also allow me to
> redouble efforts on my TC and VMT responsibilities, and I'm eager to
> be able to set course for long-awaited necessities such as
> consolidated authentication and finally putting an end to the ICLA.
> 
> To those running for PTL, remember that we're only a few days away
> from the nomination period so start polishing your platforms; I
> eagerly await the opportunity to read through them all as a regular
> voter rather than a candidate for the first time in years!

Thank you for serving for 2 years! You have elevated the already
excellent standards set by your predecessors for collaboration and
support, and the community is better off as a result of the work the
infra team has done during your tenure.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][freezer][Release-job-failures] Pre-release of openstack/freezer-web-ui failed

[Doug Hellmann]

Excerpts from jenkins's message of 2017-07-26 13:32:20 +:
> Build failed.
> 
> - freezer-web-ui-tarball 
> http://logs.openstack.org/c1/c151f2ea0c2e6e22d86b38f7916eabd2ce74c5bf/pre-release/freezer-web-ui-tarball/636/
>  : FAILURE in 2m 48s
> - freezer-web-ui-tarball-signing freezer-web-ui-tarball-signing : SKIPPED
> - freezer-web-ui-pypi-both-upload freezer-web-ui-pypi-both-upload : SKIPPED
> - freezer-web-ui-announce-release freezer-web-ui-announce-release : SKIPPED
> - propose-freezer-web-ui-update-constraints 
> propose-freezer-web-ui-update-constraints : SKIPPED
> 

It looks like there is a bad configuration setting in freezer's
setup.cfg:

  error: error in setup.cfg: command 'bdist_wheel' has no such option 
'warning_is_error'

Freezer team, please fix the issue before the RC1 deadline. I don't
think there's a need to re-do the milestone 3 tag.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][monasca][designate][requirements] late library release for monasca-statsd

[Doug Hellmann]

Excerpts from Graham Hayes's message of 2017-07-25 12:24:34 +0100:
> On 25/07/17 12:12, Doug Hellmann wrote:
> > The monasca team has requested a late release of monasca-statsd.
> > As far as I can tell, only other team using the library is designate:
> > 
> > $ whatuses monasca-statsd
> > deb-designate requirements.txt   monasca-statsd>=1.1.0 # 
> > Apache-2.0
> > designate requirements.txt   monasca-statsd>=1.1.0 # 
> > Apache-2.0
> > monasca-log-api   requirements.txt   monasca-statsd>=1.1.0 # 
> > Apache-2.0
> > monasca-notification  requirements.txt   monasca-statsd>=1.1.0 # 
> > Apache-2.0
> > requirements  global-requirements.txtmonasca-statsd>=1.1.0 # 
> > Apache-2.0
> > requirements  upper-constraints.txt  monasca-statsd===1.7.0
> > rpm-packaging global-requirements.txtmonasca-statsd>=1.1.0 # 
> > Apache-2.0
> > 
> > Can someone from the designate team take a look at the list of changes
> > in the proposed release (https://review.openstack.org/#/c/486035) and
> > give us some idea of the risk involved in releasing?
> > 
> > Doug
> 
> The difference is just some requirements updates and changing how
> the docs display the version.
> 
> The requirements match ours, and the docs won't affect us - so
> I would say it is a low risk for designate.

Thanks, Graham. We'll talk with the Monasca folks about whether the
release is critical or not, since the changes are just to the dependency
list.

Doug

> 
> ➜  monasca-statsd git:(master) git diff
> 1.7.0..6d416dd8be06587f9d8cf9f5455fd5616294f418
> diff --git a/doc/source/conf.py b/doc/source/conf.py
> index 20d22e9..9468074 100644
> --- a/doc/source/conf.py
> +++ b/doc/source/conf.py
> @@ -176,8 +176,7 @@ html_static_path = ['_static']
>  git_cmd = ["git", "log", "--pretty=format:'%ad, commit %h'",
> "--date=local",
>  "-n1"]
>  try:
> -html_last_updated_fmt = subprocess.Popen(
> -git_cmd, stdout=subprocess.PIPE).communicate()[0].decode()
> +html_last_updated_fmt =
> subprocess.check_output(git_cmd).decode('utf-8')
>  except Exception:
>  warnings.warn('Cannot get last updated time from git repository. '
>'Not setting "html_last_updated_fmt".')
> diff --git a/test-requirements.txt b/test-requirements.txt
> index bec732c..1ad4d50 100644
> --- a/test-requirements.txt
> +++ b/test-requirements.txt
> @@ -5,9 +5,9 @@ hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0
>  bandit>=1.1.0 # Apache-2.0
>  coverage!=4.4,>=4.0 # Apache-2.0
>  mock>=2.0 # BSD
> -sphinx!=1.6.1,>=1.5.1 # BSD
> +sphinx>=1.6.2 # BSD
>  oslosphinx>=4.7.0 # Apache-2.0
>  oslotest>=1.10.0 # Apache-2.0
>  os-testr>=0.8.0 # Apache-2.0
>  testrepository>=0.0.18 # Apache-2.0/BSD
> -openstackdocstheme>=1.5.0 # Apache-2.0
> +openstackdocstheme>=1.11.0 # Apache-2.0
> 
> 
> - Graham
> 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release][monasca][designate][requirements] late library release for monasca-statsd

[Doug Hellmann]

The monasca team has requested a late release of monasca-statsd.
As far as I can tell, only other team using the library is designate:

$ whatuses monasca-statsd
deb-designate requirements.txt   monasca-statsd>=1.1.0 # 
Apache-2.0
designate requirements.txt   monasca-statsd>=1.1.0 # 
Apache-2.0
monasca-log-api   requirements.txt   monasca-statsd>=1.1.0 # 
Apache-2.0
monasca-notification  requirements.txt   monasca-statsd>=1.1.0 # 
Apache-2.0
requirements  global-requirements.txtmonasca-statsd>=1.1.0 # 
Apache-2.0
requirements  upper-constraints.txt  monasca-statsd===1.7.0
rpm-packaging global-requirements.txtmonasca-statsd>=1.1.0 # 
Apache-2.0

Can someone from the designate team take a look at the list of changes
in the proposed release (https://review.openstack.org/#/c/486035) and
give us some idea of the risk involved in releasing?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][neutron] doc-migration status

[Doug Hellmann]

Excerpts from Kevin Benton's message of 2017-07-23 14:19:51 -0700:
> Yeah, the networking guide does include configuration for some of the
> sub-projects (e.g. BGP is at [1]). For the remaining ones there is work
> that needs to be done because their docs live in wiki pages.
> 
> 1.
> https://docs.openstack.org/ocata/networking-guide/config-bgp-dynamic-routing.html

OK, that's good to know. It would be good to be consistent with the
approach to the stadium projects, so we can either eliminate the list of
projects from landing pages that show things like "all of the admin
guides" or we can add the projects so users can find the docs. If
they're all covered in the networking guide, we could include that
information on the admin landing page, for example.

In the mean time, if someone from the neutron project will review
the list of "Missing URLs" on https://doughellmann.com/doc-migration/
and let me know which ones represent content included in other
documents, I can update the burndown chart generator to reflect
that.

Doug

> 
> 
> On Sun, Jul 23, 2017 at 1:32 PM, Doug Hellmann <d...@doughellmann.com>
> wrote:
> 
> > Excerpts from Kevin Benton's message of 2017-07-23 01:31:25 -0700:
> > > Yeah, I was just thinking it makes it more explicit that we haven't just
> > > skipped doing an admin guide for a particular project.
> >
> > Sure, you can do that. I don't think we want to link to all of those
> > pages from the list of admin guides, though.
> >
> > I've updated the burndown chart generator to ignore the missing
> > admin guide URLs for networking subprojects.
> >
> > I don't see configuration or installation guides for quite a few
> > of those, either. Are those also handled within the neutron main
> > tree docs?
> >
> > Doug
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [oslo.db] [relational database users] heads up for a MariaDB issue that will affect most projects

[Doug Hellmann]

Excerpts from Michael Bayer's message of 2017-07-23 16:39:20 -0400:
> Hey list -
> 
> It appears that MariaDB as of version 10.2 has made an enhancement
> that overall is great and fairly historic in the MySQL community,
> they've made CHECK constraints finally work.   For all of MySQL's
> existence, you could emit a CREATE TABLE statement that included CHECK
> constraint, but the CHECK phrase would be silently ignored; there are
> no actual CHECK constraints in MySQL.
> 
> Mariadb 10.2 has now made CHECK do something!  However!  the bad news!
>  They have decided that the CHECK constraint against a single column
> should not be implicitly dropped if you drop the column [1].   In case
> you were under the impression your SQLAlchemy / oslo.db project
> doesn't use CHECK constraints, if you are using the SQLAlchemy Boolean
> type, or the "ENUM" type without using MySQL's native ENUM feature
> (less likely), there's a simple CHECK constraint in there.
> 
> So far the Zun project has reported the first bug on Alembic [2] that
> they can't emit a DROP COLUMN for a boolean column.In [1] I've
> made my complete argument for why this decision on the MariaDB side is
> misguided.   However, be on the lookout for boolean columns that can't
> be DROPPED on some environments using newer MariaDB.  Workarounds for
> now include:
> 
> 1. when using Boolean(), set create_constraint=False
> 
> 2. when using Boolean(), make sure it has a "name" to give the
> constraint, so that later you can DROP CONSTRAINT easily
> 
> 3. if not doing #1 and #2, in order to drop the column you need to use
> the inspector (e.g. from sqlalchemy import inspect; inspector =
> inspect(engine)) and locate all the CHECK constraints involving the
> target column, and then drop them by name.

Item 3 sounds like the description of a helper function we could add to
oslo.db for use in migration scripts.

Doug

> 
> [1] https://jira.mariadb.org/browse/MDEV-4
> 
> [2] 
> https://bitbucket.org/zzzeek/alembic/issues/440/cannot-drop-boolean-column-in-mysql
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][neutron] doc-migration status

[Doug Hellmann]

Excerpts from Kevin Benton's message of 2017-07-23 01:31:25 -0700:
> Yeah, I was just thinking it makes it more explicit that we haven't just
> skipped doing an admin guide for a particular project.

Sure, you can do that. I don't think we want to link to all of those
pages from the list of admin guides, though.

I've updated the burndown chart generator to ignore the missing
admin guide URLs for networking subprojects.

I don't see configuration or installation guides for quite a few
of those, either. Are those also handled within the neutron main
tree docs?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][neutron] doc-migration status

[Doug Hellmann]

Excerpts from Akihiro Motoki's message of 2017-07-23 02:13:40 +0900:
> Hi,
> 
> I have a question on admin/ document related to the networking guide
> and would like to have advices from the documentation experts.
> 
> It seems the check site by Doug expect all project have admin/ page.
> In the case of neutron the situation is a bit special. We have the
> networking guide as admin/ document
> in the neutron repo and it covers not only neutron itself but also
> neutron stadium projects.
> It means the neutron stadium projects sometimes (often?) have no
> admin/ directory in their own repos
> in favor of adding contents to the networking guide in neutron.
> 
> Should Individual neutron stadium projects have their own admin guide
> in their repositories,
> or is it better to keep the networking guide which covers all
> networking stuff in a single guide?
> 
> What is the suggested way on the networking guide as the document expert?

If the admin guides for all of those repos are combined, then I can
modify the burndown chart generator to not count those as needed. Let me
know if that's the best approach.

Doug

> 
> Thanks,
> Akihiro
> 
> 2017-07-22 3:26 GMT+09:00 Doug Hellmann <d...@doughellmann.com>:
> > We've made huge progress, and are launching the updated landing
> > pages for docs.openstack.org as I write this. Thanks to all of the
> > contributors who have stepped up to write nearly 1,000 patches to
> > improve the health of our documentation!
> >
> > We still have around 70 URLs we expected to see after the migration
> > was complete but that produce a 404. I know some of the patches to
> > produce those pages are in progress, but please check the list at
> > https://doughellmann.com/doc-migration/ if your team is listed below
> > to ensure that nothing has been missed.
> >
> >   cinder
> >   cloudkitty
> >   congress
> >   designate
> >   heat
> >   ironic
> >   karbor
> >   keystone
> >   magnum
> >   manila
> >   murano
> >   neutron
> >   nova
> >   sahara
> >   senlin
> >   swift
> >   tacker
> >   telementry
> >   tricircle
> >   trove
> >   vitrage
> >   watcher
> >   zaqar
> >   zun
> >
> > Reply here or ping me in #openstack-docs if you have questions or need a
> > hand.
> >
> > Doug
> >
> > ___
> > OpenStack-docs mailing list
> > openstack-d...@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [docs][ptl] doc-migration status

[Doug Hellmann]

We've made huge progress, and are launching the updated landing
pages for docs.openstack.org as I write this. Thanks to all of the
contributors who have stepped up to write nearly 1,000 patches to
improve the health of our documentation!

We still have around 70 URLs we expected to see after the migration
was complete but that produce a 404. I know some of the patches to
produce those pages are in progress, but please check the list at
https://doughellmann.com/doc-migration/ if your team is listed below
to ensure that nothing has been missed.

  cinder
  cloudkitty
  congress
  designate
  heat
  ironic
  karbor
  keystone
  magnum
  manila
  murano
  neutron
  nova
  sahara
  senlin
  swift
  tacker
  telementry
  tricircle
  trove
  vitrage
  watcher
  zaqar
  zun

Reply here or ping me in #openstack-docs if you have questions or need a
hand.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [doc] dropping "draft" and series-specific publishing for docs.o.o

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-07-20 15:11:57 -0400:
> Docs team,
> 
> We have two sets of changes happening in openstack-manuals that
> will simplify managing the content there, and that will .
> 
> Over the last couple of weeks we have removed several guides from
> the repository. The remaining guides are not version-specific, which
> allows us stop creating stable branches of that repository. We will
> continue to publish from stable/newton and stable/ocata for as long
> as those versions of the guides are supported.
> 
> I am also preparing a series to patches [1] to rearrange some of
> the template pages to let us publish directly from master to docs.o.o,
> without using a separate "draft" directory that requires special
> effort at the end of a release cycle. When the series is approved
> (specifically when [2] lands), changes approved in the master branch
> will go live on the site within an hour or so of merging. They will
> no longer be published to the /drafts folder.
> 
> Both of these are changes to the current process, so we wanted to
> ensure that all contributors (and especially reviewers) were aware
> of the changes. Please keep this in mind when approving future
> changes.
> 
> The last patch in the series [3] updates the instructions for the
> end-of-release process based on these changes. I want to make sure
> these instructions are clear so that someone other than me can
> perform the steps, so I need your feedback on that patch especially.
> 
> Doug
> 
> [1] 
> https://review.openstack.org/#/q/project:openstack/openstack-manuals+topic:doc-migration/no-more-drafts
> [2] https://review.openstack.org/484971
> [3] https://review.openstack.org/485789
> 

These patches are approved and are in the post-merge queue now. The
site should be updating over the next hour or two. If you spot any
issues, please report them in #openstack-docs or as a follow-up to
this message so we can take care of them.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-doc] Attending PTG?

[Doug Hellmann]

Excerpts from Alexandra Settle's message of 2017-07-21 13:41:58 +:
> Hey everyone,
> 
> Put your hand in the air* if you’re attending the PTG!
> 
> Myself and the potential PTL will need to start planning shortly, and it 
> would be good to get an idea of numbers, and who we are planning for.
> 
> I will be looking at hosting a working bee or hack session for migration 
> related issues throughout the week.
> 
> Cheers,
> 
> Alex
> 
> *reply to this ML thread

Count me in.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [doc] dropping "draft" and series-specific publishing for docs.o.o

[Doug Hellmann]

Docs team,

We have two sets of changes happening in openstack-manuals that
will simplify managing the content there, and that will .

Over the last couple of weeks we have removed several guides from
the repository. The remaining guides are not version-specific, which
allows us stop creating stable branches of that repository. We will
continue to publish from stable/newton and stable/ocata for as long
as those versions of the guides are supported.

I am also preparing a series to patches [1] to rearrange some of
the template pages to let us publish directly from master to docs.o.o,
without using a separate "draft" directory that requires special
effort at the end of a release cycle. When the series is approved
(specifically when [2] lands), changes approved in the master branch
will go live on the site within an hour or so of merging. They will
no longer be published to the /drafts folder.

Both of these are changes to the current process, so we wanted to
ensure that all contributors (and especially reviewers) were aware
of the changes. Please keep this in mind when approving future
changes.

The last patch in the series [3] updates the instructions for the
end-of-release process based on these changes. I want to make sure
these instructions are clear so that someone other than me can
perform the steps, so I need your feedback on that patch especially.

Doug

[1] 
https://review.openstack.org/#/q/project:openstack/openstack-manuals+topic:doc-migration/no-more-drafts
[2] https://review.openstack.org/484971
[3] https://review.openstack.org/485789


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Cleaning up inactive meetbot channels

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-19 19:24:00 +:
> For those who are unaware, Freenode doesn't allow any one user to
> /join more than 120 channels concurrently. This has become a
> challenge for some of the community's IRC bots in the past year,
> most recently the "openstack" meetbot (which not only handles
> meetings but also takes care of channel logging to
> eavesdrop.openstack.org and does the nifty bug number resolution
> some people seem to like).
> 
> I have run some rudimentary analysis and come up with the following
> list of channels which have had fewer than 10 lines said by anyone
> besides a bot over the past three months:
> 
> #craton
> #openstack-api
> #openstack-app-catalog
> #openstack-bareon
> #openstack-cloudpulse
> #openstack-community
> #openstack-cue
> #openstack-diversity
> #openstack-gluon
> #openstack-gslb
> #openstack-ko
> #openstack-kubernetes
> #openstack-networking-cisco
> #openstack-neutron-release
> #openstack-opw
> #openstack-pkg
> #openstack-product
> #openstack-python3
> #openstack-quota
> #openstack-rating
> #openstack-solar
> #openstack-swauth
> #openstack-ux
> #openstack-vmware-nsx
> #openstack-zephyr
> 
> I have a feeling many of these are either no longer needed, or what
> little and infrequent conversation they get used for could just as
> easily happen in a general channel like #openstack-dev or #openstack
> or maybe in the more active channel of their parent team for some
> subteams. Who would miss these if we ceased logging/using them? Does
> anyone want to help by asking around to people who might not see
> this thread, maybe by popping into those channels and seeing if any
> of the sleeping denizens awaken and say they still want to keep it
> around?
> 
> Ultimately we should improve our meetbot deployment to support
> sharding channels across multiple bots, but that will take some time
> to implement and needs volunteers willing to work on it. In the
> meantime we're running with the meetbot present in 120 channels and
> have at least one new channel that desires logging and can't get it
> until we whittle that number down.

All of those look like good candidates for cleanup. We could easily move
#openstack-python3 to #openstack-dev.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [doc][api] need new reviewers for the api-site and developer.openstack.org

[Doug Hellmann]

Excerpts from Alexandra Settle's message of 2017-07-19 15:42:42 +:
> Hi everyone,
> 
> As you all know, we have been changing the way the documentation team 
> operates. Chief among those
> changes are reducing the workload on the shrinking team. The migration to 
> move installation, admin, and
> configuration documentation to project-team repositories is the first phase 
> of that transition. Another component on
> the documentation team's list of responsibilities is developer.openstack.org, 
> the site for consumers of
> OpenStack services to find resources to help them build their applications. 
> Finding a new team of people
> to manage that site is the next phase of shrinking the documentation team's 
> duties.
> 
> We are setting up a new review team [0] in gerrit for the openstack/api-site 
> repository and removing
> the api-site and the faafo (First App Application for OpenStack) repositories 
> from the set listed as owned by the docs
> team in governance [1].  This opens up an opportunity for a new SIG or WG 
> that is able to tackle the requirements
> of the api-site repo.
> 
> Any concerns or questions, please do not hesitate to reach out.
> 
> Cheers,
> 
> Alex
> 
> [0] https://review.openstack.org/#/c/485179/
> [1] https://review.openstack.org/#/c/485249/

Thanks, Alex. This seems like an excellent opportunity improve the
focus of the docs team and build a new dedicated team to take on
developer.openstack.org.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.config] how to deprecate a name but still have it as conf.

[Doug Hellmann]

Excerpts from Michael Bayer's message of 2017-07-18 13:21:30 -0400:
> On Tue, Jul 18, 2017 at 1:02 PM, Doug Hellmann <d...@doughellmann.com> wrote:
> 
> > Option renaming was originally meant as an operatior-facing feature
> > to handle renames for values coming from the config file, but not
> > as they are used in code.  mtreinish added
> > https://review.openstack.org/#/c/357987/ to address this for Tempest,
> > so it's possible there's a bug in the logic in oslo.config somewhere
> > (or that oslo.db's case is a new one).
> 
> OK, patch set 5 at
> https://review.openstack.org/#/c/334182/5/oslo_db/options.py shows
> what I'm trying to do to make this work, however the test case added
> in test_options still fails.   If this is supposed to "just work" then
> I hope someone can confirm that.

You found a bug related to the fact that these options are registered
using an OptGroup() not just a simple string name.
https://review.openstack.org/484897 should fix it.

> 
> Alternatively, a simple flag in DeprecatedOpt  "alias_on_conf=True"
> would be super easy here so that specific names in our DeprecatedOpt
> could be mirrored because we know projects are consuming them on conf.
> 
> >
> > That said, the options defined by a library are *NOT* part of its
> > API, and should never be used by code outside of the library. The
> > whole point of isolating options like that is to give operators a
> > way to change the way an app uses a library (drivers, credentials,
> > etc.) without the app having to know the details.  Ideally the nova
> > tests that access oslo.db configuration options directly would
> > instead use an API in oslo.db to do the same thing (that API may
> > need to be written, if it doesn't already exist).
> 
> OK, that is I suppose an option, but clearly a long and arduous one at
> this point (add new API to oslo.db, find all projects looking at
> conf., submit gerrits, somehow make sure projects never talk to
> conf. directly?   how would we ensure that?  shouldn't
> oslo.config allow the library that defines the options to plug in its
> own "private" namespace so that consuming projects don't make this
> mistake?)
> 
> >
> > At that point, only oslo.db code would refer to the option, and you
> > could use the deprecated_name and deprecated_group settings to
> > describe the move and change the references to oslo.db within the
> > library using a single patch to oslo.db.
> >
> > Doug
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][telemetry][aodh] Tag of openstack/aodh failed

[Doug Hellmann]

Excerpts from jenkins's message of 2017-07-17 12:09:25 +:
> Build failed.
> 
> - aodh-releasenotes 
> http://logs.openstack.org/bb/bbc8065ae2db109f415ff1c03f40015a75e5afe7/tag/aodh-releasenotes/1f93729/
>  : FAILURE in 2m 58s
> 

The release notes failed to build for aodh's release with the error
"Translations exist and locale_dirs missing in source/conf.py".

Unfortunately, I don't know what that means.

Fortunately, I think only the release notes build failed, but the actual
release is fine.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.config] how to deprecate a name but still have it as conf.

[Doug Hellmann]

Excerpts from Michael Bayer's message of 2017-07-18 12:34:31 -0400:
> In oslo.db, I'd like to rename the option "idle_timeout" to
> "connection_recycle_time".
> 
> Following the pattern of using DeprecatedOpt, we get this:
> 
> cfg.IntOpt('connection_recycle_time',
>default=3600,
>deprecated_opts=[cfg.DeprecatedOpt('idle_timeout',
>   group="DATABASE"),
> cfg.DeprecatedOpt('idle_timeout',
>   group="database"),
> cfg.DeprecatedOpt('sql_idle_timeout',
>   group='DEFAULT'),
> cfg.DeprecatedOpt('sql_idle_timeout',
>   group='DATABASE'),
> cfg.DeprecatedOpt('idle_timeout',
>   group='sql')],
> 
> 
> However, Nova is accessing "conf.idle_timeout" directly in
> nova/db/sqlalcemy/api.py -> _get_db_conf.  Tempest run fails.
> 
> Per the oslo.config documentation, the "deprecated_name" flag would
> create an alias on the conf. namespace.  However, this has no effect,
> even if I remove the other deprecated parameters completely:
> 
> cfg.IntOpt('connection_recycle_time',
>default=3600,
>deprecated_name='idle_timeout',
> 
> a simple unit test fails to see a value for
> conf.connection_recycle_time, including if I add
> "deprecated_group='DATABASE'" which is the group that's in this
> specific test (however this would not be a solution anyway because
> projects use different group names).
> 
> From this, it would appear that oslo.config has made it impossible to
> deprecate the name of an option because DeprecatedOpt() has no means
> of providing the value as an alias on the conf. object.   There's not
> even a way I could have projects like nova make a forwards-compatible
> change here.
> 
> Is this a bug in oslo.config or in oslo.db's usage of oslo.confg?
> 

Option renaming was originally meant as an operatior-facing feature
to handle renames for values coming from the config file, but not
as they are used in code.  mtreinish added
https://review.openstack.org/#/c/357987/ to address this for Tempest,
so it's possible there's a bug in the logic in oslo.config somewhere
(or that oslo.db's case is a new one).

That said, the options defined by a library are *NOT* part of its
API, and should never be used by code outside of the library. The
whole point of isolating options like that is to give operators a
way to change the way an app uses a library (drivers, credentials,
etc.) without the app having to know the details.  Ideally the nova
tests that access oslo.db configuration options directly would
instead use an API in oslo.db to do the same thing (that API may
need to be written, if it doesn't already exist).

At that point, only oslo.db code would refer to the option, and you
could use the deprecated_name and deprecated_group settings to
describe the move and change the references to oslo.db within the
library using a single patch to oslo.db.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] PTL nominations

[Doug Hellmann]

Excerpts from ChangBo Guo's message of 2017-07-17 23:14:54 +0800:
> Hi oslo folks,
> 
> The PTL nomination week is fast approaching [0], and as you might have
> guessed by the subject of this email, I am not planning to run for Queens,
> I'm still in the team and give some guidance about oslo PTL's daily work as
> previous PTL did before .
> It' my honor to be oslo PTL, I learned a lot  and grew quickly. It's time
> to give someone else the opportunity to grow in the amazing role of oslo PTL
> 
> [0]https://review.openstack.org/#/c/481768/4/configuration.yaml
> 

Thank you for leading the team, gcb! I'm glad to know that although
you won't be serving as PTL, you are not stepping away from the
team entirely.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] New setuptools release, and the world is broken

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-14 16:18:04 +:
> On 2017-07-14 16:05:36 + (+), Jesse Pretorius wrote:
> > On 7/14/17, 4:54 PM, "Doug Hellmann" <d...@doughellmann.com> wrote:
> [...]
> > > I wonder if we could convince the PyPA folks to allow get-pip.py
> > >to take a version argument, so we could specify which version we want 
> > > in
> > >our jobs. We would still need a way to manage that version number, but
> > >modifying get-pip.py would solve the bootstrapping issue.
> > 
> > It has been capable of this for quite some time. You can provide
> > both requirements And constraints.
> > 
> > python /opt/get-pip.py pip==9.0.1 setuptools==33.1.1 wheel==0.29.0
> > 
> > Or, far better, is to use constraints because it’ll ensure that
> > any dependent packages are also the right versions.
> > 
> > python /opt/get-pip.py pip setuptools wheel –constraint
> > http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt
> 
> Is there a mechanism to leverage this in tox or when invoking
> virtualenv? We don't run get-pip.py in most jobs because our images
> have pip/setuptools preinstalled to get around bootstrapping issues,
> though I suppose that could with some effort be moved into job
> runtime as a (very early) builder macro. Using constraints to
> control these during image generation doesn't make a whole lot of
> sense though as images are only rebuilt once a day and so tracking
> these in the constraints list won't be self-testing in that regard
> anyway.

I was thinking we would use an early stage builder to do it, too.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] New setuptools release, and the world is broken

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-07-14 15:17:52 +:
> On 2017-07-14 10:50:50 -0400 (-0400), Doug Hellmann wrote:
> > Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +:
> > > FYI if you see the following error in your job logs, you have the new 
> > > setuptools to thank:
> > > 
> > > AttributeError: Distribution instance has no attribute 'install_requires'
> > > 
> > > I’ve registered https://github.com/pypa/setuptools/issues/1086 to track 
> > > the issue, and very quickly got a response and there’s a PR up to resolve 
> > > it.
> > > 
> > > In our case we saw this when downgrading setuptools to our known, good 
> > > working version.
> > > 
> > > I’d like to suggest that we include setuptools, pip, wheel and other core 
> > > packages in the upper constraints management process and that all the 
> > > images built for jobs make use of it. The number of times that a new 
> > > release of pip/setuptools has completely ground development to a halt for 
> > > a day, sometimes more, is a little too frequent for my liking.
> > > 
> > > IIRC we’d need to just change the u-c generation output from ‘pip freeze’ 
> > > to ‘pip freeze –all’ for the output to include the versions for pip, 
> > > setuptools and wheel. Then, with that spec, pip can be installed using 
> > > u-c like so:
> > > 
> > > CURL_CMD="curl --silent --show-error --retry 5"
> > > OUTPUT_FILE="get-pip.py"
> > > ${CURL_CMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUT_FILE} ||\
> > > ${CURL_CMD} 
> > > https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py > 
> > > ${OUTPUT_FILE}
> > > 
> > > python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt
> > > 
> > > That will ensure a consistent, known good version set is installed and 
> > > will also cater for the situation where the primary URL for get-pip.py is 
> > > down (as happens sometimes).
> > > 
> > 
> > I know we made the explicit decision not to pin setuptools, but I don't
> > remember the motivation. Was it a technical decision (we can't) or
> > because it seemed like a bad idea (we want to ensure we have the
> > latest)?
> 
> Chicken and egg. Once you get to the point where pip can enforce
> constraints, you already have a version of setuptools installed. And
> as evidenced by, for example, this current bug you would just end up
> breaking on the downgrade trying to replace your existing broken
> version with whatever version is requested. Also you would need a
> separate phase to upgrade/downgrade setuptools separate from other
> packages using it.

That makes sense.

I wonder if we could convince the PyPA folks to allow get-pip.py
to take a version argument, so we could specify which version we want in
our jobs. We would still need a way to manage that version number, but
modifying get-pip.py would solve the bootstrapping issue.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] New setuptools release, and the world is broken

[Doug Hellmann]

Excerpts from Jesse Pretorius's message of 2017-07-14 08:32:48 +:
> FYI if you see the following error in your job logs, you have the new 
> setuptools to thank:
> 
> AttributeError: Distribution instance has no attribute 'install_requires'
> 
> I’ve registered https://github.com/pypa/setuptools/issues/1086 to track the 
> issue, and very quickly got a response and there’s a PR up to resolve it.
> 
> In our case we saw this when downgrading setuptools to our known, good 
> working version.
> 
> I’d like to suggest that we include setuptools, pip, wheel and other core 
> packages in the upper constraints management process and that all the images 
> built for jobs make use of it. The number of times that a new release of 
> pip/setuptools has completely ground development to a halt for a day, 
> sometimes more, is a little too frequent for my liking.
> 
> IIRC we’d need to just change the u-c generation output from ‘pip freeze’ to 
> ‘pip freeze –all’ for the output to include the versions for pip, setuptools 
> and wheel. Then, with that spec, pip can be installed using u-c like so:
> 
> CURL_CMD="curl --silent --show-error --retry 5"
> OUTPUT_FILE="get-pip.py"
> ${CURL_CMD} https://bootstrap.pypa.io/get-pip.py > ${OUTPUT_FILE} ||\
> ${CURL_CMD} 
> https://raw.githubusercontent.com/pypa/get-pip/master/get-pip.py > 
> ${OUTPUT_FILE}
> 
> python ${OUTPUT_FILE} pip setuptools wheel -c upper-constraints.txt
> 
> That will ensure a consistent, known good version set is installed and will 
> also cater for the situation where the primary URL for get-pip.py is down (as 
> happens sometimes).
> 

I know we made the explicit decision not to pin setuptools, but I don't
remember the motivation. Was it a technical decision (we can't) or
because it seemed like a bad idea (we want to ensure we have the
latest)?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [docs][all] trouble enabling warning-is-error due to "already registered" warnings

[Doug Hellmann]

tl;dr
-

A few project that use wsme.sphinxext or some other extensions that
register similar directives multiple times have been unable to turn on
the warning-is-error flag as part of the doc migration. The solution to
that problem is to add some warning suppression information to the local
sphinx build. See [1] for an example.

Why am I seeing the warning "directive 'foo' is already registered, it
will be overridden"?
--

Our doc build jobs invoke sphinx through pbr by running "tox -e venv --
python setup.py build_sphinx". Doing that, instead of sphinx-build, has
allowed us to set several sphinx configuration options consistently
across all projects without having to land a patch in every repository.
One of those options was to suppress warnings like this, which are
informational but don't actually cause us any problems.

When we updated to Sphinx 1.6.2, the integration in pbr changed in a way
that makes it impossible to pass those options through. That means we
are no longer suppressing the warnings globally. When the
warning-is-error flag is turned on, the informational warning turns into
an error and the build fails.

How do I eliminate the warning?
---

To suppress the warning, set the "suppress_warnings" option in
doc/source/conf.py to the list of names of warnings to be suppressed. By
default the list is empty. You do not want to suppress *all* warnings,
because that would make the warning-is-error flag irrelevant. It is
safe, however, to suppress warnings related to things that extensions
do, like registering directives and roles. This is what pbr used to do.

For example, in [1] I have set the option to:

suppress_warnings = [
'app.add_directive', 'app.add_role',
'app.add_generic_role', 'app.add_node',
'image.nonlocal_uri',
]

This ignores warnings related to extension actions, over which you have
no control, and tells Sphinx that it is OK to refer to images that are
not in the local repository.

Again, please be conservative about adding things to this list. Consider
it like disabling a rule in a code linter.

Doug

[1] https://review.openstack.org/#/c/483953/1/doc/source/conf.py
[2] 
http://www.sphinx-doc.org/en/stable/config.html?highlight=suppress_warnings#confval-suppress_warnings

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][doc] migration update

[Doug Hellmann]

We have recovered old tagged content from
https://docs.openstack.org/developer/$project and moved them to
https://docs.openstack.org/$project/$version. As part of this process,
we also kept any builds from the tip of the mitaka, newton, and ocata
branches using those names as $version. And finally, for the projects
that have had no doc builds since we updated the output location of the
documentation job, we moved their /developer/$project content over to
/$project/latest.

Big thanks to fungi for helping with the restoration!

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][glance] Release countdown for week R-6, July 14 - July 21

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-07-13 17:44:13 +0200:

> glance-store and instack haven't made a Pike release yet: if nothing is
> done by July 20, one release will be forced (on master HEAD) so that we
> have something to cut a stable branch from.

I have prepared a patch for a glance-store 0.21.0 release [1]. It would
be best if the glance team signed off on that before we approve it, but
either way we will ensure there is a release before the deadline next
week.

Doug

[1] https://review.openstack.org/483467

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Announcing openstack-sigs ML

[Doug Hellmann]

Excerpts from Melvin Hillsman's message of 2017-07-13 08:51:16 -0500:

>1. Start the openstack-sigs mailing list

You can subscribe by visiting
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-sigs

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove][all][tc] A proposal to rearchitect Trove

[Doug Hellmann]

Excerpts from Amrith Kumar's message of 2017-07-12 06:14:28 -0500:
> All:
> 
> First, let me thank all of you who responded and provided feedback
> on what I wrote. I've summarized what I heard below and am posting
> it as one consolidated response rather than responding to each
> of your messages and making this thread even deeper.
> 
> As I say at the end of this email, I will be setting up a session at
> the Denver PTG to specifically continue this conversation and hope
> you will all be able to attend. As soon as time slots for PTG are
> announced, I will try and pick this slot and request that you please
> attend.
> 
> 
> 
> Thierry: naming issue; call it Hoard if it does not have a migration
> path.
> 
> 
> 
> Kevin: use a container approach with k8s as the orchestration
> mechanism, addresses multiple issues including performance. Trove to
> provide containers for multiple components which cooperate to provide
> a single instance of a database or cluster. Don't put all components
> (agent, monitoring, database) in a single VM, decoupling makes
> migraiton and upgrades easier and allows trove to reuse database
> vendor supplied containers. Performance of databases in VM's poor
> compared to databases on bare-metal.
> 
> 
> 
> Doug Hellmann:
> 
> > Does "service VM" need to be a first-class thing?  Akanda creates
> > them, using a service user. The VMs are tied to a "router" which is
> > the billable resource that the user understands and interacts with
> > through the API.
> 
> Amrith: Doug, yes because we're looking not just for service VM's but all
> resources provisioned by a service. So, to Matt's comment about a
> blackbox DBaaS, the VM's, storage, snapshots, ... they should all be
> owned by the service, charged to a users quota but not visible to the
> user directly.

I still don't understand. If you have entities that represent the
DBaaS "host" or "database" or "database backup" or whatever, then
you put a quota on those entities and you bill for them. If the
database actually runs in a VM or the backup is a snapshot, those
are implementation details. You don't want to have to rewrite your
quota management or billing integration if those details change.

Doug

> 
> 
> 
> Jay:
> 
> > Frankly, I believe all of these types of services should be built
> > as applications that run on OpenStack (or other)
> > infrastructure. In other words, they should not be part of the
> > infrastructure itself.
> >
> > There's really no need for a user of a DBaaS to have access to the
> > host or hosts the DB is running on. If the user really wanted
> > that, they would just spin up a VM/baremetal server and install
> > the thing themselves.
> 
> and subsequently in follow-up with Zane:
> 
> > Think only in terms of what a user of a DBaaS really wants. At the
> > end of the day, all they want is an address in the cloud where they
> > can point their application to write and read data from.
> > ...
> > At the end of the day, I think Trove is best implemented as a hosted
> > application that exposes an API to its users that is entirely
> > separate from the underlying infrastructure APIs like
> > Cinder/Nova/Neutron.
> 
> Amrith: Yes, I agree, +1000
> 
> 
> 
> Clint (in response to Jay's proposal regarding the service making all
> resources multi-tenant) raised a concern about having multi-tenant
> shared resources. The issue is with ensuring separation between
> tenants (don't want to use the word isolation because this is database
> related).
> 
> Amrith: yes, definitely a concern and one that we don't have today
> because each DB is a VM of its own. Personally, I'd rather stick with
> that construct, one DB per VM/container/baremetal and leave that be
> the separation boundary.
> 
> 
> 
> Zane: Discomfort over throwing out working code, grass is greener on
> the other side, is there anything to salvage?
> 
> Amrith: Yes, there is certainly a 'grass is greener with a rewrite'
> fallacy. But, there is stuff that can be salvaged. The elements are
> still good, they are separable and can be used with the new
> project. Much of the controller logic however will fall by the
> wayside.
> 
> In a similar vein, Clint asks about the elements that Trove provides,
> "how has that worked out".
> 
> Amrith: Honestly, not well. Trove only provided reference elements
> suitable for development use. Never really production hardened
> ones. For example, the image elements trove provides don't bake the
> guest agent in; they assume that at VM launch, the guest agent code
> will b

[openstack-dev] [all][docs] using openstackdocstheme

[Doug Hellmann]

As projects review patches related to the migration project, please
keep in mind that only *official* projects listed as being under
TC governance [1] should use the openstackdocs theme. If your
documentation is not being published to docs.openstack.org, please
use a different theme.

Doug

[1] https://governance.openstack.org/tc/reference/projects/index.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][ironic][telemetry][dragonflow][freezer][kuryr][manila][mistral][monasca][neutron][ansible][congress][rally][senlin][storlets][zun][docs] repos without signs of migration sta

[Doug Hellmann]

Excerpts from Akihiro Motoki's message of 2017-07-11 23:56:30 +0900:
> Thanks for update the status!
> 
> > openstack/networking-midonet
> 
> The doc-migration work networking-midonet has almost completed.
> Unfortunately they do not use 'doc-migration' topic as it seems they
> already started
> the document overhaul before the doc-migration starts.
> https://review.openstack.org/#/q/topic:bug/1692788
> Hopefully midonet folks use 'doc-migration' tag for at least one patch.

OK, that's good to know.

> 
> > openstack/networking-vsphere
> 
> This is not a project under the TC governance.

I have removed it from the list.

Thanks,
Doug

> 
> Akihiro
> 
> 2017-07-11 2:26 GMT+09:00 Doug Hellmann <d...@doughellmann.com>:
> > According to the dashboard, it looks like we still have almost 100
> > repositories with documentation that have no patches with the
> > doc-migration topic, indicating that they have not started moving
> > content or updating the theme. I have tried to tag those teams in the
> > subject, but I may have missed some. Please check the list below for a
> > repo owned by your team.
> >
> > If you have completed the work and the dasbhoard script didn't pick
> > it up, please let me know so I can fix up the data.
> >
> > Doug
> >
> > openstack-dev/heat-cfnclient
> > openstack/bifrost
> > openstack/ceilometer-powervm
> > openstack/ceilometermiddleware
> > openstack/diskimage-builder
> > openstack/dragonflow
> > openstack/freezer-api
> > openstack/freezer-dr
> > openstack/freezer-web-ui
> > openstack/fuxi
> > openstack/fuxi-kubernetes
> > openstack/heat
> > openstack/heat-cfntools
> > openstack/heat-translator
> > openstack/instack
> > openstack/ironic-lib
> > openstack/karbor-dashboard
> > openstack/kolla-kubernetes
> > openstack/manila
> > openstack/manila-image-elements
> > openstack/manila-ui
> > openstack/mistral-dashboard
> > openstack/mistral-extra
> > openstack/mistral-lib
> > openstack/molteniron
> > openstack/monasca-statsd
> > openstack/monasca-transform
> > openstack/networking-hyperv
> > openstack/networking-midonet
> > openstack/networking-vsphere
> > openstack/neutron-lbaas
> > openstack/neutron-lbaas-dashboard
> > openstack/octavia-dashboard
> > openstack/openstack-ansible-apt_package_pinning
> > openstack/openstack-ansible-ceph_client
> > openstack/openstack-ansible-galera_client
> > openstack/openstack-ansible-galera_server
> > openstack/openstack-ansible-haproxy_server
> > openstack/openstack-ansible-lxc_container_create
> > openstack/openstack-ansible-lxc_hosts
> > openstack/openstack-ansible-memcached_server
> > openstack/openstack-ansible-openstack_hosts
> > openstack/openstack-ansible-openstack_openrc
> > openstack/openstack-ansible-os_aodh
> > openstack/openstack-ansible-os_barbican
> > openstack/openstack-ansible-os_ceilometer
> > openstack/openstack-ansible-os_cinder
> > openstack/openstack-ansible-os_designate
> > openstack/openstack-ansible-os_glance
> > openstack/openstack-ansible-os_gnocchi
> > openstack/openstack-ansible-os_heat
> > openstack/openstack-ansible-os_horizon
> > openstack/openstack-ansible-os_ironic
> > openstack/openstack-ansible-os_keystone
> > openstack/openstack-ansible-os_magnum
> > openstack/openstack-ansible-os_molteniron
> > openstack/openstack-ansible-os_neutron
> > openstack/openstack-ansible-os_nova
> > openstack/openstack-ansible-os_octavia
> > openstack/openstack-ansible-os_rally
> > openstack/openstack-ansible-os_sahara
> > openstack/openstack-ansible-os_swift
> > openstack/openstack-ansible-os_tempest
> > openstack/openstack-ansible-os_trove
> > openstack/openstack-ansible-pip_install
> > openstack/openstack-ansible-plugins
> > openstack/openstack-ansible-rabbitmq_server
> > openstack/openstack-ansible-repo_build
> > openstack/openstack-ansible-repo_server
> > openstack/openstack-ansible-rsyslog_client
> > openstack/openstack-ansible-rsyslog_server
> > openstack/openstack-ansible-security
> > openstack/os-net-config
> > openstack/os-win
> > openstack/osc-placement
> > openstack/oslosphinx
> > openstack/pycadf
> > openstack/python-congressclient
> > openstack/python-ironic-inspector-client
> > openstack/python-manilaclient
> > openstack/python-octaviaclient
> > openstack/python-saharaclient
> > openstack/python-tricircleclient
> > openstack/python-tripleoclient
> > ope

Re: [openstack-dev] [heat][ironic][telemetry][dragonflow][freezer][kuryr][manila][mistral][monasca][neutron][ansible][congress][rally][senlin][storlets][zun][docs] repos without signs of migration sta

[Doug Hellmann]

Excerpts from gordon chung's message of 2017-07-11 14:23:41 +:
> 
> On 10/07/17 01:26 PM, Doug Hellmann wrote:
> > openstack/ceilometer-powervm
> > openstack/ceilometermiddleware
> 
> i don't believe there are docs for these. ceilometermiddleware is a 
> simple wsgi middleware and it's usage is part of ceilometer's install 
> docs. ceilometer-powervm contains the powervm driver for ceilometer's 
> polling agent.

I've removed them from the tracking list for now, but it seems like
both are likely to have contributor documentation, at least, and
the driver would likely have installation and configuration docs,
right?

> 
> i missed this but how do we handle smaller add-on type repos like this? 
> i imagine we want to keep docs grouped by project so they are not 
> scattered across the same level.
> 
> cheers,
> 

The new "rule" for docs is "The documentation for something should
live in the same repository as the code."

Hyperlinks are easy and free, so we can use those to ensure that the
results are easy to find. We're building up a nice set of landing pages
for "all of the admin guides" and "all of the configuration references"
and so on within the openstack-manuals repository.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][ironic][telemetry][dragonflow][freezer][kuryr][manila][mistral][monasca][neutron][ansible][congress][rally][senlin][storlets][zun][docs] repos without signs of migration sta

[Doug Hellmann]

Excerpts from Renat Akhmerov's message of 2017-07-11 12:08:23 +0700:
> On 11 Jul 2017, 00:27 +0700, wrote:
> 
> > openstack/mistral-dashboard
> > openstack/mistral-extra
> 
> These two are not supposed to have docs at all. We should probably just 
> remove the “doc” folder and corresponding CI jobs.

OK, I removed them from the list of repos we are tracking.

> 
> >  openstack/mistral-lib
> 
> This should be taken care of soon.

Thanks!


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs] [pbr] [ptl] [oslo] Removing pbr's custom 'build_sphinx' setuptools command

[Doug Hellmann]

Excerpts from Anne Gentle's message of 2017-07-10 13:14:52 -0400:
> On Mon, Jul 10, 2017 at 1:11 PM, Doug Hellmann <d...@doughellmann.com> wrote:
> > Excerpts from sfinucan's message of 2017-07-10 17:20:37 +0100:
> >> On Fri, 2017-07-07 at 15:58 +0100, sfinu...@redhat.com wrote:
> >>
> >> Of these, 'project' is static and should never change, so we don't need to
> >> attempt to guess them. Version and release are dynamic but I've noticed we
> >> don't actually use them anywhere in openstackdocstheme (the version you 
> >> see in
> >> the URL is actually an artefact of the build process and nothing to do with
> >> Sphinx). The closest thing we _do_ get to including version information is 
> >> the
> >> commit ID include in header of api-ref build pages [1], which is generated 
> >> by
> >> openstackdocstheme.
> >
> > It's surprising to me that we don't include the version string anywhere
> > on the page. Is that an oversight in the theme, or was it on purpose?
> 
> The theme shows "current" - however I noticed while reviewing this
> morning the nomenclature is "latest" in the URL, so I will make a
> change. Do you suggest "latest -n.n.n" as the string value, or simply
> "latest"?

I thought we would want to insert the version, as reported in the
Sphinx config, without any reference to current or latest or the
release series.

Perhaps we can replace the static text "Project home page" in the left
sidebar with "{{project}} {{version}}" (with or without "home page")
like in https://review.openstack.org/482230 ?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [docs] migration status

[Doug Hellmann]

Progress


After more than 580 patches, the migration is moving along very
well. We do still have quite a few repositories that haven't been
updated [1], though, so please help us trim that list. It's really
important that we finish the migration before we create stable/pike
branches to avoid having to backport the content.

Recovering Old Content
--

We have started working on a tool [2] to recover "old" versions of
docs associated with tagged releases and stable branches by moving
it out of the /developer directory on the web server into the new
publishing structure. We will be coordinating with the infra team
over the next couple of weeks to move the content. I will send
another announcement when that is done.

The Theme Is Not Enough
---

Several project teams have landed the patch to move from oslosphinx
to openstackdocstheme. Please remember that that is only one of
several steps in the migration, and is not sufficient to complete
the work. The new templating system for docs.openstack.org relies
on a YAML file with booleans indicating when a project has specific
types of content (installation instructions, API reference and/or
guide, configuration reference, etc.). The flag allows the template
to insert a pre-defined URL matching the new documentation layout,
and a gate job verifies that the content exists before the link is
allowed. That means that if you have content published somewhere
other than where it is expected, it *will not be linked* from the
landing pages on docs.openstack.org.

In-Tree API Docs


We have had one or two projects who already have in-tree API
documentation that was not being published to developer.openstack.org
ask about having their links added to docs.openstack.org. That is
in process [3].

Theme Rendering Issues
--

A few folks have also raised issues with the appearance of the docs
after moving to openstackdocs theme. Good, you're reading! Please
files bugs [4] (or patches [5]) for these issues. The theme is still
being updated, and there will be lots of opportunities to address any
problems.

Doug

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-July/119483.html
[2] https://review.openstack.org/481710
[3] https://review.openstack.org/482188
[4] https://launchpad.net/openstack-manuals
[5] http://git.openstack.org/cgit/openstack/openstackdocstheme

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat][ironic][telemetry][dragonflow][freezer][kuryr][manila][mistral][monasca][neutron][ansible][congress][rally][senlin][storlets][zun][docs] repos without signs of migration startin

[Doug Hellmann]

According to the dashboard, it looks like we still have almost 100
repositories with documentation that have no patches with the
doc-migration topic, indicating that they have not started moving
content or updating the theme. I have tried to tag those teams in the
subject, but I may have missed some. Please check the list below for a
repo owned by your team.

If you have completed the work and the dasbhoard script didn't pick
it up, please let me know so I can fix up the data.

Doug

openstack-dev/heat-cfnclient
openstack/bifrost
openstack/ceilometer-powervm
openstack/ceilometermiddleware
openstack/diskimage-builder
openstack/dragonflow
openstack/freezer-api
openstack/freezer-dr
openstack/freezer-web-ui
openstack/fuxi
openstack/fuxi-kubernetes
openstack/heat
openstack/heat-cfntools
openstack/heat-translator
openstack/instack
openstack/ironic-lib
openstack/karbor-dashboard
openstack/kolla-kubernetes
openstack/manila
openstack/manila-image-elements
openstack/manila-ui
openstack/mistral-dashboard
openstack/mistral-extra
openstack/mistral-lib
openstack/molteniron
openstack/monasca-statsd
openstack/monasca-transform
openstack/networking-hyperv
openstack/networking-midonet
openstack/networking-vsphere
openstack/neutron-lbaas
openstack/neutron-lbaas-dashboard
openstack/octavia-dashboard
openstack/openstack-ansible-apt_package_pinning
openstack/openstack-ansible-ceph_client
openstack/openstack-ansible-galera_client
openstack/openstack-ansible-galera_server
openstack/openstack-ansible-haproxy_server
openstack/openstack-ansible-lxc_container_create
openstack/openstack-ansible-lxc_hosts
openstack/openstack-ansible-memcached_server
openstack/openstack-ansible-openstack_hosts
openstack/openstack-ansible-openstack_openrc
openstack/openstack-ansible-os_aodh
openstack/openstack-ansible-os_barbican
openstack/openstack-ansible-os_ceilometer
openstack/openstack-ansible-os_cinder
openstack/openstack-ansible-os_designate
openstack/openstack-ansible-os_glance
openstack/openstack-ansible-os_gnocchi
openstack/openstack-ansible-os_heat
openstack/openstack-ansible-os_horizon
openstack/openstack-ansible-os_ironic
openstack/openstack-ansible-os_keystone
openstack/openstack-ansible-os_magnum
openstack/openstack-ansible-os_molteniron
openstack/openstack-ansible-os_neutron
openstack/openstack-ansible-os_nova
openstack/openstack-ansible-os_octavia
openstack/openstack-ansible-os_rally
openstack/openstack-ansible-os_sahara
openstack/openstack-ansible-os_swift
openstack/openstack-ansible-os_tempest
openstack/openstack-ansible-os_trove
openstack/openstack-ansible-pip_install
openstack/openstack-ansible-plugins
openstack/openstack-ansible-rabbitmq_server
openstack/openstack-ansible-repo_build
openstack/openstack-ansible-repo_server
openstack/openstack-ansible-rsyslog_client
openstack/openstack-ansible-rsyslog_server
openstack/openstack-ansible-security
openstack/os-net-config
openstack/os-win
openstack/osc-placement
openstack/oslosphinx
openstack/pycadf
openstack/python-congressclient
openstack/python-ironic-inspector-client
openstack/python-manilaclient
openstack/python-octaviaclient
openstack/python-saharaclient
openstack/python-tricircleclient
openstack/python-tripleoclient
openstack/python-vitrageclient
openstack/python-zaqarclient
openstack/python-zunclient
openstack/rally
openstack/searchlight-ui
openstack/senlin
openstack/storlets
openstack/sushy
openstack/sushy-tools
openstack/tosca-parser
openstack/virtualbmc
openstack/watcher-dashboard
openstack/yaql
openstack/zun

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs] [pbr] [ptl] [oslo] Removing pbr's custom 'build_sphinx' setuptools command

[Doug Hellmann]

Excerpts from sfinucan's message of 2017-07-10 17:20:37 +0100:
> On Fri, 2017-07-07 at 15:58 +0100, sfinu...@redhat.com wrote:
> > tl;dr: pbr's 'build_spinx' derivative is broken again, and we want to just
> > remove the feature at this point. However, this is going to necessitate some
> > mechanical changes for most projects with docs and this mail serves as a
> > heads up and request for input before we proceed.
> 
> [snip]
> 
> > Here are the features that the plugin provides, along with the current
> > migration plans:
> 
> ...
> 
> > - Automatically sets project name and version using pbr's machinery
> > 
> >   Try to set this from 'openstackdocstheme'. If this isn't possible, folks 
> >   will need to add some some lines to 'conf.py' like keystoneauth does [7]
> 
> I've been looking into this particular issue further, and the more I think
> about it, the less necessary it seems. There are three configuration options
> currently set by pbr:
> 
> - project (the project name)
> - version (the full version, including alpha/beta/rc tags)
> - release (the short X.Y version)
> 
> Of these, 'project' is static and should never change, so we don't need to
> attempt to guess them. Version and release are dynamic but I've noticed we
> don't actually use them anywhere in openstackdocstheme (the version you see in
> the URL is actually an artefact of the build process and nothing to do with
> Sphinx). The closest thing we _do_ get to including version information is the
> commit ID include in header of api-ref build pages [1], which is generated by
> openstackdocstheme.

It's surprising to me that we don't include the version string anywhere
on the page. Is that an oversight in the theme, or was it on purpose?

> Given this fact, I think pbr is a providing a solution in search of problem
> here, and this feature can in fact go quietly into the night with no further
> ado.
> 
> Thoughts?
> Stephen
> 
> PS: If we really did want to include a version in the docs in the future, I
> think we could use information provided by ZUUL. I'm no ZUUL expert, but it
> seems ZUUL does have commit id info ('ZUUL_REF') and what I hope to be
> something like what 'git-describe' ('ZUUL_REFNAME'). We could simply pass 
> these
> via the '--version' parameter to 'build_sphinx' [3]. Again though, I don't
> think this is necessary.

Relying on anything outside of the repo won't work for packagers who
build from source outside of our CI system.

> PPS: I have not responded to the other replies to this mail yet because Red
> Hat's email servers have decided not to send me replies to my own posts on
> openstack-dev. I have seen them though and will reply when I figure out how to
> get mboxes.
> 
> [1] https://developer.openstack.org/api-ref/compute/
> [2] 
> https://github.com/openstack-infra/zuul/blob/8dda7c1/tools/trigger-job.py#L
> 54-L60
> [3] 
> https://github.com/openstack-infra/project-config/blob/32aff86f/jenkins/scr
> ipts/run-docs.sh#L20
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] scheduling oslosphinx for retirement at the start of queens

[Doug Hellmann]

Oslo team,

With all documentation now moving to use the openstackdocs theme,
I propose that we retire the oslosphinx repository during Queens.
We should go ahead and create the stable/pike branch at the end of
this cycle, so that we have a way to deal with bugs in existing
pike releases, but I think we can retire the repository at any point
after that.

Thoughts?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs] [pbr] [ptl] [oslo] Removing pbr's custom 'build_sphinx' setuptools command

[Doug Hellmann]

Excerpts from sfinucan's message of 2017-07-07 15:58:10 +0100:
> tl;dr: pbr's 'build_spinx' derivative is broken again, and we want to just
> remove the feature at this point. However, this is going to necessitate some
> mechanical changes for most projects with docs and this mail serves as a heads
> up and request for input before we proceed.
> 
> --
> 
> Since pretty much the beginning, pbr has provided a custom 'build_sphinx'
> setuptools command [1], which can be run like so:
> 
> python setup.py build_sphinx
> 
> This is described in the pbr docs [2] and is is based on Sphinx's own
> 'build_sphinx' command [3], which is described in the Sphinx docs [4].
> 
> Historically, this custom version has provided a couple of features that the
> upstream Sphinx extension didn't. These are outlined below. However, this
> derivative has resulted in an inordinately large number of bugs for what 
> should
> be a simple feature (#1702872, #1691129, #1681983, #1674795, #1379998,
> #1496882, ...). This is due to the lack of extension points in the upstream
> code and the way that we've been forced to extend the command (we break almost
> every time Sphinx change their code).
> 
> The latest of these bugs, #1702872, breaks a couple of things when using 
> Sphinx
> > 1.6:
> 
> - We no longer provide confoverrides when using Sphinx 1.6+, which means you
>   may see "node class 'xxx' is already registered' errors, and the project
>   name and version used in the docs may be incorrect [5]
> - The '[build_sphinx] builders' in 'setup.cfg' option is no longer read, so
>   only 'html' docs will be built (no man pages)
> - The apidoc tool won't be automatically run
> 
> Fixing these issues is possible in the short-term and won't make the thing any
> more maintainable going forward. Given the amount of issues this feature has
> caused and the fact that Sphinx has since gained many of the features we
> provided itself, we'd like to just drop the whole thing now. Some of the extra
> this feature provide are no longer necessary, but the migration paths for the
> ones that are are not great and will require mechanical changes to projects.
> 
> Here are the features that the plugin provides, along with the current
> migration plans:
> 
> - Automatically runs sphinx-apidoc [6] before building docs
> 
>   It seems that adding the 'sphinx.ext.apidoc' extension to the 'extensions'
>   list in 'doc/source/conf.py' will ensure this continues to happen
>   automatically. However, I need to check this and will get back when I have.
> 
> - Automatically runs 'autodoc', which seems to do something similar to 
> 'apidoc'
>   but in a different way (I don't know what, tbh)
> 
>   Drop this feature entirely as 'apidoc' appears to do something very similar.
> 
> - Automatically sets project name and version using pbr's machinery
> 
>   Try to set this from 'openstackdocstheme'. If this isn't possible, folks 
> will
>   need to add some some lines to 'conf.py' like keystoneauth does [7]

I'm inclined to just say these values should be set explicitly. We can
simplify it (Monty had an idea for a 1-liner that would set all of the
variables with one function call into pbr), but the project name doesn't
change so there's no real sense in doing a lot of fancy automation for
that one.

> - Supports multiple builders via the '[build_sphinx] builders' option in
>   'setup.cfg'
> 
>   This is natively supported since Sphinx 1.6. You should replace
>   '[build_sphinx] builders' with '[build_sphinx] builder' (singular) now

That would only work if we continue to invoke Sphinx using its
setuptools plugin in the gate. Is that what we want to do? We also
discussed changing the CI interface to build docs to use the "tox -e
docs" command like contributors generally run locally.

> - Historically, supported turning build warning into build falures via the
>   '[build_sphinx] warnerrors' option in 'setup.cfg'
> 
>   This is no longer supported by pbr, as Sphinx now provides a '[build_sphinx]
>   warning-is-error' option. I wrote a load of patches to fix this recently
>   using the topic 'sphinx15', but if I missed your project you need to remove
>   '[pbr] warnerrors' from 'setup.cfg' and add '[build_sphinx]
>   warning-is-error'. You should do this now.

This is also one of the doc-migration steps. Some projects that are
hitting errors caused by the missing config overrides are going to need
to wait to turn the flag on until after we figure out how to work around
those issues.

> Does anyone see any issues with this strategy going forward? If not, we're
> going to start making changes to projects using the 'sphinx16' topic and would
> appreciate reviews on these as they come in. We're not going to get to every
> project, so if you can make the necessary changes to your own project then
> please do.

Let me know if you want some help setting up a burndown chart like
https://doughellmann.com/doc-migration/. Having a tracking etherpad so
folks know whether a repo 

Re: [openstack-dev] [tc] Status update, July 7th

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-07-07 10:19:48 +0200:
> 
> == Need for a TC meeting next Tuesday ==
> 
> I propose we have a meeting next week to discuss the next steps in
> establishing the vision. I feel like we should approve it soon,
> otherwise we'll get too close to the vision date (Spring 2019)... We
> also need to wrap up the goals (selecting the two and deferring the
> others). Who is up for discussing those items at our usual meeting slot
> time on Tuesday ?

+1

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [docs][ptls] redirects for migrated docsets

[Doug Hellmann]

We've had a few folks notice that redirects from docs.o.o/$project/
to docs.o.o/$project/latest are missing. The htaccess file for those
redirects is now built from a template. To appear in the list,
projects need to be added to the docs repo data file [1]. We have
been adding projects as we add/update the landing page templates,
but you can write a patch to add your project if you want to help
out and make sure we don't miss you.  Directions are in the README.txt
in the same directory.

Doug

[1] 
http://git.openstack.org/cgit/openstack/openstack-manuals/tree/www/project-data/latest.yaml

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Wiki (was: How to deal with confusion around "hosted projects")

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-07-03 16:11:44 +0200:
> On 03/07/17 13:58 +0200, Thierry Carrez wrote:
> >Flavio Percoco wrote:
> >> Sometimes I wonder if we still need to maintain a Wiki. I guess some
> >> projects still use it but I wonder if the use they make of the Wiki could 
> >> be moved
> >> somewhere else.
> >>
> >> For example, in the TC we use it for the Agenda but I think that could be 
> >> moved
> >> to an etherpad. Things that should last forever should be documented 
> >> somewhere
> >> (project repos, governance repo in the TC case) where we can actually 
> >> monitor
> >> what goes in and easily clean up.
> >
> >This is a complete tangent, but I'll bite :) We had a thorough
> >discussion about that last year, summarized at:
> >
> >http://lists.openstack.org/pipermail/openstack-dev/2016-June/096481.html
> >
> >TL,DR; was that while most authoritative content should (and has been
> >mostly) moved off the wiki, it's still useful as a cheap publication
> >platform for teams and workgroups, somewhere between a git repository
> >with a docs job and an etherpad.
> >
> >FWIW the job of migrating authoritative things off the wiki is still
> >on-going. As an example, Thingee is spearheading the effort to move the
> >"How to Contribute" page and other first pointers to a reference website
> >(see recent thread about that).
> 
> I guess the short answer is that we hope one day we won't need it. I certainly
> do.
> 
> What would happen if we make the wiki read-only? Would that break peopl's
> workflow?
> 
> Do we know what teams modify the wiki more often and what it is they do there?
> 
> Thanks for biting :)
> Flavio
> 

The docs team is looking for operators to take over the operators guide
and move that content to the wiki (operators have said they don't want
to deal with gerrit reviews).

Please don't make the wiki read-only.

Doug


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][all][ptl] doc-migration status

[Doug Hellmann]

> On Jul 3, 2017, at 10:34 AM, Doug Hellmann <d...@doughellmann.com> wrote:
> 
> Because we still have a lot of work to do in the openstack-manuals
> repo, the documentation team has started landing patches to remove
> the content that is being migrated into project repositories. If
> you have not yet started your migration, check out the commit tagged
> "before-migration" in openstack-manuals to find the content to be
> migrated.

One of the patches we landed redirects from docs.o.o/developer/$project/ to 
docs.o.o/$project/latest/. The docs won’t exist for some projects in that 
location until they land a patch to trigger their doc build job, so if you are 
seeing a 404 review and approve a patch to fix it.

Doug


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs][all][ptl] doc-migration status

[Doug Hellmann]

> On Jul 3, 2017, at 10:34 AM, Doug Hellmann <d...@doughellmann.com> wrote:
> 
> After a busy week last week, we made some good progress with the
> documentation migration project. We still have quite a few repositories
> that haven't been started, though, so please keep working. If you
> need assistance preparing or reviewing the patches, let us know
> with a follow-up email to this thread or by asking in #openstack-docs.

Oops, that channel name should have been #openstack-doc.

Doug


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [docs][all][ptl] doc-migration status

[Doug Hellmann]

After a busy week last week, we made some good progress with the
documentation migration project. We still have quite a few repositories
that haven't been started, though, so please keep working. If you
need assistance preparing or reviewing the patches, let us know
with a follow-up email to this thread or by asking in #openstack-docs.

Because we still have a lot of work to do in the openstack-manuals
repo, the documentation team has started landing patches to remove
the content that is being migrated into project repositories. If
you have not yet started your migration, check out the commit tagged
"before-migration" in openstack-manuals to find the content to be
migrated.

As part of this work today we have also prepared a patch to move
all remaining projects to the new documentation build jobs. This
will cause any content from doc/source to be published to
docs.o.o/$project/latest instead of docs.o.o/developer/$project.
Project teams still need to reorganize this content as described
in the spec to ensure the automatically generated landing pages
point to the correct URLs.

We need to complete this migration work before creating stable
branches, to avoid having to backport the documentation changes.

Over the course of last week, we had a few questions about the
directions and about reviews.

1. Please use the gerrit topic "doc-migration" for all of the patches
related to this work. We are monitoring the available changes by
using that topic in queries, and we won't see your patches to help
with reviews if you use another topic. Tying your patch to the
blueprint using "Blueprint: doc-migration" will set the topic of
the patch to "bp/doc-migration", and the patches will not appear
in the query results.

2. Please work on the steps in the order given. We have had quite
a few patches that only apply the new theme, without reorganizing
the content for the projects. These projects will not be linked
properly from the templated landing pages, so your users will have
trouble finding your documentation.

3. Only official OpenStack projects, under TC governance, should
use the openstackdocs theme. Projects publishing their documentation
to readthedocs.org or other sites should not use the theme, and do
not need to take any of the other migration steps.

4. Reviewers, please prioritize landing the intiial content import
and ask for follow-up patches to fix issues, unless the content is
just absolutely wrong. Whitespace issues, typos, etc. should all
be fixed after the initial import of the content created by the
documentation team is complete.

5. Projects with a "deployment guide" instead of "install guide"
need to make sure they are using the openstackdocs theme and to
update the organization of the rest of their content. The deployment
guide should stay where it is, for now.

6. We've had a couple of cases where multiple people did the same
work because neither put their name next to the repository in the
list on https://etherpad.openstack.org/p/doc-migration-tracking.
We have plenty of this work to go around! Please use the etherpad
to sign up for the repo you are going to work on *before* starting
to write patches, so that we can avoid duplicating effort!

If you run into trouble or have other questions, please follow-up
here or ping us in the #openstack-doc IRC channel on Freenode.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [doc][all][ptl] doc migration issue with sphinxcontrib-pecanwsme

[Doug Hellmann]

A couple of teams have run into a problem with the
sphinxcontrib-pecanwsme extension emitting warnings which are then
treated as errors when the Sphinx flag warning-is-error is set to True.
For the time being, skip the step of setting that flag to True (but
still try to minimize the number of warnings in your doc build).

We could use some help resolving the warning, so if you're interested in
learning how sphinx extensions work ping me in #openstack-docs and I'll
try to get you started.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] How to deal with confusion around "hosted projects"

[Doug Hellmann]

Excerpts from Chris Friesen's message of 2017-06-29 09:35:01 -0600:
> On 06/29/2017 09:23 AM, Monty Taylor wrote:
> 
> > We are already WELL past where we can solve the problem you are describing.
> > Pandora's box has been opened - we have defined ourselves as an Open 
> > community.
> > Our only requirement to be official is that you behave as one of us. There 
> > is
> > nothing stopping those machine learning projects from becoming official. If 
> > they
> > did become official but were still bad software - what would we have solved?
> >
> > We have a long-time official project that currently has staffing problems. 
> > If
> > someone Googles for OpenStack DBaaS and finds Trove and then looks to see 
> > that
> > the contribution rate has fallen off recently they could get the impression 
> > that
> > OpenStack is a bunch of dead crap.
> >
> > Inclusion as an Official Project in OpenStack is not an indication that 
> > anyone
> > thinks the project is good quality. That's a decision we actively made. 
> > This is
> > the result.
> 
> I wonder if it would be useful to have a separate orthogonal status as to 
> "level 
> of stability/usefulness/maturity/quality" to help newcomers weed out projects 
> that are under TC governance but are not ready for prime time.
> 
> Chris
> 

It would be. When we made the shift away from the old incubation
process to our current 4-opens-based governance process, we said
the TC was not necessarily the right body to be making that call.
We may have the expertise for one project, but not another. We also
said that because different people could reasonably have different
opinions based on different criteria, The Market or The Community
should produce that information and share it.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

Excerpts from Andreas Jaeger's message of 2017-06-28 08:45:47 +0200:
> For those migrating from oslosphinx to openstackdocstheme, I strongly
> advise to follow the docs for openstackdocstheme 1.11 on how to set it up:
> 
> https://docs.openstack.org/openstackdocstheme/latest/
> 
> Andreas

Another work item is moving the automatically generated class
documentation that pbr produces by setting the api_doc_dir value
in the pbr section of setup.cfg.

See https://docs.openstack.org/developer/pbr/user/using.html#pbr
for details and https://review.openstack.org/#/c/478294/2/setup.cfg
for an example.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

Excerpts from Alexandra Settle's message of 2017-06-23 12:09:41 +:
> Hi everyone,
> 
> This morning (afternoon) the specification for the documentation migration 
> was merged. Thanks to all that took time to review :)
> 
> You can now view here in all its glory: 
> https://specs.openstack.org/openstack/docs-specs/specs/pike/os-manuals-migration.html
> 
> If you have any questions, feel free to shoot them to me or Doug :)
> 
> Let’s begin!
> 
> Alex

Based on some work Sean did with the nova API reference project, I've
set up a burndown chart to track our progress. It updates hourly.

https://doughellmann.com/doc-migration/

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] How to deal with confusion around "hosted projects"

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-06-28 16:50:01 +0200:
> Hi everyone,
> 
> Two weeks ago, as a result of a discussion at the Board+TC+UC workgroup
> working on "better communicating what is openstack", I started a
> thread[1] about moving away from big tent terminology. The thread went
> in a lot of directions, including discussing GitHub mirroring strategy,
> what makes projects want to be official, the need for returning to a
> past when everything was (supposedly) simpler, and naming fun.
> 
> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118368.html
> 
> Many agreed that the "Big Tent" name (as a synonym to "official
> openstack projects") is hurting more than it helps, and we should stop
> using it. The issue is, merely stopping using it won't be enough. We
> have tried, and the name sticks. You need to replace the name by
> something that sticks more, or address the root cause directly.
> 
> The central issue being discussed here is an issue of external
> perception. It's hard for newcomers to the OpenStack world to see what
> is a part of OpenStack and what's not. If you google "openstack machine
> learning", the first hits are Cognitive and Meteos, and it's impossible
> to tell that those are actually not OpenStack projects. One of those has
> been dead for 2 years -- having people think that those are official
> projects hurts all the OpenStack projects, by lowering expectations
> around what that means, in terms of quality, maintenance, or community.
> 
> The confusion mainly stems from the fact that OpenStack project
> infrastructure is open to any open source project (and it's nobody's job
> to clean up dead things). So you can find (on our wiki, on our
> mailing-list, on our cgit farm, on our gerrit, on our GitHub
> organization...) things that are actually not OpenStack, even with the
> expansive "are you one of us" definition. Arguably the most confusing
> aspect is the "openstack/" prefix in the git repository name, which
> indicates some kind of brand association.
> 
> I'd say we have two options. We can address the perception issue on the
> edges, or you can treat the root cause. Neither of those options is
> really an OpenStack  governance change (or "big tent" change) -- they
> are more about what to do with things that are actually *not* in our
> governance.
> 
> Addressing the perception on the edges means making it clearer when
> things are not official. The thread above discussed a lot of potential
> solutions. We could give unofficial things a catchy group name
> (Stackforge, Opium, Electrons...), and hope it sticks. We could find a
> way to tag all projects on GitHub that are not official, or mirror them
> to another organization, or stop mirroring them altogether. We could
> remove the openstack/ prefix from all the projects we host. We could
> actively mark all wiki pages that are not about an official project. We
> could set up a separate Gerrit or separate ML for hosted projects
> development discussions. The main issue with that approach is that it's
> a *lot* of work, and it never completely eliminates the confusion.
> 
> Removing the root cause would be a more radical move: stop offering
> hosting to non-OpenStack projects on OpenStack infrastructure
> altogether. We originally did that for a reason, though. The benefits of
> offering that service are:
> 
> 1- it lets us set up code repositories and testing infrastructure before
> a project applies to be an official OpenStack project.
> 
> 2- it lets us host things that are not openstack but which we work on
> (like abandoned Python libraries or GPL-licensed things) in a familiar
> environment
> 
> 3- it spreads "the openstack way" (Gerrit, Zuul) beyond openstack itself
> 
> I would argue that we could handle (1) and (2) within our current
> governance.
> 
> For (1) we could have an "onboarding" project team that would help
> incoming projects through the initial steps of becoming an openstack
> project. The team would act as an umbrella team, an experimental area
> for projects that have some potential to become an OpenStack project one
> day. There would be a time limit -- if after one year(?) it looks like
> you won't become an openstack project after all, the onboarding team
> would clean you up. I actually think a bit more project mentoring would
> serve us better than our current hands-free approach.
> 
> For (2) we could also have some other official project team as an
> umbrella for those deps we depend on and have to continue maintaining.
> Or we could expand Oslo's team scope to cover it. It's just a couple of
> repositories anyway.
> 
> That leaves (3). I would argue that was a nice thing to have, but its
> impact was very limited (not so many successful/alive projects in that
> category). I guess if the need is still there and people really want to
> work on this, it could be (and actually has been) set up as a parallel
> infrastructure. The confusion that stems from running it 

Re: [openstack-dev] [stable][release] Last release date vs End of Life date

[Doug Hellmann]

Excerpts from Tony Breeds's message of 2017-06-27 16:51:37 +1000:
> Hi all,
> Up 'til now we haven't set a last release date for a stable branch
> approaching end of life.  It seems like formalizing that would be a good
> thing.
> 
> This comes up as we need time to verify that said release integrates
> well (at least doesn't break) said branch.  So should we define a date
> for the last release for *libraries* services are less critical as we're
> always testing the HEAD of that branch.
> 
> I'd suggest it be 2 weeks before EOL date.  Thoughts?
> 
> Yours Tony.

That makes sense.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-06-27 17:55:57 +:
> On 2017-06-22 16:27:34 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > The new job is configured to update the docs for all repos every
> > time a patch is merged, not just when we tag releases. The server
> > projects have been working that way, but this is different for some
> > of the libraries, especially the clients.
> [...]
> 
> I think the past concern had been that since the default document
> presented was the latest one built from the master branch tip rather
> than a redirect to the documentation for the latest release, readers
> might get confused when seeing options or behaviors documented which
> didn't match the software they had downloaded.

That makes sense. The openstackdocstheme makes it easy to link to
specific versions of documentation, so we should be able to address this
concern that way. We will also have series-specific landing pages
linking directly to the appropriate guides.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

Excerpts from Alexandra Settle's message of 2017-06-08 15:17:34 +:
> Hi everyone,
> 
> Doug and I have written up a spec following on from the conversation [0] that 
> we had regarding the documentation publishing future.
> 
> Please take the time out of your day to review the spec as this affects 
> *everyone*.
> 
> See: https://review.openstack.org/#/c/472275/
> 
> I will be PTO from the 9th – 19th of June. If you have any pressing concerns, 
> please email me and I will get back to you as soon as I can, or, email Doug 
> Hellmann and hopefully he will be able to assist you.
> 
> Thanks,
> 
> Alex
> 
> [0] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117162.html

Someone left a few questions about the directions on the tracking
etherpad [1]. I tried to answer them in place, but next time *please*
post to the mailing list. I have no idea how long those questions were
there before I noticed them, and given how little time we have to do the
work I want to make sure everyone understands as soon as possible.

Doug

[1] https://etherpad.openstack.org/p/doc-migration-tracking

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

> On Jun 23, 2017, at 12:10 PM, j.kl...@cloudbau.de wrote:
> 
> Hi Doug,
> 
> first of all sorry for the late response. I am the current PTL of the 
> openstack-chef project and read the spec a while ago, but did not really 
> connect it to our project. To be honest, i am not really sure what we would 
> move and how to do it. Currently we have some wiki pages, but most of them 
> are pretty old and completely outdated since we stopped caring for them after 
> we dropped to only a few contributors and moved our focus to maintaining the 
> project code itself.
> 
> Currently we own and maintain 12 service and ops cookbook repositories:
> 
> cookbook-openstack-block-storage
> cookbook-openstack-common
> cookbook-openstack-compute
> cookbook-openstack-dashboard
> cookbook-openstack-identity
> cookbook-openstack-image
> cookbook-openstack-integration-test
> cookbook-openstack-network
> cookbook-openstack-ops-database
> cookbook-openstack-ops-messaging
> cookbook-openstack-orchestration
> cookbook-openstack-telemetry
> 
> one (not very often used) specs repository:
> 
> openstack-chef-specs
> 
>  and one repo to integrate them all:
> 
> openstack-chef-repo
> 
> All of these repos have some READMEs that contain some of the documentation 
> needed to use these. The documentation on how to use the openstack-chef 
> project as a combination of all these cookbooks, is located under 
> https://github.com/openstack/openstack-chef-repo/tree/master/doc 
> <https://github.com/openstack/openstack-chef-repo/tree/master/doc> (which 
> might already be close to the right space ?). Looking through the 
> openstack-manuals repo, i did not find any documentation specific for our 
> projects, so i think we do not need to export/import any of it. In my opinion 
> the easiest way to follow the proposed change of us would be to just move the 
> stuff we have under the ‘openstack-chef-repo’ mentioned above and add some 
> more files to follow the ‘minimal layout’. If you agree on this, i can try to 
> push a patch for it next week and add you as a reviewer?

Yes, it sounds like you only need to move things you already have around into 
the new structure. You don’t need to create empty directories or pages for 
content that doesn’t apply, so if you don’t have any configuration guide 
instructions for example then you don’t need to create the configuration/ 
directory.

If you add me as a reviewer, I’ll help make sure you have it organized as 
expected. And if you use the doc-migration tag the other folks working on the 
migration will review the patch, too.

Doug

> 
> Cheers,
> Jan
> 
> 
> On 23. June 2017 at 16:16:58, Doug Hellmann (d...@doughellmann.com 
> <mailto:d...@doughellmann.com>) wrote:
> 
>> 
>>> On Jun 23, 2017, at 8:57 AM, Renat Akhmerov <renat.akhme...@gmail.com 
>>> <mailto:renat.akhme...@gmail.com>> wrote:
>>> 
>>> I can say for Mistral . We only planned to add Mistral docs to the central 
>>> repo but didn’t do it yet. So, as far as I understand, we don’t need to 
>>> move anything. We’ll review the spec and adjust the folder structure 
>>> according to the proposed.
>> 
>> Please do review the steps in the spec. Not all of them are about moving 
>> content. There are steps for setting up the new build job so that the 
>> content will be published to the new URLs as well.
>> 
>> Doug
>> 
>>> 
>>> Thanks
>>> 
>>> Renat Akhmerov
>>> @Nokia
>>> 
>>> 23 июня 2017 г., 3:32 +0700, Doug Hellmann <d...@doughellmann.com 
>>> <mailto:d...@doughellmann.com>>, писал:
>>>> Excerpts from Alexandra Settle's message of 2017-06-08 15:17:34 +:
>>>>> Hi everyone,
>>>>> 
>>>>> Doug and I have written up a spec following on from the conversation [0] 
>>>>> that we had regarding the documentation publishing future.
>>>>> 
>>>>> Please take the time out of your day to review the spec as this affects 
>>>>> *everyone*.
>>>>> 
>>>>> See: https://review.openstack.org/#/c/472275/ 
>>>>> <https://review.openstack.org/#/c/472275/>
>>>>> 
>>>>> I will be PTO from the 9th – 19th of June. If you have any pressing 
>>>>> concerns, please email me and I will get back to you as soon as I can, 
>>>>> or, email Doug Hellmann and hopefully he will be able to assist you.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Alex
>>>>> 
>>>>> [0] 
>>>>> http://lis

Re: [openstack-dev] [ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

> On Jun 23, 2017, at 8:57 AM, Renat Akhmerov <renat.akhme...@gmail.com> wrote:
> 
> I can say for Mistral . We only planned to add Mistral docs to the central 
> repo but didn’t do it yet. So, as far as I understand, we don’t need to move 
> anything. We’ll review the spec and adjust the folder structure according to 
> the proposed.

Please do review the steps in the spec. Not all of them are about moving 
content. There are steps for setting up the new build job so that the content 
will be published to the new URLs as well.

Doug

> 
> Thanks
> 
> Renat Akhmerov
> @Nokia
> 
> 23 июня 2017 г., 3:32 +0700, Doug Hellmann <d...@doughellmann.com>, писал:
>> Excerpts from Alexandra Settle's message of 2017-06-08 15:17:34 +:
>>> Hi everyone,
>>> 
>>> Doug and I have written up a spec following on from the conversation [0] 
>>> that we had regarding the documentation publishing future.
>>> 
>>> Please take the time out of your day to review the spec as this affects 
>>> *everyone*.
>>> 
>>> See: https://review.openstack.org/#/c/472275/
>>> 
>>> I will be PTO from the 9th – 19th of June. If you have any pressing 
>>> concerns, please email me and I will get back to you as soon as I can, or, 
>>> email Doug Hellmann and hopefully he will be able to assist you.
>>> 
>>> Thanks,
>>> 
>>> Alex
>>> 
>>> [0] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117162.html
>> 
>> Andreas pointed out that the new documentation job will behave a
>> little differently from the old setup, and thought I should mention
>> it so that people aren't surprised.
>> 
>> The new job is configured to update the docs for all repos every
>> time a patch is merged, not just when we tag releases. The server
>> projects have been working that way, but this is different for some
>> of the libraries, especially the clients.
>> 
>> I will be going back and adding a step to build the docs when we
>> tag releases after the move actually starts, so that we can link
>> to docs for specific versions of projects. That change will be
>> transparent to everyone else, so I have it on the list for after
>> the migration is under way.
>> 
>> Doug
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

Excerpts from Alexandra Settle's message of 2017-06-08 15:17:34 +:
> Hi everyone,
> 
> Doug and I have written up a spec following on from the conversation [0] that 
> we had regarding the documentation publishing future.
> 
> Please take the time out of your day to review the spec as this affects 
> *everyone*.
> 
> See: https://review.openstack.org/#/c/472275/
> 
> I will be PTO from the 9th – 19th of June. If you have any pressing concerns, 
> please email me and I will get back to you as soon as I can, or, email Doug 
> Hellmann and hopefully he will be able to assist you.
> 
> Thanks,
> 
> Alex
> 
> [0] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117162.html

Andreas pointed out that the new documentation job will behave a
little differently from the old setup, and thought I should mention
it so that people aren't surprised.

The new job is configured to update the docs for all repos every
time a patch is merged, not just when we tag releases. The server
projects have been working that way, but this is different for some
of the libraries, especially the clients.

I will be going back and adding a step to build the docs when we
tag releases after the move actually starts, so that we can link
to docs for specific versions of projects. That change will be
transparent to everyone else, so I have it on the list for after
the migration is under way.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable]requirements] Bootstrapiing requirements-stable-core

[Doug Hellmann]

Excerpts from Tony Breeds's message of 2017-06-22 11:48:09 +1000:
> Hi All,
> Recently it's been clear that we need a requirements-stable team.
> Until npw that's been handled by the release managers and the
> stable-maint-core team.
> 
> With the merge of [1] The have the groundwork for that team.  I'd like
> to nominate:
> 
>  * dmllr -- Dirk Mueller
>  * prometheanfire -- Matthew Thode
>  * SeanM -- Sean McGinnis
> 
> As that initial team.  Each of them has been doing regular reviews on
> stable branches and have shown an understanding of how the stable policy
> applies to the requirements repo.
> 
> Yours Tony.
> 
> [1] https://review.openstack.org/#/c/470419/

+1 for all of them

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kuryr][Release-job-failures] Release of openstack/kuryr-tempest-plugin failed

[Doug Hellmann]

It looks like the kuryr-tempest-plugin repository has a documentation
job set up but no documentation.

http://logs.openstack.org/1e/1ee40b0f0ee4e92209b8ccff6d74f4980f6234ab/release/kuryr-tempest-plugin-docs-ubuntu-xenial/7f096fd/console.html#_2017-06-20_17_03_00_590629

--- Begin forwarded message from jenkins ---
From: jenkins 
To: release-job-failures 
Date: Tue, 20 Jun 2017 17:08:35 +
Subject: [Release-job-failures] Release of openstack/kuryr-tempest-plugin failed

Build failed.

- kuryr-tempest-plugin-tarball 
http://logs.openstack.org/1e/1ee40b0f0ee4e92209b8ccff6d74f4980f6234ab/release/kuryr-tempest-plugin-tarball/791b9b2/
 : SUCCESS in 3m 00s
- kuryr-tempest-plugin-tarball-signing 
http://logs.openstack.org/1e/1ee40b0f0ee4e92209b8ccff6d74f4980f6234ab/release/kuryr-tempest-plugin-tarball-signing/f74ada9/
 : SUCCESS in 42s
- kuryr-tempest-plugin-pypi-both-upload 
http://logs.openstack.org/1e/1ee40b0f0ee4e92209b8ccff6d74f4980f6234ab/release/kuryr-tempest-plugin-pypi-both-upload/43ac1e9/
 : SUCCESS in 30s
- kuryr-tempest-plugin-announce-release 
http://logs.openstack.org/1e/1ee40b0f0ee4e92209b8ccff6d74f4980f6234ab/release/kuryr-tempest-plugin-announce-release/65d7a48/
 : SUCCESS in 5m 07s
- propose-kuryr-tempest-plugin-update-constraints 
http://logs.openstack.org/1e/1ee40b0f0ee4e92209b8ccff6d74f4980f6234ab/release/propose-kuryr-tempest-plugin-update-constraints/8b85c19/
 : SUCCESS in 23s
- kuryr-tempest-plugin-docs-ubuntu-xenial 
http://logs.openstack.org/1e/1ee40b0f0ee4e92209b8ccff6d74f4980f6234ab/release/kuryr-tempest-plugin-docs-ubuntu-xenial/7f096fd/
 : FAILURE in 3m 32s

--- End forwarded message ---

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove][all][tc] A proposal to rearchitect Trove

[Doug Hellmann]

Excerpts from Curtis's message of 2017-06-19 18:56:25 -0600:
> On Sun, Jun 18, 2017 at 5:35 AM, Amrith Kumar  wrote:
> > Trove has evolved rapidly over the past several years, since integration in
> > IceHouse when it only supported single instances of a few databases. Today
> > it supports a dozen databases including clusters and replication.
> >
> > The user survey [1] indicates that while there is strong interest in the
> > project, there are few large production deployments that are known of (by
> > the development team).
> >
> > Recent changes in the OpenStack community at large (company realignments,
> > acquisitions, layoffs) and the Trove community in particular, coupled with a
> > mounting burden of technical debt have prompted me to make this proposal to
> > re-architect Trove.
> >
> > This email summarizes several of the issues that face the project, both
> > structurally and architecturally. This email does not claim to include a
> > detailed specification for what the new Trove would look like, merely the
> > recommendation that the community should come together and develop one so
> > that the project can be sustainable and useful to those who wish to use it
> > in the future.
> >
> > TL;DR
> >
> > Trove, with support for a dozen or so databases today, finds itself in a
> > bind because there are few developers, and a code-base with a significant
> > amount of technical debt.
> >
> > Some architectural choices which the team made over the years have
> > consequences which make the project less than ideal for deployers.
> >
> > Given that there are no major production deployments of Trove at present,
> > this provides us an opportunity to reset the project, learn from our v1 and
> > come up with a strong v2.
> >
> > An important aspect of making this proposal work is that we seek to
> > eliminate the effort (planning, and coding) involved in migrating existing
> > Trove v1 deployments to the proposed Trove v2. Effectively, with work
> > beginning on Trove v2 as proposed here, Trove v1 as released with Pike will
> > be marked as deprecated and users will have to migrate to Trove v2 when it
> > becomes available.
> >
> > While I would very much like to continue to support the users on Trove v1
> > through this transition, the simple fact is that absent community
> > participation this will be impossible. Furthermore, given that there are no
> > production deployments of Trove at this time, it seems pointless to build
> > that upgrade path from Trove v1 to Trove v2; it would be the proverbial
> > bridge from nowhere.
> >
> > This (previous) statement is, I realize, contentious. There are those who
> > have told me that an upgrade path must be provided, and there are those who
> > have told me of unnamed deployments of Trove that would suffer. To this, all
> > I can say is that if an upgrade path is of value to you, then please commit
> > the development resources to participate in the community to make that
> > possible. But equally, preventing a v2 of Trove or delaying it will only
> > make the v1 that we have today less valuable.
> >
> > We have learned a lot from v1, and the hope is that we can address that in
> > v2. Some of the more significant things that I have learned are:
> >
> > - We should adopt a versioned front-end API from the very beginning; making
> > the REST API versioned is not a ‘v2 feature’
> >
> > - A guest agent running on a tenant instance, with connectivity to a shared
> > management message bus is a security loophole; encrypting traffic,
> > per-tenant-passwords, and any other scheme is merely lipstick on a security
> > hole
> >
> > - Reliance on Nova for compute resources is fine, but dependence on Nova VM
> > specific capabilities (like instance rebuild) is not; it makes things like
> > containers or bare-metal second class citizens
> >
> > - A fair portion of what Trove does is resource orchestration; don’t
> > reinvent the wheel, there’s Heat for that. Admittedly, Heat wasn’t as far
> > along when Trove got started but that’s not the case today and we have an
> > opportunity to fix that now
> >
> > - A similarly significant portion of what Trove does is to implement a
> > state-machine that will perform specific workflows involved in implementing
> > database specific operations. This makes the Trove taskmanager a stateful
> > entity. Some of the operations could take a fair amount of time. This is a
> > serious architectural flaw
> >
> > - Tenants should not ever be able to directly interact with the underlying
> > storage and compute used by database instances; that should be the default
> > configuration, not an untested deployment alternative
> >
> 
> As an operator I wouldn't run Trove as it is, unless I absolutely had to.
> 
> I think it is a good idea to reboot the project. I really think the
> concept of "service VMs" should be a thing. I'm not sure where the
> OpenStack community has landed on that, my fault for not paying close
> 

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
> So I'm trying to figure out how to actually use it.
> 
> We (and any other container based deploy..) will run into some
> chicken/egg problem - you need to deploy container to generate big
> yaml with defaults, then you need to overload it with your

The config schema file (the "big YAML with defaults") should be part of
the packaged software, so the deployment tool shouldn't need to generate
it unless you're handling drivers that are not included in tree.

> configurations, validate if they're not deprecated, run container with

It doesn't do it today, but the thing that converts the input data to
the INI file could automatically translate old option names to their new
names.

> this ansible role (or module...really doesn't matter), spit out final

Why does the config file need to be generated inside a container?

> confg, lay it down, deploy container again. And that will have to be
> done for every host class (as configs might differ host to host). Imho
> a bit too much for this to be appealing (but I might be wrong). I'd
> much rather have:
> 1. Yaml as input to oslo.config instead of broken ini

I'm not opposed to switching to YAML, but it's a bit more involved than
just adding support in the parser. All of the work that has been done on
generating sample default files and documentation needs to be updated to
support YAML. We need a migration path to move everyone from INI to
YAML. And we need to update devstack and all of its plugins to edit the
new file format. There are probably more tasks involved in the
migration. I'm dealing with a couple of other projects right now, and
don't have time to plan all of that out myself. If someone else wants to
pick it up, I can help with reviews on the spec and code changes.

> 2. Validator to throw an error if one of our regular,
> template-rendered, configs is deprecated
> 
> We can run this validator in gate to have quick feedback when
> something gets deprecated.
> 
> Thoughts?
> Michal
> 
> On 16 June 2017 at 13:24, Emilien Macchi  wrote:
> > On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský  wrote:
> >> On 15.6.2017 19:06, Emilien Macchi wrote:
> >>>
> >>> I missed [tripleo] tag.
> >>>
> >>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi 
> >>> wrote:
> 
>  If you haven't followed the "Configuration management with etcd /
>  confd" thread [1], Doug found out that using confd to generate
>  configuration files wouldn't work for the Cinder case where we don't
>  know in advance of the deployment what settings to tell confd to look
>  at.
>  We are still looking for a generic way to generate *.conf files for
>  OpenStack, that would be usable by Deployment tools and operators.
>  Right now, Doug and I are investigating some tooling that would be
>  useful to achieve this goal.
> 
>  Doug has prototyped an Ansible role that would generate configuration
>  files by consumming 2 things:
> 
>  * Configuration schema, generated by Ben's work with Machine Readable
>  Sample Config.
> $ oslo-config-generator --namespace cinder --format yaml >
>  cinder-schema.yaml
> 
>  It also needs: https://review.openstack.org/#/c/474306/ to generate
>  some extra data not included in the original version.
> 
>  * Parameters values provided in config_data directly in the playbook:
>  config_data:
>    DEFAULT:
>  transport_url: rabbit://user:password@hostname
>  verbose: true
> 
>  There are 2 options disabled by default but which would be useful for
>  production environments:
>  * Set to true to always show all configuration values:
>  config_show_defaults
>  * Set to true to show the help text: config_show_help: true
> 
>  The Ansible module is available on github:
>  https://github.com/dhellmann/oslo-config-ansible
> 
>  To try this out, just run:
> $ ansible-playbook ./playbook.yml
> 
>  You can quickly see the output of cinder.conf:
>   https://clbin.com/HmS58
> 
> 
>  What are the next steps:
> 
>  * Getting feedback from Deployment Tools and operators on the concept
>  of this module.
> Maybe this module could replace what is done by Kolla with
>  merge_configs and OpenStack Ansible with config_template.
>  * On the TripleO side, we would like to see if this module could
>  replace the Puppet OpenStack modules that are now mostly used for
>  generating configuration files for containers.
> A transition path would be having Heat to generate Ansible vars
>  files and give it to this module. We could integrate the playbook into
>  a new task in the composable services, something like
> "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>  also driven 

Re: [openstack-dev] [tc] Status update, Jun 16

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-06-16 11:17:30 +0200:

> == Need for a TC meeting next Tuesday ==
> 
> In order to make progress on the Pike goal selection, I think a
> dedicated IRC meeting will be necessary. We have a set of valid goals
> proposed already: we need to decide how many we should have, and which
> ones. Gerrit is not great to have that ranking discussion, so I think we
> should meet to come up with a set, and propose it on the mailing-list
> for discussion. We could use the regular meeting slot on Tuesday,
> 20:00utc. How does that sound ?
> 

+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][qa][glance] some recent tempest problems

[Doug Hellmann]

Excerpts from Ghanshyam Mann's message of 2017-06-16 23:05:08 +0900:
> On Fri, Jun 16, 2017 at 10:57 PM, Sean Dague  wrote:
> > On 06/16/2017 09:51 AM, Sean McGinnis wrote:
> >>>
> >>> It would be useful to provide detailed examples. Everything is trade
> >>> offs, and having the conversation in the abstract is very difficult to
> >>> understand those trade offs.
> >>>
> >>>  -Sean
> >>>
> >>
> >> We've had this issue in Cinder and os-brick. Usually around Ceph, but if
> >> you follow the user survey, that's the most popular backend.
> >>
> >> The problem we see is the tempest test that covers this is non-voting.
> >> And there have been several cases so far where this non-voting job does
> >> not pass, due to a legitimate failure, but the tempest patch merges anyway.
> >>
> >>
> >> To be fair, these failures usually do point out actual problems that need
> >> to be fixed. Not always, but at least in a few cases. But instead of it
> >> being addressed first to make sure there is no disruption, it's suddenly
> >> a blocking issue that holds up everything until it's either reverted, 
> >> skipped,
> >> or the problem is resolved.
> >>
> >> Here's one recent instance: https://review.openstack.org/#/c/471352/
> >
> > Sure, if ceph is the primary concern, that feels like it should be a
> > reasonable specific thing to fix. It's not a grand issue, it's a
> > specific mismatch on what configs should be common.
> 
> yea, we had such cases and decided to have blacklist of tests not
> suitable for ceph. ceph job will exclude the tests failing on ceph.
> Jon is working on this - https://review.openstack.org/#/c/459774/
> 
> This approach solve the problem without limiting tests scope. [1]
> 
> ..1 http://lists.openstack.org/pipermail/openstack-dev/2017-May/116172.html
> 
> -gmann

Is ceph behaving in an unexpected way or are the tests are making
implicit assumptions that might also cause trouble for other backends
if these tests ever make it into the suite used by the interop team?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Moving away from "big tent" terminology

[Doug Hellmann]

Excerpts from gordon chung's message of 2017-06-15 20:24:06 +:
> 
> On 15/06/17 03:23 PM, Doug Hellmann wrote:
> >
> > We are very open with our hosting, allowing projects that have not
> > yet, and may never, sign up to be governed by the TC to use our
> > infrastructure services. We expect them to be related in some way,
> > but we have even imported projects when we've taken over maintenance
> > (several Oslo libs fall into this category, as do a few others like
> > mox3 and sqlalchemy-migrate). With the move away from stackforge,
> > and other changes in that hosting (that were made for good reasons
> > to make the infra team's lives easier and to make it simpler for a
> > project to join the set of governed projects), we have removed most
> > of the other technical signals about which projects are in that
> > "official" list and which are not.  We did not at the same time
> > remove all of the people in the world who want to understand what
> > is, and what is not, "in" OpenStack.
> 
> i see, so this is less an existential question of 'what is openstack' 
> and more 'how to differentiate governance projects from a random repo 
> created last weekend'
> 
> this might have been just me, but big tent was exactly 'big tent == 
> governance' so when i read 'moving away from "big tent"' i think 'what 
> is this *new* thing we're moving to and if we're redefining this new 
> thing, what for?'. it seems this is not the case.

No. We're trying to pick new words, because there continues to be
confusion about the old words.

> > And for the record, from the TC's perspective, being a governed
> > project has nothing to do with whether the participants are sponsors
> > of the foundation.
> 
> sorry, i probably wasn't clear, i simply noticed that it was a corporate 
> sponsor that was misusing the 'big tent' name so was just thinking we 
> could easily tell them, that's not what it means. wasn't suggesting 
> anything else by sponsor comment.

You'd think it would be that easy. A surprising number of folks
within the community don't really understand the old naming either,
though (see the rest of this thread for examples).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][glance][nova][neutron][horizon][cinder][osc][swift][manila][telemetry][heat][ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-06-12 11:43:25 -0400:
> I added subject tags for the projects most affected by this change. It
> would be good to have the PTLs or liaisons from those teams review the
> spec so there are no surprises when we start moving files around.

I have set up patches for oslo.config, glance, glance client,
python-openstackclient, and horizon for folks to use as examples
[1]. I've also updated the tracking etherpad [2] with some more
detailed directions.

I hope the examples will answer any remaining questions about the
plan and PTLs will sign-off on the spec so we can move forward in
earnest next week.

Doug

[1] https://review.openstack.org/#/q/topic:doc-migration
[2] https://etherpad.openstack.org/p/doc-migration-tracking

> 
> Excerpts from Alexandra Settle's message of 2017-06-08 15:17:34 +:
> > Hi everyone,
> > 
> > Doug and I have written up a spec following on from the conversation [0] 
> > that we had regarding the documentation publishing future.
> > 
> > Please take the time out of your day to review the spec as this affects 
> > *everyone*.
> > 
> > See: https://review.openstack.org/#/c/472275/
> > 
> > I will be PTO from the 9th – 19th of June. If you have any pressing 
> > concerns, please email me and I will get back to you as soon as I can, or, 
> > email Doug Hellmann and hopefully he will be able to assist you.
> > 
> > Thanks,
> > 
> > Alex
> > 
> > [0] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117162.html
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][qa][glance] some recent tempest problems

[Doug Hellmann]

Excerpts from Brian Rosmaita's message of 2017-06-15 13:04:39 -0400:
> This isn't a glance-specific problem though we've encountered it quite
> a few times recently.
> 
> Briefly, we're gating on Tempest jobs that tempest itself does not
> gate on.  This leads to a situation where new tests can be merged in
> tempest, but wind up breaking our gate. We aren't claiming that the
> added tests are bad or don't provide value; the problem is that we
> have to drop everything and fix the gate.  This interrupts our current
> work and forces us to prioritize bugs to fix based not on what makes
> the most sense for the project given current priorities and resources,
> but based on whatever we can do to get the gates un-blocked.
> 
> As we said earlier, this situation seems to be impacting multiple projects.
> 
> One solution for this is to change our gating so that we do not run
> any Tempest jobs against Glance repositories that are not also gated
> by Tempest.  That would in theory open a regression path, which is why
> we haven't put up a patch yet.  Another way this could be addressed is
> by the Tempest team changing the non-voting jobs causing this
> situation into voting jobs, which would prevent such changes from
> being merged in the first place.  The key issue here is that we need
> to be able to prioritize bugs based on what's most important to each
> project.
> 
> We want to be clear that we appreciate the work the Tempest team does.
> We abhor bugs and want to squash them too.  The problem is just that
> we're stretched pretty thin with resources right now, and being forced
> to prioritize bug fixes that will get our gate un-blocked is
> interfering with our ability to work on issues that may have a higher
> impact on end users.
> 
> The point of this email is to find out whether anyone has a better
> suggestion for how to handle this situation.
> 
> Thanks!
> 
> Erno Kuvaja
> Glance Release Czar
> 
> Brian Rosmaita
> Glance PTL
> 

Asymmetric gating definitely has a way of introducing these problems.

Which jobs are involved?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][fuel] Making Fuel a hosted project

[Doug Hellmann]

Excerpts from Jay Pipes's message of 2017-06-15 12:06:53 -0400:
> On 06/15/2017 11:59 AM, Thierry Carrez wrote:
> > Jay Pipes wrote:
> >> While I personally agree that Fuel should be moved out of the official
> >> projects list, I'd like to point out that Triple-O is virtually entirely
> >> a Red Hat project:
> >>
> >> http://stackalytics.com/?module=tripleo-group
> >> http://stackalytics.com/?module=tripleo-group=commits
> >>
> >> so the fact that a project is entirely run by a single vendor or "has
> >> popularity outside one vendor's customer base" has not been and
> >> continues not to be a deciding factor on whether something is an
> >> official OpenStack project or not.
> > 
> > Right, it's certainly not sufficient reason. The main difference between
> > the two is that activity in TripleO is actually growing, so there is
> > still a chance that it may attract a more diverse base in the future.
> 
> Sure, it may very well be. My point was that team diversity isn't a 
> defining characteristic of "officialness" in OpenStack. If it was, we'd 
> have far fewer "official" projects.
> 
> Best,
> -jay
> 

Right. We said that we would not block projects from joining just
because the contributors were mostly coming from one source *because* we
wanted to let teams attract new contributors, and we were told that some
companies would only allow their employees to work on official projects.

In my mind, it is far more important that the project doesn't seem
very healthy/active right now. I would be happy to hear that I am
wrong in that impression, though.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Moving away from "big tent" terminology

[Doug Hellmann]

Excerpts from gordon chung's message of 2017-06-15 18:56:22 +:
> 
> On 15/06/17 02:05 PM, Davanum Srinivas wrote:
> > Example from https://www.meetup.com/openstack/events/237621777/
> > "Platform9 recently open-sourced Project Mors and VM HA as part of the
> > OpenStack Big Tent initiative."
> 
> ah i see, i imagine you could correct those who are corporate sponsors 
> (and send in the suits for those who aren't. j/k).
> 
> it seems like we want to drop "big tent" because marketers ruined it but 
> i'm still not sure we've formalised why we need a replacement (not 
> saying we don't).
> 

One of the *most* common complaints the TC gets from outside the
contributor community is that people do not understand what projects
are part of OpenStack and what parts are not. We have a clear
definition of that in our minds (the projects that have said they
want to be part of OpenStack, and agreed to put themselves under
TC governance, with all of the policies that implies). That definition
is so trivial to say, that it seems like a tautology.  However,
looking in from the outside of the community, that definition isn't
helpful.

We are very open with our hosting, allowing projects that have not
yet, and may never, sign up to be governed by the TC to use our
infrastructure services. We expect them to be related in some way,
but we have even imported projects when we've taken over maintenance
(several Oslo libs fall into this category, as do a few others like
mox3 and sqlalchemy-migrate). With the move away from stackforge,
and other changes in that hosting (that were made for good reasons
to make the infra team's lives easier and to make it simpler for a
project to join the set of governed projects), we have removed most
of the other technical signals about which projects are in that
"official" list and which are not.  We did not at the same time
remove all of the people in the world who want to understand what
is, and what is not, "in" OpenStack.

So, we need to find a way to answer that question. As Thierry said,
one way is to have 2 names to describe the 2 states. Big Tent used
to be one such name, except that we have learned that term was
unclear to a bunch of people (insert joke about how hard naming
things can be here).

Other communities don't seem to have this problem, because they either
don't host projects that are not part of their umbrella, or they don't
have projects moving in and out of governance so often.

And for the record, from the TC's perspective, being a governed
project has nothing to do with whether the participants are sponsors
of the foundation. The minute it does, I will step down. As part
of continuing to have a healthy community, we want to attract new
members, regardless of whether they are coming with money (and in
fact being an individual member is free).  We look at whether
projects (a) ask to be and (b) follow the guidelines we've set down.
We do look at who is contributing, but only when we consider the
various team diversity tags, none of which are required to be a
governed project.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][fuel] Making Fuel a hosted project

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-06-15 10:48:21 +0200:
> Hi everyone,
> 
> Part of reducing OpenStack perceived complexity is to cull projects that
> have not delivered on their initial promises. Those are always difficult
> discussions, but we need to have them. In this email I'd like to discuss
> whether we should no longer consider Fuel an official OpenStack project,
> and turn it into a hosted (unofficial) project.
> 
> Fuel originated at Mirantis as their OpenStack installer. It was
> proposed as an official OpenStack project in July 2015 and approved in
> November 2015. The promise at that time was that making it official
> would drive other organizations to participate in its development and
> turn it into the one generic OpenStack installer that everyone wanted.
> Fuel was not a small endeavor: in Mitaka and Newton it represented more
> commits than Nova.
> 
> The Fuel team fully embraced open collaboration, but failed to attract
> other organizations. Mitaka and Newton were still 96% the work of
> Mirantis. In my view, while deployment/packaging tools sit at the
> periphery of the "OpenStack" map, they make sense as official OpenStack
> teams if they create an open collaboration playing field and attract
> multiple organizations. Otherwise they are just another opinionated
> install tool that happens to be blessed with an "official" label.
> 
> Since October 2016, Fuel's activity has dropped, following the gradual
> disengagement of its main sponsor. Comparing activity in the 5 first
> months of the year, there was a 68% drop between 2016 and 2017, the
> largest of any official OpenStack project. The Fuel team hasn't met on
> IRC for the last 3 months. Activity dropped from ~990 commits/month (Apr
> 2016, Aug 2016) to 52 commits in April 2017 and 25 commits in May 2017.
> And there are unsolved issues around licensing that have been lingering
> for the last 6 months.
> 
> I think that, despite the efforts of the Fuel team, Fuel did not become
> what we hoped when we made it official: a universal installer that would
> be used across the board. It was worth a try, I'm happy that we tried,
> but I think it's time to stop considering it a part of "OpenStack"
> proper and make it a hosted project. It can of course continue its
> existence as an unofficial project hosted on OpenStack infrastructure.
> 
> Thoughts ?
> 

IIRC, they are hosting their release artifacts on a Mirantis server,
too. I agree, the project was never fully "upstreamed" in the way we
hoped.

+1 for changing the project status.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][glance][nova][neutron][horizon][cinder][osc][swift][manila][telemetry][heat][ptls][all][tc][docs] Documentation migration spec

[Doug Hellmann]

I added subject tags for the projects most affected by this change. It
would be good to have the PTLs or liaisons from those teams review the
spec so there are no surprises when we start moving files around.

Excerpts from Alexandra Settle's message of 2017-06-08 15:17:34 +:
> Hi everyone,
> 
> Doug and I have written up a spec following on from the conversation [0] that 
> we had regarding the documentation publishing future.
> 
> Please take the time out of your day to review the spec as this affects 
> *everyone*.
> 
> See: https://review.openstack.org/#/c/472275/
> 
> I will be PTO from the 9th – 19th of June. If you have any pressing concerns, 
> please email me and I will get back to you as soon as I can, or, email Doug 
> Hellmann and hopefully he will be able to assist you.
> 
> Thanks,
> 
> Alex
> 
> [0] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117162.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][requirements][all] removing global pins for linters

[Doug Hellmann]

Excerpts from Sean Dague's message of 2017-06-12 07:09:12 -0400:
> On 06/11/2017 11:30 AM, Doug Hellmann wrote:
> > The recent thread about updating the pylint version in
> > global-requirements.txt raised an issue because it was trying to
> > update the pylint version for all projects using it, but some teams
> > were not ready for the new tests in the latest version. I thought
> > we had dealt with that kind of case in the past by treating linter
> > projects like pylint and flake8 as special cases, and leaving them
> > out of the global requirements list. The requirements repo has a
> > separate file (blacklist.txt) for projects that should not be synced
> > into repositories and tested against the global-requirements.txt
> > list, and pylint is included there along with several other linter
> > tools.
> > 
> > I'm not sure why the linters were also being added to
> > global-requirements.txt, but it seems like a mistake. I have proposed
> > a patch [2] to remove them, which should allow projects that want
> > to update pylint to do so while not forcing everyone to update at
> > the same time. If we find issues with the requirements sync after
> > removing the entries from the global list, we should fix the syncing
> > scripts so we can keep the linters blacklisted.
> > 
> > Doug
> > 
> > [1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118085.html
> > [2] https://review.openstack.org/473094
> 
> Are you sure that works as expected? My understanding is that the
> requirements enforcement jobs only let you set requirements lines to
> what are in that file. So that effectively prevents anyone from changing
> the linters lines ever (see
> http://logs.openstack.org/69/473369/1/check/gate-nova-requirements/b425844/console.html)
> 
> -Sean
> 

Thanks. https://review.openstack.org/473402 should take care of that.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][requirements][all] removing global pins for linters

[Doug Hellmann]

The recent thread about updating the pylint version in
global-requirements.txt raised an issue because it was trying to
update the pylint version for all projects using it, but some teams
were not ready for the new tests in the latest version. I thought
we had dealt with that kind of case in the past by treating linter
projects like pylint and flake8 as special cases, and leaving them
out of the global requirements list. The requirements repo has a
separate file (blacklist.txt) for projects that should not be synced
into repositories and tested against the global-requirements.txt
list, and pylint is included there along with several other linter
tools.

I'm not sure why the linters were also being added to
global-requirements.txt, but it seems like a mistake. I have proposed
a patch [2] to remove them, which should allow projects that want
to update pylint to do so while not forcing everyone to update at
the same time. If we find issues with the requirements sync after
removing the entries from the global list, we should fix the syncing
scripts so we can keep the linters blacklisted.

Doug

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118085.html
[2] https://review.openstack.org/473094

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] Stepping down from core

[Doug Hellmann]

Excerpts from Roman Podoliaka's message of 2017-06-11 17:32:49 +0300:
> Hi all,
> 
> I recently changed job and hasn't been able to devote as much time to
> oslo.db as it is expected from a core reviewer. I'm no longer working
> on OpenStack, so you won't see me around much.
> 
> Anyway, it's been an amazing experience to work with all of you! Best
> of luck! And see ya at various PyCon's around the world! ;)
> 
> Thanks,
> Roman
> 

Thanks for your help launching oslo.db and making it so useful,
Roman.  We'll miss your contributions!

Good luck with your new job,
Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-06-09 16:52:25 +:
> On Fri, Jun 9, 2017 at 11:30 AM Britt Houser (bhouser) 
> wrote:
> 
> > How does confd run inside the container?  Does this mean we’d need some
> > kind of systemd in every container which would spawn both confd and the
> > real service?  That seems like a very large architectural change.  But
> > maybe I’m misunderstanding it.
> >
> >
> Copying part of my reply to Doug's email:
> 
> 1. Run confd + openstack service in side the container. My concern in this
> case
> would be that we'd have to run 2 services inside the container and structure
> things in a way we can monitor both services and make sure they are both
> running. Nothing impossible but one more thing to do.
> 
> 2. Run confd `-onetime` and then run the openstack service.
> 
> 
> I either case, we could run confd as part of the entrypoint and have it run
> in
> background for the case #1 or just run it sequentially for case #2.

I think all of this is moot unless we can solve the case where we don't
know in advance of the deployment what settings to tell confd to look at
(what I've been calling the "cinder case", since that's where I saw it
come up first).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements][mistral][tripleo][horizon][nova][releases] release models for projects tracked in global-requirements.txt

[Doug Hellmann]

Excerpts from Alex Schultz's message of 2017-06-09 10:54:16 -0600:
> I ran into a case where I wanted to add python-tripleoclient to
> test-requirements for tripleo-heat-templates but it's not in the
> global requirements. In looking into adding this, I noticed that
> python-tripleoclient and tripleo-common are not
> cycle-with-intermediary either. Should/can we update these as well?
> tripleo-common is already in the global requirements but I guess since
> we've been releasing non-prerelease versions fairly regularly with the
> milestones it hasn't been a problem.

Yes, let's get all of the tripleo team's libraries onto the
cycle-with-intermediary release model.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Call for check: is your project ready for pylint 1.7.1?

[Doug Hellmann]

Excerpts from Akihiro Motoki's message of 2017-06-09 03:53:34 +0900:
> Hi all,
> 
> Is your project ready for pylint 1.7.1?
> If you use pylint in your pep8 job, it is worth checked.
> 
> Our current version of pylint is 1.4.5 but it is not safe in python 3.5.
> The global-requirements update was merged once [1],
> However, some projects (at least neutron) are not ready for pylint
> 1.7.1 and it was reverted [2].
> it is reasonable to give individual projects time to cope with pylint 1.7.1.
> 
> I believe bumping pylint version to 1.7.1 (or later) is the right
> direction in long term.
> I would suggest to make your project ready for pylint 1.7.1 soon (two
> weeks or some?)
> You can disable new rules in pylint 1.7.1 temporarily and clean up
> your code later
> as neutron does [3]. As far as I checked, most rules are reasonable
> and worth enabled.
> 
> Thanks,
> Akihiro Motoki
> 
> [1] https://review.openstack.org/#/c/470800/
> [2] https://review.openstack.org/#/c/471756/
> [3] https://review.openstack.org/#/c/471763/
> 

I thought we had linters in a list that didn't require the versions
to be synced across projects, to allow projects to update at their
own pace. Did we undo that work?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release][glance][barbican][telemetry][keystone][designate][congress][magnum][searchlight][swift][tacker] unreleased libraries

[Doug Hellmann]

We have several teams with library deliverables that haven't seen
any releases at all yet this cycle. Please review the list below,
and if there are changes on master since the last release prepare
a release request.  Remember that because of the way our CI system
works, patches that land in libraries are not used in tests for
services that use the libs unless the library has a release and the
constraints list is updated.

Doug

glance-store
instack
pycadf
python-barbicanclient
python-ceilometerclient
python-congressclient
python-designateclient
python-keystoneclient
python-magnumclient
python-searchlightclient
python-swiftclient
python-tackerclient
requestsexceptions

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [release][barbican][congress][designate][neutron][zaqar] missing pike-2 milestone releases

[Doug Hellmann]

We have several projects with deliverables following the
cycle-with-milestones release model without pike 2 releases. Please
check the list below and prepare those release requests as soon as
possible. Remember that this milestone is date-based, not feature-based,
so unless your gate is completely broken there is no reason to wait to
tag the milestone.

Doug

barbican
congress
designate-dashboard
designate
networking-bagpipe
networking-bgpvpn
networking-midonet
networking-odl
networking-ovn
networking-sfc
neutron-dynamic-routing
neutron-fwaas
neutron
zaqar-ui
zaqar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Doug Hellmann's message of 2017-06-07 14:00:38 -0400:
> Excerpts from Emilien Macchi's message of 2017-06-07 16:42:13 +0200:
> > On Wed, Jun 7, 2017 at 3:31 PM, Doug Hellmann <d...@doughellmann.com> wrote:
> > >
> > > On Jun 7, 2017, at 7:20 AM, Emilien Macchi <emil...@redhat.com> wrote:
> > >
> > > I'm also wondering if we could use oslo-config-generate directly to
> > > generate confd templates, with a new format. So we would have ini,
> > > yaml, json and confd.
> > > "confd" format would be useful when building rpms that we ship in
> > > containers.
> > > "yaml" format would be useful for installers to expose the options
> > > directly to the User Interface, so we know which params OpenStack
> > > provide and we could re-use the data to push it into etcd.
> > >
> > > Would it make sense?
> > >
> > >
> > > I did think about making oslo-config-generator also take the YAML file as
> > > input instead of scanning plugins, and then including all the output 
> > > formats
> > > in the single command. I haven’t looked to see how much extra complexity
> > > that would add.
> > 
> > Do you mean taking the YAML file that we generate with Ben's work
> > (which would include the parameters values, added by some other
> > tooling maybe)?
> > 
> > I see 2 options at least:
> > 
> > * Let installers to feed etcd with the parameters by using this etcd
> > namespace $CUSTOM_PREFIX + /project/section/parameter (example
> > /node1/keystone/DEFAULT/debug).
> >   And patch oslo.config to be able to generate confd templates with
> > all the options (and ship the template in the package)
> >   I like this option because it provides a way for operators to learn
> > about all possible options in the configuration, with documentation
> > and default values.
> > 
> > * Also let installers to feed etcd but use a standard template like
> > you showed me last week (credits to you for the code):
> > http://paste.openstack.org/show/2KZUQsWYpgrcG2K8TDcE/
> >I like this option because nothing has to be done in oslo.config,
> > since we use a standard template for all OpenStack configs (see the
> > paste ^)
> > 
> > Thoughts?

[My apologies, I sent this reply directly to Emilien the first time.]

> There are 2 problems with using the generic template.
> 
> 1. In order for confd to work, you have to give it a list of all of the
>keys in etcd that it should monitor, and that list is
>application-specific.
> 
> 2. Not all of our configuration values are simple strings or numbers.
>We have options for managing lists of values, and there is even
>an Opt class for loading a dictionary for some reason. So,
>rendering the value in the template will depend on the type of
>the option.
> 
> Given those constraints, it makes sense to generate a custom template
> for each set of options. We need to generate the confd file anyway, and
> the template can have the correct logic for rendering mult-value
> options.
> 

> One further problem I don't know how to address yet is the applications
> that use dynamic sections in configuration files. I think Cinder
> is still the primary example of this, but other apps may use that
> ability.  I don't know how to tell confd that it needs to look at
> the keys in those groups, since we don't know the names in advance.

The more I think about dealing with configuration files, the more
I think this is going to be the killer issue. If an application
doesn't know what sections go in the file, it can't monitor the
right parts of etcd or any other database looking at individual
settings.

The configmap approach assumes that something publishes the entire
INI file, which at least moves the problem outside of the container
to a place where we've already implemented the logic to deal with
the dynamic aspect of the configuration files.

Using configmap to inject config files, we gain the ability to have
a full accurate INI file but lose the ability to monitor the file
for updates and have mutable options.  Given that we're running the
service in a container, and starting a new container is easy, maybe
that's fine.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:

> Unless I'm missing something, to use confd with an OpenStack deployment on
> k8s, we'll have to do something like this:
> 
> * Deploy confd in every node where we may want to run a pod (basically
> wvery node)

Oh, no, no. That's not how it works at all.

confd runs *inside* the containers. It's input files and command line
arguments tell it how to watch for the settings to be used just for that
one container instance. It does all of its work (reading templates,
watching settings, HUPing services, etc.) from inside the container.

The only inputs confd needs from outside of the container are the
connection information to get to etcd. Everything else can be put
in the system package for the application.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

> On Jun 8, 2017, at 4:29 PM, Fox, Kevin M  wrote:
> 
> That is possible. But, a yaml/json driver might still be good, regardless of 
> the mechanism used to transfer the file.
> 
> So the driver abstraction still might be useful.

Why would it be useful to have oslo.config read files in more than one format?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Emilien Macchi's message of 2017-06-08 22:20:34 +0200:
> On Thu, Jun 8, 2017 at 8:49 PM, Doug Hellmann <d...@doughellmann.com> wrote:
> > Excerpts from Steven Dake (stdake)'s message of 2017-06-08 17:51:48 +:
> >> Doug,
> >>
> >> In short, a configmap takes a bunch of config files, bundles them in a 
> >> kubernetes object called a configmap, and then ships them to etcd.  When a 
> >> pod is launched, the pod mounts the configmaps using tmpfs and the raw 
> >> config files are available for use by the openstack services.
> >
> > That sounds like what confd does. Something puts data into one of
> > several possible databases. confd takes it out and writes it to
> > file(s) when the container starts. The app in the container reads
> > the file(s).
> >
> > It sounds like configmaps would work well, too, it just doesn't
> > sound like a fundamentally different solution.
> 
> Sorry for my lack of knowledge in ConfigMap but I'm trying to see how
> we could bring pieces together.
> Doug and I are currently investigating how oslo.config can be useful
> to generate the parameters loaded by the application at startup,
> without having to manage config with Puppet or Ansible.
> 
> If I understand correctly (and if not, please correct me, and maybe
> propose something), we could use oslo.config to generate a portion of
> ConfigMap (that can be imported in another ConfigMap iiuc) where we
> would have parameters for one app.
> 
> Example with Keystone:
> 
>   apiVersion: v1
>   kind: ConfigMap
>   metadata:
> name: keystone-config
> namespace: DEFAULT
>   data:
> debug: true
> rpc_backend: rabbit
> ... (parameters generated by oslo.config, and data fed by installers)
> 
> So iiuc we would give this file to k8s when deploying pods. Parameters
> values would be automatically pushed into etcd, and used when
> generating the configuration. Am I correct? (I need to understand if
> we need to manually manage etcd key/values).
> 
> In that case, what deployments tools (like Kolla, TripleO, etc) would
> expect from OpenStack to provide (tooling in oslo.config to generate
> ConfigMap? etc.
> 
> Thanks for your help,

Based on [1] I think the idea is to write the entire config file
for the service outside of the container, upload it to the configmap,
then configure the pod to create a volume and write the configmap
contents to the volume before launching the container. It's sort of like
nova's file-injection feature.

The approach seems appealing, although I don't fully understand the
issues others have raised with adding volumes to containers.

Doug

[1] 
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#populate-a-volume-with-data-stored-in-a-configmap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Fox, Kevin M's message of 2017-06-08 20:08:25 +:
> Yeah, I think k8s configmaps might be a good config mechanism for k8s based 
> openstack deployment.
> 
> One feature that might help which is related to the etcd plugin would be a 
> yaml/json plugin. It would allow more native looking configmaps.

We have at least 2 mechanisms for getting config files into containers
without such significant changes to oslo.config.  At this point I'm
not sure it's necessary to do the driver work at all.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Steven Dake (stdake)'s message of 2017-06-08 17:51:48 +:
> Doug,
> 
> In short, a configmap takes a bunch of config files, bundles them in a 
> kubernetes object called a configmap, and then ships them to etcd.  When a 
> pod is launched, the pod mounts the configmaps using tmpfs and the raw config 
> files are available for use by the openstack services.

That sounds like what confd does. Something puts data into one of
several possible databases. confd takes it out and writes it to
file(s) when the container starts. The app in the container reads
the file(s).

It sounds like configmaps would work well, too, it just doesn't
sound like a fundamentally different solution.

Doug

> 
> Operating on configmaps is much simpler and safer than using a different 
> backing database for the configuration data.
> 
> Hope the information helps.
> 
> Ping me in #openstack-kolla if you have more questions.
> 
> Regards
> -steve
> 
> -Original Message-
> From: Doug Hellmann <d...@doughellmann.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: Thursday, June 8, 2017 at 10:12 AM
> To: openstack-dev <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla]  
>   [helm] Configuration management with etcd / confd
> 
> Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> > On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> > >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> > >>On 06.06.2017 18:08, Emilien Macchi wrote:
> > >>>Another benefit is that confd will generate a configuration file when
> > >>>the application will start. So if etcd is down *after* the app
> > >>>startup, it shouldn't break the service restart if we don't ask confd
> > >>>to re-generate the config. It's good for operators who were concerned
> > >>>about the fact the infrastructure would rely on etcd. In that case, 
> we
> > >>>would only need etcd at the initial deployment (and during lifecycle
> > >>>actions like upgrades, etc).
> > >>>
> > >>>The downside is that in the case of containers, they would still have
> > >>>a configuration file within the container, and the whole goal of this
> > >>>feature was to externalize configuration data and stop having
> > >>>configuration files.
> > >>
> > >>It doesn't look a strict requirement. Those configs may (and should) 
> be
> > >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> > >>something what *does* make embedded configs a strict requirement?..
> > >
> > >mmh, one thing I liked about this effort was possibility of stop 
> bind-mounting
> > >config files into the containers. I'd rather find a way to not need any
> > >bindmount and have the services get their configs themselves.
> > 
> > Probably sent too early!
> > 
> > If we're not talking about OpenStack containers running in a COE, I 
> guess this
> > is fine. For k8s based deployments, I think I'd prefer having installers
> > creating configmaps directly and use that. The reason is that depending 
> on files
> > that are in the host is not ideal for these scenarios. I hate this idea 
> because
> > it makes deployments inconsistent and I don't want that.
> > 
> > Flavio
> > 
> 
> I'm not sure I understand how a configmap is any different from what is
> proposed with confd in terms of deployment-specific data being added to
> a container before it launches. Can you elaborate on that?
> 
> Doug
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev