Re: [Openstack-operators] Packaging sample config versions

[Jeremy Stanley]

On 2015-02-01 00:52:21 +0100 (+0100), Thomas Goirand wrote:
[...]
> Upstream authors decided to remove the sample config files for a
> reason, which is: there's no way to provide a valid sample config
> file because it depends on the version of the installed libs.
[...]

That's sort of true, but more specifically the projects which
removed them did so because there was no way to test that they were
kept up to date without introducing spontaneous breakage into the
development workflow.

Since configuration options can be introduced by new versions of
other libraries, any tests in the projects for which the
configuration was published would start rejecting unrelated commits
when that happened. Since these projects removed the tests to
confirm whether the sample configs being published were
correct/current, they config files themselves were also removed
rather than continue publishing something which they knew would get
further and further out of sync with reality.
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 01/29/2015 12:30 AM, Tom Fifield wrote:
> Actually, many folks I spoke to didn't really care about this.

(computer) science isn't about voting.

On 01/29/2015 01:55 AM, Fischer, Matt wrote:
> Agreed completely. I know it wpn¹t be 100% perfect, but its 95% and
> right now I have 0%.

5% is enough to create big bugs.

Upstream authors decided to remove the sample config files for a reason,
which is: there's no way to provide a valid sample config file because
it depends on the version of the installed libs. You're now deciding to
ignore that reason. So basically, you're deciding you know better than
everyone else. You'll get into issues and troubles. I just hope you know
about it, and wont complain when things break badly.

Thomas


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Fischer, Matt]

On 1/28/15, 4:45 PM, "Mathieu Gagné"  wrote:

>On 2015-01-28 6:30 PM, Tom Fifield wrote:
>> On 29/01/15 07:28, Thomas Goirand wrote:
>>> On 01/27/2015 11:00 PM, Tom Fifield wrote:
 Hi all,

 Based on Gustavo's excellent work below, talking with many ops, and
 after a brief chats with Jeremey and a few other TC folks, here's what
 I'd propose as an end goal:

 * A git repository that has raw, sample configs in it for each project
 that will be automagically updated

 * Raw configs distributed in the tar files we make as part of the
release

 Does that seem acceptable for us all?
>>>
>>> It is not. Since it has been already discuss, but we're still not
>>>having
>>> any counter point of argumentation, I shall repeat myself, until sanity
>>> gets restored.
>>>
>>> You are still *not* addressing the main issue: the .sample config files
>>> *must* match your environment and the version of the different libs on
>>> which a given service is going to run. Therefore, any attempt to go
>>>back
>>> to the previous situation (where we have already pre-built config
>>>files)
>>> can be considered a grave regression.
>>
>> Actually, many folks I spoke to didn't really care about this.
>>
>>
>
>I'm very well aware that configs "exported" by 3rd party libraries might
>not reflect exactly what I have in my own environment. This is a
>downside many operators are willing to accept to get back a simple
>sample config file.
>
>Many operators (myself included, of course) used to rely on the sample
>config file found in the repository of the project and it did a *great*
>job. Now it's gone and all proposed/alternative solutions just doesn't
>cut it for us *operators*, not packagers, developers, etc.

Agreed completely. I know it wpn¹t be 100% perfect, but its 95% and right
now I have 0%.

(please ignore the cruft below)...



This E-mail and any of its attachments may contain Time Warner Cable 
proprietary information, which is privileged, confidential, or subject to 
copyright belonging to Time Warner Cable. This E-mail is intended solely for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient of this E-mail, you are hereby notified that any 
dissemination, distribution, copying, or action taken in relation to the 
contents of and attachments to this E-mail is strictly prohibited and may be 
unlawful. If you have received this E-mail in error, please notify the sender 
immediately and permanently delete the original and any copy of this E-mail and 
any printout.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Mathieu Gagné]

On 2015-01-28 6:30 PM, Tom Fifield wrote:

On 29/01/15 07:28, Thomas Goirand wrote:

On 01/27/2015 11:00 PM, Tom Fifield wrote:

Hi all,

Based on Gustavo's excellent work below, talking with many ops, and
after a brief chats with Jeremey and a few other TC folks, here's what
I'd propose as an end goal:

* A git repository that has raw, sample configs in it for each project
that will be automagically updated

* Raw configs distributed in the tar files we make as part of the release

Does that seem acceptable for us all?


It is not. Since it has been already discuss, but we're still not having
any counter point of argumentation, I shall repeat myself, until sanity
gets restored.

You are still *not* addressing the main issue: the .sample config files
*must* match your environment and the version of the different libs on
which a given service is going to run. Therefore, any attempt to go back
to the previous situation (where we have already pre-built config files)
can be considered a grave regression.


Actually, many folks I spoke to didn't really care about this.




I'm very well aware that configs "exported" by 3rd party libraries might 
not reflect exactly what I have in my own environment. This is a 
downside many operators are willing to accept to get back a simple 
sample config file.


Many operators (myself included, of course) used to rely on the sample 
config file found in the repository of the project and it did a *great* 
job. Now it's gone and all proposed/alternative solutions just doesn't 
cut it for us *operators*, not packagers, developers, etc.


I proposed this exact same solution back when Cinder proposed removing 
its own sample config file [1], I was just too busy/lazy to follow up 
with my own idea. Now someone is doing it for real, I'm happy.


[1] https://review.openstack.org/#/c/96581/

--
Mathieu

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Tom Fifield]

On 29/01/15 07:28, Thomas Goirand wrote:
> On 01/27/2015 11:00 PM, Tom Fifield wrote:
>> Hi all,
>>
>> Based on Gustavo's excellent work below, talking with many ops, and
>> after a brief chats with Jeremey and a few other TC folks, here's what
>> I'd propose as an end goal:
>>
>> * A git repository that has raw, sample configs in it for each project
>> that will be automagically updated
>>
>> * Raw configs distributed in the tar files we make as part of the release
>>
>> Does that seem acceptable for us all?
> 
> It is not. Since it has been already discuss, but we're still not having
> any counter point of argumentation, I shall repeat myself, until sanity
> gets restored.
> 
> You are still *not* addressing the main issue: the .sample config files
> *must* match your environment and the version of the different libs on
> which a given service is going to run. Therefore, any attempt to go back
> to the previous situation (where we have already pre-built config files)
> can be considered a grave regression.

Actually, many folks I spoke to didn't really care about this.


Regards,


Tom


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 01/27/2015 11:00 PM, Tom Fifield wrote:
> Hi all,
> 
> Based on Gustavo's excellent work below, talking with many ops, and
> after a brief chats with Jeremey and a few other TC folks, here's what
> I'd propose as an end goal:
> 
> * A git repository that has raw, sample configs in it for each project
> that will be automagically updated
> 
> * Raw configs distributed in the tar files we make as part of the release
> 
> Does that seem acceptable for us all?

It is not. Since it has been already discuss, but we're still not having
any counter point of argumentation, I shall repeat myself, until sanity
gets restored.

You are still *not* addressing the main issue: the .sample config files
*must* match your environment and the version of the different libs on
which a given service is going to run. Therefore, any attempt to go back
to the previous situation (where we have already pre-built config files)
can be considered a grave regression.

Shall I remind everyone that if there's a config option that shouldn't
be there, the daemons will refuse to start?

I don't see what the problem is, really. We have a perfectly valid
system using oslo-config-generator. Here's an example from Ceilometer
Kilo beta 1, in my debian/rules file, in the override_dh_install target:

oslo-config-generator --output-file
$(CURDIR)/debian/ceilometer-common/usr/share/ceilometer-common/ceilometer.conf
\
--namespace ceilometer \
--namespace oslo.db \
--namespace oslo.messaging \
--namespace keystonemiddleware.auth_token

This works perfectly, and is very easy to implement on anyone doing
packaging decently.

Now, let's say there's a new version of oslo.db that adds a new
configuration option, but Debian didn't upgrade oslo.db yet. Then the
ceilometer.conf there available online will be *WRONG*.

Please, just don't do it, and force everyone to use
oslo-config-generator, as they should. What's bad is to use stuff from
/openstack/common (ie: oslo-incubator, like it can be seen in
the heat for Kilo beta 1), and that's the thing we shall try to kill
before the final version of Kilo is out.

Cheers,

Thomas Goirand (zigo)


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[George Shuklin]

Yes!

Just had have discussion about this with my colleague yesterday.

Seems be perfect solution.

On 01/28/2015 12:00 AM, Tom Fifield wrote:

Hi all,

Based on Gustavo's excellent work below, talking with many ops, and
after a brief chats with Jeremey and a few other TC folks, here's what
I'd propose as an end goal:


* A git repository that has raw, sample configs in it for each project
that will be automagically updated

* Raw configs distributed in the tar files we make as part of the release


Does that seem acceptable for us all?


Regards,



Tom



On 21/01/15 13:22, gustavo panizzo (gfa) wrote:


On 12/18/2014 09:57 AM, Jeremy Stanley wrote:

4. Set up a service that periodically regenerates sample
configuration and tracks it over time. This attempts to address the
stated desire to be able to see how sample configurations change,
but note that this is a somewhat artificial presentation since there
are a lot of variables (described earlier) influencing the contents
of such samples--any attempt to render it as a linear/chronological
series could be misleading.


i've setup a github repo where i dump sample config files for the
projects that autogenerate them, because i know nothing about rpm build
tools i only do it for debian and ubuntu packages.

if you build your deb packages you can use my, very simple and basic,
scripts to autogenerate the sample config files.


the repo is here

https://github.com/gfa/os-sample-configs

i will happily move to osops or other community repo



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Tom Fifield]

Hi all,

Based on Gustavo's excellent work below, talking with many ops, and
after a brief chats with Jeremey and a few other TC folks, here's what
I'd propose as an end goal:


* A git repository that has raw, sample configs in it for each project
that will be automagically updated

* Raw configs distributed in the tar files we make as part of the release


Does that seem acceptable for us all?


Regards,



Tom



On 21/01/15 13:22, gustavo panizzo (gfa) wrote:
> 
> 
> On 12/18/2014 09:57 AM, Jeremy Stanley wrote:
>> 4. Set up a service that periodically regenerates sample
>> configuration and tracks it over time. This attempts to address the
>> stated desire to be able to see how sample configurations change,
>> but note that this is a somewhat artificial presentation since there
>> are a lot of variables (described earlier) influencing the contents
>> of such samples--any attempt to render it as a linear/chronological
>> series could be misleading.
> 
> 
> i've setup a github repo where i dump sample config files for the
> projects that autogenerate them, because i know nothing about rpm build
> tools i only do it for debian and ubuntu packages.
> 
> if you build your deb packages you can use my, very simple and basic,
> scripts to autogenerate the sample config files.
> 
> 
> the repo is here
> 
> https://github.com/gfa/os-sample-configs
> 
> i will happily move to osops or other community repo
> 


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[gustavo panizzo (gfa)]

On 12/18/2014 09:57 AM, Jeremy Stanley wrote:

4. Set up a service that periodically regenerates sample
configuration and tracks it over time. This attempts to address the
stated desire to be able to see how sample configurations change,
but note that this is a somewhat artificial presentation since there
are a lot of variables (described earlier) influencing the contents
of such samples--any attempt to render it as a linear/chronological
series could be misleading.



i've setup a github repo where i dump sample config files for the 
projects that autogenerate them, because i know nothing about rpm build 
tools i only do it for debian and ubuntu packages.


if you build your deb packages you can use my, very simple and basic, 
scripts to autogenerate the sample config files.



the repo is here

https://github.com/gfa/os-sample-configs

i will happily move to osops or other community repo

--
1AE0 322E B8F7 4717 BDEA BF1D 44BB 1BA7 9F6C 6333

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Anne Gentle]

On Fri, Dec 19, 2014 at 9:18 AM, Thomas Bechtold  wrote:
>
> On 18.12.2014 14:03, Thomas Goirand wrote:
> > Jeremy,
> >
> > Thanks *A LOT* for writing this up. This is very helpful.
> >
> > On 12/18/2014 09:57 AM, Jeremy Stanley wrote:
> >> During the first half of yesterday's cross-project meeting, we went
> >> through the sample configuration packaging/publishing topic to get a
> >> better idea of what options are open to us. Many thanks to all who
> >> attended. The meeting summary with a link to the full discussion
> >> logs can be found here:
> >>
> >>  http://eavesdrop.openstack.org/meetings/crossproject/2014/crossproject.2014-12-16-21.01.html
> >
> >>
> >> We spent a fair amount of time discussing the current configuration
> >> model and the challenges presented by its design, particularly that
> >> the presence or absence of different libraries (and differing
> >> versions thereof) influence what configuration options are available
> >> in a service, the values to which they can be set, and in some cases
> >> those to which they default when otherwise omitted. I don't want to
> >> go into further detail on that within this thread, but feel
> >> obligated to point out that this model is to some extent the result
> >> of earlier operational complaints about having to modify too many
> >> different configuration files to set up a single service.
> >>
> >> We debated the merits and drawbacks of a number of options proposed
> >> here and within the meeting. Before I go into them I'm compelled to
> >> remind everyone that none of these comes without some cost in
> >> development effort and ongoing management overhead, and ask that
> >> anyone who expresses a preference for one or more to include
> >> concrete use case descriptions. Things we can consider implementing:
> >>
> >> 1. Standardize on a common mechanism across all projects to generate
> >> sample configuration files. This should be able to run within a
> >> global system context, not just within a virtualenv via tox.
> >
> > Yes please! I'm already using what tox does, instead of tox itself. IMO,
> > this should go into oslo.config (or some kind of lib like this).
>
> There is already oslo-config-generator (
> http://docs.openstack.org/developer/oslo.config/generator.html ). That
> should imho be used and is already in use by some projects.
>
>
This is how we're making the Configuration Reference each release [1] but
for swift, we wrote a special scraper. [2]

There are some requirements for any of the solutions so we don't lose what
we already have with the Configuration Reference:
- version shown in the output
- indication of new, changed, and deprecated options
- default values and units
- meaningful descriptions for each setting [3]

Thanks,
Anne

1. http://docs.openstack.org/juno/config-reference/content/
2.
http://git.openstack.org/cgit/openstack/openstack-doc-tools/tree/autogenerate_config_docs/extract_swift_flags.py
3.
http://docs.openstack.org/juno/config-reference/content/container-sync-realms-configuration.html


> >> 2. Provide a solution which runs within the scope of each project's
> >> setup.py to generate sample configuration and include it in any
> >> sdist tarball or Python wheel. This would have the added benefit
> >> that people installing via pip from PyPI or just retrieving official
> >> tarballs would get copies of sample configuration from the timeframe
> >> when they were generated. As this complicates sdist generation
> >> (because it requires installation of required and optional libraries
> >> used by the service), it probably needs to be easy to enable and
> >> disable.
> >
> > As you know, I don't care about the sdist tarballs, but I do want
> > "python setup.py install" to generate the config files. Otherwise, a
> > "python setup.py config-file" or something similar would do, as long as
> > it is:
> > 1/ Documented
> > 2/ Consistent across all of OpenStack
>
> Yes. And generating the sample-config during the sdist run and including
> it into the sdist or wheel file doesn't fix the issue with the
> consistency of libs.
>
> >> 3. Design a Sphinx plug-in or other similar solution to generate and
> >> include sample configuration files within the developer
> >> documentation of each project. Since this documentation is
> >> automatically updated and published, it would provide a stable
> >> location where interested parties can view and download these files
> >> without needing to manually generate or extract them from an
> >> archive.
> >
> > This doesn't fix the issue with the consistency of libs.
> >
> >> 4. Set up a service that periodically regenerates sample
> >> configuration and tracks it over time. This attempts to address the
> >> stated desire to be able to see how sample configurations change,
> >> but note that this is a somewhat artificial presentation since there
> >> are a lot of variables (described earlier) influencing the contents
> >> of such samples--any attempt to render it as a linear/chronolog

Re: [Openstack-operators] Packaging sample config versions

[Thomas Bechtold]

On 18.12.2014 14:03, Thomas Goirand wrote:
> Jeremy,
> 
> Thanks *A LOT* for writing this up. This is very helpful.
> 
> On 12/18/2014 09:57 AM, Jeremy Stanley wrote:
>> During the first half of yesterday's cross-project meeting, we went
>> through the sample configuration packaging/publishing topic to get a
>> better idea of what options are open to us. Many thanks to all who
>> attended. The meeting summary with a link to the full discussion
>> logs can be found here:
>>
>> > http://eavesdrop.openstack.org/meetings/crossproject/2014/crossproject.2014-12-16-21.01.html
>>  >
>>
>> We spent a fair amount of time discussing the current configuration
>> model and the challenges presented by its design, particularly that
>> the presence or absence of different libraries (and differing
>> versions thereof) influence what configuration options are available
>> in a service, the values to which they can be set, and in some cases
>> those to which they default when otherwise omitted. I don't want to
>> go into further detail on that within this thread, but feel
>> obligated to point out that this model is to some extent the result
>> of earlier operational complaints about having to modify too many
>> different configuration files to set up a single service.
>>
>> We debated the merits and drawbacks of a number of options proposed
>> here and within the meeting. Before I go into them I'm compelled to
>> remind everyone that none of these comes without some cost in
>> development effort and ongoing management overhead, and ask that
>> anyone who expresses a preference for one or more to include
>> concrete use case descriptions. Things we can consider implementing:
>>
>> 1. Standardize on a common mechanism across all projects to generate
>> sample configuration files. This should be able to run within a
>> global system context, not just within a virtualenv via tox.
> 
> Yes please! I'm already using what tox does, instead of tox itself. IMO,
> this should go into oslo.config (or some kind of lib like this).

There is already oslo-config-generator (
http://docs.openstack.org/developer/oslo.config/generator.html ). That
should imho be used and is already in use by some projects.

>> 2. Provide a solution which runs within the scope of each project's
>> setup.py to generate sample configuration and include it in any
>> sdist tarball or Python wheel. This would have the added benefit
>> that people installing via pip from PyPI or just retrieving official
>> tarballs would get copies of sample configuration from the timeframe
>> when they were generated. As this complicates sdist generation
>> (because it requires installation of required and optional libraries
>> used by the service), it probably needs to be easy to enable and
>> disable.
> 
> As you know, I don't care about the sdist tarballs, but I do want
> "python setup.py install" to generate the config files. Otherwise, a
> "python setup.py config-file" or something similar would do, as long as
> it is:
> 1/ Documented
> 2/ Consistent across all of OpenStack

Yes. And generating the sample-config during the sdist run and including
it into the sdist or wheel file doesn't fix the issue with the
consistency of libs.

>> 3. Design a Sphinx plug-in or other similar solution to generate and
>> include sample configuration files within the developer
>> documentation of each project. Since this documentation is
>> automatically updated and published, it would provide a stable
>> location where interested parties can view and download these files
>> without needing to manually generate or extract them from an
>> archive.
> 
> This doesn't fix the issue with the consistency of libs.
> 
>> 4. Set up a service that periodically regenerates sample
>> configuration and tracks it over time. This attempts to address the
>> stated desire to be able to see how sample configurations change,
>> but note that this is a somewhat artificial presentation since there
>> are a lot of variables (described earlier) influencing the contents
>> of such samples--any attempt to render it as a linear/chronological
>> series could be misleading.
> 
> Same issue.
> 
>> Anyway, this is just an attempt to level-set and spur the discussion
>> onward to actionable solutions rather than continuing to debate in
>> the abstract. Hopefully it takes us in a good direction.
> 
> Let's just hope we'll experience consistency.
> 

Cheers,

Tom

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

Jeremy,

Thanks *A LOT* for writing this up. This is very helpful.

On 12/18/2014 09:57 AM, Jeremy Stanley wrote:
> During the first half of yesterday's cross-project meeting, we went
> through the sample configuration packaging/publishing topic to get a
> better idea of what options are open to us. Many thanks to all who
> attended. The meeting summary with a link to the full discussion
> logs can be found here:
> 
>  http://eavesdrop.openstack.org/meetings/crossproject/2014/crossproject.2014-12-16-21.01.html
>  >
> 
> We spent a fair amount of time discussing the current configuration
> model and the challenges presented by its design, particularly that
> the presence or absence of different libraries (and differing
> versions thereof) influence what configuration options are available
> in a service, the values to which they can be set, and in some cases
> those to which they default when otherwise omitted. I don't want to
> go into further detail on that within this thread, but feel
> obligated to point out that this model is to some extent the result
> of earlier operational complaints about having to modify too many
> different configuration files to set up a single service.
> 
> We debated the merits and drawbacks of a number of options proposed
> here and within the meeting. Before I go into them I'm compelled to
> remind everyone that none of these comes without some cost in
> development effort and ongoing management overhead, and ask that
> anyone who expresses a preference for one or more to include
> concrete use case descriptions. Things we can consider implementing:
> 
> 1. Standardize on a common mechanism across all projects to generate
> sample configuration files. This should be able to run within a
> global system context, not just within a virtualenv via tox.

Yes please! I'm already using what tox does, instead of tox itself. IMO,
this should go into oslo.config (or some kind of lib like this).

> 2. Provide a solution which runs within the scope of each project's
> setup.py to generate sample configuration and include it in any
> sdist tarball or Python wheel. This would have the added benefit
> that people installing via pip from PyPI or just retrieving official
> tarballs would get copies of sample configuration from the timeframe
> when they were generated. As this complicates sdist generation
> (because it requires installation of required and optional libraries
> used by the service), it probably needs to be easy to enable and
> disable.

As you know, I don't care about the sdist tarballs, but I do want
"python setup.py install" to generate the config files. Otherwise, a
"python setup.py config-file" or something similar would do, as long as
it is:
1/ Documented
2/ Consistent across all of OpenStack

> 3. Design a Sphinx plug-in or other similar solution to generate and
> include sample configuration files within the developer
> documentation of each project. Since this documentation is
> automatically updated and published, it would provide a stable
> location where interested parties can view and download these files
> without needing to manually generate or extract them from an
> archive.

This doesn't fix the issue with the consistency of libs.

> 4. Set up a service that periodically regenerates sample
> configuration and tracks it over time. This attempts to address the
> stated desire to be able to see how sample configurations change,
> but note that this is a somewhat artificial presentation since there
> are a lot of variables (described earlier) influencing the contents
> of such samples--any attempt to render it as a linear/chronological
> series could be misleading.

Same issue.

> Anyway, this is just an attempt to level-set and spur the discussion
> onward to actionable solutions rather than continuing to debate in
> the abstract. Hopefully it takes us in a good direction.

Let's just hope we'll experience consistency.

Thomas Goirand (zigo)


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Jeremy Stanley]

On 2014-12-18 01:57:20 + (+), Jeremy Stanley wrote:
[...]
> Anyway, this is just an attempt to level-set and spur the
> discussion onward to actionable solutions rather than continuing
> to debate in the abstract. Hopefully it takes us in a good
> direction.

I meant to add that as an outcome of further discussion, any
cross-project recommendation arrived at should proceed into a draft
specification in the openstack/openstack-specs repository. We'll
need a volunteer to start that process as well and attempt to refine
subsequent feedback.
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Jeremy Stanley]

During the first half of yesterday's cross-project meeting, we went
through the sample configuration packaging/publishing topic to get a
better idea of what options are open to us. Many thanks to all who
attended. The meeting summary with a link to the full discussion
logs can be found here:

http://eavesdrop.openstack.org/meetings/crossproject/2014/crossproject.2014-12-16-21.01.html
 >

We spent a fair amount of time discussing the current configuration
model and the challenges presented by its design, particularly that
the presence or absence of different libraries (and differing
versions thereof) influence what configuration options are available
in a service, the values to which they can be set, and in some cases
those to which they default when otherwise omitted. I don't want to
go into further detail on that within this thread, but feel
obligated to point out that this model is to some extent the result
of earlier operational complaints about having to modify too many
different configuration files to set up a single service.

We debated the merits and drawbacks of a number of options proposed
here and within the meeting. Before I go into them I'm compelled to
remind everyone that none of these comes without some cost in
development effort and ongoing management overhead, and ask that
anyone who expresses a preference for one or more to include
concrete use case descriptions. Things we can consider implementing:

1. Standardize on a common mechanism across all projects to generate
sample configuration files. This should be able to run within a
global system context, not just within a virtualenv via tox.

2. Provide a solution which runs within the scope of each project's
setup.py to generate sample configuration and include it in any
sdist tarball or Python wheel. This would have the added benefit
that people installing via pip from PyPI or just retrieving official
tarballs would get copies of sample configuration from the timeframe
when they were generated. As this complicates sdist generation
(because it requires installation of required and optional libraries
used by the service), it probably needs to be easy to enable and
disable.

3. Design a Sphinx plug-in or other similar solution to generate and
include sample configuration files within the developer
documentation of each project. Since this documentation is
automatically updated and published, it would provide a stable
location where interested parties can view and download these files
without needing to manually generate or extract them from an
archive.

4. Set up a service that periodically regenerates sample
configuration and tracks it over time. This attempts to address the
stated desire to be able to see how sample configurations change,
but note that this is a somewhat artificial presentation since there
are a lot of variables (described earlier) influencing the contents
of such samples--any attempt to render it as a linear/chronological
series could be misleading.

Anyway, this is just an attempt to level-set and spur the discussion
onward to actionable solutions rather than continuing to debate in
the abstract. Hopefully it takes us in a good direction.
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[James Page]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi George

On 16/12/14 01:38, George Shuklin wrote:
> Can you say why you didn't ship 2013.2.4? It was available in
> announced support lifecycle for cloudarchive, and you just stops to
> do anything with 2013.2.3 (havana) somewhere in the middle of the
> summer. It was really bad, because few important fixes for neutron
> were landed in 2013.2.4 and was not available in 2013.2.3 (f.e. 
> https://bugs.launchpad.net/neutron/+bug/1240849)

Canonical eol'ed support for the Havana Cloud Archive 3 months after
the release of Icehouse as described at [0]; this pre-dates the
2013.2.4 stable release (Sept 2014) which is why it was never shipped.

[0] https://wiki.ubuntu.com/ServerTeam/CloudArchive

- -- 
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUkgBdAAoJEL/srsug59jDziIP/1CdYS1QtpR0QA9p7CxD2YiC
SWrPTjKliH/P+XeCHgJicUe3bIYu62ZguM3IqzZaXD8OdxSP450KCNAnARkIdDm7
5/ovWKdOF3rp1HGQQtLHyaqLlMrSOs3boVHCBgfFdpkmuFw0OPyG3TW4pPy4kWfb
yOlDaq9NMEsH7kU3uBiokNk9ze6YwMcoakseQ50IqZIm0sSurSvVklR+LytMVcVL
ZJzavZUS+PVNfoqVzhzScOSktMyUCd1xRmTf2BsNHD5PZeazgYpywz7GWQLYhmqG
Br6WElXJOYpAKZFzAu1YEZs2Lg5gFBPo07s4o6pinUgzd9+oGiCfFTa/ikL4iZvK
BTCRoD5qApgWveeDWTP35DqToCanuT6pPtKRQPyfVXuzsVoTvAM7RS5Ywc/YcTSO
RCbc2lKf+qKtGPzSNCVldAUrZCKMwGdTKHoWUWzcRhzPBGWdLNGt7YoBKbX8C1e6
4c0FJ0olqauWy5ScOAfF0yeOyRobWjqkaQPkTzozOqvWjlrb6RPL18T6nWpo0thx
+Vmi+Q42RqYxCNlSvMTduYqCncA/OwUS9JXKDc30wqJJWC0n0iJDn+mpmWc1qfxf
1cpP3K0qKg+WlI5Y28hlm+y96JRBPQpRDWstIf4I2sCg3ihoZdOWLnmc1a+ovJ1H
PuYSd8AekmInYjVYOm1U
=19cw
-END PGP SIGNATURE-

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 12/16/2014 12:33 AM, Kris G. Lindgren wrote:
> That¹s - great.  But other people will want it to be based on RHEL -
> others Ubuntu, some people will want it on SLES.  I understand debian
> makes your life easier, but at Paris is was already determined that a one
> size fits all approach wont work.  Whatever is done needs to handle both
> deb packages and rpms.

Sure! Do your RPMs if you like, though I'd be happy to have new
contributors to the Debian side.

>>> I guess I can get that for staying on trunk.  But when openstack does a
>>> stable release... Shouldn't the client versions be pretty well defined?
>>> And not just >= 1.4 - but should be capped to some specific version?
>>
>> No, clients should *never* be capped. And in fact, over the years, I've
>> often had to express my strong opinion about the fact that no Python
>> module should ever receive a higher bound cap. Indeed, upgrading a
>> Python module in a distribution could be motivated by something
>> completely unrelated to OpenStack.
> 
> Which is one point of view as a packager for debian you need to care about
> that stuff.  For other people building openstack packages (ME) they go
> onto a machine that are basically only function to house specific version
> of openstack and since I am using rhel - the likelyhood that the python
> versions changing because of something they did its pretty small. Either
> way it looks like their were some recent commits to try to add version
> capping to gate: 
> https://github.com/openstack/requirements/commit/ad967b1cdb23181de818 - so
> apparently I am not the only one thinking this way ;-).

This kind of capping is fine. What's not fine is to cap let's say
Django<1.7 or SQLAlchemy<=0.7.99 (to take real examples of the past
which I had to deal with) and believe that OpenStack is alone, and the
distribution will adapt. Even in the Red Hat world, that's not what is
happening.

>> Then having a look to it, this Anvil repository seems very much Red Hat
>> centric. I'm not really convince that moving all the spec files into a
>> single repository helps (I took the opposite approach of having one git
>> repository per package, like everyone is doing in Debian anyway). And
>> also it seems to consider only core project packages, which is IMO a
>> very small part of the needed work. Also, it seems to mix deployment
>> stuff and packaging, which doesn't seem the best approach either.
> 
> That¹s because it figures out the ptyhon dependenices and packages them
> for me.  Which was already stated another mainling list. Only a few
> python modules have problems being packaged this way - the others can be
> packaged using a pretty standard template. I can tell you that in 1+ year
> of deploying openstack from my own packages I have built exactly 0 python
> dependent packages (and I am not using the RDO repo to avoid building them
> either). Then again - I don¹t have to deal with distributions level
> issues since my servers that get openstack packages are basically single
> purpose.

Sure, you may achieve some result the way you do which may be ok for
what you do. Thought it will always be a more limited scope by doing
things this way. Also, having packages within a distribution like Debian
ensure there's a better quality, thanks to the multiple tests that are
done archive wide, a single bug report place for users not only of
OpenStack, and more peer review.

> I believe this is why their is also desires to put openstack
> into venv's for each service or do containerization around an openstack
> service. Their are tools out there that do exactly this (giftwrap,anvil)
> I am sure of some others that I am forgetting at this point.

Outch! :)
Feel free to do this way if you like, but I wont be the one who would
recommend upgrading 15 venv in a single machine when a security issue is
raised.

>> As much as I know, there's no automated tool that can do the manual work
>> of figuring out (build-)dependencies correctly. This has to be a manual
>> work, because of the fact one needs to check what version is currently
>> available on the current stable distribution and sometimes just avoid
>> version dependencies when possible.
> 
> Again - For me Anvil does this, other people have created giftwrap and I
> am sure their are some other projects that auto figure out dependencies
> builds them and packages them.  But with anvil over the past year I have
> built exactly 0 python dependent packages by hand. I have updated exactly
> 0 spec files for those updated dependancies as well. Anvil figures out
> the requirements across all the openstack projects you are building. It
> then, if need be, fixes conflicting version requirements (I am sure that
> you know about this in the past where one project wanted a version that
> conflicted with another project). Then it builds and packages the
> dependent packages, then the openstack specific packages. In the end I
> have two repos, the repo with just the core packages in it and then a

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 12/16/2014 09:33 AM, George Shuklin wrote:
> Nope. I've only done stuff with debian-jenkins-glue. But I have some
> experience on backporting patches from icehouse to havana (it still in
> production and still need fixes). I can research/fix something specific
> and local.

Oh, if you're good with back-porting, then when the Icehouse is
officially end-of-life upstream, you could join the team working on its
extended support. I'll be doing the distribution coordination for
security fixing. Is this some area you'd like to work on?

Otherwise, there's room for just *any* packaging work. From the smallest
Python module, to backporting key packages (like libvirt, Qemu, Ceph,
and so on...), or even working on core packages and testing. Just take
your pick... Basically, I'd accept *any* help, and I will adapt to make
it comfy for you to help! :)

Cheers,

Thomas Goirand (zigo)


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[George Shuklin]

Oh!

Ubuntu's guy.

Can you say why you didn't ship 2013.2.4? It was available in announced 
support lifecycle for cloudarchive, and you just stops to do anything 
with 2013.2.3 (havana) somewhere in the middle of the summer. It was 
really bad, because few important fixes for neutron were landed in 
2013.2.4 and was not available in 2013.2.3 (f.e. 
https://bugs.launchpad.net/neutron/+bug/1240849)


(about configs)
Actual config file does not matter - we have own chef'ed configs, 
created from scratch & databags. All I wanted were to have no unexpected 
questions during installation. And I completely forgot about 
non-interactive mode for dpkg.


On 12/15/2014 07:46 PM, James Page wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi Thomas

On 15/12/14 08:49, Thomas Goirand wrote:

and ubuntu just put files

in proper places without changing configs.

Ahem... Ubuntu simply doesn't care much about config files. See
what they ship for Nova and Cinder. I wouldn't say "without
changing configs" in this case.

Just for the record, we (Ubuntu) ship either a minimal functional
configuration files or the upstream sample configuration files with
sufficient patching to be functional.

We made the choice to specifically not include debconf for
configuration file generation; in terms of where we wanted to focus
the resources we have working on packaging, debconf was not high on
the list of priorities as the majority of users are deploying across
multiple servers and are using some sort of configuration management
tool; maintaining debconf for OpenStack packages added overhead we did
not want to take on.

So we do care about configuration files... just not in the same way
you do for Debian!


- -- 
James Page

Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUjx55AAoJEL/srsug59jDcQQP/1wFJdmf038q7Fwjgfev0iTJ
3j8IXgsz6OLi1jSxsIYzCGvME6U6SMpfV6edX/X637nHxr1gYg00npz51xJjbVap
c33T5MxWkq6L/e0MdwEX4OqEejOiBuayw61h1+9Kg+BlkYJI+o/GZ+soMGbWAu/T
xKIwzGLYRjr8s7ToeKxv8HYwiyrdeMjwaHb2Ontpyh8psPMprOwyXeV91shGxdY6
RN6Fw3NGLAUyl2JsyCDAeeg6UZ1Cb5tnHfoZg1Tu0MGl+VrSLUy7KvPGf5vNXT8l
XmE9IiYSFlW70b4jliuE44LhR4p/PCfQXBV5a60PUJWIWxtjcVSEb+MtwzZpSIQm
tg9+q2dIJlbWUGCLaE39wJfNuZoUPNoFCOxruAYkwZ3DGnozt36XYDL5jyCmMfX3
jnbZ11YfmBa0ZILUmlwQ6ZRVL3BWxKsFVLPauq0u79fhF0H5i6lpVmx7S4LEt5GE
EwJH2yhAnx4qzzl9x6y/wyCcHh+0JgdYWIYGlYylGkw27mfMsEtowQfHypf7cB3l
5dwA+d/NWZb2fKX/vPvrTCIPJuN7czWTZDVATOSjCwoZMfafMa/66E+LvCD5GZ+S
8etVt4uXY6yWOxrvyw411/gNtXvKbuoUpLky+skdjjNGcnB24ZQQ4W8mOq1yNMQc
1WO0buP8OUF2FItrg0JP
=GxCK
-END PGP SIGNATURE-

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[George Shuklin]

On 12/15/2014 10:49 AM, Thomas Goirand wrote:

and ubuntu just put files
in proper places without changing configs.

Ahem... Ubuntu simply doesn't care much about config files. See what
they ship for Nova and Cinder. I wouldn't say "without changing configs"
in this case.


We using chef for
configuration, so ubuntu approach is better

It's not better or worse, it's exactly the same as for Debian, as the
Debian package will *never* change something you modified in a config
file, as per Debian policy (if they do, then it's a bug you shall report
to the tracker).

Thank you.

It's rather unexpected, but I'll take this in account for next 
installation. I'm not a big fan of ubuntu maintanance policy (they 
basically dropped it 2 month prior announced date), and I prefer use of 
debian where possible. Now I see it's ok with openstack too, and it's 
good. I think is some kind of implicit FUD, because I was absolutely 
sure that Canonical/RH packaging and Debian is far in the tail of the 
process. It is not true and I'm happy.

Anyway, I'm ready to help but have no idea how (within my limits).

Do you have any experience building 3rd party CIs on OpenStack infra?

Nope. I've only done stuff with debian-jenkins-glue. But I have some 
experience on backporting patches from icehouse to havana (it still in 
production and still need fixes). I can research/fix something specific 
and local.


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[James Page]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi Thomas

On 15/12/14 08:49, Thomas Goirand wrote:
>> and ubuntu just put files
>>> in proper places without changing configs.
> Ahem... Ubuntu simply doesn't care much about config files. See
> what they ship for Nova and Cinder. I wouldn't say "without
> changing configs" in this case.

Just for the record, we (Ubuntu) ship either a minimal functional
configuration files or the upstream sample configuration files with
sufficient patching to be functional.

We made the choice to specifically not include debconf for
configuration file generation; in terms of where we wanted to focus
the resources we have working on packaging, debconf was not high on
the list of priorities as the majority of users are deploying across
multiple servers and are using some sort of configuration management
tool; maintaining debconf for OpenStack packages added overhead we did
not want to take on.

So we do care about configuration files... just not in the same way
you do for Debian!


- -- 
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUjx55AAoJEL/srsug59jDcQQP/1wFJdmf038q7Fwjgfev0iTJ
3j8IXgsz6OLi1jSxsIYzCGvME6U6SMpfV6edX/X637nHxr1gYg00npz51xJjbVap
c33T5MxWkq6L/e0MdwEX4OqEejOiBuayw61h1+9Kg+BlkYJI+o/GZ+soMGbWAu/T
xKIwzGLYRjr8s7ToeKxv8HYwiyrdeMjwaHb2Ontpyh8psPMprOwyXeV91shGxdY6
RN6Fw3NGLAUyl2JsyCDAeeg6UZ1Cb5tnHfoZg1Tu0MGl+VrSLUy7KvPGf5vNXT8l
XmE9IiYSFlW70b4jliuE44LhR4p/PCfQXBV5a60PUJWIWxtjcVSEb+MtwzZpSIQm
tg9+q2dIJlbWUGCLaE39wJfNuZoUPNoFCOxruAYkwZ3DGnozt36XYDL5jyCmMfX3
jnbZ11YfmBa0ZILUmlwQ6ZRVL3BWxKsFVLPauq0u79fhF0H5i6lpVmx7S4LEt5GE
EwJH2yhAnx4qzzl9x6y/wyCcHh+0JgdYWIYGlYylGkw27mfMsEtowQfHypf7cB3l
5dwA+d/NWZb2fKX/vPvrTCIPJuN7czWTZDVATOSjCwoZMfafMa/66E+LvCD5GZ+S
8etVt4uXY6yWOxrvyw411/gNtXvKbuoUpLky+skdjjNGcnB24ZQQ4W8mOq1yNMQc
1WO0buP8OUF2FItrg0JP
=GxCK
-END PGP SIGNATURE-

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Kris G. Lindgren]

On 12/13/14, 8:09 AM, "Thomas Goirand"  wrote:

>On 12/12/2014 02:17 AM, Kris G. Lindgren wrote:
>>> Why do you think it's a good idea to restart doing the work of
>>> distributions by yourself? Why not joining a common effort?
>> 
>> Well for a lot of reasons.  Some of us carry our own patch sets
>
>My employer (ie: Mirantis) also has its own patch set, however, we can
>only work together on "community packages", so I don't really see your
>point here.
>
>>- some of
>> us want to run certain services on trunk.
>
>I also would like to have packages following trunk. One of the reason
>would be to do gating on trunk, working together with the infra team. I
>only lack time to build that CI for the moment, but it shouldn't be very
>hard to build.
>
>> But mainly the bigger push is
>> that if we come up with a standard set of openstack packages - then we
>>can
>> build a standard set of tooling around dealing/installing those
>>packages.
>
>It'd be super nice if the standard would be what's in Debian. I am
>convinced that Debian is the correct place to make it happen, as it's
>not bound to any company or agenda.
That¹s - great.  But other people will want it to be based on RHEL -
others Ubuntu, some people will want it on SLES.  I understand debian
makes your life easier, but at Paris is was already determined that a one
size fits all approach wont work.  Whatever is done needs to handle both
deb packages and rpms.
>
>> The point of this is that we are trying to build that common effort and
>> not leave it up to the distro's to solve all the problems?
>
>How isn't what's happening in Debian not a "common effort"? It's all
>community based. I'm largely not interested in anything RPM based, so
>others could work on that part if they wish.

I never said it wasn't - its just for *ME* what you are doing for debian
packages has very little carry over to what I have to do for rpm packages.
 
>
>>> On 12/09/2014 11:23 AM, Fischer, Matt wrote:
 Not only is the sample
 config a useful reference, but having it in a git repo allows me to
 see when new options were added or defaults were changed.
>>>
>>> Yeah, though if you've been doing packages for a long time, you do have
>>> this information. At least, I do have it for all the last stable
>>>releases.
>> 
>> Great - some of us also are busy actually running clouds here...
>
>I wasn't meaning that "I know" (hopefully, I'm not that pretending). But
>rather "the Git knows" or "the Debian archive knows". In other words,
>the distribution has a history of all .conf.sample over time, which
>makes it possible to know.
>
>>> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
 [...] sample config and replace it with some tox script. Which may or
 may not run on your current operating system without going to pip to
 get newer versions of stuff.
>>>
>>> Excuse my words, but that's bullshit. If your operating system doesn't
>>> have a version high enough for one of your build-dependency, then this
>>> should be fixed. It should be your work as a package maintainer. For
>>> example, if we need a newer version of tox (which is *not* the case, as
>>> tox isn't really needed for generating config files), then the tox
>>> package should be updated. It's as simple as that. And if you want this
>>> to happen, then I would strongly suggest you to get involved in Debian
>>> and/or Fedora packaging, because that's where things happen.
>> 
>> I guess from my point of view - not providing a sample config file for
>> your project is bullshit - unless said project can work without any
>>config
>> file at all (even then its still bullshit).  Providing a script to do
>>it -
>> ok that¹s live-able.  Providing documentation to use tox that installs
>>all
>> the projects deps into a venv then generates a script from those deps
>> which don't match what on the system.  That¹s bullshit as well.
>
>I shouldn't have used "bullshit", I can see you took it badly. Sorry.
>That wasn't aimed to create a strong reaction.
>
>What I wanted to say was that the fact you can't run a specific command
>in your distribution because of an outdated Python module is wrong,
>simply because it's the work of a package maintainer to do updates if
>it's needed.
>
>And no, you don't need a venv. I use tox.ini files as a kind of
>packaging documentation, so I know what shell command to use, and
>everything else happens in debian/rules, with build-dependencies, not
>venv stuff.
>
>>> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>>> Let me give you an example. In most (if not *all* projects) you will
>>> find some [keystone:auth_token] section. These are directly derived
>>>from
>>> what's in python-keystoneclient. Indeed, the output .conf.sample file
>>> will depend on that... Now let's say the maintainers of
>>> python-keystoneclient would like to update something (let's say, update
>>> a comment on one of the directives, or simply a *new* directive),
>>> obviously, you want that change to propagate on

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 12/14/2014 06:39 AM, George Shuklin wrote:
> Btw: we talking about debian packages or ubuntu?

Debian, not Ubuntu.

> They are differ -
> debian heavily relies on answers to debconfig

You mean debconf. And no, it doesn't "rely on", it's completely
optional, and the packages *must* be able to be installed in a
non-interactive way (as per the Debian policy).

> and ubuntu just put files
> in proper places without changing configs.

Ahem... Ubuntu simply doesn't care much about config files. See what
they ship for Nova and Cinder. I wouldn't say "without changing configs"
in this case.

> We using chef for
> configuration, so ubuntu approach is better

It's not better or worse, it's exactly the same as for Debian, as the
Debian package will *never* change something you modified in a config
file, as per Debian policy (if they do, then it's a bug you shall report
to the tracker).

> (when we starts doing
> openstack that was on of deciding factors between debian and ubuntu).

Then you decided on the wrong grounds.

On 12/14/2014 09:03 AM, George Shuklin wrote:
> Well, 'preseed' is just more work

But it's completely optional. And also, the openstack-meta-packages
source package provides all the facility for you (see the
"openstack-deploy" package which contains the preseed lib).

> noninteractive dpkg - is much better.

Then use:
DEBIAN_FRONTEND=noninteractive apt-get install 

and the Debian packages are all non-interactive as well.

> Anyway, I'm ready to help but have no idea how (within my limits).

Do you have any experience building 3rd party CIs on OpenStack infra?

Thomas Goirand (zigo)


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[George Shuklin]

On 12/14/2014 01:27 AM, Jeremy Stanley wrote:

On 2014-12-14 00:39:41 +0200 (+0200), George Shuklin wrote:
[...]

debian heavily relies on answers to debconfig, and ubuntu just put
files in proper places without changing configs. We using chef for
configuration, so ubuntu approach is better (when we starts doing
openstack that was on of deciding factors between debian and
ubuntu).

Those debconf prompts are merely for convenience during interactive
installs. You can preseed them if you like and/or set dpkg to
noninteractive, and still just put config files in place like you do
in Ubuntu. I don't see why that specifically would be a mark against
Debian's OpenStack packages.
Well, 'preseed' is just more work. noninteractive dpkg - is much better. 
Shame on me I didn't remember about this during selection of platform.


Anyway, I'm ready to help but have no idea how (within my limits).


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Jeremy Stanley]

On 2014-12-14 00:39:41 +0200 (+0200), George Shuklin wrote:
[...]
> debian heavily relies on answers to debconfig, and ubuntu just put
> files in proper places without changing configs. We using chef for
> configuration, so ubuntu approach is better (when we starts doing
> openstack that was on of deciding factors between debian and
> ubuntu).

Those debconf prompts are merely for convenience during interactive
installs. You can preseed them if you like and/or set dpkg to
noninteractive, and still just put config files in place like you do
in Ubuntu. I don't see why that specifically would be a mark against
Debian's OpenStack packages.
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[George Shuklin]

On 12/13/2014 05:13 PM, Thomas Goirand wrote:

If I can help somehow, I'm ready to do something, but What should I
do, exactly?

There's a lot that can be done. If you like working on CI stuff, then
you could help me with building the package validation CI which I'm
trying to (re-)work. All of this is currently inside the debian/juno of
the openstack-meta-packages (in the openstack-tempest-ci package, which
uses the openstack-deploy package).

In the past, I saw *A LOT* of CIs, and most of them were written in a
very dirty way. In fact, it's easy to write a CI, but it's very hard to
write it well. I'm not saying my approach is perfect, but IMO it's
moving toward the good direction.
All CIs are dirty pile of bash scripts. Some of them have enough dirt to 
give birth to new life. Which in turn starts  civilization, inventing 
computers and start doing own CI.



For the moment, the packaged CI can do a full all-in-one deployment from
scratch (starting with an empty VM), install and configure tempest, and
run the Keystone tempest unit tests. I'm having issues with nova-compute
using Qemu, and also the Neutron setup. But once that's fixed, I hope to
be able to run most tempest tests. The next step will be to run on a
multi-node setup.

So, if you want to help on that, and as it seems you like doing CI
stuff, you're more welcome to do so.

Once we have this, then we could start building a repository with
everything from trunk. And when that is done, starting the effort of
building a 3rd party CI to do package validation on the gate.

Your thoughts?

Oops, I don't feel I can't respond on this in smart way. I'm do not know 
some of the stuff (like tempest). It's better if you give one concrete 
area to work (something of scale of normal issue from tracker).


Btw: we talking about debian packages or ubuntu? They are differ - 
debian heavily relies on answers to debconfig, and ubuntu just put files 
in proper places without changing configs. We using chef for 
configuration, so ubuntu approach is better (when we starts doing 
openstack that was on of deciding factors between debian and ubuntu).


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Jeremy Stanley]

On 2014-12-13 23:16:56 +0800 (+0800), Thomas Goirand wrote:
[...]
> Therefore, could you please forward my thought on the meeting,
> which is: python setup.py install/sdist should be running the
> "tools/config/generate_sample.sh -b . -p nova -o etc/nova" thing
> automatically.

I'll be happy to do so. I also think this would be an excellent
solution on which to attempt to standardize (e.g., make sure that
any generated sdist contains auto-compiled sample configuration),
but think we should consider taking it further: embedding the sample
configuration in the developer docs which are auto-built for each
new change merged as well. This would provide a stable location from
which the sample can be referenced without needing to unpack a
tarball or manually rerun the generator.
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Jeremy Stanley]

On 2014-12-13 23:13:44 +0800 (+0800), Thomas Goirand wrote:
[...]
> Once we have this, then we could start building a repository with
> everything from trunk. And when that is done, starting the effort of
> building a 3rd party CI to do package validation on the gate.

This sounds like a great idea--it's basically what the Smokestack CI
was doing to spot early regressions on trunk for Red Hat's RPMs (I
haven't kept up with its status, so not sure if this is still
happening).
-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 12/13/2014 04:59 AM, Jeremy Stanley wrote:
> On a related note, this topic has been added to the agenda[1] for
> Tuesday's Cross-Project Meeting (December 16, 21:00 UTC in the
> #openstack-meeting channel on the Freenode IRC network). If you're
> passionate about the issue then please come help work out a
> consistent solution we can recommend to all projects for future
> releases.
> 
> [1] 
> https://wiki.openstack.org/wiki/Meetings/CrossProjectMeeting#Proposed_agenda

Thanks for this.

I'm afraid I wont be able to attend, since I'm back in the Chinese
timezone until the 29th of this month.

Therefore, could you please forward my thought on the meeting, which is:
python setup.py install/sdist should be running the
"tools/config/generate_sample.sh -b . -p nova -o etc/nova" thing
automatically.

Cheers,

Thomas Goirand (zigo)


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 12/13/2014 04:30 AM, George Shuklin wrote:
> I do some tiny CI in my company: repackaging ubuntu
> packages with debian-jenkins-glue (plus backported patches
> icehouse->havana)

Ah, that's interesting! :)

> If I can help somehow, I'm ready to do something, but What should I
> do, exactly?

There's a lot that can be done. If you like working on CI stuff, then
you could help me with building the package validation CI which I'm
trying to (re-)work. All of this is currently inside the debian/juno of
the openstack-meta-packages (in the openstack-tempest-ci package, which
uses the openstack-deploy package).

In the past, I saw *A LOT* of CIs, and most of them were written in a
very dirty way. In fact, it's easy to write a CI, but it's very hard to
write it well. I'm not saying my approach is perfect, but IMO it's
moving toward the good direction.

For the moment, the packaged CI can do a full all-in-one deployment from
scratch (starting with an empty VM), install and configure tempest, and
run the Keystone tempest unit tests. I'm having issues with nova-compute
using Qemu, and also the Neutron setup. But once that's fixed, I hope to
be able to run most tempest tests. The next step will be to run on a
multi-node setup.

So, if you want to help on that, and as it seems you like doing CI
stuff, you're more welcome to do so.

Once we have this, then we could start building a repository with
everything from trunk. And when that is done, starting the effort of
building a 3rd party CI to do package validation on the gate.

Your thoughts?

Cheers,

Thomas Goirand (zigo)


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

On 12/12/2014 02:17 AM, Kris G. Lindgren wrote:
>> Why do you think it's a good idea to restart doing the work of
>> distributions by yourself? Why not joining a common effort?
> 
> Well for a lot of reasons.  Some of us carry our own patch sets 

My employer (ie: Mirantis) also has its own patch set, however, we can
only work together on "community packages", so I don't really see your
point here.

>- some of
> us want to run certain services on trunk.

I also would like to have packages following trunk. One of the reason
would be to do gating on trunk, working together with the infra team. I
only lack time to build that CI for the moment, but it shouldn't be very
hard to build.

> But mainly the bigger push is
> that if we come up with a standard set of openstack packages - then we can
> build a standard set of tooling around dealing/installing those packages.

It'd be super nice if the standard would be what's in Debian. I am
convinced that Debian is the correct place to make it happen, as it's
not bound to any company or agenda.

> The point of this is that we are trying to build that common effort and
> not leave it up to the distro's to solve all the problems?

How isn't what's happening in Debian not a "common effort"? It's all
community based. I'm largely not interested in anything RPM based, so
others could work on that part if they wish.

>> On 12/09/2014 11:23 AM, Fischer, Matt wrote:
>>> Not only is the sample
>>> config a useful reference, but having it in a git repo allows me to
>>> see when new options were added or defaults were changed.
>>
>> Yeah, though if you've been doing packages for a long time, you do have
>> this information. At least, I do have it for all the last stable releases.
> 
> Great - some of us also are busy actually running clouds here...

I wasn't meaning that "I know" (hopefully, I'm not that pretending). But
rather "the Git knows" or "the Debian archive knows". In other words,
the distribution has a history of all .conf.sample over time, which
makes it possible to know.

>> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>>> [...] sample config and replace it with some tox script. Which may or
>>> may not run on your current operating system without going to pip to
>>> get newer versions of stuff.
>>
>> Excuse my words, but that's bullshit. If your operating system doesn't
>> have a version high enough for one of your build-dependency, then this
>> should be fixed. It should be your work as a package maintainer. For
>> example, if we need a newer version of tox (which is *not* the case, as
>> tox isn't really needed for generating config files), then the tox
>> package should be updated. It's as simple as that. And if you want this
>> to happen, then I would strongly suggest you to get involved in Debian
>> and/or Fedora packaging, because that's where things happen.
> 
> I guess from my point of view - not providing a sample config file for
> your project is bullshit - unless said project can work without any config
> file at all (even then its still bullshit).  Providing a script to do it -
> ok that’s live-able.  Providing documentation to use tox that installs all
> the projects deps into a venv then generates a script from those deps
> which don't match what on the system.  That’s bullshit as well.

I shouldn't have used "bullshit", I can see you took it badly. Sorry.
That wasn't aimed to create a strong reaction.

What I wanted to say was that the fact you can't run a specific command
in your distribution because of an outdated Python module is wrong,
simply because it's the work of a package maintainer to do updates if
it's needed.

And no, you don't need a venv. I use tox.ini files as a kind of
packaging documentation, so I know what shell command to use, and
everything else happens in debian/rules, with build-dependencies, not
venv stuff.

>> On 12/09/2014 12:01 PM, Kris G. Lindgren wrote:
>> Let me give you an example. In most (if not *all* projects) you will
>> find some [keystone:auth_token] section. These are directly derived from
>> what's in python-keystoneclient. Indeed, the output .conf.sample file
>> will depend on that... Now let's say the maintainers of
>> python-keystoneclient would like to update something (let's say, update
>> a comment on one of the directives, or simply a *new* directive),
>> obviously, you want that change to propagate on all projects, but *only*
>> if we have a new version of python-keystoneclient.
> 
> I guess I can get that for staying on trunk.  But when openstack does a
> stable release... Shouldn't the client versions be pretty well defined?
> And not just >= 1.4 - but should be capped to some specific version?

No, clients should *never* be capped. And in fact, over the years, I've
often had to express my strong opinion about the fact that no Python
module should ever receive a higher bound cap. Indeed, upgrading a
Python module in a distribution could be motivated by something
completely unrelated to OpenStack.

For 

Re: [Openstack-operators] Packaging sample config versions

[Jeremy Stanley]

On a related note, this topic has been added to the agenda[1] for
Tuesday's Cross-Project Meeting (December 16, 21:00 UTC in the
#openstack-meeting channel on the Freenode IRC network). If you're
passionate about the issue then please come help work out a
consistent solution we can recommend to all projects for future
releases.

[1] https://wiki.openstack.org/wiki/Meetings/CrossProjectMeeting#Proposed_agenda

-- 
Jeremy Stanley

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[George Shuklin]

Real hard work, man. Thanks.

I think the problem is that most of ops does not know enough about whole 
stuff needed. I do some tiny CI in my company: repackaging ubuntu 
packages with debian-jenkins-glue (plus backported patches 
icehouse->havana), but I can't say I understand all stuff happens under 
the hood of pbr/tox/etc.  And I spend about two weeks trying and 
learning in the process before 1st build was done.


If I can help somehow, I'm ready to do something, but What should I 
do, exactly?


On 12/11/2014 06:29 PM, Thomas Goirand wrote:

Hi,

FYI, I'm the person behind all Debian packages of OpenStack. I'm working
full-time on packaging OpenStack for Debian, and I've been doing so for
the last few years (I started to get involved in the Cactus release,
which is years ago now...). It's been a long time I wanted to reply to
this thread, and I'm finally finding the time in (the very long trip on)
the plane today.

tl;dr: The generation of config files is a non-issue which doesn't need
any tox or pip stuff. Please join me in the packaging effort in Debian
and let's work together.

Longer version:

Before I start quoting and replying, I'd like to point out a few things.
For many years, I've been trying to push for working on packages within
the community, with Debian in mind. It is my view that all the work has
to happen in Debian, since Ubuntu derives from it, and that this is
where everything happens anyway.

During this last Juno cycle, I've done absolutely *ALL* of the Python
module dependency packaging by myself, with the help of nobody. Zero,
nada, black-hole help from anyone. Even guys from Canonical didn't help,
they just synched absolutely all of the packages I made into the last
version of Ubuntu, and that's it. When I complained about it to both
Mark Shuttleworth and James Page, then the answer was "Thanks!!!".

It's been a long long time that I would like to be able to track trunk
in my package git repositories, and be able to do gating using packages.
But given the amount of work that it would represent, and the fact that
I'd be alone working on all of that, I currently gave up, until the
company that currently pays for my salary (Mirantis, since last
September) gets someone else on-board so that we may finally be a team
of more than a single person.

At the same time, I know that on the Red Hat side, they are only "2.75
persons" (if you count 0.75 for Alan Pevec, which is also doing some
Keystone work). I'd also consider this as understaffed.

So, when I have heard from my colleagues that a small group of people
from the operators mailing list thought it was a good idea to do another
set of packages of their own, I found it both amusing, irritating, and
huge time waste.

Why do you think it's a good idea to restart doing the work of
distributions by yourself? Why not joining a common effort?

I'd be more than happy to have some more contributors. We could open a
new branch on git.debian.org so that we track stuff from trunk (let's
call it debian/trunk instead of debian/kilo?). I already have all the
infrastructure to build using Jenkins, cowbuilder, etc, and
automatically package repositories for both Debian and Ubuntu. The only
thing that would be needed would be enough workforce so that we maintain
this at the same time as trunk, and fix issues as we see them (like
adding a new {build,runtime}-dependency when there's a new one, upgrade
versions when needed, and things like that).

Note that the Debian packaging is already happening in the open on
git.debian.org (ie: service managed by http://alioth.debian.org), and it
is very easy to add new contributors.

Ok, let me quote what I've seen in this thread now...

On 12/09/2014 10:46 AM, Kris G. Lindgren wrote:

Hello all,

Since somewhere around icehouse projects have started to stop shipping
sample configuration files with their projects and instead create a
README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox –egenconfig

The problem that I am running into is that tox –egenconfig now requires
a newer version of tox than what is available for the build distro:
[root@localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

First of all, you do *not* need tox to generate the config files. In
fact, you should assign yourself to *not* use tox or pip when building a
package. If you follow what tox -egenconfig (by reading the tox.ini),
you will find out it does:

bash tools/config/generate_sample.sh -b . -p nova -o etc/nova

that's the only thing you need to run, not tox!


I know I can do a pip install blah and blindly hope that I get
everything installed locally on my build system and that I don’t install
something that conflicts with the system but that seems like a less
than desirable solution.

Once you figure out what command tox -egenconfig, and what the
build-depende

Re: [Openstack-operators] Packaging sample config versions

[Joshua Harlow]

Example of this @ http://paste.ubuntu.com/9483214/

For those that are interested (and yes I know the conflict resolver
isn't super...)

-Josh

Kris G. Lindgren wrote:
> Anvil figures out the build dependencies for me - it also figures out if
> those deps are satisfiable via yum, if not downloads the version from pip
> and packages that.  So for me requirements is pretty trivial.  But to this
> point and your previous one where you ran everything on a vm to figure out
> WTF it was doing.  The point is now to get a sample config I need to as
> part of the package build process - build/install all the*RUNTIME*  deps
> for that service on the box so I can generate a config file.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Kris G. Lindgren]

Please see inline.

TL;DR - Thanks for the bash script and a kinda round about way of how you
got your configs for Debian.  You are missing the point that the packaging
tooling that we want to build - should make your life as a the only
package maintainer for Debian easier.

 
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


On 12/11/14, 9:29 AM, "Thomas Goirand"  wrote:

>Hi,
>
>FYI, I'm the person behind all Debian packages of OpenStack. I'm working
>full-time on packaging OpenStack for Debian, and I've been doing so for
>the last few years (I started to get involved in the Cactus release,
>which is years ago now...). It's been a long time I wanted to reply to
>this thread, and I'm finally finding the time in (the very long trip on)
>the plane today.
>
>tl;dr: The generation of config files is a non-issue which doesn't need
>any tox or pip stuff. Please join me in the packaging effort in Debian
>and let's work together.
>
>Longer version:
>
>Before I start quoting and replying, I'd like to point out a few things.
>For many years, I've been trying to push for working on packages within
>the community, with Debian in mind. It is my view that all the work has
>to happen in Debian, since Ubuntu derives from it, and that this is
>where everything happens anyway.
>
>During this last Juno cycle, I've done absolutely *ALL* of the Python
>module dependency packaging by myself, with the help of nobody. Zero,
>nada, black-hole help from anyone. Even guys from Canonical didn't help,
>they just synched absolutely all of the packages I made into the last
>version of Ubuntu, and that's it. When I complained about it to both
>Mark Shuttleworth and James Page, then the answer was "Thanks!!!".
>
>It's been a long long time that I would like to be able to track trunk
>in my package git repositories, and be able to do gating using packages.
>But given the amount of work that it would represent, and the fact that
>I'd be alone working on all of that, I currently gave up, until the
>company that currently pays for my salary (Mirantis, since last
>September) gets someone else on-board so that we may finally be a team
>of more than a single person.
>
>At the same time, I know that on the Red Hat side, they are only "2.75
>persons" (if you count 0.75 for Alan Pevec, which is also doing some
>Keystone work). I'd also consider this as understaffed.
>
>So, when I have heard from my colleagues that a small group of people
>from the operators mailing list thought it was a good idea to do another
>set of packages of their own, I found it both amusing, irritating, and
>huge time waste.
>
>Why do you think it's a good idea to restart doing the work of
>distributions by yourself? Why not joining a common effort?

Well for a lot of reasons.  Some of us carry our own patch sets - some of
us want to run certain services on trunk.  But mainly the bigger push is
that if we come up with a standard set of openstack packages - then we can
build a standard set of tooling around dealing/installing those packages.
The point of this is that we are trying to build that common effort and
not leave it up to the distro's to solve all the problems?
>
>I'd be more than happy to have some more contributors. We could open a
>new branch on git.debian.org so that we track stuff from trunk (let's
>call it debian/trunk instead of debian/kilo?). I already have all the
>infrastructure to build using Jenkins, cowbuilder, etc, and
>automatically package repositories for both Debian and Ubuntu. The only
>thing that would be needed would be enough workforce so that we maintain
>this at the same time as trunk, and fix issues as we see them (like
>adding a new {build,runtime}-dependency when there's a new one, upgrade
>versions when needed, and things like that).
>
>Note that the Debian packaging is already happening in the open on
>git.debian.org (ie: service managed by http://alioth.debian.org), and it
>is very easy to add new contributors.

That’s cool.  I don't use Debian/Ubuntu.

>
>Ok, let me quote what I've seen in this thread now...
>
>On 12/09/2014 10:46 AM, Kris G. Lindgren wrote:
>> Hello all,
>> 
>> Since somewhere around icehouse projects have started to stop shipping
>> sample configuration files with their projects and instead create a
>> README-sample.conf.txt that usually contains something like:
>> To generate the sample nova.conf file, run the following
>> command from the top level of the nova directory:
>> 
>> tox ­egenconfig
>> 
>> The problem that I am running into is that tox ­egenconfig now requires
>> a newer version of tox than what is available for the build distro:
>> [root@localhost ceilometer-2014.2.1]# tox -egenconfig
>> ERROR: tox version is 1.4.2, required is at least 1.6
>
>First of all, you do *not* need tox to generate the config files. In
>fact, you should assign yourself to *not* use tox or pip when building a
>package. If you follow what tox -egenconfig (by reading the tox.in

Re: [Openstack-operators] Packaging sample config versions

[Fischer, Matt]

Top-posting because the quoting is getting pretty deep:

>At first glance, I don't know what's going on there, so I'll investigate 
>further and comment in the bugs. Thanks Matt for logging.
>
>I'm not sure I exactly understand the difference between sample configs and 
>example configs - my best guess is:
>sample configs: complete .conf file with some options enabled that will work 
>and all the rest of the options in the file but commented out including 
>defaults and description?
>example configs: working .conf file that doesn't contain all the options.
>
>Is that accurate?

Yes, that’s right. I see example configs as useful for something like “how do I 
configure keystone to talk to LDAP”, whereas sample configs (aka complete 
configs) are useful for me when I want to know whether puppet is changing a 
default. I’d say we inherit 75% or more default values, and my “upstream” for 
defaults is not only the projects but the puppet code.

If you have better terminology, I’m happy to switch, sorry for the potential 
confusion.

Thanks.


From: Anne Gentle mailto:a...@openstack.org>>
Date: Thursday, December 11, 2014 at 9:38 AM
To: Matt Fischer 
mailto:matthew.fisc...@twcable.com>>
Cc: Michael Dorman mailto:mdor...@godaddy.com>>, 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
mailto:openstack-operators@lists.openstack.org>>
Subject: Re: [Openstack-operators] Packaging sample config versions



On Wed, Dec 10, 2014 at 11:16 AM, Fischer, Matt 
mailto:matthew.fisc...@twcable.com>> wrote:


From: Anne Gentle mailto:a...@openstack.org>>
Date: Wednesday, December 10, 2014 at 8:41 AM
To: Michael Dorman mailto:mdor...@godaddy.com>>
Cc: 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
mailto:openstack-operators@lists.openstack.org>>
Subject: Re: [Openstack-operators] Packaging sample config versions



On Tue, Dec 9, 2014 at 11:25 AM, Michael Dorman 
mailto:mdor...@godaddy.com>> wrote:
Well I think we can all agree this is an irritation.  But how are others
actually dealing with this problem?  (Maybe it’s less complicated in
Ubuntu.)

The sense I get is that most people using Anvil, or other custom-ish
packaging tools, are also running config management which handles
generating the config files, anyway.  So you don’t so much care about the
contents of the config file shipped with the package.

Is that accurate for most people?  Or are folks doing some other magic to
get a good config file in the packages?

>The docs team -- reallly, Matt Kassawara -- regularly logs bugs for packagers 
>to put in better, working, default config files.
>
>We do generate documentation for all the configurations across projects that 
>use oslo.config (and even for swift, which doesn't). So you can rely on this 
>reference:
>http://docs.openstack.org/juno/config-reference/content/

I don’t see full sample configs in here, just more focused “example” configs, 
am I missing it? Does the generated content include everything from config.py 
in each project? I’m asking because I’ve seen a few places where the documented 
defaults and whats in the code do not match.

When I upgrade puppet versions, like I’m doing now from the icehouse puppet 
branches, its very important for me to know:

 1.  what has a new default value
 2.  whether we were previously using the old default value
 3.  what lines change/move/got renamed
 4.  what is deprecated

The docs do a good job of most of these, but I cannot beat a well maintained 
full sample config.

Here are a few small examples of places where the docs and code didn’t match 
for Juno, hence my questions about how the docs were generated:

https://bugs.launchpad.net/openstack-manuals/+bug/1380778
https://bugs.launchpad.net/openstack-manuals/+bug/1380813


At first glance, I don't know what's going on there, so I'll investigate 
further and comment in the bugs. Thanks Matt for logging.

I'm not sure I exactly understand the difference between sample configs and 
example configs - my best guess is:
sample configs: complete .conf file with some options enabled that will work 
and all the rest of the options in the file but commented out including 
defaults and description?
example configs: working .conf file that doesn't contain all the options.

Is that accurate?

Note: I also filed two bugs against keystone itself for having default values 
that were deprecated, so its not perfect either.

>You can also see new, updated, and deprecated options for each service, such 
>as:
>http://docs.openstack.org/juno/config-reference/content/nova-conf-changes-juno.html
>
>I don't believe our reference document was what encouraged devs to take the 
>sample config generation out-of-tree, but I am letting you know you

Re: [Openstack-operators] Packaging sample config versions

[Anne Gentle]

On Wed, Dec 10, 2014 at 11:16 AM, Fischer, Matt  wrote:

>
>
>   From: Anne Gentle 
> Date: Wednesday, December 10, 2014 at 8:41 AM
> To: Michael Dorman 
> Cc: "openstack-operators@lists.openstack.org" <
> openstack-operators@lists.openstack.org>
> Subject: Re: [Openstack-operators] Packaging sample config versions
>
>
>
> On Tue, Dec 9, 2014 at 11:25 AM, Michael Dorman 
> wrote:
>
>> Well I think we can all agree this is an irritation.  But how are others
>> actually dealing with this problem?  (Maybe it’s less complicated in
>> Ubuntu.)
>>
>> The sense I get is that most people using Anvil, or other custom-ish
>> packaging tools, are also running config management which handles
>> generating the config files, anyway.  So you don’t so much care about the
>> contents of the config file shipped with the package.
>>
>> Is that accurate for most people?  Or are folks doing some other magic to
>> get a good config file in the packages?
>>
>
>  >The docs team -- reallly, Matt Kassawara -- regularly logs bugs for
> packagers to put in better, working, default config files.
> >
> >We do generate documentation for all the configurations across projects
> that use oslo.config (and even for swift, which doesn't). So you can rely
> on this reference:
> >http://docs.openstack.org/juno/config-reference/content/
>
>  I don’t see full sample configs in here, just more focused “example”
> configs, am I missing it? Does the generated content include everything
> from config.py in each project? I’m asking because I’ve seen a few places
> where the documented defaults and whats in the code do not match.
>
>  When I upgrade puppet versions, like I’m doing now from the icehouse
> puppet branches, its very important for me to know:
>
>1. what has a new default value
>2. whether we were previously using the old default value
>3. what lines change/move/got renamed
>4. what is deprecated
>
>  The docs do a good job of most of these, but I cannot beat a well
> maintained full sample config.
>
>  Here are a few small examples of places where the docs and code didn’t
> match for Juno, hence my questions about how the docs were generated:
>
>  https://bugs.launchpad.net/openstack-manuals/+bug/1380778
>  https://bugs.launchpad.net/openstack-manuals/+bug/1380813
>
>
At first glance, I don't know what's going on there, so I'll investigate
further and comment in the bugs. Thanks Matt for logging.

I'm not sure I exactly understand the difference between sample configs and
example configs - my best guess is:
sample configs: complete .conf file with some options enabled that will
work and all the rest of the options in the file but commented out
including defaults and description?
example configs: working .conf file that doesn't contain all the options.

Is that accurate?


>  Note: I also filed two bugs against keystone itself for having default
> values that were deprecated, so its not perfect either.
>
>  >You can also see new, updated, and deprecated options for each service,
> such as:
> >
> http://docs.openstack.org/juno/config-reference/content/nova-conf-changes-juno.html
>  >
> >I don't believe our reference document was what encouraged devs to take
> the sample config generation out-of-tree, but I am letting you know your
> best option besides troubleshooting generating it yourself.
> >
> >Anne
>
>
>>
>> Mike
>>
>>
>>
>>
>>
>> On 12/9/14, 5:02 PM, "Kris G. Lindgren"  wrote:
>>
>> >So more to my point on the latest version of RHEL and doing: yum install
>> >tox -egenconfig
>> >
>> >ceilometer-2014.2.1]# tox -egenconfig
>> >ERROR: tox version is 1.4.2, required is at least 1.6
>> >
>> >
>> >nova-2014.2.1]# tox -egenconfig
>> >ERROR: tox version is 1.4.2, required is at least 1.6
>> >
>> >
>> >glance-2014.2.1]# tox -egenconfig
>> >ERROR: tox version is 1.4.2, required is at least 1.6
>> >
>> >
>> >[root@localhost ~]# pip install --update tox
>> >(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
>> >1.4.14)
>> >
>> >glance-2014.2.1]# tox -egenconfig
>> >genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
>> >genconfig installdeps:
>> >-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
>> >-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
>> >ERROR: invocation failed (exit code 1), logfile:
>> >/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genc

Re: [Openstack-operators] Packaging sample config versions

[Thomas Goirand]

Hi,

FYI, I'm the person behind all Debian packages of OpenStack. I'm working
full-time on packaging OpenStack for Debian, and I've been doing so for
the last few years (I started to get involved in the Cactus release,
which is years ago now...). It's been a long time I wanted to reply to
this thread, and I'm finally finding the time in (the very long trip on)
the plane today.

tl;dr: The generation of config files is a non-issue which doesn't need
any tox or pip stuff. Please join me in the packaging effort in Debian
and let's work together.

Longer version:

Before I start quoting and replying, I'd like to point out a few things.
For many years, I've been trying to push for working on packages within
the community, with Debian in mind. It is my view that all the work has
to happen in Debian, since Ubuntu derives from it, and that this is
where everything happens anyway.

During this last Juno cycle, I've done absolutely *ALL* of the Python
module dependency packaging by myself, with the help of nobody. Zero,
nada, black-hole help from anyone. Even guys from Canonical didn't help,
they just synched absolutely all of the packages I made into the last
version of Ubuntu, and that's it. When I complained about it to both
Mark Shuttleworth and James Page, then the answer was "Thanks!!!".

It's been a long long time that I would like to be able to track trunk
in my package git repositories, and be able to do gating using packages.
But given the amount of work that it would represent, and the fact that
I'd be alone working on all of that, I currently gave up, until the
company that currently pays for my salary (Mirantis, since last
September) gets someone else on-board so that we may finally be a team
of more than a single person.

At the same time, I know that on the Red Hat side, they are only "2.75
persons" (if you count 0.75 for Alan Pevec, which is also doing some
Keystone work). I'd also consider this as understaffed.

So, when I have heard from my colleagues that a small group of people
from the operators mailing list thought it was a good idea to do another
set of packages of their own, I found it both amusing, irritating, and
huge time waste.

Why do you think it's a good idea to restart doing the work of
distributions by yourself? Why not joining a common effort?

I'd be more than happy to have some more contributors. We could open a
new branch on git.debian.org so that we track stuff from trunk (let's
call it debian/trunk instead of debian/kilo?). I already have all the
infrastructure to build using Jenkins, cowbuilder, etc, and
automatically package repositories for both Debian and Ubuntu. The only
thing that would be needed would be enough workforce so that we maintain
this at the same time as trunk, and fix issues as we see them (like
adding a new {build,runtime}-dependency when there's a new one, upgrade
versions when needed, and things like that).

Note that the Debian packaging is already happening in the open on
git.debian.org (ie: service managed by http://alioth.debian.org), and it
is very easy to add new contributors.

Ok, let me quote what I've seen in this thread now...

On 12/09/2014 10:46 AM, Kris G. Lindgren wrote:
> Hello all,
> 
> Since somewhere around icehouse projects have started to stop shipping
> sample configuration files with their projects and instead create a
> README-sample.conf.txt that usually contains something like:
> To generate the sample nova.conf file, run the following
> command from the top level of the nova directory:
> 
> tox –egenconfig
> 
> The problem that I am running into is that tox –egenconfig now requires
> a newer version of tox than what is available for the build distro:
> [root@localhost ceilometer-2014.2.1]# tox -egenconfig
> ERROR: tox version is 1.4.2, required is at least 1.6

First of all, you do *not* need tox to generate the config files. In
fact, you should assign yourself to *not* use tox or pip when building a
package. If you follow what tox -egenconfig (by reading the tox.ini),
you will find out it does:

bash tools/config/generate_sample.sh -b . -p nova -o etc/nova

that's the only thing you need to run, not tox!

> I know I can do a pip install blah and blindly hope that I get
> everything installed locally on my build system and that I don’t install
> something that conflicts with the system but that seems like a less
> than desirable solution.

Once you figure out what command tox -egenconfig, and what the
build-dependencies are, it's actually fine.

> So how are people who are doing packaging
> handling this? Are you now building a venv per package for tox just to
> generate a sample configuration?

Last time I did use a VM to figure out what was doing on with "tox
-egenconfig". Then once it was done, I just copied the nova.conf file
into the debian folder of the nova package, though next time, I'll just
do things properly like I just wrote above: find out the correct
build-depends, and hard-wire the generation of the /conf.sample fil

Re: [Openstack-operators] Packaging sample config versions

[Fischer, Matt]

From: Anne Gentle mailto:a...@openstack.org>>
Date: Wednesday, December 10, 2014 at 8:41 AM
To: Michael Dorman mailto:mdor...@godaddy.com>>
Cc: 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
mailto:openstack-operators@lists.openstack.org>>
Subject: Re: [Openstack-operators] Packaging sample config versions



On Tue, Dec 9, 2014 at 11:25 AM, Michael Dorman 
mailto:mdor...@godaddy.com>> wrote:
Well I think we can all agree this is an irritation.  But how are others
actually dealing with this problem?  (Maybe it’s less complicated in
Ubuntu.)

The sense I get is that most people using Anvil, or other custom-ish
packaging tools, are also running config management which handles
generating the config files, anyway.  So you don’t so much care about the
contents of the config file shipped with the package.

Is that accurate for most people?  Or are folks doing some other magic to
get a good config file in the packages?

>The docs team -- reallly, Matt Kassawara -- regularly logs bugs for packagers 
>to put in better, working, default config files.
>
>We do generate documentation for all the configurations across projects that 
>use oslo.config (and even for swift, which doesn't). So you can rely on this 
>reference:
>http://docs.openstack.org/juno/config-reference/content/

I don’t see full sample configs in here, just more focused “example” configs, 
am I missing it? Does the generated content include everything from config.py 
in each project? I’m asking because I’ve seen a few places where the documented 
defaults and whats in the code do not match.

When I upgrade puppet versions, like I’m doing now from the icehouse puppet 
branches, its very important for me to know:

 1.  what has a new default value
 2.  whether we were previously using the old default value
 3.  what lines change/move/got renamed
 4.  what is deprecated

The docs do a good job of most of these, but I cannot beat a well maintained 
full sample config.

Here are a few small examples of places where the docs and code didn’t match 
for Juno, hence my questions about how the docs were generated:

https://bugs.launchpad.net/openstack-manuals/+bug/1380778
https://bugs.launchpad.net/openstack-manuals/+bug/1380813

Note: I also filed two bugs against keystone itself for having default values 
that were deprecated, so its not perfect either.

>You can also see new, updated, and deprecated options for each service, such 
>as:
>http://docs.openstack.org/juno/config-reference/content/nova-conf-changes-juno.html
>
>I don't believe our reference document was what encouraged devs to take the 
>sample config generation out-of-tree, but I am letting you know your best 
>option besides troubleshooting generating it yourself.
>
>Anne


Mike





On 12/9/14, 5:02 PM, "Kris G. Lindgren" 
mailto:klindg...@godaddy.com>> wrote:

>So more to my point on the latest version of RHEL and doing: yum install
>tox -egenconfig
>
>ceilometer-2014.2.1]# tox -egenconfig
>ERROR: tox version is 1.4.2, required is at least 1.6
>
>
>nova-2014.2.1]# tox -egenconfig
>ERROR: tox version is 1.4.2, required is at least 1.6
>
>
>glance-2014.2.1]# tox -egenconfig
>ERROR: tox version is 1.4.2, required is at least 1.6
>
>
>[root@localhost ~]# pip install --update tox
>(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
>1.4.14)
>
>glance-2014.2.1]# tox -egenconfig
>genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
>genconfig installdeps:
>-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
>-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
>ERROR: invocation failed (exit code 1), logfile:
>/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
>ERROR: actionid=genconfig
>
>
>Running setup.py install for MySQL-python
>
>   /usr/include/mysql/my_config_x86_64.h:654:2: error: #error
> MUST be included first!
> #error  MUST be included first!
>  ^
>error: command 'gcc' failed with exit status 1
>
>__ summary
>__
>ERROR:   genconfig: could not install deps
>[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
>-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
>InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/p
>i
>p install --allow-all-external --allow-insecure netaddr -U
>-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
>-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
>/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
>1)
>
>
>
>So a few things to p

Re: [Openstack-operators] Packaging sample config versions

[Anne Gentle]

On Tue, Dec 9, 2014 at 11:25 AM, Michael Dorman  wrote:

> Well I think we can all agree this is an irritation.  But how are others
> actually dealing with this problem?  (Maybe it’s less complicated in
> Ubuntu.)
>
> The sense I get is that most people using Anvil, or other custom-ish
> packaging tools, are also running config management which handles
> generating the config files, anyway.  So you don’t so much care about the
> contents of the config file shipped with the package.
>
> Is that accurate for most people?  Or are folks doing some other magic to
> get a good config file in the packages?
>

The docs team -- reallly, Matt Kassawara -- regularly logs bugs for
packagers to put in better, working, default config files.

We do generate documentation for all the configurations across projects
that use oslo.config (and even for swift, which doesn't). So you can rely
on this reference:
http://docs.openstack.org/juno/config-reference/content/

You can also see new, updated, and deprecated options for each service,
such as:
http://docs.openstack.org/juno/config-reference/content/nova-conf-changes-juno.html

I don't believe our reference document was what encouraged devs to take the
sample config generation out-of-tree, but I am letting you know your best
option besides troubleshooting generating it yourself.

Anne


>
> Mike
>
>
>
>
>
> On 12/9/14, 5:02 PM, "Kris G. Lindgren"  wrote:
>
> >So more to my point on the latest version of RHEL and doing: yum install
> >tox -egenconfig
> >
> >ceilometer-2014.2.1]# tox -egenconfig
> >ERROR: tox version is 1.4.2, required is at least 1.6
> >
> >
> >nova-2014.2.1]# tox -egenconfig
> >ERROR: tox version is 1.4.2, required is at least 1.6
> >
> >
> >glance-2014.2.1]# tox -egenconfig
> >ERROR: tox version is 1.4.2, required is at least 1.6
> >
> >
> >[root@localhost ~]# pip install --update tox
> >(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
> >1.4.14)
> >
> >glance-2014.2.1]# tox -egenconfig
> >genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
> >genconfig installdeps:
> >-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
> >-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
> >ERROR: invocation failed (exit code 1), logfile:
> >/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
> >ERROR: actionid=genconfig
> >
> >
> >Running setup.py install for MySQL-python
> >
> >   /usr/include/mysql/my_config_x86_64.h:654:2: error: #error
> > MUST be included first!
> > #error  MUST be included first!
> >  ^
> >error: command 'gcc' failed with exit status 1
> >
> >__ summary
> >__
> >ERROR:   genconfig: could not install deps
> >[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
> >-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
> >InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/p
> >i
> >p install --allow-all-external --allow-insecure netaddr -U
> >-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
> >-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
> >/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
> >1)
> >
> >
> >
> >So a few things to point out in order to even get tox -egenconfig I had to
> >update the system packages versions using pip.  Since we have other python
> >packages using virtualenv I have no idea if the updated venvironment
> >package is going to break those systems or not.  So the included
> >script/command is already a barrier to getting a sample config.  2) tox
> >fails to even build all the deps - it happens to be exactly failing at
> >mysql in both nova/cinder/glance/keystone 3) It's installing it own
> >versions of python libraries that solve the dependencies that are then
> >going to be used to generate the configuration.  If the configuration is
> >so dynamic that getting a different version of oslo.config could generate
> >a sample configuration that wont work on my system then how am I suppose
> >to deal with:
> >Tox installed version:
> >oslo.config-1.5.0
> >
> >
> >System installed version:
> >python-oslo-config-1.3.0
> >
> >
> >
> >
> >Also python-libvrit failed to build because I don¹t have libvrit installed
> >on this system.  So am I to assume that there are no libvrit options
> >(which we both know is false)?
> >Now I can get a example config - that wont work with my system - per what
> >everyone else has been saying.  Also, at what point would the average user
> >just say "F it"? - because at the point I feel like if I was an average
> >user - I would be there right now.
> >
> >
> >Kris Lindgren
> >Senior Linux Systems Engineer
> >GoDaddy, LLC.
> >
> >
> >On 12/9/14, 8:14 AM, "Mathieu Gagné"  wrote:
> >
> >>On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:
> >>>
> >>>   I don¹t thin

Re: [Openstack-operators] Packaging sample config versions

[Michael Dorman]

Well I think we can all agree this is an irritation.  But how are others 
actually dealing with this problem?  (Maybe it’s less complicated in 
Ubuntu.)

The sense I get is that most people using Anvil, or other custom-ish 
packaging tools, are also running config management which handles 
generating the config files, anyway.  So you don’t so much care about the 
contents of the config file shipped with the package.

Is that accurate for most people?  Or are folks doing some other magic to 
get a good config file in the packages?

Mike





On 12/9/14, 5:02 PM, "Kris G. Lindgren"  wrote:

>So more to my point on the latest version of RHEL and doing: yum install
>tox -egenconfig
>
>ceilometer-2014.2.1]# tox -egenconfig
>ERROR: tox version is 1.4.2, required is at least 1.6
>
>
>nova-2014.2.1]# tox -egenconfig
>ERROR: tox version is 1.4.2, required is at least 1.6
>
>
>glance-2014.2.1]# tox -egenconfig
>ERROR: tox version is 1.4.2, required is at least 1.6
>
>
>[root@localhost ~]# pip install --update tox
>(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
>1.4.14)
>
>glance-2014.2.1]# tox -egenconfig
>genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
>genconfig installdeps:
>-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
>-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
>ERROR: invocation failed (exit code 1), logfile:
>/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
>ERROR: actionid=genconfig
>
>
>Running setup.py install for MySQL-python
>
>   /usr/include/mysql/my_config_x86_64.h:654:2: error: #error
> MUST be included first!
> #error  MUST be included first!
>  ^
>error: command 'gcc' failed with exit status 1
>
>__ summary
>__
>ERROR:   genconfig: could not install deps
>[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
>-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
>InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/p
>i
>p install --allow-all-external --allow-insecure netaddr -U
>-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
>-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
>/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
>1)
>
>
>
>So a few things to point out in order to even get tox -egenconfig I had to
>update the system packages versions using pip.  Since we have other python
>packages using virtualenv I have no idea if the updated venvironment
>package is going to break those systems or not.  So the included
>script/command is already a barrier to getting a sample config.  2) tox
>fails to even build all the deps - it happens to be exactly failing at
>mysql in both nova/cinder/glance/keystone 3) It's installing it own
>versions of python libraries that solve the dependencies that are then
>going to be used to generate the configuration.  If the configuration is
>so dynamic that getting a different version of oslo.config could generate
>a sample configuration that wont work on my system then how am I suppose
>to deal with: 
>Tox installed version:
>oslo.config-1.5.0
>
>
>System installed version:
>python-oslo-config-1.3.0
>
>
>
>
>Also python-libvrit failed to build because I don¹t have libvrit installed
>on this system.  So am I to assume that there are no libvrit options
>(which we both know is false)?
>Now I can get a example config - that wont work with my system - per what
>everyone else has been saying.  Also, at what point would the average user
>just say "F it"? - because at the point I feel like if I was an average
>user - I would be there right now.
>
> 
>Kris Lindgren
>Senior Linux Systems Engineer
>GoDaddy, LLC.
>
>
>On 12/9/14, 8:14 AM, "Mathieu Gagné"  wrote:
>
>>On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:
>>>
>>>   I don¹t think its too much to ask for each project to include a 
>>>script
>>> that will build a venv that includes tox and the other relevant deps to
>>> build the sample configuration.
>>
>>This is already the case. Back then, I did the work of documenting how
>>you could generate the sample config files for each projects (I cared
>>about):
>>http://blog.mgagne.ca/generating-sample-config-files-in-openstack/
>>
>>You can see that the process isn't streamlined. Each project has its own
>>particularities. Some projects don't use the (un)official standard "tox
>>-egenconfig" command, I patched some projects to make it less a pain.
>>
>>And my thoughts about sample config files:
>>http://blog.mgagne.ca/where-are-the-sample-config-files/
>>
>>I wrote it after Cinder proposed removing its sample config file. They
>>abandoned the patch at that time but now sample config file is gone.
>>
>>It's not an easy problem because core libraries used by OpenStack
>>projects also use oslo.config and configs requ

Re: [Openstack-operators] Packaging sample config versions

[Kris G. Lindgren]

So more to my point on the latest version of RHEL and doing: yum install
tox -egenconfig

ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6


nova-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6


glance-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6


[root@localhost ~]# pip install --update tox
(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
1.4.14)

glance-2014.2.1]# tox -egenconfig
genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
genconfig installdeps:
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile:
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
ERROR: actionid=genconfig


Running setup.py install for MySQL-python

   /usr/include/mysql/my_config_x86_64.h:654:2: error: #error
 MUST be included first!
 #error  MUST be included first!
  ^
error: command 'gcc' failed with exit status 1

__ summary
__
ERROR:   genconfig: could not install deps
[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/pi
p install --allow-all-external --allow-insecure netaddr -U
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
1)



So a few things to point out in order to even get tox -egenconfig I had to
update the system packages versions using pip.  Since we have other python
packages using virtualenv I have no idea if the updated venvironment
package is going to break those systems or not.  So the included
script/command is already a barrier to getting a sample config.  2) tox
fails to even build all the deps - it happens to be exactly failing at
mysql in both nova/cinder/glance/keystone 3) It's installing it own
versions of python libraries that solve the dependencies that are then
going to be used to generate the configuration.  If the configuration is
so dynamic that getting a different version of oslo.config could generate
a sample configuration that wont work on my system then how am I suppose
to deal with: 
Tox installed version:
oslo.config-1.5.0


System installed version:
python-oslo-config-1.3.0




Also python-libvrit failed to build because I don¹t have libvrit installed
on this system.  So am I to assume that there are no libvrit options
(which we both know is false)?
Now I can get a example config - that wont work with my system - per what
everyone else has been saying.  Also, at what point would the average user
just say "F it"? - because at the point I feel like if I was an average
user - I would be there right now.

 
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


On 12/9/14, 8:14 AM, "Mathieu Gagné"  wrote:

>On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:
>>
>>   I don¹t think its too much to ask for each project to include a script
>> that will build a venv that includes tox and the other relevant deps to
>> build the sample configuration.
>
>This is already the case. Back then, I did the work of documenting how
>you could generate the sample config files for each projects (I cared
>about):
>http://blog.mgagne.ca/generating-sample-config-files-in-openstack/
>
>You can see that the process isn't streamlined. Each project has its own
>particularities. Some projects don't use the (un)official standard "tox
>-egenconfig" command, I patched some projects to make it less a pain.
>
>And my thoughts about sample config files:
>http://blog.mgagne.ca/where-are-the-sample-config-files/
>
>I wrote it after Cinder proposed removing its sample config file. They
>abandoned the patch at that time but now sample config file is gone.
>
>It's not an easy problem because core libraries used by OpenStack
>projects also use oslo.config and configs required by those libraries
>are part of the main configurations required for a project to even work.
>([database]/connection for instance) You just can't ignore those configs
>when generating the sample config file.
>
>(sorry for self-promotion, I didn't want to rewrite my thoughts on this
>list)
>
>-- 
>Mathieu


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Mathieu Gagné]

On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:


  I don’t think its too much to ask for each project to include a script
that will build a venv that includes tox and the other relevant deps to
build the sample configuration.


This is already the case. Back then, I did the work of documenting how 
you could generate the sample config files for each projects (I cared 
about):

http://blog.mgagne.ca/generating-sample-config-files-in-openstack/

You can see that the process isn't streamlined. Each project has its own 
particularities. Some projects don't use the (un)official standard "tox 
-egenconfig" command, I patched some projects to make it less a pain.


And my thoughts about sample config files:
http://blog.mgagne.ca/where-are-the-sample-config-files/

I wrote it after Cinder proposed removing its sample config file. They 
abandoned the patch at that time but now sample config file is gone.


It's not an easy problem because core libraries used by OpenStack 
projects also use oslo.config and configs required by those libraries 
are part of the main configurations required for a project to even work. 
([database]/connection for instance) You just can't ignore those configs 
when generating the sample config file.


(sorry for self-promotion, I didn't want to rewrite my thoughts on this 
list)


--
Mathieu

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Kris G. Lindgren]

Matt,

I completely agree.

Nova stopped shipping the file starting in icehouse, ceilometer started in 
Juno, I see commits now in pretty much every project that remove the sample 
config and replace it with some tox script.   Which may or may not run on your 
current operating system – without going to pip to get newer versions of stuff.

I see that whats being said is that a  valid sample configuration file depends 
on the version of the other dependent libraries that are installed on the 
system as well.  I assume this is talking about different versions of 
oslo.config and oslo.messaging take different configuration options and as such 
we can't generate a generic sample config file because we don’t know what 
versions of dependent libraries you actually have installed.

This seems to have taken a problem that’s been created on the development side 
and simply punted over the wall to let document writers/operators/packagers 
deal with:  As this sample configuration file changes independently from the 
project, this check failed on the gate time to time, as this file  needed to be 
updated manually [1].  I guess I reject the assertion that the sample 
configuration file is independent of the project.  Just because some options 
change depending on the oslo.config version doesn't mean the configuration 
options that are specific to that project change as well…

Anyway – back to packaging.  The problem that I have with this change is that 
in order to package a sample configuration I need to basically do the 
following: 1.) get the current build/commit run 2.) run python build 3.) strip 
away the relevant built parts of the package 4.) install on the build machine 
all the python runtime deps that I am going to need for this package 5.) 
install tox and other packages as needed to generate a sample configuration 6.) 
generate sample config 7.) build a new package with the exact same requirements 
as what I installed in step 4 8.) add sample configuration generated in step  6 
to the package.  Then I need to make sure I also package all of the 
python-versions that was used in step 4, making sure that I don’t have 
conflicting system level dependencies from other openstack projects.

Now anvil can do a lot of this for me… but this seems overly complicated.

 I don’t think its too much to ask for each project to include a script that 
will build a venv that includes tox and the other relevant deps to build the 
sample configuration.   IMHO, its also ok if the sample configuration is not 
100% perfect re: config options for dependencies – call that out in the config 
file and we can look up what the correct config parameters are based upon our 
configuration environment.  But there is no execuse to not include an example 
config that at least has the config options relevant to project.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


From: , Matt 
mailto:matthew.fisc...@twcable.com>>
Date: Monday, December 8, 2014 at 8:23 PM
To: "Kris G. Lindgren" mailto:klindg...@godaddy.com>>, 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
mailto:openstack-operators@lists.openstack.org>>
Subject: Re: [Openstack-operators] Packaging sample config versions

I also find this issue really really annoying. Not only is the sample config a 
useful reference, but having it in a git repo allows me to see when new options 
were added or defaults were changed. Otherwise you have to rely on the docs 
which are generally wrong. When I was looking at some of the Juno transitions, 
I’ve already filed a few doc bugs about wrong defaults and config options. 
Assuming a project enforces that the sample config is up-to-date when commits 
are made, it’s by far the best source of this info. As for why? Well I’ve 
brought this up a few times before, and there was even a bug filed 
(https://bugs.launchpad.net/nova/+bug/1301519) but it was marked as “Won’t 
Fix”. I don’t really know the reason. This is one of my larger annoyances as an 
operator…

I do know that some projects, like nova and cinder, still ship the file.

Maybe someone can setup a cron job to do a pull, build the sample configs and 
post them to git hub every day.

From: "Kris G. Lindgren" mailto:klindg...@godaddy.com>>
Date: Monday, December 8, 2014 at 7:46 PM
To: 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
mailto:openstack-operators@lists.openstack.org>>
Subject: [Openstack-operators] Packaging sample config versions

Hello all,

Since somewhere around icehouse projects have started to stop shipping sample 
configuration files with their projects and instead create a 
README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox 

Re: [Openstack-operators] Packaging sample config versions

[Fischer, Matt]

>>
>> I do know that some projects, like nova and cinder, still ship the file.
>
>Cinder I believe just kicked that out (for better or worse),
>
>https://github.com/openstack/cinder/commit/62a72a7574b18f7
>
>One less. I don't think nova does either anymore...
>

Oops I meant keystone! Nova was one of the first places I noticed it was
gone.


This E-mail and any of its attachments may contain Time Warner Cable 
proprietary information, which is privileged, confidential, or subject to 
copyright belonging to Time Warner Cable. This E-mail is intended solely for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient of this E-mail, you are hereby notified that any 
dissemination, distribution, copying, or action taken in relation to the 
contents of and attachments to this E-mail is strictly prohibited and may be 
unlawful. If you have received this E-mail in error, please notify the sender 
immediately and permanently delete the original and any copy of this E-mail and 
any printout.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Mathieu Gagné]

On 2014-12-08 10:23 PM, Fischer, Matt wrote:


I do know that some projects, like nova and cinder, still ship the file.



Nova does not ship the config file anymore:
https://review.openstack.org/#/c/81588/
https://github.com/openstack/nova/blob/master/etc/nova/README-nova.conf.txt

Cinder recently removed its file too:
https://review.openstack.org/#/c/139511/
https://github.com/openstack/cinder/commit/62a72a7574b18f74e8fceab700c2e831f6ca8961

--
Mathieu

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging sample config versions

[Joshua Harlow]

Fischer, Matt wrote:

I also find this issue really really annoying. Not only is the sample
config a useful reference, but having it in a git repo allows me to see
when new options were added or defaults were changed. Otherwise you have
to rely on the docs which are generally wrong. When I was looking at
some of the Juno transitions, I’ve already filed a few doc bugs about
wrong defaults and config options. Assuming a project enforces that the
sample config is up-to-date when commits are made, it’s by far the best
source of this info. As for why? Well I’ve brought this up a few times
before, and there was even a bug filed
(https://bugs.launchpad.net/nova/+bug/1301519
<https://bugs.launchpad.net/nova/+bug/1301519>) but it was marked as
“Won’t Fix”. I don’t really know the reason. This is one of my larger
annoyances as an operator…

I do know that some projects, like nova and cinder, still ship the file.


Cinder I believe just kicked that out (for better or worse),

https://github.com/openstack/cinder/commit/62a72a7574b18f7

One less. I don't think nova does either anymore...

Personally I don't know how to solve this without changes around the 
usage of oslo.config and its pervasiveness (and how usage of oslo.config 
by libraries can alter the valid configuration of a project just by 
installing a different library version); which is a pretty big change 
all over the place.


Perhaps this goes back to the point that I've seen on the developers 
list in: 
http://lists.openstack.org/pipermail/openstack-dev/2014-December/052150.html 
(perhaps projects shouldn't use libraries that use oslo.config and those 
projects should *only* use oslo.config in there top-level project and 
configure libraries via some other mechanism that doesn't cause these 
kinds of dynamically changing configuration issues).


Something to think about...

-Josh



Maybe someone can setup a cron job to do a pull, build the sample
configs and post them to git hub every day.

From: "Kris G. Lindgren" mailto:klindg...@godaddy.com>>
Date: Monday, December 8, 2014 at 7:46 PM
To: "openstack-operators@lists.openstack.org
<mailto:openstack-operators@lists.openstack.org>"
mailto:openstack-operators@lists.openstack.org>>
Subject: [Openstack-operators] Packaging sample config versions

Hello all,

Since somewhere around icehouse projects have started to stop shipping
sample configuration files with their projects and instead create a
README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox –egenconfig

The problem that I am running into is that tox –egenconfig now requires
a newer version of tox than what is available for the build distro:
[root@localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

I know I can do a pip install blah and blindly hope that I get
everything installed locally on my build system and that I don’t install
something that conflicts with the system… but that seems like a less
than desirable solution. So how are people who are doing packaging
handling this? Are you now building a venv per package for tox just to
generate a sample configuration? Shouldn't this be part of the python
build/install steps per project?

It seems redhat/rdo is simply including a sample configuration that they
generated somehow.

What are you other packagers doing?

Also, is it just me or does this seem wrong? Most of the commits that
made this change seem to indicate it was because the sample config file
was separate from the project and that it was breaking gate when it
wasn't kept up to date. Shouldn't this be something that each project
generates? This seemed to work for 8 previous releases – now its too
difficult? I don’t get the motivations behind this change.

Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


This E-mail and any of its attachments may contain Time Warner Cable
proprietary information, which is privileged, confidential, or subject
to copyright belonging to Time Warner Cable. This E-mail is intended
solely for the use of the individual or entity to which it is addressed.
If you are not the intended recipient of this E-mail, you are hereby
notified that any dissemination, distribution, copying, or action taken
in relation to the contents of and attachments to this E-mail is
strictly prohibited and may be unlawful. If you have received this
E-mail in error, please notify the sender immediately and permanently
delete the original and any copy of this E-mail and any printout.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-ope

Re: [Openstack-operators] Packaging sample config versions

[Fischer, Matt]

I also find this issue really really annoying. Not only is the sample config a 
useful reference, but having it in a git repo allows me to see when new options 
were added or defaults were changed. Otherwise you have to rely on the docs 
which are generally wrong. When I was looking at some of the Juno transitions, 
I’ve already filed a few doc bugs about wrong defaults and config options. 
Assuming a project enforces that the sample config is up-to-date when commits 
are made, it’s by far the best source of this info. As for why? Well I’ve 
brought this up a few times before, and there was even a bug filed 
(https://bugs.launchpad.net/nova/+bug/1301519) but it was marked as “Won’t 
Fix”. I don’t really know the reason. This is one of my larger annoyances as an 
operator…

I do know that some projects, like nova and cinder, still ship the file.

Maybe someone can setup a cron job to do a pull, build the sample configs and 
post them to git hub every day.

From: "Kris G. Lindgren" mailto:klindg...@godaddy.com>>
Date: Monday, December 8, 2014 at 7:46 PM
To: 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
mailto:openstack-operators@lists.openstack.org>>
Subject: [Openstack-operators] Packaging sample config versions

Hello all,

Since somewhere around icehouse projects have started to stop shipping sample 
configuration files with their projects and instead create a 
README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox –egenconfig

The problem that I am running into is that tox –egenconfig now requires a newer 
version of tox than what is available for the build distro:
[root@localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

I know I can do a pip install blah and blindly hope that I get everything 
installed locally on my build system and that I don’t install something that 
conflicts with the system… but that seems like a less than desirable solution.  
So how are people who are doing packaging handling this?  Are you now building 
a venv per package for tox just to generate a sample configuration?  Shouldn't 
this be part of the python build/install steps per project?

It seems redhat/rdo is simply including a sample configuration that they 
generated somehow.

What are you other packagers doing?

Also, is it just me or does this seem wrong?  Most of the commits that made 
this change seem to indicate it was because the sample config file was separate 
from the project and that it was breaking gate when it wasn't kept up to date.  
Shouldn't this be something that each project generates?  This seemed to work 
for 8 previous releases – now its too difficult?  I don’t get the motivations 
behind this change.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


This E-mail and any of its attachments may contain Time Warner Cable 
proprietary information, which is privileged, confidential, or subject to 
copyright belonging to Time Warner Cable. This E-mail is intended solely for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient of this E-mail, you are hereby notified that any 
dissemination, distribution, copying, or action taken in relation to the 
contents of and attachments to this E-mail is strictly prohibited and may be 
unlawful. If you have received this E-mail in error, please notify the sender 
immediately and permanently delete the original and any copy of this E-mail and 
any printout.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Packaging sample config versions

[Kris G. Lindgren]

Hello all,

Since somewhere around icehouse projects have started to stop shipping sample 
configuration files with their projects and instead create a 
README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox -egenconfig

The problem that I am running into is that tox -egenconfig now requires a newer 
version of tox than what is available for the build distro:
[root@localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

I know I can do a pip install blah and blindly hope that I get everything 
installed locally on my build system and that I don't install something that 
conflicts with the system... but that seems like a less than desirable 
solution.  So how are people who are doing packaging handling this?  Are you 
now building a venv per package for tox just to generate a sample 
configuration?  Shouldn't this be part of the python build/install steps per 
project?

It seems redhat/rdo is simply including a sample configuration that they 
generated somehow.

What are you other packagers doing?

Also, is it just me or does this seem wrong?  Most of the commits that made 
this change seem to indicate it was because the sample config file was separate 
from the project and that it was breaking gate when it wasn't kept up to date.  
Shouldn't this be something that each project generates?  This seemed to work 
for 8 previous releases - now its too difficult?  I don't get the motivations 
behind this change.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators