Re: [openstack-dev] [tripleo] let's talk (development) environment deployment tooling and workflows

[Alex Schultz]

On Wed, Sep 21, 2016 at 9:00 AM, John Trowbridge  wrote:
>
>
>
> On 09/19/2016 01:21 PM, Steven Hardy wrote:
> > Hi Alex,
> >
> > Firstly, thanks for this detailed feedback - it's very helpful to have
> > someone with a fresh perspective look at the day-1 experience for TripleO,
> > and while some of what follows are "know issues", it's great to get some
> > perspective on them, as well as ideas re how we might improve things.
> >
> > On Thu, Sep 15, 2016 at 09:09:24AM -0600, Alex Schultz wrote:
> >> Hi all,
> >>
> >> I've recently started looking at the various methods for deploying and
> >> developing tripleo.  What I would like to bring up is the current
> >> combination of the tooling for managing the VM instances and the
> >> actual deployment method to launch the undercloud/overcloud
> >> installation.  While running through the various methods and reading
> >> up on the documentation, I'm concerned that they are not currently
> >> flexible enough for a developer (or operator for that matter) to be
> >> able to setup the various environment configurations for testing
> >> deployments and doing development.  Additionally I ran into issues
> >> just trying get them working at all so this probably doesn't help when
> >> trying to attract new contributors as well.  The focus of this email
> >> and of my experience seems to relate with workflow-simplification
> >> spec[0].  I would like to share my experiences with the various
> >> tooling available and raise some ideas.
> >>
> >> Example Situation:
> >>
> >> For example, I have a laptop with 16G of RAM and an SSD and I'd like
> >> to get started with tripleo.  How can I deploy tripleo?
> >
> > So, this is probably problem #1, because while I have managed to deploy a
> > minimal TripleO environment on a laptop with 16G of RAM, I think it's
> > pretty widely known that it's not really enough (certainly with our default
> > configuration, which has unfortunately grown over time as more and more
> > things got integrated).
> >
> > I see two options here:
> >
> > 1. Document the reality (which is really you need a physical machine with
> > at least 32G RAM unless you're prepared to deal with swapping).
> >
> > 2. Look at providing a "TripleO lite" install option, which disables some
> > services (both on the undercloud and default overcloud install).
> >
> > Either of these are defintely possible, but (2) seems like the best
> > long-term solution (although it probably means another CI job).
> >
> >> Tools:
> >>
> >> instack:
> >>
> >> I started with the tripleo docs[1] that reference using the instack
> >> tools for virtual environment creation while deploying tripleo.   The
> >> docs say you need at least 12G of RAM[2].  The docs lie (step 7[3]).
> >> So after basically shutting everything down and letting it deploy with
> >> all my RAM, the deployment fails because the undercloud runs out of
> >> RAM and OOM killer kills off heat.  This was not because I had reduced
> >> the amount of ram for the undercloud node or anything.  It was because
> >> by default, 6GB of RAM with no swap is configured for the undercloud
> >> (not sure if this is a bug?).  So I added a swap file to the
> >> undercloud and continued. My next adventure was having the overcloud
> >> deployment fail because lack of memory as puppet fails trying to spawn
> >> a process and gets denied.  The instack method does not configure swap
> >> for the VMs that are deployed and the deployment did not work with 5GB
> >> RAM for each node.  So for a full 16GB I was unable to follow the
> >> documentation and use instack to successfully deploy.  At this point I
> >> switched over to trying to use tripleo-quickstart.  Eventually I was
> >> able to figure out a configuration with instack to get it to deploy
> >> when I figured out how to enable swap for the overcloud deployment.
> >
> > Yeah, so this definitely exposes that we need to update the docs, and also
> > provide an easy install-time option to enable swap on all-the-things for
> > memory contrained environments.
> >
> >> tripleo-quickstart:
> >>
> >> The next thing I attempted to use was the tripleo-quickstart[4].
> >> Following the directions I attempted to deploy against my localhost.
> >> I turns out that doesn't work as expected since ansible likes to do
> >> magic when dealing with localhost[5].  Ultimately I was unable to get
> >> it working against my laptop locally because I ran into some libvirt
> >> issues.  But I was able to get it to work when I pointed it at a
> >> separate machine.  It should be noted that tripleo-quickstart creates
> >> an undercloud with swap which was nice because then it actually works,
> >> but is an inconsistent experience depending on which tool you used for
> >> your deployment.
> >
> > Yeah, so while a lot of folks have good luck with tripleo-quickstart, it
> > has the disadvantage of not currently being the tool used in upstream
> > TripleO CI (which folks have looked at fixing, but 

Re: [openstack-dev] [tripleo] let's talk (development) environment deployment tooling and workflows

[John Trowbridge]

On 09/19/2016 01:21 PM, Steven Hardy wrote:
> Hi Alex,
> 
> Firstly, thanks for this detailed feedback - it's very helpful to have
> someone with a fresh perspective look at the day-1 experience for TripleO,
> and while some of what follows are "know issues", it's great to get some
> perspective on them, as well as ideas re how we might improve things.
> 
> On Thu, Sep 15, 2016 at 09:09:24AM -0600, Alex Schultz wrote:
>> Hi all,
>>
>> I've recently started looking at the various methods for deploying and
>> developing tripleo.  What I would like to bring up is the current
>> combination of the tooling for managing the VM instances and the
>> actual deployment method to launch the undercloud/overcloud
>> installation.  While running through the various methods and reading
>> up on the documentation, I'm concerned that they are not currently
>> flexible enough for a developer (or operator for that matter) to be
>> able to setup the various environment configurations for testing
>> deployments and doing development.  Additionally I ran into issues
>> just trying get them working at all so this probably doesn't help when
>> trying to attract new contributors as well.  The focus of this email
>> and of my experience seems to relate with workflow-simplification
>> spec[0].  I would like to share my experiences with the various
>> tooling available and raise some ideas.
>>
>> Example Situation:
>>
>> For example, I have a laptop with 16G of RAM and an SSD and I'd like
>> to get started with tripleo.  How can I deploy tripleo?
> 
> So, this is probably problem #1, because while I have managed to deploy a
> minimal TripleO environment on a laptop with 16G of RAM, I think it's
> pretty widely known that it's not really enough (certainly with our default
> configuration, which has unfortunately grown over time as more and more
> things got integrated).
> 
> I see two options here:
> 
> 1. Document the reality (which is really you need a physical machine with
> at least 32G RAM unless you're prepared to deal with swapping).
> 
> 2. Look at providing a "TripleO lite" install option, which disables some
> services (both on the undercloud and default overcloud install).
> 
> Either of these are defintely possible, but (2) seems like the best
> long-term solution (although it probably means another CI job).
> 
>> Tools:
>>
>> instack:
>>
>> I started with the tripleo docs[1] that reference using the instack
>> tools for virtual environment creation while deploying tripleo.   The
>> docs say you need at least 12G of RAM[2].  The docs lie (step 7[3]).
>> So after basically shutting everything down and letting it deploy with
>> all my RAM, the deployment fails because the undercloud runs out of
>> RAM and OOM killer kills off heat.  This was not because I had reduced
>> the amount of ram for the undercloud node or anything.  It was because
>> by default, 6GB of RAM with no swap is configured for the undercloud
>> (not sure if this is a bug?).  So I added a swap file to the
>> undercloud and continued. My next adventure was having the overcloud
>> deployment fail because lack of memory as puppet fails trying to spawn
>> a process and gets denied.  The instack method does not configure swap
>> for the VMs that are deployed and the deployment did not work with 5GB
>> RAM for each node.  So for a full 16GB I was unable to follow the
>> documentation and use instack to successfully deploy.  At this point I
>> switched over to trying to use tripleo-quickstart.  Eventually I was
>> able to figure out a configuration with instack to get it to deploy
>> when I figured out how to enable swap for the overcloud deployment.
> 
> Yeah, so this definitely exposes that we need to update the docs, and also
> provide an easy install-time option to enable swap on all-the-things for
> memory contrained environments.
> 
>> tripleo-quickstart:
>>
>> The next thing I attempted to use was the tripleo-quickstart[4].
>> Following the directions I attempted to deploy against my localhost.
>> I turns out that doesn't work as expected since ansible likes to do
>> magic when dealing with localhost[5].  Ultimately I was unable to get
>> it working against my laptop locally because I ran into some libvirt
>> issues.  But I was able to get it to work when I pointed it at a
>> separate machine.  It should be noted that tripleo-quickstart creates
>> an undercloud with swap which was nice because then it actually works,
>> but is an inconsistent experience depending on which tool you used for
>> your deployment.
> 
> Yeah, so while a lot of folks have good luck with tripleo-quickstart, it
> has the disadvantage of not currently being the tool used in upstream
> TripleO CI (which folks have looked at fixing, but it's not yet happened).
> 
> The original plan was for tripleo-quickstart to completely replace the
> instack-virt-setup workflow:
> 
> https://blueprints.launchpad.net/tripleo/+spec/tripleo-quickstart
> 
> But for a variety of reasons, we never 

Re: [openstack-dev] [tripleo] let's talk (development) environment deployment tooling and workflows

[Alex Schultz]

On Mon, Sep 19, 2016 at 11:21 AM, Steven Hardy  wrote:

> Hi Alex,
>
> Firstly, thanks for this detailed feedback - it's very helpful to have
> someone with a fresh perspective look at the day-1 experience for TripleO,
> and while some of what follows are "know issues", it's great to get some
> perspective on them, as well as ideas re how we might improve things.
>
> On Thu, Sep 15, 2016 at 09:09:24AM -0600, Alex Schultz wrote:
> > Hi all,
> >
> > I've recently started looking at the various methods for deploying and
> > developing tripleo.  What I would like to bring up is the current
> > combination of the tooling for managing the VM instances and the
> > actual deployment method to launch the undercloud/overcloud
> > installation.  While running through the various methods and reading
> > up on the documentation, I'm concerned that they are not currently
> > flexible enough for a developer (or operator for that matter) to be
> > able to setup the various environment configurations for testing
> > deployments and doing development.  Additionally I ran into issues
> > just trying get them working at all so this probably doesn't help when
> > trying to attract new contributors as well.  The focus of this email
> > and of my experience seems to relate with workflow-simplification
> > spec[0].  I would like to share my experiences with the various
> > tooling available and raise some ideas.
> >
> > Example Situation:
> >
> > For example, I have a laptop with 16G of RAM and an SSD and I'd like
> > to get started with tripleo.  How can I deploy tripleo?
>
> So, this is probably problem #1, because while I have managed to deploy a
> minimal TripleO environment on a laptop with 16G of RAM, I think it's
> pretty widely known that it's not really enough (certainly with our default
> configuration, which has unfortunately grown over time as more and more
> things got integrated).
>
> I see two options here:
>
> 1. Document the reality (which is really you need a physical machine with
> at least 32G RAM unless you're prepared to deal with swapping).
>
> 2. Look at providing a "TripleO lite" install option, which disables some
> services (both on the undercloud and default overcloud install).
>
> Either of these are defintely possible, but (2) seems like the best
> long-term solution (although it probably means another CI job).
>
>
Yea I think 1 is an ok short term and 2 is the ideal solution.  I think we
really need to do a full evaluation of what services run and where to make
sure they are also properly tunable. I think it might help with some of
this memory utilization as well.


> > Tools:
> >
> > instack:
> >
> > I started with the tripleo docs[1] that reference using the instack
> > tools for virtual environment creation while deploying tripleo.   The
> > docs say you need at least 12G of RAM[2].  The docs lie (step 7[3]).
> > So after basically shutting everything down and letting it deploy with
> > all my RAM, the deployment fails because the undercloud runs out of
> > RAM and OOM killer kills off heat.  This was not because I had reduced
> > the amount of ram for the undercloud node or anything.  It was because
> > by default, 6GB of RAM with no swap is configured for the undercloud
> > (not sure if this is a bug?).  So I added a swap file to the
> > undercloud and continued. My next adventure was having the overcloud
> > deployment fail because lack of memory as puppet fails trying to spawn
> > a process and gets denied.  The instack method does not configure swap
> > for the VMs that are deployed and the deployment did not work with 5GB
> > RAM for each node.  So for a full 16GB I was unable to follow the
> > documentation and use instack to successfully deploy.  At this point I
> > switched over to trying to use tripleo-quickstart.  Eventually I was
> > able to figure out a configuration with instack to get it to deploy
> > when I figured out how to enable swap for the overcloud deployment.
>
> Yeah, so this definitely exposes that we need to update the docs, and also
> provide an easy install-time option to enable swap on all-the-things for
> memory contrained environments.
>

Even for non-memory constrained environments, having some would help
alleviate some of these random errors due to memory issues.


>
> > tripleo-quickstart:
> >
> > The next thing I attempted to use was the tripleo-quickstart[4].
> > Following the directions I attempted to deploy against my localhost.
> > I turns out that doesn't work as expected since ansible likes to do
> > magic when dealing with localhost[5].  Ultimately I was unable to get
> > it working against my laptop locally because I ran into some libvirt
> > issues.  But I was able to get it to work when I pointed it at a
> > separate machine.  It should be noted that tripleo-quickstart creates
> > an undercloud with swap which was nice because then it actually works,
> > but is an inconsistent experience depending on which tool you used for
> > 

Re: [openstack-dev] [tripleo] let's talk (development) environment deployment tooling and workflows

[Steven Hardy]

Hi Alex,

Firstly, thanks for this detailed feedback - it's very helpful to have
someone with a fresh perspective look at the day-1 experience for TripleO,
and while some of what follows are "know issues", it's great to get some
perspective on them, as well as ideas re how we might improve things.

On Thu, Sep 15, 2016 at 09:09:24AM -0600, Alex Schultz wrote:
> Hi all,
> 
> I've recently started looking at the various methods for deploying and
> developing tripleo.  What I would like to bring up is the current
> combination of the tooling for managing the VM instances and the
> actual deployment method to launch the undercloud/overcloud
> installation.  While running through the various methods and reading
> up on the documentation, I'm concerned that they are not currently
> flexible enough for a developer (or operator for that matter) to be
> able to setup the various environment configurations for testing
> deployments and doing development.  Additionally I ran into issues
> just trying get them working at all so this probably doesn't help when
> trying to attract new contributors as well.  The focus of this email
> and of my experience seems to relate with workflow-simplification
> spec[0].  I would like to share my experiences with the various
> tooling available and raise some ideas.
> 
> Example Situation:
> 
> For example, I have a laptop with 16G of RAM and an SSD and I'd like
> to get started with tripleo.  How can I deploy tripleo?

So, this is probably problem #1, because while I have managed to deploy a
minimal TripleO environment on a laptop with 16G of RAM, I think it's
pretty widely known that it's not really enough (certainly with our default
configuration, which has unfortunately grown over time as more and more
things got integrated).

I see two options here:

1. Document the reality (which is really you need a physical machine with
at least 32G RAM unless you're prepared to deal with swapping).

2. Look at providing a "TripleO lite" install option, which disables some
services (both on the undercloud and default overcloud install).

Either of these are defintely possible, but (2) seems like the best
long-term solution (although it probably means another CI job).

> Tools:
> 
> instack:
> 
> I started with the tripleo docs[1] that reference using the instack
> tools for virtual environment creation while deploying tripleo.   The
> docs say you need at least 12G of RAM[2].  The docs lie (step 7[3]).
> So after basically shutting everything down and letting it deploy with
> all my RAM, the deployment fails because the undercloud runs out of
> RAM and OOM killer kills off heat.  This was not because I had reduced
> the amount of ram for the undercloud node or anything.  It was because
> by default, 6GB of RAM with no swap is configured for the undercloud
> (not sure if this is a bug?).  So I added a swap file to the
> undercloud and continued. My next adventure was having the overcloud
> deployment fail because lack of memory as puppet fails trying to spawn
> a process and gets denied.  The instack method does not configure swap
> for the VMs that are deployed and the deployment did not work with 5GB
> RAM for each node.  So for a full 16GB I was unable to follow the
> documentation and use instack to successfully deploy.  At this point I
> switched over to trying to use tripleo-quickstart.  Eventually I was
> able to figure out a configuration with instack to get it to deploy
> when I figured out how to enable swap for the overcloud deployment.

Yeah, so this definitely exposes that we need to update the docs, and also
provide an easy install-time option to enable swap on all-the-things for
memory contrained environments.

> tripleo-quickstart:
> 
> The next thing I attempted to use was the tripleo-quickstart[4].
> Following the directions I attempted to deploy against my localhost.
> I turns out that doesn't work as expected since ansible likes to do
> magic when dealing with localhost[5].  Ultimately I was unable to get
> it working against my laptop locally because I ran into some libvirt
> issues.  But I was able to get it to work when I pointed it at a
> separate machine.  It should be noted that tripleo-quickstart creates
> an undercloud with swap which was nice because then it actually works,
> but is an inconsistent experience depending on which tool you used for
> your deployment.

Yeah, so while a lot of folks have good luck with tripleo-quickstart, it
has the disadvantage of not currently being the tool used in upstream
TripleO CI (which folks have looked at fixing, but it's not yet happened).

The original plan was for tripleo-quickstart to completely replace the
instack-virt-setup workflow:

https://blueprints.launchpad.net/tripleo/+spec/tripleo-quickstart

But for a variety of reasons, we never quite got to that - we may need a
summit discussion on the path forward here.

For me (as an upstream developer) it really boils down to the CI usage
issue - at all times I want to 

[openstack-dev] [tripleo] let's talk (development) environment deployment tooling and workflows

[Alex Schultz]

Hi all,

I've recently started looking at the various methods for deploying and
developing tripleo.  What I would like to bring up is the current
combination of the tooling for managing the VM instances and the
actual deployment method to launch the undercloud/overcloud
installation.  While running through the various methods and reading
up on the documentation, I'm concerned that they are not currently
flexible enough for a developer (or operator for that matter) to be
able to setup the various environment configurations for testing
deployments and doing development.  Additionally I ran into issues
just trying get them working at all so this probably doesn't help when
trying to attract new contributors as well.  The focus of this email
and of my experience seems to relate with workflow-simplification
spec[0].  I would like to share my experiences with the various
tooling available and raise some ideas.

Example Situation:

For example, I have a laptop with 16G of RAM and an SSD and I'd like
to get started with tripleo.  How can I deploy tripleo?

Tools:

instack:

I started with the tripleo docs[1] that reference using the instack
tools for virtual environment creation while deploying tripleo.   The
docs say you need at least 12G of RAM[2].  The docs lie (step 7[3]).
So after basically shutting everything down and letting it deploy with
all my RAM, the deployment fails because the undercloud runs out of
RAM and OOM killer kills off heat.  This was not because I had reduced
the amount of ram for the undercloud node or anything.  It was because
by default, 6GB of RAM with no swap is configured for the undercloud
(not sure if this is a bug?).  So I added a swap file to the
undercloud and continued. My next adventure was having the overcloud
deployment fail because lack of memory as puppet fails trying to spawn
a process and gets denied.  The instack method does not configure swap
for the VMs that are deployed and the deployment did not work with 5GB
RAM for each node.  So for a full 16GB I was unable to follow the
documentation and use instack to successfully deploy.  At this point I
switched over to trying to use tripleo-quickstart.  Eventually I was
able to figure out a configuration with instack to get it to deploy
when I figured out how to enable swap for the overcloud deployment.


tripleo-quickstart:

The next thing I attempted to use was the tripleo-quickstart[4].
Following the directions I attempted to deploy against my localhost.
I turns out that doesn't work as expected since ansible likes to do
magic when dealing with localhost[5].  Ultimately I was unable to get
it working against my laptop locally because I ran into some libvirt
issues.  But I was able to get it to work when I pointed it at a
separate machine.  It should be noted that tripleo-quickstart creates
an undercloud with swap which was nice because then it actually works,
but is an inconsistent experience depending on which tool you used for
your deployment.

Thoughts:

What these two methods showed me is that the deployment of tripleo is
not exactly a foolproof thing and that there are a lot of assumptions
that are being handled by the both of these tools.  My initial goal to
start this conversation around tooling and workflows was to bring the
idea of separation of the (virtual) environment configuration from the
actual deployment of tripleo as well as identifying places for
improvement as a way to speed up development and deployment testing.
I believe there are a few reasons why this can be beneficial.

The first reason is that as a developer, I would like to simplify the
development environment creation process and be able to draw the line
between environment and actual deployment tool.  By developing and
documenting a working development/deployment workflow, we can simplify
the onboarding experience as well as possibly accelerating the
existing development processes by reducing the time spent messing with
creating environments.  Does tripleo need to manage creation of VMs to
deploy on? The answer is probably no.  As the end user will want to
deploy tripleo on his or her gear, the focus for tripleo probably
should be on improving that process.  Now this doesn't mean that we
can't write stuff to do this, as it's important for development and
testing.  I'm not sure this is a core part of what should be
'tripleo'.

Another reason why I think this is important is as we talk about
creating different scenarios for CI[6] to improve testing, it would
also be useful for a developer or qa engineer to be able to test
different environmental configurations that would be more realistic of
actual deployment scenarios without having to hunt down multiple
machines or configure actual hardware networking.  For example,
creating environments using multiple networks, changing NICs,
providing different sized nodes based on roles, etc can all be done
virtually.  While tripleo-quickstart has some of these options, it is
mixed in with the tripleo