Re: [openstack-dev] [nova] Why don't we unbind ports or terminate volume connections on shelve offload?

[Andrew Laski]

On Thu, Apr 13, 2017, at 12:45 PM, Matt Riedemann wrote:
> This came up in the nova/cinder meeting today, but I can't for the life 
> of me think of why we don't unbind ports or terminate the connection 
> volumes when we shelve offload an instance from a compute host.
> 
> When you unshelve, if the instance was shelved offloaded, the conductor 
> asks the scheduler for a new set of hosts to build the instance on 
> (unshelve it). That could be a totally different host.
> 
> So am I just missing something super obvious? Or is this the most latent 
> bug ever?

It's at the very least a hack, and may be a bug depending on what
behaviour is being seen while the instance is offloaded or unshelved.

The reason that networks and volumes are left in place is because it
is/was the only way to prevent them from being used by another instance
and causing a subsequent unshelve to fail. During the unshelve operation
it is expected that they will then be shifted over to the new host the
instance lands on if it switches hosts.

This is similar to how resize is handled.  From an implementation point
of view you can think of shelve as being a really really long
resize/migration operation.

There very well may be issues with this approach.

> 
> -- 
> 
> Thanks,
> 
> Matt
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Stepping back

[Andrew Laski]

I should have sent this weeks ago but I'm a bad person who forgets
common courtesy. My employment situation has changed in a way that does
not afford me the necessary time to remain a regular contributor to
Nova, or the broader OpenStack community. So it is with regret that I
announce that I will no longer be actively involved in the project.

Fortunately, for those of you reading this, the Nova community is full
of wonderful and talented individuals who have picked up the work that
I'm not able to continue. Primarily this means parts of the cellsv2
effort, for which I am extremely grateful.

It has been a true pleasure working with you all these past few years
and I'm thankful to have had the opportunity. As I've told people many
times when they ask me what it's like to work on an open source project
like this: working on proprietary software exposes you to smart people
but you're limited to the small set of people within an organization,
working on a project like this exposed me to smart people from many
companies and many parts of the world. I have learned a lot working with
you all. Thanks.

I will continue to lurk in #openstack-nova, and try to stay minimally
involved as time allows, so feel free to ping me there.

-Laski

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova]Question about unit tests for shelve/unshleve

[Andrew Laski]

On Sun, Oct 16, 2016, at 07:11 AM, Rikimaru Honjo wrote:
> Hi all,
> 
> I have a question about unit tests of nova.
> (I found this question when I fixed a bug about shelve.[1])
> 
> "nova.tests.unit.compute.test_shelve.ShelveComputeAPITestCase" has
> test cases for "nova.compute.api.API.shelve()/unshelve()".
> But "nova.tests.unit.compute.test_compute_api._ComputeAPIUnitTestMixIn"
> also has test cases for same methods.
> 
> Is their purpose duplicated?
> And, can I organize them if their purpose is duplicated?

I just looked at them briefly and they're not exactly duplicates. It
appears that test_shelve.py has more functional tests and
test_compute_api.py is more unit tests. But it would be nice to have
them all in the same place.

> 
> FYI, I think that we should consolidate them into
> "nova.tests.unit.compute.test_compute_api._ComputeAPIUnitTestMixIn".
> Because it is inherited to some test classes.
> 

Personally I would prefer consolidating them into test_shelve.py because
_ComputeAPIUnitTestMixin is a giant class and it can be hard to discover
where something is tested. I like having the features tested in a
dedicated test file.

> 
> [1]: https://bugs.launchpad.net/nova/+bug/1588657
> 
> Best regards,
> -- 
> Rikimaru Honjo
> E-mail:honjo.rikim...@po.ntts.co.jp
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Why do we have project specific hacking rules?

[Andrew Laski]

On Sun, Oct 2, 2016, at 11:02 AM, Matt Riedemann wrote:
> On 10/2/2016 5:47 AM, Amrith Kumar wrote:
> > It was my understanding that hacking rules were like the 'Ten 
> > Commandments', the 'Four Opens'; things that were universally true across 
> > all projects and an attempt to bring standardization to all OpenStack code.
> >
> > How come we then have extensive project specific hacking rules? Why not 
> > make these "nova-specific" rules OpenStack wide?
> >
> > I looked at the checks.py file that Matt linked to below, and I can't see 
> > anything really "nova-specific"; i.e. all of them would translate just fine 
> > to Trove. Is there some reason they can't become OpenStack wide rules?

For some of them there are good reasons not to make them OpenStack wide
checks.

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n155
no db usage in virt driver.

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n168
no db session in public db api

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n201
virt drivers aren't importing modules from other virt drivers

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n220
virt drivers aren't using config options from other virt drivers

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n406
api microversion decorator is first decorator when used

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n616
check that nova specific utility method is used rather than
greenthread|eventlet.spawn|spawn_n

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n643
config options are registered in a central place

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n667
policy options are registered in a central place

http://git.openstack.org/cgit/openstack/nova/tree/nova/hacking/checks.py?h=stable/newton#n681
a nova specific context.can() method must be used for policy enforcement
rather than policy.enforce|authorize

All of these checks are either very specific to Nova or are
philosophical choices that have been made in Nova but don't need to be
shared by other projects.

And as Matt points out below some of the checks not listed above depend
on specific paths being used in the check which makes it more difficult
to share those checks.

A further hindrance to moving these checks to be OpenStack wide is that
each check violation is a failure. So in order to roll a new check out
across projects it requires a lot of churn in each project which
violates the checks. I would prefer to leave it up to each project to
make the decision to incorporate a new check and take the review load.
Ultimately what I think sounds good would be a central listing of checks
that are in use across projects and then leave it up to each project to
replicate those that they like.


> >
> > -amrith
> >
> >> -Original Message-
> >> From: Ihar Hrachyshka [mailto:ihrac...@redhat.com]
> >> Sent: Sunday, October 02, 2016 5:25 AM
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> 
> >> Subject: Re: [openstack-dev] [all][i18n] do we need translation mark for
> >> strings in tests?
> >>
> >> Matt Riedemann  wrote:
> >>
> >>> On 10/1/2016 5:49 PM, Matt Riedemann wrote:
>  No you shouldn't need to mark strings for translation in test code. I
>  believe we have a hacking rule for marking LOG.info/warning/error
>  messages for translation but it should skip test directories.
> >>>
> >>> Ah I guess the hacking rule is nova-specific:
> >>>
> >>>
> >> https://github.com/openstack/nova/blob/2851ceaed3010c19f42e308be637b952eda
> >> b092a/nova/hacking/checks.py#L342
> >>
> >> We have a similar one in neutron; but note that it does not explicitly
> >> FAIL
> >> on translation markers in test code; it instead fails on no markers in
> >> production code, which is different different.
> >>
> >> Ihar
> >>
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> Well for one thing the specific one I pointed out has hard-coded nova 
> paths in it which might not work for other projects.
> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> OpenStack Development Mailing List (not fo

Re: [openstack-dev] os-loganalyze, project log parsing, or ...

[Andrew Laski]

On Wed, Sep 28, 2016, at 07:15 AM, Sean Dague wrote:
> On 09/27/2016 06:19 PM, Andrew Laski wrote:
> 
> > 
> > I totally understand where you're coming from. I just see it
> > differently.
> > 
> > The way it was done did not affect any other projects, and the plumbing
> > used is something that could have easily be left in. And as luck would
> > have it there's another patch up to use that same plumbing so it's
> > probably going in anyways.
> > 
> > I completely agree that before expanding this to be in any way cross
> > project it's worth figuring out a better way to do it. But at this point
> > I don't feel comfortable enough with a long term vision to tackle that.
> > I would much prefer to experiment in a small way before moving forward.
> 
> Ok, so what if we go forward with your existing patch, but agree only to
> replace the post_test_hook on Nova specific jobs, like the placement
> one. (The change proposed here -
> https://review.openstack.org/#/c/376537/6/jenkins/jobs/nova.yaml). That
> should give enough data and experimentation, and it show up for every
> nova patch.
> 
> Before migrating this kind of approach to integrated gate jobs, we
> revisit a way to make sure that multiple projects can plumb content like
> this. Be it, expanding the test hook model, or a common post hook
> project that could have core members from multiple teams, or some
> generic yaml thing (though I agree, that's a lot harder to wrap my head
> around).

Deal.

> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] os-loganalyze, project log parsing, or ...

[Andrew Laski]

On Tue, Sep 27, 2016, at 05:54 PM, Sean Dague wrote:
> On 09/27/2016 05:45 PM, Andrew Laski wrote:
> > 
> > 
> > On Tue, Sep 27, 2016, at 04:43 PM, Matthew Treinish wrote:
> > 
> >>>>
> >>>> I definitely can see the value in having machine parsable log stats in
> >>>> our
> >>>> artifacts, but I'm not sure where project specific pieces would come
> >>>> from. But,
> >>>> given that hypothetical I would say as long as you made those pieces
> >>>> configurable (like a yaml syntax to search for patterns by log file or
> >>>> something) and kept a generic framework/tooling for parsing the log stats
> >>>> I
> >>>> think it's still a good fit for a QA or Infra project. Especially if you
> >>>> think
> >>>> whatever pattern you're planning to use is something other projects would
> >>>> want
> >>>> to reuse.
> >>>
> >>> My concern here is that I want to go beyond simple pattern matching. I
> >>> want to be able to maintain state while parsing to associate log lines
> >>> with events that came before. The project specific bits I envision are
> >>> the logic to handle that, but I don't think yaml is expressive enough
> >>> for it. I came up with a quick example at
> >>> http://paste.openstack.org/show/583160/ . That's Nova specific and
> >>> beyond my capability to express in yaml or elastic-recheck.
> >>
> >> That's pretty simple to do with yaml too. Especially since it's tied to a
> >> single
> >> regex. For example, something roughly like:
> >>
> >> http://paste.openstack.org/show/583165/
> >>
> >> would use yaml to make it a generic framework. 
> > 
> > Okay, I see that it can be done for this example. But I look at that
> > yaml and all I see is that after a couple of definitions:
> > 
> > count_api_stuffs:
> >   - name: Instance Boots
> > regex = '(req-\S+).*Starting instance'
> > log_file = n-cpu.txt
> >   - name: Volume something
> > regex = '(req-\S+).*Random Cinder Event to Count'
> > log_file = c-api.txt
> > time_api_event:
> >-name: Foo
> >regex = 
> > another_thing:
> > yet_another:
> > 
> > there's no context to what each of these does and the logic is competely
> > decoupled. I guess I'm just personally biased to wanting to see it done
> > in code to help my understanding.
> > 
> > If yaml is what makes this acceptable then maybe I'll rework to do it
> > that way eventually. What I was hoping to do was start small, prove the
> > concept in a project while maintaining flexibility, and then look at
> > expanding for general use. Essentially the oslo model. I'm a little
> > surprised that there's so much resistance to that.
> 
> I think the crux of the resistance isn't to starting small. It's to
> doing all the gate plumbing for a thing in one tree, then do the replumb
> later. And it mostly comes from having to connect all those pieces then
> stage them back out in a non breaking way later.
> 
> If this was stuff where the execution control was also embedded in
> tox.ini, so changes in calling were easily addressed there, then it
> could be run in a pretty quiet controlled experiment.
> 
> But it pretty much starts multi project from day one, because it needs
> specific things (and guaruntees) from devstack and devstack-gate from
> the get go. So thinking about this as a multi project effort from the
> get go seems worth while.

I totally understand where you're coming from. I just see it
differently.

The way it was done did not affect any other projects, and the plumbing
used is something that could have easily be left in. And as luck would
have it there's another patch up to use that same plumbing so it's
probably going in anyways.

I completely agree that before expanding this to be in any way cross
project it's worth figuring out a better way to do it. But at this point
I don't feel comfortable enough with a long term vision to tackle that.
I would much prefer to experiment in a small way before moving forward.

> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] os-loganalyze, project log parsing, or ...

[Andrew Laski]

On Tue, Sep 27, 2016, at 04:43 PM, Matthew Treinish wrote:

> > > 
> > > I definitely can see the value in having machine parsable log stats in
> > > our
> > > artifacts, but I'm not sure where project specific pieces would come
> > > from. But,
> > > given that hypothetical I would say as long as you made those pieces
> > > configurable (like a yaml syntax to search for patterns by log file or
> > > something) and kept a generic framework/tooling for parsing the log stats
> > > I
> > > think it's still a good fit for a QA or Infra project. Especially if you
> > > think
> > > whatever pattern you're planning to use is something other projects would
> > > want
> > > to reuse.
> > 
> > My concern here is that I want to go beyond simple pattern matching. I
> > want to be able to maintain state while parsing to associate log lines
> > with events that came before. The project specific bits I envision are
> > the logic to handle that, but I don't think yaml is expressive enough
> > for it. I came up with a quick example at
> > http://paste.openstack.org/show/583160/ . That's Nova specific and
> > beyond my capability to express in yaml or elastic-recheck.
> 
> That's pretty simple to do with yaml too. Especially since it's tied to a
> single
> regex. For example, something roughly like:
> 
> http://paste.openstack.org/show/583165/
> 
> would use yaml to make it a generic framework. 

Okay, I see that it can be done for this example. But I look at that
yaml and all I see is that after a couple of definitions:

count_api_stuffs:
  - name: Instance Boots
regex = '(req-\S+).*Starting instance'
log_file = n-cpu.txt
  - name: Volume something
regex = '(req-\S+).*Random Cinder Event to Count'
log_file = c-api.txt
time_api_event:
   -name: Foo
   regex = 
another_thing:
yet_another:

there's no context to what each of these does and the logic is competely
decoupled. I guess I'm just personally biased to wanting to see it done
in code to help my understanding.

If yaml is what makes this acceptable then maybe I'll rework to do it
that way eventually. What I was hoping to do was start small, prove the
concept in a project while maintaining flexibility, and then look at
expanding for general use. Essentially the oslo model. I'm a little
surprised that there's so much resistance to that.


> 
> That's what I was trying to get across before, what might seem project
> specific
> is just a search pattern or combination of them for a log file. These
> patterns
> are generic enough we can extract out the project specific pieces and it
> could
> be made configurable to enable reuse between projects. Especially since
> most of
> the logging format should be standardized. (this would further expose
> where
> things differ too)
> 
> So yeah I still think this makes sense as a QA or Infra project. Although
> I'm
> wondering if there is a better way to harvest this info from a run.
> 
> As an aside it does feel like we should have a better mechanism for most
> of
> this. Like for counting booted instances I normally use the qemu log
> files, for
> example:
> 
> http://logs.openstack.org/49/373249/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b0159bf/logs/libvirt/qemu/
> 
> which shows how many instances we booted during that run. But I guess it
> really depends on what we're looking for. So there probably isn't an
> easier
> answer.
> 
> -Matt Treinish
> 
> > > > > 
> > > > > I would caution against doing it as a one off in a project repo 
> > > > > doesn't
> > > > > seem
> > > > > like the best path forward for something like this. We actually tried 
> > > > > to
> > > > > do
> > > > > something similar to that in the past inside the tempest repo:
> > > > > 
> > > > > http://git.openstack.org/cgit/openstack/tempest/tree/tools/check_logs.py
> > > > > 
> > > > > and
> > > > > 
> > > > > http://git.openstack.org/cgit/openstack/tempest/tree/tools/find_stack_traces.py
> > > > > 
> > > > > all it did was cause confusion because no one knew where the output 
> > > > > was
> > > > > coming
> > > > > from. Although, the output from those tools was also misleading, which
> > > > > was
> > > > > likely a bigger problm. So this probably won't be an issue if you add 
> > > > > a
> > > > > json
> > > > > output to the jobs.
> > > > > 
> > > > > I also wonder if the JSONFormatter from oslo.log:
> > > > > 
> > > > > http://docs.openstack.org/developer/oslo.log/api/formatters.html#oslo_log.formatters.JSONFormatter
> > > > > 
> > > > > would be useful here. We can proabbly turn that on if it makes things
> > > > > easier.
> > > > > 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1

Re: [openstack-dev] os-loganalyze, project log parsing, or ...

[Andrew Laski]

On Tue, Sep 27, 2016, at 02:40 PM, Matthew Treinish wrote:
> On Tue, Sep 27, 2016 at 01:03:35PM -0400, Andrew Laski wrote:
> > 
> > 
> > On Tue, Sep 27, 2016, at 12:39 PM, Matthew Treinish wrote:
> > > On Tue, Sep 27, 2016 at 11:36:07AM -0400, Andrew Laski wrote:
> > > > Hello all,
> > > > 
> > > > Recently I noticed that people would look at logs from a Zuul née
> > > > Jenkins CI run and comment something like "there seem to be more
> > > > warnings in here than usual." And so I thought it might be nice to
> > > > quantify that sort of thing so we didn't have to rely on gut feelings.
> > > > 
> > > > So I threw together https://review.openstack.org/#/c/376531 which is a
> > > > script that lives in the Nova tree, gets called from a devstack-gate
> > > > post_test_hook, and outputs an n-stats.json file which can be seen at
> > > > http://logs.openstack.org/06/375106/8/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/e103612/logs/n-stats.json.
> > > > This provides just a simple way to compare two runs and spot large
> > > > changes between them. Perhaps later things could get fancy and these
> > > > stats could be tracked over time. I am also interested in adding stats
> > > > for things that are a bit project specific like how long (max, min, med)
> > > > it took to boot an instance, or what's probably better to track is how
> > > > many operations that took for some definition of an operation.
> > > > 
> > > > I received some initial feedback that this might be a better fit in the
> > > > os-loganalyze project so I took a look over there. So I cloned the
> > > > project to take a look and quickly noticed
> > > > http://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/README.rst#n13.
> > > > That makes me think it would not be a good fit there because what I'm
> > > > looking to do relies on parsing the full file, or potentially multiple
> > > > files, in order to get useful data.
> > > > 
> > > > So my questions: does this seem like a good fit for os-loganalyze? If
> > > > not is there another infra/QA project that this would be a good fit for?
> > > > Or would people be okay with a lone project like Nova implementing this
> > > > in tree for their own use?
> > > > 
> > > 
> > > I think having this in os-loganalyze makes sense since we use that for
> > > visualizing the logs already. It also means we get it for free on all the
> > > log
> > > files. But, if it's not a good fit for a technical reason then I think
> > > creating
> > > another small tool under QA or infra would be a good path forward. Since
> > > there
> > > really isn't anything nova specific in that.
> > 
> > There's nothing Nova specific atm because I went for low hanging fruit.
> > But if the plan is to have Nova specific, Cinder specific, Glance
> > specific, etc... things in there do people still feel that a QA/infra
> > tool is the right path forward. That's my only hesitation here.
> 
> Well I think that raises more questions, what do you envision the nova
> specific
> bits would be. The only thing I could see would be something that looks
> for
> specific log messages or patterns in the logs. Which feels like exactly
> what
> elastic-recheck does?

I'm thinking beyond single line things. An example could be a parser
that can calculate the timing between the first log message seen for a
request-id and the last, or could count the number of log lines
associated with each instance boot perhaps even broken down by log
level. Things that require both an understanding of how to correlate
groups of log lines with specific events(instance boot), and being able
to calculate stats for groups of log lines(debug log line count by
request-id).

I have only a rudimentary familiarity with elastic-recheck but my
understanding is that doing anything that looks at multiple lines like
that is either complex or not really possible.


> 
> I definitely can see the value in having machine parsable log stats in
> our
> artifacts, but I'm not sure where project specific pieces would come
> from. But,
> given that hypothetical I would say as long as you made those pieces
> configurable (like a yaml syntax to search for patterns by log file or
> something) and kept a generic framework/tooling for parsing the log stats
> I
> think it's still a good fit for a QA or Infra project. Especially if you
> think
> what

Re: [openstack-dev] os-loganalyze, project log parsing, or ...

[Andrew Laski]

On Tue, Sep 27, 2016, at 12:39 PM, Matthew Treinish wrote:
> On Tue, Sep 27, 2016 at 11:36:07AM -0400, Andrew Laski wrote:
> > Hello all,
> > 
> > Recently I noticed that people would look at logs from a Zuul née
> > Jenkins CI run and comment something like "there seem to be more
> > warnings in here than usual." And so I thought it might be nice to
> > quantify that sort of thing so we didn't have to rely on gut feelings.
> > 
> > So I threw together https://review.openstack.org/#/c/376531 which is a
> > script that lives in the Nova tree, gets called from a devstack-gate
> > post_test_hook, and outputs an n-stats.json file which can be seen at
> > http://logs.openstack.org/06/375106/8/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/e103612/logs/n-stats.json.
> > This provides just a simple way to compare two runs and spot large
> > changes between them. Perhaps later things could get fancy and these
> > stats could be tracked over time. I am also interested in adding stats
> > for things that are a bit project specific like how long (max, min, med)
> > it took to boot an instance, or what's probably better to track is how
> > many operations that took for some definition of an operation.
> > 
> > I received some initial feedback that this might be a better fit in the
> > os-loganalyze project so I took a look over there. So I cloned the
> > project to take a look and quickly noticed
> > http://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/README.rst#n13.
> > That makes me think it would not be a good fit there because what I'm
> > looking to do relies on parsing the full file, or potentially multiple
> > files, in order to get useful data.
> > 
> > So my questions: does this seem like a good fit for os-loganalyze? If
> > not is there another infra/QA project that this would be a good fit for?
> > Or would people be okay with a lone project like Nova implementing this
> > in tree for their own use?
> > 
> 
> I think having this in os-loganalyze makes sense since we use that for
> visualizing the logs already. It also means we get it for free on all the
> log
> files. But, if it's not a good fit for a technical reason then I think
> creating
> another small tool under QA or infra would be a good path forward. Since
> there
> really isn't anything nova specific in that.

There's nothing Nova specific atm because I went for low hanging fruit.
But if the plan is to have Nova specific, Cinder specific, Glance
specific, etc... things in there do people still feel that a QA/infra
tool is the right path forward. That's my only hesitation here.

> 
> I would caution against doing it as a one off in a project repo doesn't
> seem
> like the best path forward for something like this. We actually tried to
> do
> something similar to that in the past inside the tempest repo:
> 
> http://git.openstack.org/cgit/openstack/tempest/tree/tools/check_logs.py
> 
> and
> 
> http://git.openstack.org/cgit/openstack/tempest/tree/tools/find_stack_traces.py
> 
> all it did was cause confusion because no one knew where the output was
> coming
> from. Although, the output from those tools was also misleading, which
> was
> likely a bigger problm. So this probably won't be an issue if you add a
> json
> output to the jobs.
> 
> I also wonder if the JSONFormatter from oslo.log:
> 
> http://docs.openstack.org/developer/oslo.log/api/formatters.html#oslo_log.formatters.JSONFormatter
> 
> would be useful here. We can proabbly turn that on if it makes things
> easier.
> 
> -Matt Treinish
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] os-loganalyze, project log parsing, or ...

[Andrew Laski]

On Tue, Sep 27, 2016, at 11:59 AM, Ian Cordasco wrote:
> On Tue, Sep 27, 2016 at 10:36 AM, Andrew Laski  wrote:
> > Hello all,
> >
> > Recently I noticed that people would look at logs from a Zuul née
> > Jenkins CI run and comment something like "there seem to be more
> > warnings in here than usual." And so I thought it might be nice to
> > quantify that sort of thing so we didn't have to rely on gut feelings.
> >
> > So I threw together https://review.openstack.org/#/c/376531 which is a
> > script that lives in the Nova tree, gets called from a devstack-gate
> > post_test_hook, and outputs an n-stats.json file which can be seen at
> > http://logs.openstack.org/06/375106/8/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/e103612/logs/n-stats.json.
> > This provides just a simple way to compare two runs and spot large
> > changes between them. Perhaps later things could get fancy and these
> > stats could be tracked over time. I am also interested in adding stats
> > for things that are a bit project specific like how long (max, min, med)
> > it took to boot an instance, or what's probably better to track is how
> > many operations that took for some definition of an operation.
> >
> > I received some initial feedback that this might be a better fit in the
> > os-loganalyze project so I took a look over there. So I cloned the
> > project to take a look and quickly noticed
> > http://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/README.rst#n13.
> > That makes me think it would not be a good fit there because what I'm
> > looking to do relies on parsing the full file, or potentially multiple
> > files, in order to get useful data.
> >
> > So my questions: does this seem like a good fit for os-loganalyze? If
> > not is there another infra/QA project that this would be a good fit for?
> > Or would people be okay with a lone project like Nova implementing this
> > in tree for their own use?
> 
> My first instinct was that this could be tracked along with the Health
> project but after a little thought log warnings aren't necessarily an
> indication of the health of the project.
> 
> It might make sense to start this out in Nova to see how useful it
> ends up being, but I can see Glance being interested in this at some
> point if it ends up being useful to Nova.

Yeah, starting it in Nova to gauge it's usefulness was my goal. I could
see growing a common framework out of this that projects could pull in,
like an oslo project though I'm not sure if oslo would be the right fit,
and then having each project provide specific parsers for their data.


> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] os-loganalyze, project log parsing, or ...

[Andrew Laski]

Hello all,

Recently I noticed that people would look at logs from a Zuul née
Jenkins CI run and comment something like "there seem to be more
warnings in here than usual." And so I thought it might be nice to
quantify that sort of thing so we didn't have to rely on gut feelings.

So I threw together https://review.openstack.org/#/c/376531 which is a
script that lives in the Nova tree, gets called from a devstack-gate
post_test_hook, and outputs an n-stats.json file which can be seen at
http://logs.openstack.org/06/375106/8/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/e103612/logs/n-stats.json.
This provides just a simple way to compare two runs and spot large
changes between them. Perhaps later things could get fancy and these
stats could be tracked over time. I am also interested in adding stats
for things that are a bit project specific like how long (max, min, med)
it took to boot an instance, or what's probably better to track is how
many operations that took for some definition of an operation.

I received some initial feedback that this might be a better fit in the
os-loganalyze project so I took a look over there. So I cloned the
project to take a look and quickly noticed
http://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/README.rst#n13.
That makes me think it would not be a good fit there because what I'm
looking to do relies on parsing the full file, or potentially multiple
files, in order to get useful data.

So my questions: does this seem like a good fit for os-loganalyze? If
not is there another infra/QA project that this would be a good fit for?
Or would people be okay with a lone project like Nova implementing this
in tree for their own use?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Centralizing policy (was: Anyone interested in writing a policy generator sphinx extension?)

[Andrew Laski]

On Thu, Sep 22, 2016, at 01:34 PM, Alexander Makarov wrote:
> Andrew,
> 
> the idea is to shift existing RBAC implementation:
> currently policy is enforced in the service (Nova, for instance)
> against the result of token validation, which is, in general, an access 
> check;
> I'm thinking about performing policy enforcement along with access check
> in a single operation and only if necessary -
> not every operation is protected and requires token validation,
> though now keystone middleware validates a token on every request.

There's a lot of information necessary to determine which policy checks
to run, and even more information necessary in order to run the policy
check.

For example: When booting an instance there are policies that are run
based on the data included in the POST body of the request. Should
Keystone be passed that POST body and contain logic to determine which
checks to run? Or would Nova pass a list of checks to Keystone and say
"do these policies pass?" I don't see an advantage to either of those.

Also, a lot of policy checks rely on data that must be looked up from
the Nova db. Checking policy for deletion of an instance requires
looking up that instance to check the context user_id/project_id against
the instance user_id/project_id. So Nova would need to do that lookup
and pass that information to Keystone to use for a policy check. Which
also means that Keystone can't do the check at the time of token
validation because at that point it doesn't have enough data for the
policy check.


> 
> AFAIK Nova is using some custom logic to change local policies at
> run-time,
> so I assume there may be a benefit in dynamic centralized storage 
> managed via API,
> so that Horizon can even provide a UI for that.

There is no special mechanism in Nova for changing policies at run time.
Policy defaults are now embedded in the Nova code so it's possible to
run without a policy file, and it's possible to override those policies
with a policy file. But once Nova is running the only way to update
policies is to update the policy file. That works the same in every
project because the reload mechanism is handled automatically in
oslo.policy.

Furthermore, what's more interesting to Horizon I think is the question
of what is a user allowed to do with the API. And that question requires
more information than just policy to answer. An example is that it's
possible to resize an instance to a smaller size if the instance is on a
Xen hypervisor. So in order for Horizon to allow/disallow that option
for a user it needs information that Nova is in possession of, and
Keystone never will be. That's why the Nova community has been
discussing the idea of exposing "capabilities" in the Nova API which be
let a user know what they can do based on policy settings and many other
factors.

> 
> There are many questions in the matter, and my main is:
> if we do RBAC in OpenStack the best way?

I'm sure we don't do RBAC in the best way. However, just like quota
enforcement, I think it's something that is going to need to be handled
in each project individually. But unlike quota administration I don't
see the benefit of centralizing policy administration. It's fairly
static data, and on the occasions that it needs to change it can be
updated with a configuration management solution and the live policy
reloading that oslo.policy has.

> 
> 
> On 21.09.2016 20:16, Andrew Laski wrote:
> >
> > On Wed, Sep 21, 2016, at 12:02 PM, Joshua Harlow wrote:
> >> Andrew Laski wrote:
> >>> However, I have asked twice now on the review what the benefit of doing
> >>> this is and haven't received a response so I'll ask here. The proposal
> >>> would add additional latency to nearly every API operation in a service
> >>> and in return what do they get? Now that it's possible to register sane
> >>> policy defaults within a project most operators do not even need to
> >>> think about policy for projects that do that. And any policy changes
> >>> that are necessary are easily handled by a config management system.
> >>>
> >>> I would expect to see a pretty significant benefit in exchange for
> >>> moving policy control out of Nova, and so far it's not clear to me what
> >>> that would be.
> >> One way to do this is to setup something like etc.d or zookeeper and
> >> have policy files be placed into certain 'keys' in there by keystone,
> >> then consuming projects would 'watch' those keys for being changed (and
> >> get notified when they are changed); the project would then reload its
> >> policy when the other service (keystone) write a new

Re: [openstack-dev] [oslo][nova] Anyone interested in writing a policy generator sphinx extension?

[Andrew Laski]

On Wed, Sep 21, 2016, at 04:18 PM, Joshua Harlow wrote:
> Andrew Laski wrote:
> >
> > On Wed, Sep 21, 2016, at 03:18 PM, Joshua Harlow wrote:
> >> Andrew Laski wrote:
> >>> On Wed, Sep 21, 2016, at 12:02 PM, Joshua Harlow wrote:
> >>>> Andrew Laski wrote:
> >>>>> However, I have asked twice now on the review what the benefit of doing
> >>>>> this is and haven't received a response so I'll ask here. The proposal
> >>>>> would add additional latency to nearly every API operation in a service
> >>>>> and in return what do they get? Now that it's possible to register sane
> >>>>> policy defaults within a project most operators do not even need to
> >>>>> think about policy for projects that do that. And any policy changes
> >>>>> that are necessary are easily handled by a config management system.
> >>>>>
> >>>>> I would expect to see a pretty significant benefit in exchange for
> >>>>> moving policy control out of Nova, and so far it's not clear to me what
> >>>>> that would be.
> >>>> One way to do this is to setup something like etc.d or zookeeper and
> >>>> have policy files be placed into certain 'keys' in there by keystone,
> >>>> then consuming projects would 'watch' those keys for being changed (and
> >>>> get notified when they are changed); the project would then reload its
> >>>> policy when the other service (keystone) write a new key/policy.
> >>>>
> >>>> https://coreos.com/etcd/docs/latest/api.html#waiting-for-a-change
> >>>>
> >>>> or
> >>>> https://zookeeper.apache.org/doc/r3.4.5/zookeeperProgrammers.html#ch_zkWatches
> >>>>
> >>>> or (pretty sure consul has something similar),
> >>>>
> >>>> This is pretty standard stuff folks :-/ and it's how afaik things like
> >>>> https://github.com/skynetservices/skydns work (and more), and it would
> >>>> avoid that 'additional latency' (unless the other service is adjusting
> >>>> the policy key every millisecond, which seems sorta unreasonable).
> >>> Sure. Or have Keystone be a frontend for ansible/puppet/chef/ What's
> >>> not clear to me in any of this is what's the benefit to having Keystone
> >>> as a fronted to policy configuration/changes, or be involved in any real
> >>> way with authorization decisions? What issue is being solved by getting
> >>> Keystone involved?
> >>>
> >> I don't understand the puppet/chef connection, can u clarify.
> >>
> >> If I'm interpreting it right, I would assume it's the same reason that
> >> something like 'skydns' exists over etcd; to provide a useful API that
> >> focuses on the dns particulars that etcd will of course not have any
> >> idea about. So I guess the keystone API could(?)/would(?) then focus on
> >> policy particulars as its value-add.
> >>
> >> Maybe now I understand what u mean by puppet/chef, in that you are
> >> asking why isn't skydns (for example) just letting/invoking
> >> puppet/chef/ansible to distribute/send-out dns (dnsmasq) files? Is that
> >> your equivalent question?
> >
> > I'm focused on Nova/Keystone/OpenStack here, I'm sure skydns has good
> > reasons for their technical choices and I'm in no place to question
> > them.
> >
> > I'm trying to understand the value-add that Keystone could provide here.
> > Policy configuration is fairly static so I'm not understanding the
> > desire to put an API on top of it. But perhaps I'm missing the use case
> > here which is why I've been asking.
> >
> > My ansible/puppet/chef comparison was just that those are ways to
> > distribute static files and would work just as well as something built
> > on top of etcd/zookeeper. I'm not really concerned about how it's
> > implemented though. I'm just trying to understand if the desire is to
> > have Keystone handle this so that deployers don't need to work with
> > their configuration management system to configure policy files, or is
> > there something more here?
> >
> >
> 
> Gotcha, thanks for explaining.
> 
> I'll let others comment, but my semi-useful/semi-baked thoughts around 
> this are as a user I would want to:
> 
> #1 Can I q

Re: [openstack-dev] [oslo][nova] Anyone interested in writing a policy generator sphinx extension?

[Andrew Laski]

On Wed, Sep 21, 2016, at 03:18 PM, Joshua Harlow wrote:
> Andrew Laski wrote:
> >
> > On Wed, Sep 21, 2016, at 12:02 PM, Joshua Harlow wrote:
> >> Andrew Laski wrote:
> >>> However, I have asked twice now on the review what the benefit of doing
> >>> this is and haven't received a response so I'll ask here. The proposal
> >>> would add additional latency to nearly every API operation in a service
> >>> and in return what do they get? Now that it's possible to register sane
> >>> policy defaults within a project most operators do not even need to
> >>> think about policy for projects that do that. And any policy changes
> >>> that are necessary are easily handled by a config management system.
> >>>
> >>> I would expect to see a pretty significant benefit in exchange for
> >>> moving policy control out of Nova, and so far it's not clear to me what
> >>> that would be.
> >> One way to do this is to setup something like etc.d or zookeeper and
> >> have policy files be placed into certain 'keys' in there by keystone,
> >> then consuming projects would 'watch' those keys for being changed (and
> >> get notified when they are changed); the project would then reload its
> >> policy when the other service (keystone) write a new key/policy.
> >>
> >> https://coreos.com/etcd/docs/latest/api.html#waiting-for-a-change
> >>
> >> or
> >> https://zookeeper.apache.org/doc/r3.4.5/zookeeperProgrammers.html#ch_zkWatches
> >>
> >> or (pretty sure consul has something similar),
> >>
> >> This is pretty standard stuff folks :-/ and it's how afaik things like
> >> https://github.com/skynetservices/skydns work (and more), and it would
> >> avoid that 'additional latency' (unless the other service is adjusting
> >> the policy key every millisecond, which seems sorta unreasonable).
> >
> > Sure. Or have Keystone be a frontend for ansible/puppet/chef/ What's
> > not clear to me in any of this is what's the benefit to having Keystone
> > as a fronted to policy configuration/changes, or be involved in any real
> > way with authorization decisions? What issue is being solved by getting
> > Keystone involved?
> >
> 
> I don't understand the puppet/chef connection, can u clarify.
> 
> If I'm interpreting it right, I would assume it's the same reason that 
> something like 'skydns' exists over etcd; to provide a useful API that 
> focuses on the dns particulars that etcd will of course not have any 
> idea about. So I guess the keystone API could(?)/would(?) then focus on 
> policy particulars as its value-add.
> 
> Maybe now I understand what u mean by puppet/chef, in that you are 
> asking why isn't skydns (for example) just letting/invoking 
> puppet/chef/ansible to distribute/send-out dns (dnsmasq) files? Is that 
> your equivalent question?

I'm focused on Nova/Keystone/OpenStack here, I'm sure skydns has good
reasons for their technical choices and I'm in no place to question
them.

I'm trying to understand the value-add that Keystone could provide here.
Policy configuration is fairly static so I'm not understanding the
desire to put an API on top of it. But perhaps I'm missing the use case
here which is why I've been asking.

My ansible/puppet/chef comparison was just that those are ways to
distribute static files and would work just as well as something built
on top of etcd/zookeeper. I'm not really concerned about how it's
implemented though. I'm just trying to understand if the desire is to
have Keystone handle this so that deployers don't need to work with
their configuration management system to configure policy files, or is
there something more here?


> 
> -Josh
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Anyone interested in writing a policy generator sphinx extension?

[Andrew Laski]

On Wed, Sep 21, 2016, at 12:02 PM, Joshua Harlow wrote:
> Andrew Laski wrote:
> > However, I have asked twice now on the review what the benefit of doing
> > this is and haven't received a response so I'll ask here. The proposal
> > would add additional latency to nearly every API operation in a service
> > and in return what do they get? Now that it's possible to register sane
> > policy defaults within a project most operators do not even need to
> > think about policy for projects that do that. And any policy changes
> > that are necessary are easily handled by a config management system.
> >
> > I would expect to see a pretty significant benefit in exchange for
> > moving policy control out of Nova, and so far it's not clear to me what
> > that would be.
> 
> One way to do this is to setup something like etc.d or zookeeper and 
> have policy files be placed into certain 'keys' in there by keystone, 
> then consuming projects would 'watch' those keys for being changed (and 
> get notified when they are changed); the project would then reload its 
> policy when the other service (keystone) write a new key/policy.
> 
> https://coreos.com/etcd/docs/latest/api.html#waiting-for-a-change
> 
> or 
> https://zookeeper.apache.org/doc/r3.4.5/zookeeperProgrammers.html#ch_zkWatches
> 
> or (pretty sure consul has something similar),
> 
> This is pretty standard stuff folks :-/ and it's how afaik things like 
> https://github.com/skynetservices/skydns work (and more), and it would 
> avoid that 'additional latency' (unless the other service is adjusting 
> the policy key every millisecond, which seems sorta unreasonable).

Sure. Or have Keystone be a frontend for ansible/puppet/chef/ What's
not clear to me in any of this is what's the benefit to having Keystone
as a fronted to policy configuration/changes, or be involved in any real
way with authorization decisions? What issue is being solved by getting
Keystone involved?


> 
> -Josh
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Anyone interested in writing a policy generator sphinx extension?

[Andrew Laski]

On Wed, Sep 21, 2016, at 11:05 AM, Alexander Makarov wrote:
> What if policy will be manageable using RESTful API?
> I'd like to validate the idea to handle policies in keystone or 
> affiliated service: https://review.openstack.org/#/c/325326/

As Matt said, that's unrelated to what he's asking about.

However, I have asked twice now on the review what the benefit of doing
this is and haven't received a response so I'll ask here. The proposal
would add additional latency to nearly every API operation in a service
and in return what do they get? Now that it's possible to register sane
policy defaults within a project most operators do not even need to
think about policy for projects that do that. And any policy changes
that are necessary are easily handled by a config management system.

I would expect to see a pretty significant benefit in exchange for
moving policy control out of Nova, and so far it's not clear to me what
that would be.


> 
> On 21.09.2016 17:49, Matt Riedemann wrote:
> > Nova has policy defaults in code now and we can generate the sample 
> > using oslopolicy-sample-generator but we'd like to get the default 
> > policy sample in the Nova developer documentation also, like we have 
> > for nova.conf.sample.
> >
> > I see we use the sphinxconfiggen extension for building the 
> > nova.conf.sample in our docs, but I don't see anything like that for 
> > generating docs for a sample policy file.
> >
> > Has anyone already started working on that, or is interested in 
> > working on that? I've never written a sphinx extension before but I'm 
> > guessing it could be borrowed a bit from how sphinxconfiggen was 
> > written in oslo.config.
> >
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] ops meetup feedback

[Andrew Laski]

Excellent writeup, thanks. Some comments inline.


On Tue, Sep 20, 2016, at 09:20 AM, Sean Dague wrote:
> 
> 
> Performance Bottlenecks
> ---
> 
> * scheduling issues with Ironic - (this is a bug we got through during
>   the week after the session)
> * live snapshots actually end up performance issue for people
> 
> The workarounds config group was not well known, and everyone in the
> room wished we advertised that a bit more. The solution for snapshot
> performance is in there.
> 
> There were also general questions about what scale cells should be
> considered at.
> 
> ACTION: we should make sure workarounds are advertised better
> ACTION: we should have some document about "when cells"?

This is a difficult question to answer because "it depends." It's akin
to asking "how many nova-api/nova-conductor processes should I run?"
Well, what hardware is being used, how much traffic do you get, is it
bursty or sustained, are instances created and left alone or are they
torn down regularly, do you prune your database, what version of rabbit
are you using, etc...

I would expect the best answer(s) to this question are going to come
from the operators themselves. What I've seen with cellsv1 is that
someone will decide for themselves that they should put no more than X
computes in a cell and that information filters out to other operators.
That provides a starting point for a new deployment to tune from.

>  
> 
> Policy
> --
> 
> How are you customizing policy? People were largely making policy
> changes to protect their users that didn't really understand cloud
> semantics. Turning off features that they thought would confuse them
> (like pause). The large number of VM states is confusing, and not
> clearly useful for end users, and they would like simplification.
> 
> Ideally policy could be set on a project by project admin, because
> they would like to delegate that responsibility down.
> 
> No one was using the user_id based custom policy (yay!).
> 
> There was desire that flavors could be RBAC locked down, which was
> actually being done via policy hacks right now. Providers want to
> expose some flavors (especially those with aggregate affinity) to only
> some projects.
> 
> People were excited about the policy in code effort, only concern was
> that the defacto documentation of what you could change wouldn't be in
> the sample config.
> 
> ACTION: ensure there is policy config reference now that the sample
> file is empty

We have the "genpolicy" tox target which mimics the "genconfig" target.
It's similar to the old sample except guaranteed to be up to date, and
can include comments. Is that sufficient?


> ACTION: flavor RBAC is a thing most of the room wanted, is there a
> taker on spec / implementation?

This isn't the most clear spec ever but we have an approved backlog spec
for flavor-classes which was looking to get into this area
http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/backlog/approved/flavor-class.rst
. That would add a grouping mechanism for flavors which could then be
used for policy checks. I think having that in place would minimize the
policy explosion that might happen if policy could be attached to
individual flavors.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] No cellsv2 meeting today

[Andrew Laski]

Since we're closing in on RC1 which needs peoples attention and nothing
cells related affects that we're going to skip the cells meeting today.
The next planned meeting is September 28th at 1700UTC. Thanks.

-Andrew

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch instance with Floating IP

[Andrew Laski]

On Wed, Sep 7, 2016, at 08:41 PM, Fox, Kevin M wrote:
> Interesting. It kind of sounds like your proposing a sort of... openstack
> standard library api api? (yes, double apis) Where you can build up an
> api using other api calls and users can rely on those standard
> overarching api's to be there?

What I was thinking was just a service that can interact with multiple
projects for simple API requests. If we take the example from elsewhere
in this thread of pre-creating a port, assigning a floating-ip to it,
and then booting an instance with that port that's currently multiple
calls to two different services. I would love to see a service that can
take all of that info in one request and then make the multiple calls
necessary on behalf of the user. I additionally would expect it to be
able to list resources, delete resources, whatever the user needs.

My grand idea would be that users do not interact with
Nova/Cinder/Neutron directly unless they have specific complex needs and
instead use a new API proxy/orchestration service. For a long time it's
been said that this should be handled for a user on the client side.
That works great for users who wish to use a client that does that, but
there are many users out there who do not use something like
openstackclient, for various reasons.

> 
> Thanks,
> Kevin
> ________
> From: Andrew Laski [and...@lascii.com]
> Sent: Wednesday, September 07, 2016 4:34 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch
> instance with Floating IP
> 
> On Wed, Sep 7, 2016, at 06:54 PM, Martin Millnert wrote:
> > On Thu, 2016-09-08 at 09:34 +1200, Adrian Turjak wrote:
> > > This is exactly what something like Minstral would be fantastic for.
> > > Multi-project workflow.
> > > Rather than try and build logic like this directly into nova, looks
> > > at extending something like Minstral with a basic easy to call task.
> >
> > I have API- and code-reading homework to do here - but based on input
> > on IRC, and efforts thus far, the ideal solution is not really to bolt
> > a humongous piece of logic onto Nova. Rather it is appears to be first
> > and foremost Neutron that need a little bit more intelligence regarding
> > its virtual networks and the intentions of its operators.
> >
> > From what I can tell, Mistral is a very useful project for scheduling
> > recurring tasks or similar, which there is a definite need of in a
> > mature deployment.
> >
> > But I disagree with the "solve it with a new project"-approach here:
> >
> >  1) "launch my instance" is as core as it gets in OpenStack,
> >
> >  2) "launch my instance with Internet" is approximately identically as
> > core, just a bit difficult for $reasons and not fully supported today,
> >
> >  3) core functionality should IMO require as few API calls as possible,
> > to as few components as possible, while keeping REST data models etc.
> > intact, [1][2]
> 
> I agree that it should require as few API calls as possible but maybe we
> disagree about what core functionality is. Or to put it another way,
> what is core functionality depends on your perspective.
> 
> I subscribe to the plumbing and porcelain approach
> (https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain)
> and believe that Nova should be part of the plumbing. So while I fully
> agree with a small number of API calls to do simple tasks I don't
> believe that orchestrating network setups is core functionality in Nova
> but is core to OpenStack.
> 
> >
> >  4) there are timing concerns with adding Internet to an instance today
> > in OpenStack, since it in some cases needs to happen somewhere between
> > the events "network port created" and "instance launched", in order for
> > the instance to actually have Internet when it boots,
> >
> >  5) Scheduling "Launch instance with Internet" or "Add Internet to
> > instance" via something like Mistral would additionally either, A) add
> > a significant delay to the boot time, or B) not even fulfill the
> > objective of having Internet once instance powers on,
> >
> >  6) To replicate the logic of shade's "get me online" functions, a
> > large amount of code is required, and depending on how you go about it,
> > you also risk duplicating logic already in e.g. Nova or Neutron.
> >
> > Best regards,
> > Martin
> >
> > [1] "A designer knows he has achieved perfection not when there is
> > nothing left to add, but when there is nothing left to

Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch instance with Floating IP

[Andrew Laski]

On Thu, Sep 8, 2016, at 07:18 AM, Sean Dague wrote:
> On 09/07/2016 07:34 PM, Andrew Laski wrote:
> > 
> > 
> > On Wed, Sep 7, 2016, at 06:54 PM, Martin Millnert wrote:
> >> On Thu, 2016-09-08 at 09:34 +1200, Adrian Turjak wrote:
> >>  3) core functionality should IMO require as few API calls as possible,
> >> to as few components as possible, while keeping REST data models etc.
> >> intact, [1][2]
> > 
> > I agree that it should require as few API calls as possible but maybe we
> > disagree about what core functionality is. Or to put it another way,
> > what is core functionality depends on your perspective.
> > 
> > I subscribe to the plumbing and porcelain approach
> > (https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain)
> > and believe that Nova should be part of the plumbing. So while I fully
> > agree with a small number of API calls to do simple tasks I don't
> > believe that orchestrating network setups is core functionality in Nova
> > but is core to OpenStack.
> 
> Personally, I think that the role of Nova is to give you a functional
> compute unit. If you can't talk to that over the network, that doesn't
> seem very functional to me. For complicated setups it's fine that you
> need to do complicated things, but "I would like a working server with
> network" doesn't feel like it's a complicated ask from the user.

I'd really like to agree here, however it seems that it is actually a
somewhat complicated ask from the user due to the diverse network setups
used in practice.

But I was responding to the idea of booting an instance with a
floating-ip which to me goes beyond setting up a simple default network
for an instance. And I think therein lies the problem, there seems to be
some disagreement as to what a simple default network setup should be.

> 
> And more importantly, if there isn't some firm line there, the ability
> to do that between clouds is going to be somewhat limited, because
> exactly how you need to configure each cloud's neutron manually to work
> with their physical network setup is going to be different.
> 
> You end up with the shade situation now, which isn't a great user
> experience.
> 
> I do get the concern about Nova scope creep into orchestration. However
> it's not like "orchestration" has a bright line anywhere. And I think
> that basic working networking for requested computes isn't something we
> should require another service for.
> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch instance with Floating IP

[Andrew Laski]

On Thu, Sep 8, 2016, at 05:48 AM, Kevin Benton wrote:
> Why don't you just pre-create the port and associate the floating IP
> before booting the instance?

Right. It's not difficult to do at all but there's a strong desire to
have this be encompassed in a single API call. I think that's a great
idea, but I don't think it belongs in Nova.

>
> On Wed, Sep 7, 2016 at 5:58 PM, Adrian Turjak
>  wrote:
>> Double api sounds a little terrifying when really there are
>> probably so
>>  many different ways you can already solve this using the services we
>>  already have.
>>
>>  The thing I don't get is Martin's requirement that "an instance must
>>  have internet on boot" and that to do that he must have a
>>  floating ip
>>  assigned to it. Internet on boot I can understand because of
>>  preconfigured images and snapshots starting internal services
>>  and tasks
>>  on boot, but why is floating ip on boot such a hard requirement? Is
>>  adding a floating ip before boot even possible with Nova right
>>  now (I
>>  ask as I've never looked into it)?
>>
>>  Unless I'm missing something, you can easily setup a private network
>>  with internet access, boot your instance on that, and then add a
>>  floating ip. The instance will have internet on boot, and then
>>  be given
>>  a floating ip. Does that not solve your problem and can that not be
>>  orchestrated with the current range of services/tools we have?
>>
>>
>> On 08/09/16 12:41, Fox, Kevin M wrote:
>>  > Interesting. It kind of sounds like your proposing a sort of...
>>  > openstack standard library api api? (yes, double apis) Where you
>>  > can build up an api using other api calls and users can rely on
>>  > those standard overarching api's to be there?
>>  >
>>  > Thanks,
>>  > Kevin
>>  > 
>>  > From: Andrew Laski [and...@lascii.com]
>>  > Sent: Wednesday, September 07, 2016 4:34 PM
>>  > To: openstack-dev@lists.openstack.org
>>  > Subject: Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch
>>  > instance with Floating IP
>>  >
>>  > On Wed, Sep 7, 2016, at 06:54 PM, Martin Millnert wrote:
>>  >> On Thu, 2016-09-08 at 09:34 +1200, Adrian Turjak wrote:
>>  >>> This is exactly what something like Minstral would be fantastic
>>  >>> for.
>>  >>> Multi-project workflow.
>>  >>> Rather than try and build logic like this directly into nova,
>>  >>> looks
>>  >>> at extending something like Minstral with a basic easy to call
>>  >>> task.
>>  >> I have API- and code-reading homework to do here - but based on
>>  >> input
>>  >> on IRC, and efforts thus far, the ideal solution is not really to
>>  >> bolt
>>  >> a humongous piece of logic onto Nova. Rather it is appears to be
>>  >> first
>>  >> and foremost Neutron that need a little bit more intelligence
>>  >> regarding
>>  >> its virtual networks and the intentions of its operators.
>>  >>
>>  >> From what I can tell, Mistral is a very useful project for
>>  >> scheduling
>>  >> recurring tasks or similar, which there is a definite need of
>>  >> in a
>>  >> mature deployment.
>>  >>
>>  >> But I disagree with the "solve it with a new project"-approach
>>  >> here:
>>  >>
>>  >>  1) "launch my instance" is as core as it gets in OpenStack,
>>  >>
>>  >>  2) "launch my instance with Internet" is approximately
>>  >> identically as
>>  >> core, just a bit difficult for $reasons and not fully supported
>>  >> today,
>>  >>
>>  >>  3) core functionality should IMO require as few API calls as
>>  >> possible,
>>  >> to as few components as possible, while keeping REST data models
>>  >> etc.
>>  >> intact, [1][2]
>>  > I agree that it should require as few API calls as possible but
>>  > maybe we
>>  > disagree about what core functionality is. Or to put it another
>>  > way,
>>  > what is core functionality depends on your perspective.
>>  >
>>  > I subscribe to the plumbing and porcelain approach
>>  > (https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain)
>>  > and believe that Nova should be pa

Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch instance with Floating IP

[Andrew Laski]

On Wed, Sep 7, 2016, at 06:33 PM, Adrian Turjak wrote:
> Heat is a bit of overkill for something as tiny as this.

I agree. And though it's been a while since I've looked at Heat I didn't
get the impression that it was designed for this sort of usage: a simple
API call that orchestrates networking and instance setup that then
allows you to interact with Nova/Neutron directly for further actions.

> I was mainly responding to Andrew Laski's comment talking about a
> service exactly like Minstral, for smaller cross project orchestration
> of finite tasks.
> On 8/09/2016 10:12 AM, "Fox, Kevin M"  wrote:
>  >
>  > I've typically used heat to do this too. As it has a nice way out
>  > of the box to to undo the actions it does.
>  >
>  > Thanks,
>  > Kevin
>  > 
>  > From: Adrian Turjak [adri...@catalyst.net.nz]
>  > Sent: Wednesday, September 07, 2016 2:34 PM
>  > To: OpenStack Development Mailing List (not for usage questions)
>  > Subject: Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch
>  > instance with Floating IP
>  >
>  > This is exactly what something like Minstral would be fantastic
>  > for. Multi-project workflow.
>  >
>  > Rather than try and build logic like this directly into nova, looks
>  > at extending something like Minstral with a basic easy to call
>  > task.
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch instance with Floating IP

[Andrew Laski]

On Wed, Sep 7, 2016, at 06:54 PM, Martin Millnert wrote:
> On Thu, 2016-09-08 at 09:34 +1200, Adrian Turjak wrote:
> > This is exactly what something like Minstral would be fantastic for.
> > Multi-project workflow.
> > Rather than try and build logic like this directly into nova, looks
> > at extending something like Minstral with a basic easy to call task.
> 
> I have API- and code-reading homework to do here - but based on input
> on IRC, and efforts thus far, the ideal solution is not really to bolt
> a humongous piece of logic onto Nova. Rather it is appears to be first
> and foremost Neutron that need a little bit more intelligence regarding
> its virtual networks and the intentions of its operators.
> 
> From what I can tell, Mistral is a very useful project for scheduling
> recurring tasks or similar, which there is a definite need of in a
> mature deployment.
> 
> But I disagree with the "solve it with a new project"-approach here:
> 
>  1) "launch my instance" is as core as it gets in OpenStack,
> 
>  2) "launch my instance with Internet" is approximately identically as
> core, just a bit difficult for $reasons and not fully supported today,
> 
>  3) core functionality should IMO require as few API calls as possible,
> to as few components as possible, while keeping REST data models etc.
> intact, [1][2]

I agree that it should require as few API calls as possible but maybe we
disagree about what core functionality is. Or to put it another way,
what is core functionality depends on your perspective.

I subscribe to the plumbing and porcelain approach
(https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain)
and believe that Nova should be part of the plumbing. So while I fully
agree with a small number of API calls to do simple tasks I don't
believe that orchestrating network setups is core functionality in Nova
but is core to OpenStack.

> 
>  4) there are timing concerns with adding Internet to an instance today
> in OpenStack, since it in some cases needs to happen somewhere between
> the events "network port created" and "instance launched", in order for
> the instance to actually have Internet when it boots,
> 
>  5) Scheduling "Launch instance with Internet" or "Add Internet to
> instance" via something like Mistral would additionally either, A) add
> a significant delay to the boot time, or B) not even fulfill the
> objective of having Internet once instance powers on,
> 
>  6) To replicate the logic of shade's "get me online" functions, a
> large amount of code is required, and depending on how you go about it,
> you also risk duplicating logic already in e.g. Nova or Neutron.
> 
> Best regards,
> Martin
> 
> [1] "A designer knows he has achieved perfection not when there is
> nothing left to add, but when there is nothing left to take away."
>   -- Antoine de Saint-Exupery
> [2] https://tools.ietf.org/rfcdiff?url2=draft-heitz-idr-large-community
> -02.txt (example of commendable improvement almost unheard of within
> the IETF)
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon][OSC][Nova][Neutron] Launch instance with Floating IP

[Andrew Laski]

On Wed, Sep 7, 2016, at 04:31 AM, Martin Millnert wrote:
> Hi Matt,
> 
> On Tue, 2016-09-06 at 16:39 -0500, Matt Riedemann wrote:
> > Floating IPs aren't required in OpenStack deployments, and anyone 
> > running with cells v1 (Rackspace, GoDaddy, NeCTAR, CERN) doesn't use
> > or 
> > support them, at least without patching Nova. So I'm not sure what 
> > 'industry standard' is being referred to here.
> 
> Nod. There are many models, I acknowledge that as well. When referring
> to 'industry standard', I refer to e.g. default EC2 VPC or GCE non-
> legacy network behaviour. I didn't mean to imply that this model is the
> only one, not even the best one, just the most widely used one at
> large.
> 
> > >   Problem:
> > >  - nova: we're not adding anything
> > 
> > Correct, nova provides the APIs to do this already, something sitting
> > on top just needs to use them to orchestrate your use case.
> 
> It exists in terms of multiple calls, yes. My e-mail is about what the
> best multi-project solution to improving the total logic required to
> achieve the goal, within OpenStack. It's not clear the answer is to
> improve Nova's server create API, but it is one very obvious candidate
> solution.
> 
> > The get-me-a-network work is complete with the 2.37 API microversion
> > in Nova and the 6.0.0 python-novaclient release. You can test it out
> > today. 
> > However, it does not allocate and auto-assign a floating IP.
> 
> I'd argue that what a typical user is most interested in is not, in
> fact, in "having a network", but, "having internet access".
> 
> Nova has the POST /servers 'fixed_ip' and 'networks' keys. Neither one
> of them, as you point out, deals with auto-allocating-and-assigning
> FIPs, which is fine in and of itself - Rome wasn't built in one day
> either.
> 
> I.e. today,
>  A) 'networks=auto' != "get me online",
>  B) 'networks=auto' == "get me an openstack network interface".
> 
> B is a subset of A, but A is not a subset of B.
> 
> Assuming,
>  1) 'networks' is definitely meant to mean just what it's called, and does 
> today, and
>  2) "get me online" is a desirable feature,
> 
> then it is actually the case that we're missing an option like
> "public_ip=auto" or similar to complete the picture.
> In the deployments you mention above, it'd have virtually nothing to do.
> In others, for example FIP, it'd have only little to do.
> 
> Then, the combination networks=auto, public_ip(v4)=auto would equal
> "getmeonline".
> 
> > > So how can one solve an OpenStack cross-project problem like
> this,
> > > possibly without having to implement an artificial
> > > superintelligence
> > > first?
> > 
> > Orchestrate on top of Nova's APIs, either in your own CLI, or UI, or 
> > maybe even Heat would support this.
> 
> Right, the total amount of options are thus:
> 1) Introduce a new, custom API service to proxy for Nova, and fork
> Horizon to handle it,

I would change this slightly and say that my preference would be to have
a new API service which can handle cross project orchestration needs.
Things like what you're discussing, or creating a volume and booting
from it, and many other things that require a client to touch multiple
services. I believe there's a gap here that Heat doesn't fill and that
encourages people to think that adding orchestration to Nova is the
right thing to do.

> 2) Patch Nova and do UI fixes in Horizon, but do not submit it
> upstream,
> 3) Patch Nova and do UI fixes in Horizon, and submit it upstream,
> 4) Make Horizon stateful and do UI changes, but do not submit it
> upstream,
> 5) Make Horizon stateful and do UI changes, and submit it upstream
> 
> I'm sure there are more enumerations of this, but Heat is not one of
> them. :-)
> 
> To me, from the above, option 3 seems the one that involves the least
> amount of complexity, which already there is a good indication of being
> the right choice.
> 
> Best regards,
> Martin
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Add support for tag instances when booting

[Andrew Laski]

On Wed, Sep 7, 2016, at 05:15 AM, Zhenyu Zheng wrote:
> Hi All,
>
> Tags for servers are supported in microversion 2.26, but currently we
> can only add tags to
> instances that are already existed in the cloud, that is, we can not
> set tags to instances
> when we boot the instances. User will have to first find the instances
> and then add tags
> with another API call. This is not user-friendly enough when user
> doing bulk boot, it will be
> not practical to add tags for those instances one by one afterwards.
>
> So, I think it will be good to add support for tag instances when
> booting them, I have registered
> a BP for O:
> https://blueprints.launchpad.net/nova/+spec/support-tag-instance-when-boot
> https://review.openstack.org/#/c/366469/
>
> Comments?

I haven't looked at the spec, and probably won't until at least RC1 is
done, but I think this is a good idea. A few cores noticed that this was
missing a little bit ago and already agreed that it would be a good
addition in O.

>
> Thanks,
>
> Zhenyu Zheng
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

[Andrew Laski]

On Tue, Aug 30, 2016, at 09:55 AM, lebre.adr...@free.fr wrote:
> 
> 
> - Mail original -
> > De: "Andrew Laski" 
> > À: openstack-dev@lists.openstack.org
> > Envoyé: Mardi 30 Août 2016 15:03:35
> > Objet: Re: [openstack-dev] [all][massively 
> > distributed][architecture]Coordination between actions/WGs
> > 
> > 
> > 
> > On Tue, Aug 30, 2016, at 05:36 AM, lebre.adr...@free.fr wrote:
> > > Dear all
> > > 
> > > Sorry my lack of reactivity, I 've been out for the few last days.
> > > 
> > > According to the different replies, I think we should enlarge the
> > > discussion and not stay on the vCPE use-case, which is clearly
> > > specific
> > > and represents only one use-case among the ones we would like to
> > > study.
> > > For instance we are in touch with NRENs in France and Poland that
> > > are
> > > interested to deploy up to one rack in each of their largest PoP in
> > > order
> > > to provide a distributed IaaS platform  (for further informations
> > > you can
> > > give a look to the presentation we gave during the last summit [1]
> > > [2]).
> > > 
> > > The two questions were:
> > > 1./ Understand whether the fog/edge computing use case is in the
> > > scope of
> > > the Architecture WG and if not, do we need a massively distributed
> > > WG?
> > 
> > Besides the question of which WG this might fall under is the
> > question
> > of how any of the work groups are going to engage with the project
> > communities. There is a group of developers pushing forward on
> > cellsv2
> > in Nova there should be some level of engagement between them and
> > whomever is discussing the fog/edge computing use case. To me it
> > seems
> > like there's some level of overlap between the efforts even if
> > cellsv2
> > is not a full solution. But whatever conversations are taking place
> > about fog/edge or large scale distributed use cases seem  to be
> > happening in channels that I am not aware of, and I haven't heard any
> > other cells developers mention them either.
> > 
> 
> I can only agree !
> Actually we organised an informal exchange with Sylvain Bauza in July in
> order to get additional information regarding the Cell V2
> architecture/implementation.  From our point of view, such changes in the
> code can help us toward our ultimate goal of managing remote DCs in an
> efficient manner (i.e by mitigating for instance the inter-sites
> traffic). 
> 
> 
> > So let's please find a way for people who are interested in these use
> > cases to talk to the developers who are working on similar things.
> 
> What is your proposal ? any particular ideas in mind?  

I am generally aware of things that are discussed in the weekly Nova IRC
meeting, on the ML with a [Nova] tag, and in proposed specs. Using those
forums as part of these discussions would be my recommendation. Or at
the very least use those forums to advertise that there is discussion
happening elsewhere.

The reality is that in order for any discussion to turn into tangible
work it needs to end up as a proposed spec. That can be the start of a
discussion or a summary of a discussion but it really needs to be a part
of the lifecycle of any discussion. Often from there it can branch out
into ML discussions or summit discussions. But specs are a good contact
point between Nova developers and people who have use cases for Nova. It
is important to note that spec proposals should be backed by someone
willing to do the work, which doesn't necessarily need to be the person
proposing the spec.


> 
> Ad_rien_
> 
> > 
> > 
> > > 2./ How can we coordinate our actions with the ones performed in
> > > the
> > > Architecture WG?
> > > 
> > > Regarding 1./, according to the different reactions, I propose to
> > > write a
> > > first draft in an etherpard to present the main goal of the
> > > Massively
> > > distributed WG and how people interested by such discussions can
> > > interact
> > > (I will paste the link to the etherpad by tomorrow).
> > > 
> > > Regarding 2./,  I mentioned the Architecture WG because we do not
> > > want to
> > > develop additional software layers like Tricircle or other
> > > solutions (at
> > > least for the moment).
> > > The goal of the WG is to conduct studies and experiments to
> > > identify to
> > > what extent current mechanisms can satisfy the needs of such a
> > >

Re: [openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

[Andrew Laski]

On Tue, Aug 30, 2016, at 05:36 AM, lebre.adr...@free.fr wrote:
> Dear all 
> 
> Sorry my lack of reactivity, I 've been out for the few last days.
> 
> According to the different replies, I think we should enlarge the
> discussion and not stay on the vCPE use-case, which is clearly specific
> and represents only one use-case among the ones we would like to study.
> For instance we are in touch with NRENs in France and Poland that are
> interested to deploy up to one rack in each of their largest PoP in order
> to provide a distributed IaaS platform  (for further informations you can
> give a look to the presentation we gave during the last summit [1] [2]).
> 
> The two questions were: 
> 1./ Understand whether the fog/edge computing use case is in the scope of
> the Architecture WG and if not, do we need a massively distributed WG? 

Besides the question of which WG this might fall under is the question
of how any of the work groups are going to engage with the project
communities. There is a group of developers pushing forward on cellsv2
in Nova there should be some level of engagement between them and
whomever is discussing the fog/edge computing use case. To me it seems
like there's some level of overlap between the efforts even if cellsv2
is not a full solution. But whatever conversations are taking place
about fog/edge or large scale distributed use cases seem  to be
happening in channels that I am not aware of, and I haven't heard any
other cells developers mention them either.

So let's please find a way for people who are interested in these use
cases to talk to the developers who are working on similar things.


> 2./ How can we coordinate our actions with the ones performed in the
> Architecture WG? 
> 
> Regarding 1./, according to the different reactions, I propose to write a
> first draft in an etherpard to present the main goal of the Massively
> distributed WG and how people interested by such discussions can interact
> (I will paste the link to the etherpad by tomorrow). 
> 
> Regarding 2./,  I mentioned the Architecture WG because we do not want to
> develop additional software layers like Tricircle or other solutions (at
> least for the moment). 
> The goal of the WG is to conduct studies and experiments to identify to
> what extent current mechanisms can satisfy the needs of such a massively
> distributed use-cases and what are the missing elements.  
> 
> I don't want to give to many details in the present mail in order to stay
> as consice as possible (details will be given in the proposal).
> 
> Best regards, 
> Adrien 
> 
> [1] https://youtu.be/1oaNwDP661A?t=583 (please just watch the use-case
> introduction ;  the distribution of the DB  was one possible revision of
> Nova and according to the cell V2 changes it is probably now deprecated). 
> [2] https://hal.inria.fr/hal-01320235
> 
> - Mail original -
> > De: "Peter Willis" 
> > À: "OpenStack Development Mailing List (not for usage questions)" 
> > 
> > Envoyé: Mardi 30 Août 2016 11:24:00
> > Objet: Re: [openstack-dev] [all][massively 
> > distributed][architecture]Coordination between actions/WGs
> > 
> > 
> > 
> > Colleagues,
> > 
> > 
> > An interesting discussion, the only question appears to be whether
> > vCPE is a suitable use case as the others do appear to be cloud use
> > cases. Lots of people assume CPE == small residential devices
> > however CPE covers a broad spectrum of appliances. Some of our
> > customers' premises are data centres, some are HQs, some are
> > campuses, some are branches. For residential CPE we use the
> > Broadband Forum's CPE Wide Area Network management protocol
> > (TR-069), which may be easier to modify to handle virtual
> > machines/containers etc. than to get OpenStack to scale to millions
> > of nodes. However that still leaves us with the need to manage a
> > stack of servers in thousands of telephone exchanges, central
> > offices or even cell-sites, running multiple work loads in a
> > distributed fault tolerant manner.
> > 
> > 
> > Best Regards,
> > Peter.
> > 
> > 
> > On Tue, Aug 30, 2016 at 4:48 AM, joehuang < joehu...@huawei.com >
> > wrote:
> > 
> > 
> > Hello, Jay,
> > 
> > > The Telco vCPE and Mobile "Edge cloud" (hint: not a cloud) use
> > > cases
> > 
> > Do you mean Mobile Edge Computing for Mobile "Edge cloud"? If so,
> > it's cloud. The introduction slides [1] can help you to learn the
> > use cases quickly, there are lots of material in ETSI website[2].
> > 
> > [1]
> > http://www.etsi.org/images/files/technologies/MEC_Introduction_slides__SDN_World_Congress_15-10-14.pdf
> > [2]
> > http://www.etsi.org/technologies-clusters/technologies/mobile-edge-computing
> > 
> > And when we talk about massively distributed cloud, vCPE is only one
> > of the scenarios( now in argue - ing ), but we can't forget that
> > there are other scenarios like vCDN, vEPC, vIMS, MEC, IoT etc.
> > Architecture level discussion is still necessary to see if current
> > design and new propos

Re: [openstack-dev] [Nova] Support specified volume_type when boot instance, do we like it?

[Andrew Laski]

On Mon, Aug 29, 2016, at 09:06 AM, Jordan Pittier wrote:
>
>
> On Mon, Aug 29, 2016 at 8:50 AM, Zhenyu Zheng
>  wrote:
>> Hi, all
>>
>> Currently we have customer demands about adding parameter
>> "volume_type" to --block-device to provide the support of specified
>> storage backend to boot instance. And I find one newly drafted
>> Blueprint that aiming to address the same feature:
>> https://blueprints.launchpad.net/nova/+spec/support-boot-instance-set-store-type
>> ;
>>
>> As I know this is kind of "proxy" feature for cinder and we don't
>> like it in general, but as the boot from volume functional was
>> already there, so maybe it is OK to support another parameter?
>>
>> So, my question is that what are your opinions about this in general?
>> Do you like it or it will not be able to got approved at all?
>>
>> Thanks,
>>
>> Kevin Zheng
>
> Hi,
> I think it's not a great idea. Not only for the reason you mention,
> but also because the "nova boot" command is already way to complicated
> with way to many options. IMO we should only add support for new
> features, not "features" we can have by other means, just for
> convenience.

I completely agree with this. However I have some memory of us
saying(in Austin?) that adding volume_type would be acceptable since
it's a clear oversight in the list of parameters for specifying a block
device. So while I greatly dislike Nova creating volumes and would
rather users pass in pre-created volume ids I would support adding this
parameter. I do not support continuing to add parameters if Cinder adds
parameters though.


>
>
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Reconciling flavors and block device mappings

[Andrew Laski]

On Mon, Aug 29, 2016, at 08:42 AM, Sylvain Bauza wrote:
> 
> 
> Le 29/08/2016 14:20, Jay Pipes a écrit :
> > On 08/29/2016 05:11 AM, Sylvain Bauza wrote:
> >> Le 29/08/2016 13:25, Jay Pipes a écrit :
> >>> On 08/26/2016 09:20 AM, Ed Leafe wrote:
> >>>> On Aug 25, 2016, at 3:19 PM, Andrew Laski  wrote:
> >>>>
> >>>>> One other thing to note is that while a flavor constrains how much
> >>>>> local
> >>>>> disk is used it does not constrain volume size at all. So a user can
> >>>>> specify an ephemeral/swap disk <= to what the flavor provides but can
> >>>>> have an arbitrary sized root disk if it's a remote volume.
> >>>>
> >>>> This kind of goes to the heart of the argument against flavors being
> >>>> the sole source of truth for a request. As cloud evolves, we keep
> >>>> packing more and more stuff into a concept that was originally meant
> >>>> to only divide up resources that came bundled together (CPU, RAM, and
> >>>> local disk). This hasn’t been a good solution for years, and the
> >>>> sooner we start accepting that a request can be much more complex
> >>>> than a flavor can adequately express, the better.
> >>>>
> >>>> If we have decided that remote volumes are a good thing (I don’t
> >>>> think there’s any argument there), then we should treat that part of
> >>>> the request as being as fundamental as a flavor. We need to make the
> >>>> scheduler smarter so that it doesn’t rely on flavor as being the only
> >>>> source of truth.
> >>>>
> >>>> The first step to improving Nova is admitting we have a problem. :)
> >>>
> >>> FWIW, I agree with you on the above. The issue I had with the proposed
> >>> patches was that they would essentially be a hack for a short period
> >>> of time until the resource providers work standardized the way that
> >>> DISK_GB resources were tracked -- including for providers of shared
> >>> disk storage.
> >>>
> >>> I've long felt that flavors as a concept should be, as Chris so
> >>> adeptly wrote, "UI furniture" and should be decomposed into their
> >>> requisite lists of resource amounts, required traits and preferred
> >>> traits and that those decomposed parts are what should be passed to
> >>> the Compute API, not a flavor ID.
> >>>
> >>> But again, we're actively changing all this code in the resource
> >>> providers and qualitative traits patches so I warned about adding more
> >>> code that was essentially just a short-lived hack. I'd be OK adding
> >>> the hack code if there were some big bright WARNINGs placed in there
> >>> that likely the code would be removed in Ocata.
> >>>
> >>
> >> While :
> >> #1 the first change about setting root_gb equals 0 in RequestSpec for
> >> making sure BFV instances are correctly using the DiskFilter is fine by
> >> me having it merged with a TODO/FIXME saying that the code would be
> >> expired once the scheduler uses the resource providers,
> >> #2, then the second patch about trying to look at the BDMs for
> >> DiskFilter is very wrong by me, because the Compute API shouldn't accept
> >> IMHO to ask for a flavor *and* a BDM with a related disk size different
> >> from the flavor. AFAIT, we should return a 40x (probably 409 or 400) for
> >> that instead of accepting it silently.
> >
> > Well, a flavor is always required when launching an instance. I wasn't 
> > aware until recently that one could "override" the root GB (or 
> > eph/swap) sizes in the API. Apparently, the API allows it, though, 
> > even though the code ignores whatever was passed as the override 
> > value. If the API supports it, I think the code should probably be 
> > changed to override the size values to whatever the user entered, no?
> >
> I'm very sad to see this possible. FWIW, I think that the flavor should 
> always be the source of truth for knowing the asked disk size (if no, 
> imagine all the conditionals in code...) and we should somehow reconcile 
> what the flavor provided and what the user explicitly asked for.

At the heart of all of this is the fact that we've allowed people to
believe two different things: flavors are the source of truth, bdms can
be used to override flavors. But bdms only half override f

Re: [openstack-dev] [Nova] Reconciling flavors and block device mappings

[Andrew Laski]

On Mon, Aug 29, 2016, at 07:25 AM, Jay Pipes wrote:
> On 08/26/2016 09:20 AM, Ed Leafe wrote:
> > On Aug 25, 2016, at 3:19 PM, Andrew Laski  wrote:
> >
> >> One other thing to note is that while a flavor constrains how much local
> >> disk is used it does not constrain volume size at all. So a user can
> >> specify an ephemeral/swap disk <= to what the flavor provides but can
> >> have an arbitrary sized root disk if it's a remote volume.
> >
> > This kind of goes to the heart of the argument against flavors being the 
> > sole source of truth for a request. As cloud evolves, we keep packing more 
> > and more stuff into a concept that was originally meant to only divide up 
> > resources that came bundled together (CPU, RAM, and local disk). This 
> > hasn’t been a good solution for years, and the sooner we start accepting 
> > that a request can be much more complex than a flavor can adequately 
> > express, the better.
> >
> > If we have decided that remote volumes are a good thing (I don’t think 
> > there’s any argument there), then we should treat that part of the request 
> > as being as fundamental as a flavor. We need to make the scheduler smarter 
> > so that it doesn’t rely on flavor as being the only source of truth.
> >
> > The first step to improving Nova is admitting we have a problem. :)
> 
> FWIW, I agree with you on the above. The issue I had with the proposed 
> patches was that they would essentially be a hack for a short period of 
> time until the resource providers work standardized the way that DISK_GB 
> resources were tracked -- including for providers of shared disk storage.
> 
> I've long felt that flavors as a concept should be, as Chris so adeptly 
> wrote, "UI furniture" and should be decomposed into their requisite 
> lists of resource amounts, required traits and preferred traits and that 
> those decomposed parts are what should be passed to the Compute API, not 
> a flavor ID.
> 
> But again, we're actively changing all this code in the resource 
> providers and qualitative traits patches so I warned about adding more 
> code that was essentially just a short-lived hack. I'd be OK adding the 
> hack code if there were some big bright WARNINGs placed in there that 
> likely the code would be removed in Ocata.

I'd like to clarify that there are two parts to the patches proposed.
One part is about bdm overrides influencing the scheduler, and the other
part is about proper resource tracking. I've attempted to punt on the
resource tracking part in this thread because I agree that we have a
proper solution on the way and the current proposals are workarounds.
There is some value in the workarounds though as they could be
backported to Mitaka.

> 
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Reconciling flavors and block device mappings

[Andrew Laski]

On Mon, Aug 29, 2016, at 08:20 AM, Jay Pipes wrote:
> On 08/29/2016 05:11 AM, Sylvain Bauza wrote:
> > Le 29/08/2016 13:25, Jay Pipes a écrit :
> >> On 08/26/2016 09:20 AM, Ed Leafe wrote:
> >>> On Aug 25, 2016, at 3:19 PM, Andrew Laski  wrote:
> >>>
> >>>> One other thing to note is that while a flavor constrains how much
> >>>> local
> >>>> disk is used it does not constrain volume size at all. So a user can
> >>>> specify an ephemeral/swap disk <= to what the flavor provides but can
> >>>> have an arbitrary sized root disk if it's a remote volume.
> >>>
> >>> This kind of goes to the heart of the argument against flavors being
> >>> the sole source of truth for a request. As cloud evolves, we keep
> >>> packing more and more stuff into a concept that was originally meant
> >>> to only divide up resources that came bundled together (CPU, RAM, and
> >>> local disk). This hasn’t been a good solution for years, and the
> >>> sooner we start accepting that a request can be much more complex
> >>> than a flavor can adequately express, the better.
> >>>
> >>> If we have decided that remote volumes are a good thing (I don’t
> >>> think there’s any argument there), then we should treat that part of
> >>> the request as being as fundamental as a flavor. We need to make the
> >>> scheduler smarter so that it doesn’t rely on flavor as being the only
> >>> source of truth.
> >>>
> >>> The first step to improving Nova is admitting we have a problem. :)
> >>
> >> FWIW, I agree with you on the above. The issue I had with the proposed
> >> patches was that they would essentially be a hack for a short period
> >> of time until the resource providers work standardized the way that
> >> DISK_GB resources were tracked -- including for providers of shared
> >> disk storage.
> >>
> >> I've long felt that flavors as a concept should be, as Chris so
> >> adeptly wrote, "UI furniture" and should be decomposed into their
> >> requisite lists of resource amounts, required traits and preferred
> >> traits and that those decomposed parts are what should be passed to
> >> the Compute API, not a flavor ID.
> >>
> >> But again, we're actively changing all this code in the resource
> >> providers and qualitative traits patches so I warned about adding more
> >> code that was essentially just a short-lived hack. I'd be OK adding
> >> the hack code if there were some big bright WARNINGs placed in there
> >> that likely the code would be removed in Ocata.
> >>
> >
> > While :
> > #1 the first change about setting root_gb equals 0 in RequestSpec for
> > making sure BFV instances are correctly using the DiskFilter is fine by
> > me having it merged with a TODO/FIXME saying that the code would be
> > expired once the scheduler uses the resource providers,
> > #2, then the second patch about trying to look at the BDMs for
> > DiskFilter is very wrong by me, because the Compute API shouldn't accept
> > IMHO to ask for a flavor *and* a BDM with a related disk size different
> > from the flavor. AFAIT, we should return a 40x (probably 409 or 400) for
> > that instead of accepting it silently.
> 
> Well, a flavor is always required when launching an instance. I wasn't 
> aware until recently that one could "override" the root GB (or eph/swap) 
> sizes in the API. Apparently, the API allows it, though, even though the 
> code ignores whatever was passed as the override value. If the API 
> supports it, I think the code should probably be changed to override the 
> size values to whatever the user entered, no?

That's the question here. There's a lot of desire to have the overrides
be the values used in scheduling, since currently the flavor values are
used, but making that change potentially impacts how flavors are packed
onto hosts for some deployers. The main thing I want to get out of this
thread is if that's okay.

The sense I get so far is that it's okay to make the change to have bdm
values be passed to the scheduler, via RequestSpec modifications, and
deployers can rely upon CPU/RAM constraints to determine packing. So the
main thing to do is have good release notes about the change.

> 
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [Nova] Reconciling flavors and block device mappings

[Andrew Laski]

On Fri, Aug 26, 2016, at 11:01 AM, John Griffith wrote:
>
>
> On Fri, Aug 26, 2016 at 7:37 AM, Andrew Laski
>  wrote:
>>
>>
>> On Fri, Aug 26, 2016, at 03:44
>> AM,kostiantyn.volenbovs...@swisscom.com
>>  wrote:
>> > Hi,
>>  > option 1 (=that's what patches suggest) sounds totally fine.
>>  > Option 3 > Allow block device mappings, when present, to mostly
>>  > determine instance  packing sounds like option 1+additional logic
>>  > (=keyword 'mostly') I think I miss to understand the part of
>>  > 'undermining the purpose of the flavor' Why new behavior might
>>  > require one more parameter to limit number of instances of host?
>>  > Isn't it that those VMs will be under control of other flavor
>>  > constraints, such as CPU and RAM anyway and those will be the ones
>>  > controlling 'instance packing'?
>>
>> Yes it is possible that CPU and RAM could be controlling instance
>>  packing. But my understanding is that since those are often
>>  oversubscribed
> I don't understand why the oversubscription ratio matters here?
>

My experience is with environments where the oversubscription was used
to be a little loose with how many vCPUs were allocated or how much RAM
was allocated but disk was strictly controlled.

>
>
>
>> while disk is not that it's actually the disk amounts
>>  that control the packing on some environments.
> Maybe an explanation of what you mean by "packing" here.  Customers
> that I've worked with over the years have used CPU and Mem as their
> levers and the main thing that they care about in terms of how many
> Instances go on a Node.  I'd like to learn more about why that's wrong
> and that disk space is the mechanism that deployers use for this.
>

By packing I just mean the various ways that different flavors fit on a
host. A host may be designed to hold 1 xlarge, or 2 large, or 4 mediums,
or 1 large and 2 mediums, etc... The challenge I see here is that the
constraint can be managed by using CPU or RAM or disk or some
combination of the three. For deployers just using disk the above
patches will change behavior for them.

It's not wrong to use CPU/RAM, but it's not what everyone is doing. One
purpose of this email was to gauge if it would be acceptable to only use
CPU/RAM for packing.


>
>
>> But that is a sub option
>>  here, just document that disk amounts should not be used to
>>  determine
>>  flavor packing on hosts and instead CPU and RAM must be used.
>>
>>  > Does option 3 covers In case someone relied on eg. flavor root
>>  > disk for disk volume booted from volume - and now instance packing
>>  > will change once patches are implemented?
>>
>> That's the goal. In a simple case of having hosts with 16 CPUs,
>> 128GB of
>>  RAM and 2TB of disk and a flavor with VCPU=4, RAM=32GB,
>>  root_gb=500GB,
>>  swap/ephemeral=0 the deployer is stating that they want only 4
>>  instances
>>  on that host.
> How do you arrive at that logic?  What if they actually wanted a
> single VCPU=4,RAM=32GB,root_gb=500 but then they wanted the remaining
> resources split among Instances that were all 1 VCPU, 1 G ram and a 1
> G root disk?

My example assumes the one stated flavor. But if they have a smaller
flavor then more than 4 instances would fit.

>
>> If there is CPU and RAM oversubscription enabled then by
>>  using volumes a user could end up with more than 4 instances on that
>>  host. So a max_instances=4 setting could solve that. However I don't
>>  like the idea of adding a new config, and I think it's too
>>  simplistic to
>>  cover more complex use cases. But it's an option.
>
> I would venture to guess that most Operators would be sad to read
> that.  So rather than give them an explicit lever that does exactly
> what they want clearly and explicitly we should make it as complex as
> possible and have it be the result of a 4 or 5 variable equation?  Not
> to mention it's completely dynamic (because it seems like
> lots of clouds have more than one flavor).

Is that lever exactly what they want? That's part of what I'd like to
find out here. But currently it's possible to setup a situation where 1
large flavor or 4 small flavors fit on a host. So would the
max_instances=4 setting be desired? Keeping in mind that if the above
patches merged 4 large flavors could be put on that host if they only
use remote volumes and aren't using proper CPU/RAM limits.

I probably was not clear enough in my original description or made some
bad assumptions. The concern I have is that 

Re: [openstack-dev] [Openstack-operators] [Nova] Reconciling flavors and block device mappings

[Andrew Laski]

On Fri, Aug 26, 2016, at 03:44 AM, kostiantyn.volenbovs...@swisscom.com
wrote:
> Hi, 
> option 1 (=that's what patches suggest) sounds totally fine.
> Option 3 > Allow block device mappings, when present, to mostly determine
> instance  packing 
> sounds like option 1+additional logic (=keyword 'mostly') 
> I think I miss to understand the part of 'undermining the purpose of the
> flavor'
> Why new behavior might require one more parameter to limit number of
> instances of host? 
> Isn't it that those VMs will be under control of other flavor
> constraints, such as CPU and RAM anyway and those will be the ones
> controlling 'instance packing'?

Yes it is possible that CPU and RAM could be controlling instance
packing. But my understanding is that since those are often
oversubscribed while disk is not that it's actually the disk amounts
that control the packing on some environments.  But that is a sub option
here, just document that disk amounts should not be used to determine
flavor packing on hosts and instead CPU and RAM must be used.

> Does option 3 covers In case someone relied on eg. flavor root disk for
> disk volume booted from volume - and now instance packing will change
> once patches are implemented?

That's the goal. In a simple case of having hosts with 16 CPUs, 128GB of
RAM and 2TB of disk and a flavor with VCPU=4, RAM=32GB, root_gb=500GB,
swap/ephemeral=0 the deployer is stating that they want only 4 instances
on that host. If there is CPU and RAM oversubscription enabled then by
using volumes a user could end up with more than 4 instances on that
host. So a max_instances=4 setting could solve that. However I don't
like the idea of adding a new config, and I think it's too simplistic to
cover more complex use cases. But it's an option.

> 
> BR, 
> Konstantin
> 
> > -Original Message-
> > From: Andrew Laski [mailto:and...@lascii.com]
> > Sent: Thursday, August 25, 2016 10:20 PM
> > To: openstack-dev@lists.openstack.org
> > Cc: openstack-operat...@lists.openstack.org
> > Subject: [Openstack-operators] [Nova] Reconciling flavors and block device
> > mappings
> > 
> > Cross posting to gather some operator feedback.
> > 
> > There have been a couple of contentious patches gathering attention recently
> > about how to handle the case where a block device mapping supersedes flavor
> > information. Before moving forward on either of those I think we should 
> > have a
> > discussion about how best to handle the general case, and how to handle any
> > changes in behavior that results from that.
> > 
> > There are two cases presented:
> > 
> > 1. A user boots an instance using a Cinder volume as a root disk, however 
> > the
> > flavor specifies root_gb = x where x > 0. The current behavior in Nova is 
> > that the
> > scheduler is given the flavor root_gb info to take into account during 
> > scheduling.
> > This may disqualify some hosts from receiving the instance even though that 
> > disk
> > space  is not necessary because the root disk is a remote volume.
> > https://review.openstack.org/#/c/200870/
> > 
> > 2. A user boots an instance and uses the block device mapping parameters to
> > specify a swap or ephemeral disk size that is less than specified on the 
> > flavor.
> > This leads to the same problem as above, the scheduler is provided 
> > information
> > that doesn't match the actual disk space to be consumed.
> > https://review.openstack.org/#/c/352522/
> > 
> > Now the issue: while it's easy enough to provide proper information to the
> > scheduler on what the actual disk consumption will be when using block 
> > device
> > mappings that undermines one of the purposes of flavors which is to control
> > instance packing on hosts. So the outstanding question is to what extent 
> > should
> > users have the ability to use block device mappings to bypass flavor 
> > constraints?
> > 
> > One other thing to note is that while a flavor constrains how much local 
> > disk is
> > used it does not constrain volume size at all. So a user can specify an
> > ephemeral/swap disk <= to what the flavor provides but can have an arbitrary
> > sized root disk if it's a remote volume.
> > 
> > Some possibilities:
> > 
> > Completely allow block device mappings, when present, to determine instance
> > packing. This is what the patches above propose and there's a strong desire 
> > for
> > this behavior from some folks. But changes how many instances may fit on a
> > host which could be undes

Re: [openstack-dev] [nova] VM console for VMware instances

[Andrew Laski]

On Fri, Aug 26, 2016, at 04:16 AM, Radoslav Gerganov wrote:
> On 25.08.2016 18:25, Andrew Laski wrote:
> > Is there a reason this has not been proposed to the Nova project, or
> > have I missed that? I looked for a proposal and did not see one.
> > 
> 
> The main reason I developed this out of tree is that reviewing patches
> in Nova takes forever.  For example this patch[1] that propose changes
> in this part of the code base has been in review for 2 years.

It's true that sometimes a patch gets overlooked for a long period of
time. If you notice that happening please bring it to our attention in
the #openstack-nova channel or bring it up in open discussion in a
weekly nova meeting.

> 
> > I see that there's support in Nova and python-novaclient for this
> > feature, but the actual proxy is not in the Nova tree. In situations
> > like this, where there's in-tree code to support an out of tree feature,
> > we typically deprecate and remove that code unless there's a plan to
> > move all of the components into the project. 
> 
> I don't think this is the case for console proxies.  The RDP console
> proxy is also developed out of tree, in C++[2].  We can't expect that
> all vendors will commit their proxy implementations in the Nova tree
> for various technical reasons.  In the VMware case, there are no
> technical reasons which prevent putting mksproxy in the Nova tree, I
> decided to start its development out of tree only because reviewing
> patches in Nova takes forever.

Okay. It's possible that there's a different policy in place for console
proxies that I am not aware of, and I don't see anything like that
documented in our docs.

A console proxy written in a different language is a special case where
we're not going to include that in tree. But as much as possible we like
to ensure that Nova services all adhere to the same guidelines and
standards and we've found that having these components in tree has
worked to allow us to do so. For example a major concern is upgrade
support and if we change the API that Nova uses to communicate with its
console proxies we can ensure compatibility with in-tree proxies but may
break a third party proxy and not be able to support those users.

> 
> > Is there a plan to move this proxy into Nova?
> > 
> 
> I will propose adding mksproxy in the Nova tree for the next release
> and if it is accepted, I will deprecate nova-mksproxy.

Thanks.

> 
> [1] https://review.openstack.org/#/c/115483/
> [2] https://cloudbase.it/freerdp-html5-proxy-windows/
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Reconciling flavors and block device mappings

[Andrew Laski]

Cross posting to gather some operator feedback.

There have been a couple of contentious patches gathering attention
recently about how to handle the case where a block device mapping
supersedes flavor information. Before moving forward on either of those
I think we should have a discussion about how best to handle the general
case, and how to handle any changes in behavior that results from that.

There are two cases presented:

1. A user boots an instance using a Cinder volume as a root disk,
however the flavor specifies root_gb = x where x > 0. The current
behavior in Nova is that the scheduler is given the flavor root_gb info
to take into account during scheduling. This may disqualify some hosts
from receiving the instance even though that disk space  is not
necessary because the root disk is a remote volume.
https://review.openstack.org/#/c/200870/

2. A user boots an instance and uses the block device mapping parameters
to specify a swap or ephemeral disk size that is less than specified on
the flavor. This leads to the same problem as above, the scheduler is
provided information that doesn't match the actual disk space to be
consumed. https://review.openstack.org/#/c/352522/

Now the issue: while it's easy enough to provide proper information to
the scheduler on what the actual disk consumption will be when using
block device mappings that undermines one of the purposes of flavors
which is to control instance packing on hosts. So the outstanding
question is to what extent should users have the ability to use block
device mappings to bypass flavor constraints?

One other thing to note is that while a flavor constrains how much local
disk is used it does not constrain volume size at all. So a user can
specify an ephemeral/swap disk <= to what the flavor provides but can
have an arbitrary sized root disk if it's a remote volume.

Some possibilities:

Completely allow block device mappings, when present, to determine
instance packing. This is what the patches above propose and there's a
strong desire for this behavior from some folks. But changes how many
instances may fit on a host which could be undesirable to some.

Keep the status quo. It's clear that is undesirable based on the bug
reports and proposed patches above.

Allow block device mappings, when present, to mostly determine instance
packing. By that I mean that the scheduler only takes into account local
disk that would be consumed, but we add additional configuration to Nova
which limits the number of instance that can be placed on a host. This
is a compromise solution but I fear that a single int value does not
meet the needs of deployers wishing to limit instances on a host. They
want it to take into account cpu allocations and ram and disk, in short
a flavor :)

And of course there may be some other unconsidered solution. That's
where you, dear reader, come in.

Thoughts?

-Andrew


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][API] Need naming suggestions for "capabilities"

[Andrew Laski]

On Thu, Aug 25, 2016, at 12:22 PM, Everett Toews wrote:
> Top posting with general comment...
>
> It sounds like there's some consensus in Nova-land around these traits
> (née "capabilities"). The API Working Group [4] is
> also aware of similar efforts in Cinder [1][2] and Glance [3].

To be clear, we're looking at exposing both traits and capabilities in
Nova. This puts us in a weird spot where I think our concept of traits
aligns with cinders capabilities, but I don't see any match for the Nova
concept of capabilities. So I'm still open to naming suggestions but I
think capabilities most accurately describes what it is. Dean has it
right, I think, that what we really have are 'api capabilities' and
'host capabilities'. But people will end up just using 'capabilities'
and cause confusion.


>
> If these are truly the same concepts being discussed across projects,
> it would be great to see consistency in the APIs and have the projects
> come together under a new guideline. I encourage the projects and
> people to propose such a guideline and for someone to step up and
> champion it. Seems like good fodder for a design session proposal at
> the upcoming summit.

Here's what all of these different things look like to me:

Cinder is looking to expose hardware capabilities. This pretty closely
aligns with what traits are intending to do in Nova. This answers the
question of "can I create a resource that needs/does X in this
deployment?" However in Nova we ultimately want users to be able to
specify which traits they want for their instance. That may be embedded
in a flavor or arbitrarily specified in the request but a trait is not
implicitly available to all resources like it seems it is in Cinder. We
assume there could be a heterogeneous environment so without requesting
a trait there's no guarantee of getting it.

Nova capabilities are intended to answer the question of "as user Y with
resource X what can I do with it?" This is dependent on user
authorization, hardware "traits" where the resource lives, and service
version. I didn't see an analog to this in any of the proposals below.
And one major difference between this and the other proposals is that,
if possible, we would like the response to map to the API action that
will perform that capability. So if a user can perform a resize on their
instance the response might include 'POST .../servers//action -d
resize' or whatever form we come up with.

The Glance concept of value discovery maps closely to what Nova
capabilities are in intent in that it answers the question of "what
can I do in this API request that will be valid?" But the scope is
completely different in that it doesn't answer the question of which
API requests can be made, just what values can be used in this
specific call.


Given the above I find that I don't have the imagination required to
consolidate those into a consistent API concept that can be shared
across projects. Cinder capabilities and Nova traits could potentially
work, but the rest seem too different to me. And if we change traits-
>capabilities then we should find another name for what is currently
Nova capabilities.

-Andrew

>
> Cheers,
> Everett
>
> [1] https://review.openstack.org/#/c/306930/
> [2] https://review.openstack.org/#/c/350310/
> [3]  
> https://specs.openstack.org/openstack/glance-specs/specs/mitaka/approved/image-import/image-import-refactor.html#value-discovery
> [4] http://specs.openstack.org/openstack/api-wg/
>
>
>> On Aug 16, 2016, at 3:16 AM, Sylvain Bauza  wrote:
>>
>>
>>
>> Le 15/08/2016 22:59, Andrew Laski a écrit :
>>
>>> On Mon, Aug 15, 2016, at 10:33 AM, Jay Pipes wrote:
>>>
>>>> On 08/15/2016 09:27 AM, Andrew Laski wrote:
>>>>
>>>>> Currently in Nova we're discussion adding a "capabilities" API to
>>>>> expose
>>>>>  to users what actions they're allowed to take, and having compute
>>>>>  hosts
>>>>>  expose "capabilities" for use by the scheduler. As much fun as it
>>>>>  would
>>>>>  be to have the same term mean two very different things in
>>>>>  Nova to
>>>>>  retain some semblance of sanity let's rename one or both of these
>>>>>  concepts.
>>>>>
>>>>>  An API "capability" is going to be an action, or URL, that a
>>>>>  user is
>>>>>  allowed to use. So "boot an instance" or "resize this instance"
>>>>>  are
>>>>>  capabilities from the API point of view. Whether or not a user
>>>>>  ha

Re: [openstack-dev] [nova] VM console for VMware instances

[Andrew Laski]

On Thu, Aug 25, 2016, at 09:50 AM, Radoslav Gerganov wrote:
> Hi,
> 
> If you want to use the MKS console for VMware instances, it's now
> possible with the nova-mksproxy[1].
> There is a devstack plugin, so simply add this in your local.conf:
> 
>   [[local|localrc]]
>   enable_plugin nova-mksproxy https://github.com/openstack/nova-mksproxy
> 
> the CLI command for getting a console URL is:
> 
>   $ nova get-mks-console 
> 
> This is the preferred console type for VMware instances because it
> doesn't require any configuration
> changes on the hypervisor (whereas VNC requires opening network ports).
> 
> Any comments/feedback is welcome.
> 
> -Rado
> 
> [1] https://github.com/openstack/nova-mksproxy

Is there a reason this has not been proposed to the Nova project, or
have I missed that? I looked for a proposal and did not see one.

I see that there's support in Nova and python-novaclient for this
feature, but the actual proxy is not in the Nova tree. In situations
like this, where there's in-tree code to support an out of tree feature,
we typically deprecate and remove that code unless there's a plan to
move all of the components into the project. Is there a plan to move
this proxy into Nova?

> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][API] Need naming suggestions for "capabilities"

[Andrew Laski]

On Mon, Aug 15, 2016, at 10:33 AM, Jay Pipes wrote:
> On 08/15/2016 09:27 AM, Andrew Laski wrote:
> > Currently in Nova we're discussion adding a "capabilities" API to expose
> > to users what actions they're allowed to take, and having compute hosts
> > expose "capabilities" for use by the scheduler. As much fun as it would
> > be to have the same term mean two very different things in Nova to
> > retain some semblance of sanity let's rename one or both of these
> > concepts.
> >
> > An API "capability" is going to be an action, or URL, that a user is
> > allowed to use. So "boot an instance" or "resize this instance" are
> > capabilities from the API point of view. Whether or not a user has this
> > capability will be determined by looking at policy rules in place and
> > the capabilities of the host the instance is on. For instance an
> > upcoming volume multiattach feature may or may not be allowed for an
> > instance depending on host support and the version of nova-compute code
> > running on that host.
> >
> > A host "capability" is a description of the hardware or software on the
> > host that determines whether or not that host can fulfill the needs of
> > an instance looking for a home. So SSD or x86 could be host
> > capabilities.
> > https://github.com/jaypipes/os-capabilities/blob/master/os_capabilities/const.py
> > has a list of some examples.
> >
> > Some possible replacement terms that have been thrown out in discussions
> > are features, policies(already used), grants, faculties. But none of
> > those seemed to clearly fit one concept or the other, except policies.
> >
> > Any thoughts on this hard problem?
> 
> I know, naming is damn hard, right? :)
> 
> After some thought, I think I've changed my mind on referring to the 
> adjectives as "capabilities" and actually think that the term 
> "capabilities" is better left for the policy-like things.
> 
> My vote is the following:
> 
> GET /capabilities <-- returns a set of *actions* or *abilities* that the 
> user is capable of performing
> 
> GET /traits <-- returns a set of *adjectives* or *attributes* that may 
> describe a provider of some resource

Traits sounds good to me.

> 
> I can rename os-capabilities to os-traits, which would make Sean Mooney 
> happy I think and also clear up the terminology mismatch.
> 
> Thoughts?
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova][API] Need naming suggestions for "capabilities"

[Andrew Laski]

Currently in Nova we're discussion adding a "capabilities" API to expose
to users what actions they're allowed to take, and having compute hosts
expose "capabilities" for use by the scheduler. As much fun as it would
be to have the same term mean two very different things in Nova to
retain some semblance of sanity let's rename one or both of these
concepts.

An API "capability" is going to be an action, or URL, that a user is
allowed to use. So "boot an instance" or "resize this instance" are
capabilities from the API point of view. Whether or not a user has this
capability will be determined by looking at policy rules in place and
the capabilities of the host the instance is on. For instance an
upcoming volume multiattach feature may or may not be allowed for an
instance depending on host support and the version of nova-compute code
running on that host.

A host "capability" is a description of the hardware or software on the
host that determines whether or not that host can fulfill the needs of
an instance looking for a home. So SSD or x86 could be host
capabilities.
https://github.com/jaypipes/os-capabilities/blob/master/os_capabilities/const.py
has a list of some examples.

Some possible replacement terms that have been thrown out in discussions
are features, policies(already used), grants, faculties. But none of
those seemed to clearly fit one concept or the other, except policies.

Any thoughts on this hard problem?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Possible REST API design change for get-me-a-network (2.37)

[Andrew Laski]

On Thu, Aug 11, 2016, at 06:54 PM, Chris Friesen wrote:
> On 08/11/2016 03:53 PM, Matt Riedemann wrote:
> > I wanted to bring this up for awareness since we're getting close to feature
> > freeze and want consensus before it gets too late.
> >
> > Ken'ichi brought up a good question on my REST API change for the 2.37
> > microversion:
> >
> > https://review.openstack.org/#/c/316398/
> >
> > The way I had written this was to just add special auto/none values for the
> > networks 'uuid' field in the server create request schema.
> >
> > The catch with using auto/none is that they can't be specified with any 
> > other
> > values, like other networks, or a port, or really anything else. It's just a
> > list with a single entry and that's either uuid=auto or uuid=none.
> >
> > Ken'ichi's point was, why not just make "networks" in this case map to 
> > 'auto' or
> > 'none' or the list that exists today.
> >
> > I like the idea, it's cleaner and it probably allows moving some of the
> > validation from the REST API code into the json schema (I think, not totally
> > sure about that yet).
> >
> > It is a change for how networks are requested today so there would be some
> > conditional logic change pushed on the client - my tempest test change and
> > novaclient changes would have to be updated for sure.
> >
> > So I'm looking for input on that idea before we get too late, is that 
> > difference
> > worth the work and syntax change in how the client requests a network when
> > creating a server?
> 
> I like the idea...having magic values for 'uuid' that aren't actually
> uuids and 
> magically don't work with other parameters is sort of gross.

+1. It's cleaner and better represents what's being requested IMO.

> 
> Chris
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Some thoughts on API microversions

[Andrew Laski]

On Thu, Aug 4, 2016, at 11:40 AM, John Garbutt wrote:
> On 4 August 2016 at 16:28, Edward Leafe  wrote:
> > On Aug 4, 2016, at 8:18 AM, Andrew Laski  wrote:
> >
> >> This gets to the point I'm trying to make. We don't guarantee old
> >> behavior in all cases at which point users can no longer rely on
> >> microversions to signal non breaking changes. And where we do guarantee
> >> old behavior sometimes we do it artificially because the only signal we
> >> have is microversions and that's the contract we're trying to adhere to.
> >
> > I've always understood microversions to be a way to prevent breaking an 
> > automated tool when we change either the input or output of our API. Its 
> > benefit was less clear for the case of adding a new API, since there is no 
> > chance of breaking something that would never call it. We also accept that 
> > a bug fix doesn't require a microversion bump, as users should *never* be 
> > expecting a 5xx response, so not only does fixing that not need a bump, but 
> > such fixes can be backported to affect all microversions.
> >
> > The idea that by specifying a distinct microversion would somehow guarantee 
> > an immutable behavior, though, is simply not the case. We discussed this at 
> > length at the midcycle regarding the dropping of the nova-network code; 
> > once that's dropped, there won't be any way to get that behavior no matter 
> > what microversion you specify. It's gone. We signal this with deprecation 
> > notices, release notes, etc., and it's up to individuals to move away from 
> > using that behavior during this deprecation period. A new microversion will 
> > never help anyone who doesn't follow these signals.
> >
> > In the case that triggered this thread [0], the change was completely on 
> > the server side of things; no change to either the request or response of 
> > the API. It simply allowed a failed resize to be recovered more easily. 
> > That's a behavior change, not an API change, and frankly, I can't imagine 
> > anyone who would ever *want* the old behavior of leaving an instance in an 
> > error state. To me, that's not very different than fixing a 5xx response, 
> > as it is correcting an error on the server side.
> >
> 
> The problem is was thinking about is, how do you know if a cloud
> supports that new behaviour? For me, a microversion does help to
> advertise that. Its probably a good example of where its not important
> enough to add a new capability to tell people thats possible.

I do see this as a capability though. I've been thinking of capabilities
as an answer to the question of "what can I do with this resource?" So a
capability query to an instance that errored during resize might
currently just return ['delete', 'call admin(joking)'] and assuming we
relax the restriction it would return ['delete', 'revert_resize'].

> 
> That triggers the follow up question, of is that important in this
> case, could you just make the call and see if it works?

Sure. Until we have more discoverability in the API this is the reality
of what users need to do due to things like policy checks.

What I'm aiming for is discoverability that works well for users. The
current situation is a new microversion means go check the docs or
release notes and where I'd like to be is a new microversion means check
the provided API schemas, and a new/removed capability expresses a
change in behavior. And if there are other types of changes users should
be aware of that we think about the right mechanism for exposing it. All
I'm saying is all we have is a hammer, is everything we're using it on
really a nail? :)

> 
> Thanks,
> johnthetubaguy
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Some thoughts on API microversions

[Andrew Laski]

On Thu, Aug 4, 2016, at 08:20 AM, Sean Dague wrote:
> On 08/03/2016 08:54 PM, Andrew Laski wrote:
> > I've brought some of these thoughts up a few times in conversations
> > where the Nova team is trying to decide if a particular change warrants
> > a microversion. I'm sure I've annoyed some people by this point because
> > it wasn't germane to those discussions. So I'll lay this out in it's own
> > thread.
> > 
> > I am a fan of microversions. I think they work wonderfully to express
> > when a resource representation changes, or when different data is
> > required in a request. This allows clients to make the same request
> > across multiple clouds and expect the exact same response format,
> > assuming those clouds support that particular microversion. I also think
> > they work well to express that a new resource is available. However I do
> > think think they have some shortcomings in expressing that a resource
> > has been removed. But in short I think microversions work great for
> > expressing that there have been changes to the structure and format of
> > the API.
> > 
> > I think microversions are being overused as a signal for other types of
> > changes in the API because they are the only tool we have available. The
> > most recent example is a proposal to allow the revert_resize API call to
> > work when a resizing instance ends up in an error state. I consider
> > microversions to be problematic for changes like that because we end up
> > in one of two situations:
> > 
> > 1. The microversion is a signal that the API now supports this action,
> > but users can perform the action at any microversion. What this really
> > indicates is that the deployment being queried has upgraded to a certain
> > point and has a new capability. The structure and format of the API have
> > not changed so an API microversion is the wrong tool here. And the
> > expected use of a microversion, in my opinion, is to demarcate that the
> > API is now different at this particular point.
> > 
> > 2. The microversion is a signal that the API now supports this action,
> > and users are restricted to using it only on or after that microversion.
> > In many cases this is an artificial constraint placed just to satisfy
> > the expectation that the API does not change before the microversion.
> > But the reality is that if the API change was exposed to every
> > microversion it does not affect the ability I lauded above of a client
> > being able to send the same request and receive the same response from
> > disparate clouds. In other words exposing the new action for all
> > microversions does not affect the interoperability story of Nova which
> > is the real use case for microversions. I do recognize that the
> > situation may be more nuanced and constraining the action to specific
> > microversions may be necessary, but that's not always true.
> > 
> > In case 1 above I think we could find a better way to do this. And I
> > don't think we should do case 2, though there may be special cases that
> > warrant it.
> > 
> > As possible alternate signalling methods I would like to propose the
> > following for consideration:
> > 
> > Exposing capabilities that a user is allowed to use. This has been
> > discussed before and there is general agreement that this is something
> > we would like in Nova. Capabilities will programatically inform users
> > that a new action has been added or an existing action can be performed
> > in more cases, like revert_resize. With that in place we can avoid the
> > ambiguous use of microversions to do that. In the meantime I would like
> > the team to consider not using microversions for this case. We have
> > enough of them being added that I think for now we could just wait for
> > the next microversion after a capability is added and document the new
> > capability there.
> 
> The problem with this approach is that the capability add isn't on a
> microversion boundary, as long as we continue to believe that we want to
> support CD deployments this means people can deploy code with the
> behavior change, that's not documented or signaled any way.

The fact that the capability add isn't on a microversion boundary is
exactly my point. There's no need for it to be in many cases. But it
would only apply for capability adds which don't affect the
interoperability of multiple deployments.

The signaling would come from the ability to query the capabilities
listing. A change in what that listing returns indicates a behavior
change.

Another reason I

[openstack-dev] [Nova] Some thoughts on API microversions

[Andrew Laski]

I've brought some of these thoughts up a few times in conversations
where the Nova team is trying to decide if a particular change warrants
a microversion. I'm sure I've annoyed some people by this point because
it wasn't germane to those discussions. So I'll lay this out in it's own
thread.

I am a fan of microversions. I think they work wonderfully to express
when a resource representation changes, or when different data is
required in a request. This allows clients to make the same request
across multiple clouds and expect the exact same response format,
assuming those clouds support that particular microversion. I also think
they work well to express that a new resource is available. However I do
think think they have some shortcomings in expressing that a resource
has been removed. But in short I think microversions work great for
expressing that there have been changes to the structure and format of
the API.

I think microversions are being overused as a signal for other types of
changes in the API because they are the only tool we have available. The
most recent example is a proposal to allow the revert_resize API call to
work when a resizing instance ends up in an error state. I consider
microversions to be problematic for changes like that because we end up
in one of two situations:

1. The microversion is a signal that the API now supports this action,
but users can perform the action at any microversion. What this really
indicates is that the deployment being queried has upgraded to a certain
point and has a new capability. The structure and format of the API have
not changed so an API microversion is the wrong tool here. And the
expected use of a microversion, in my opinion, is to demarcate that the
API is now different at this particular point.

2. The microversion is a signal that the API now supports this action,
and users are restricted to using it only on or after that microversion.
In many cases this is an artificial constraint placed just to satisfy
the expectation that the API does not change before the microversion.
But the reality is that if the API change was exposed to every
microversion it does not affect the ability I lauded above of a client
being able to send the same request and receive the same response from
disparate clouds. In other words exposing the new action for all
microversions does not affect the interoperability story of Nova which
is the real use case for microversions. I do recognize that the
situation may be more nuanced and constraining the action to specific
microversions may be necessary, but that's not always true.

In case 1 above I think we could find a better way to do this. And I
don't think we should do case 2, though there may be special cases that
warrant it.

As possible alternate signalling methods I would like to propose the
following for consideration:

Exposing capabilities that a user is allowed to use. This has been
discussed before and there is general agreement that this is something
we would like in Nova. Capabilities will programatically inform users
that a new action has been added or an existing action can be performed
in more cases, like revert_resize. With that in place we can avoid the
ambiguous use of microversions to do that. In the meantime I would like
the team to consider not using microversions for this case. We have
enough of them being added that I think for now we could just wait for
the next microversion after a capability is added and document the new
capability there.

Secondly we could consider some indicator that exposes how new the code
in a deployment is. Rather than using microversions as a proxy to
indicate that a deployment has hit a certain point perhaps there could
be a header that indicates the date of the last commit in that code.
That's not an ideal way to implement it but hopefully it makes it clear
what I'm suggesting. Some marker that a user can use to determine that a
new behavior is to be expected, but not one that's more intended to
signal structural API changes.

Thoughts?

-Andrew

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does this api modification need Microversion?

[Andrew Laski]

On Wed, Aug 3, 2016, at 02:24 PM, Chris Friesen wrote:
> On 08/03/2016 10:53 AM, Andrew Laski wrote:
> > I think the discussion about whether or not this needs a microversion is 
> > missing
> > the bigger question of whether or not this should be in the API to begin 
> > with.
> 
> I don't think it's actually in the API.

It's allowing the revert_resize API call to be used in more situations
by relaxing the vm_state checks that occur. I didn't see anything that
calls that code in the event of an error, it just leaves it open for
users to call it.

> 
> > If it's safe to rollback from this error state why not just do that
> > automatically in Nova?
> 
> I think we do for other cases, my understanding is that the issue is
> whether we 
> would need a new microversion for the API because we're changing the 
> user-visible behaviour of the migrate/resize operation when something
> goes 
> wrong.  (IE rolling back to an active state on the source rather than
> staying in 
> an ERROR state.)
> 
> > To me it seems like adding a microversion because a policy rule was
> > changed.
> 
> I believe this is exactly what it is.
> 
>  > I know we should have some sort of signal here for users, but I think
> > we need to look at different ways to signal this type of change.
> 
> How?  The microversion is (currently) the only way the client has of
> determining 
> server behaviour.

Yes, currently this is the only mechanism we have. My preference would
be to be a bit lax on signaling  while we get other mechanisms in place.
For this case I think it would make sense to wait until we have
capability exposure in the API to present a strong signal that a user
could revert from a resize with an instance in an ERROR state. In the
meantime we could use documentation and piggyback on the next
microversion to say "If you're using an API with at least version 2.x
then this capability is available." 

This is driven by my desire to avoid cruft because when we do have
capabilities exposed in the API then we're left with with this odd
microversion that signals the same thing as capabilities. I think
microversions make a lot of sense for signaling different
representations that are available in the API, e.g. an instance looks
one way at version 2.3 and looks different at 2.5. The use of a
microversion to signal something that could apply across every version
of the API makes the meaning of a microversion imprecise. I would love
to be able to explain to someone that microversions mean x. But the
reality is they have no precise definition. All they signal is
"something changed, and it may or may not be visible to you and I can't
tell you what it is, please go check the docs." /rant

> 
> Chris
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does this api modification need Microversion?

[Andrew Laski]

I think the discussion about whether or not this needs a microversion is
missing the bigger question of whether or not this should be in the API
to begin with. If it's safe to rollback from this error state why not
just do that automatically in Nova? If it's proposed for the API because
it's not considered safe I don't agree it should be in the API. This is
not an API that's restricted to admins by default.

However if this is going to be exposed in the API I lean towards this
not needing a microversion. It's a new policy in the usage of the API,
not a change to the API. To me it seems like adding a microversion
because a policy rule was changed. I know we should have some sort of
signal here for users, but I think we need to look at different ways to
signal this type of change.

-Andrew


On Tue, Aug 2, 2016, at 02:11 AM, han.ro...@zte.com.cn wrote:
> patchset url: https://review.openstack.org/#/c/334747/
>
>
>
>
> Allow "revert_resize" to recover error instance after resize/migrate.
>
> When resize/migrate instance, if error occurs on source compute node,
> instance state can rollback to active currently. But if error occurs
> in "finish_resize" function on destination compute node, the instance
> state would not rollback to active.
>
> This patch is to rollback instance state from error to active when
> resize or migrate action failed on destination compute node..
>
>
>
>
> Best,
> Rong Han
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] [RFC] ResourceProviderTags - Manage Capabilities with ResourceProvider

[Andrew Laski]

On Mon, Aug 1, 2016, at 08:08 AM, Jay Pipes wrote:
> On 07/31/2016 10:03 PM, Alex Xu wrote:
> > 2016-07-28 22:31 GMT+08:00 Jay Pipes  > >:
> >
> > On 07/20/2016 11:25 PM, Alex Xu wrote:
> >
> > One more for end users: Capabilities Discovery API, it should be
> > 'GET
> > /resource_providers/tags'. Or a proxy API from nova to the placement
> > API?
> >
> >
> > I would imagine that it should be a `GET
> > /resource-providers/{uuid}/capabilities` call on the placement API,
> > only visible to cloud administrators.
> >
> > When the end-user request a capability which doesn't support by the
> > cloud, the end-user needs to wait for a moment after sent boot request
> > due to we use async call in nova, then he get an instance with error
> > status. The error info is no valid host. If this is the only way for
> > user to discover the capabilities in the cloud, that sounds bad. So we
> > need an API for the end-user to discover the Capabilities which are
> > supported in the cloud, the end-user can query this API before send boot
> > request.
> 
> Ah, yes, totally agreed. I'm not sure if that is something that we'd 
> want to put as a normal-end-user-callable API endpoint in the placement 
> API, but certainly we could do something like this in the placement API:
> 
>   GET /capabilities
> 
> Would return a list of capability strings representing the distinct set 
> of capabilities that any resource provider in the system exposed. It 
> would not give the user any counts of resource providers that expose the 
> capabilities, nor would it provide any information regarding which 
> resource providers had any available inventory for a consumer to use.

This is what I had imagined based on the midcycle discussion of this
topic. Just information about what is possible to request, and no
information about what is available.

> 
> Nova could then either have a proxy API call that would add the normal 
> end-user interface to that information or completely hide it from end 
> users via the existing flavors interface?

Please no more proxy APIs :)

> 
> Thoughts?
> 
> Best,
> -jay
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] No cellsv2 meeting this week

[Andrew Laski]

Since the majority of us met in person last week and there doesn't seem to be 
anything that needs discussion right now I am not planning to have a meeting 
this week. In it's place please feel free to do some reviews :)__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] policy in code update

[Andrew Laski]

Here's a quick update on where we have ended up so far on the work to
define default policy rules in Nova code.

The majority of the effort has merged so now all used policy checks are
defined in code, thanks Claudiu for doing most of this work. The sample
policy file is now empty
http://git.openstack.org/cgit/openstack/nova/tree/etc/nova/policy.json?id=f6f4003dfdac83d1decad56cdf39258e9dc75ec0
and Devstack is running with this policy file in Zuul jobs. Interested
people can look through the patches in
https://blueprints.launchpad.net/nova/+spec/policy-in-code to see what
the effort looked like.

Devs wishing to add a new policy check should be aware that it is now
done by registering a policy check in nova/policies/* just like it would
be done for a config option. And then "context.can(...)" is used for
performing the actual policy check. This will raise an exception if an
attempt is made to use a policy rule which is not registered.

There is a new "tox -e genpolicy" target which will generate a full
policy sample file for devs/deployers who wish to see which policy rules
are available.

As far as what's left there are a few tools that would be useful to
have. There is a WIP proposal for a CLI tool which will generate a list
of policies that a given context and target will pass
(https://review.openstack.org/#/c/322944/). There is another WIP
proposal which will expose two oslo.policy console scripts
(https://review.openstack.org/#/c/335667/). One script compares a policy
file against the registered rules and informs the user of configured
rules that match the defaults. The other script outputs the effective
rules in use which are a merge of the registered rules and configured
rules.

Next steps are to look at a good way to expose this information in an
API so that users can precheck their credentials against policies in
order to know what they are allowed or not allowed to do. This will not
be done in Newton.

Thanks to everyone involved in coding and reviewing this work.

-Andrew

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Plugins for all

[Andrew Laski]

On Thu, Jul 14, 2016, at 03:21 PM, Hayes, Graham wrote:
> I just proposed a review to openstack/governance repo [0] that aims
> to have everything across OpenStack be plugin based for all cross
> project interaction, or allow all projects access to the same internal
> APIs and I wanted to give a bit of background on my motivation, and how
> it came about.

Which internal APIs are you referring to here? Are you limiting this to
internal APIs within Tempest/Devstack that projects can use for
testing/deployment, or are you referring to things like the nova-compute
service which has an internal RPC API?

I feel like I'm missing some context when reading this email because
there are a lot of references to plugins with no clear definition, that
I can see, of what exactly you mean by that.

> 
> Coming from a smaller project, I can see issues for new projects,
> smaller projects, and projects that may not be seen as "important".
> 
> As a smaller project trying to fit into cross project initiatives,
> (and yes, make sure our software looks at least OK in the
> Project Navigator) the process can be difficult.
> 
> A lot of projects / repositories have plugin interfaces, but also
> have project integrations in tree, that do not follow the plugin
> interface. This makes it difficult to see what a plugin can, and
> should do.
> 
> When we moved to the big tent, we wanted as a community to move to
> a flatter model, removing the old integrated status.
> 
> Unfortunately we still have areas when some projects are more equal -
> there is a lingering set of projects who were integrated at the point
> in time that we moved, and have preferential status.
> 
> A lot of the effects are hard to see, and are not insurmountable, but
> do cause projects to re-invent the wheel.
> 
> For example, quotas - there is no way for a project that is not nova, 
> neutron, cinder to hook into the standard CLI, or UI for setting
> quotas. They can be done as either extra commands
> (openstack dns quota set --foo bar) or as custom panels, but not
> the way other quotas get set.

Can you provide more clarity on how "openstack dns quota set --foo bar"
differs from setting a quota on Nova/Neutron/Cinder?

> 
> Tempest plugins are another example. Approximately 30 of the 36
> current plugins are using resources that are not supposed to be
> used, and are an unstable interface. Projects in tree in tempest
> are at a much better position, as any change to the internal
> API will have to be fixed before the gate merges, but other
> out of tree plugins are in a place where they can be broken at any
> point.

Has there been an attempt to elevate these internal interfaces into
stable and publicly consumable interfaces? Was there resistance to such
an effort?

> 
> None of this is meant to single out projects, or teams. A lot
> of the projects that are in this situation have inordinate amounts of
> work placed on them by the big-tent, and I can emphasize with why things 
> are this way. These were the examples that currently stick out
> in my mind, and I think we have come to a point where we need to make
> a change as a community.
> 
> By moving to a "plugins for all" model, these issues are reduced.
> It undoubtedly will cause more, but it is closer to our goal
> of Recognizing all our community is part of OpenStack, and
> differentiate projects by tags.
> 
> This won't be a change that happens tomorrow, next week, or even next
> cycle, but think as a goal, we should start moving in this direction
> as soon as we can, and start building momentum.

Is this just a matter of nobody is doing this work, or are there claims
that some projects are special and should have access that other
projects do not?

> 
> Thanks,
> 
> Graham
> 
> 0 - https://review.openstack.org/342366
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Questions about instance actions' update and finish

[Andrew Laski]

On Wed, Jun 29, 2016, at 11:11 PM, Matt Riedemann wrote:
> On 6/29/2016 10:10 PM, Matt Riedemann wrote:
> > On 6/29/2016 6:40 AM, Andrew Laski wrote:
> >>
> >>
> >>
> >> On Tue, Jun 28, 2016, at 09:27 PM, Zhenyu Zheng wrote:
> >>> How about I sync updated_at and created_at in my patch, and leave the
> >>> finish to the other BP, by this way, I can use updated_at for the
> >>> timestamp filter I added and it don't need to change again once the
> >>> finish BP is complete.
> >>
> >> Sounds good to me.
> >>
> >
> > It's been a long day so my memory might be fried, but the options we
> > talked about in the API meeting were:
> >
> > 1. Setting updated_at = created_at when the instance action record is
> > created. Laski likes this, I'm not crazy about it, especially since we
> > don't do that for anything else.

I would actually like for us to do this generally. I have the same
thinking as Ed does elsewhere in this thread, the creation of a record
is an update of that record. So take my comments as applying to Nova
overall and not just this issue.

> >
> > 2. Update the instance action's updated_at when instance action events
> > are created. I like this since the instance action is like a parent
> > resource and the event is the child, so when we create/modify an event
> > we can consider it an update to the parent. Laski thought this might be
> > weird UX given we don't expose instance action events in the REST API
> > unless you're an admin. This is also probably not something we'd do for
> > other related resources like server groups and server group members (but
> > we don't page on those either right now).

Right. My concern is just that the ordering of actions can change based
on events happening which are not visible to the user. However thinking
about it further we don't really allow multiple actions at once, except
for a few special cases like delete, so this may not end up affecting
any ordering as actions are mostly serial. I think this is a fine
solution for the issue at hand. I just think #1 is a more general
solution.

> >
> > 3. Order the results by updated_at,created_at so that if updated_at
> > isn't set for older records, created_at will be used. I think we all
> > agreed in the meeting to do this regardless of #1 or #2 above.
> >

+1

> 
> Oh and
> 
> #4. Sean Dague needs to come back from leadership training camp in 
> Michigan and make these kind of API decisions for us.

+2

> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] No cells meeting today

[Andrew Laski]

Sorry for the late notice but based on an informal poll of the usual
attendees due to the impending feature freeze everyone is busy with
other things, and there wouldn't be much of anything to discuss this
week. We will resume next week at the normal time.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Questions about instance actions' update and finish

[Andrew Laski]

 
 
 
On Tue, Jun 28, 2016, at 09:27 PM, Zhenyu Zheng wrote:
> How about I sync updated_at and created_at in my patch, and leave the
> finish to the other BP, by this way, I can use updated_at for the
> timestamp filter I added and it don't need to change again once the
> finish BP is complete.
 
Sounds good to me.
 
>
> On Tue, Jun 28, 2016 at 8:28 PM, Andrew Laski
>  wrote:
>> __
>>
>>
>>
>>
>> On Tue, Jun 28, 2016, at 03:26 AM, Zhenyu Zheng wrote:
>>> Hi all,
>>>
>>> I'm working on add pagination and timestamp filter for os-instance-
>>> actions API:
>>> https://review.openstack.org/#/c/326326/
>>> As Alex_xu pointed out that it will be better to filter by
>>> `updated_at` for timestamp filter which is reasonable, but when I
>>> tried to modify the patch I found out that:
>>>
>>> 1. The current APIs only called _record_action_start
>>>(objects.InstanceAction.action_start) and never call
>>>action_finish, so
>>> the field of `finish_time` is always empty in instance_actions
>>> table;
>>
>>
>> There was a spec proposed to address this, though I don't believe it
>> was approved for Newton. So for now you have to assume this will
>> continue to be empty.
>>
>>
>>>
>>> 2. The updated_at field is also empty, should we sync the updated_at
>>>time to the created_at time when we create the action and also
>>>update it whenever the action status changed, e.g finished.
>>
>>
>> When a finish_time is recorded that should definitely also update
>> updated_at. I would be in favor of having updated_at set when the
>> instance action is created. I've never fully understood why Nova
>> doesn't do that generally.
>>
>>>
>>> Thanks,
>>> Kevin Zheng
>>> ___-
>>> _
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-
>>> requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> ___-
>> ___
>>  OpenStack Development Mailing List (not for usage questions)
>>  Unsubscribe: OpenStack-dev-
>>  requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we require the 'next' link for paging, always?

[Andrew Laski]

On Tue, Jun 28, 2016, at 02:09 PM, Matt Riedemann wrote:
> On 6/27/2016 9:57 PM, Alex Xu wrote:
> > Matt, thanks for the email, I +1 on the next page link. Then user
> > needn't build up link by themself.
> >
> > I also have one more question after review those pagination patches:
> >
> > As those pagination proposes change the default sort order. So should we
> > keep the sort order for old version API?
> > I think it should be yes. For instance-actions, it should keep the
> > order sorted by 'create_at' column in old versionAPI. The new version
> > API will sort the result by 'updated_at' column.
> 
> This sounds reasonable since we were already ordering on created_at for 
> instance actions:
> 
> https://github.com/openstack/nova/blob/0c5ff5057edcf1f9ab55a559804a5c0c6a8158b2/nova/db/sqlalchemy/api.py#L5976
> 
> > But the question for keypairs and hypervisors, looks like they
> > didn't specify the sort order explicitly, so I think they should depend
> > on the DB implementation. Should we keep the old version API unspecified
> > sort order?
> 
> I think Andrew Laski made a comment about this in the keypairs change 
> also, that since we never explicitly guaranteed a sort order for these 
> before and never documented anything to that effect, enforcing a new 
> sort order should not be a problem, regardless of the microversion. The 
> natural sort order should be the auto-incrementing id but that's not 
> guaranteed. And the hypervisors pagination change is using id for 
> sorting. The keypairs change is sorting by name. I think at the summit 
> we even told Pavel to push a separate change before the microversion to 
> start ordering keypairs since we weren't going to support sort keys.
> 
> So I don't have a strong opinion either way. If it's just a difference 
> in which DB API method gets called per the microversion, that seems 
> simple enough to keep it working as it did before the pagination changes 
> are added. If it makes the code much more complicated though, then I'd 
> probably not try to retrofit the ordering, or lack thereof.

I think the most important thing is some sort of generally stable
ordering so two requests with no intervening change return the same
ordering. But I also don't view it as a contract that we have to ensure
through a change, either a microversion or a data migration. If we're
stable before the change and stable afterwards then some temporary
instability should be okay as long as we return the correct data.

If users need a completely predictable sort order then we need to offer
a sort_key parameter. Though I'm not suggesting we do that.

> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Questions about instance actions' update and finish

[Andrew Laski]

 
 
 
On Tue, Jun 28, 2016, at 03:26 AM, Zhenyu Zheng wrote:
> Hi all,
>
> I'm working on add pagination and timestamp filter for os-instance-
> actions API:
> https://review.openstack.org/#/c/326326/
> As Alex_xu pointed out that it will be better to filter by
> `updated_at` for timestamp filter which is reasonable, but when I
> tried to modify the patch I found out that:
>
> 1. The current APIs only called _record_action_start
>(objects.InstanceAction.action_start) and never call
>action_finish, so
> the field of `finish_time` is always empty in instance_actions table;
 
There was a spec proposed to address this, though I don't believe it
was approved for Newton. So for now you have to assume this will
continue to be empty.
 
>
> 2. The updated_at field is also empty, should we sync the updated_at
>time to the created_at time when we create the action and also
>update it whenever the action status changed, e.g finished.
 
When a finish_time is recorded that should definitely also update
updated_at. I would be in favor of having updated_at set when the
instance action is created. I've never fully understood why Nova doesn't
do that generally.
 
>
> Thanks,
> Kevin Zheng
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Andrew Laski]

On Tue, Jun 14, 2016, at 05:36 AM, Julien Danjou wrote:
> On Sun, Jun 12 2016, Monty Taylor wrote:
> 
> > I will call hogwash on that. That we install files such as that which
> > are actually code has been a deployer nightmare since day one, and the
> > solution is not to make pip smarter, it's to stop installing such files.
> >
> > NOW - if it's quite literally a file that just exists better as a file
> > and not direct python code - that's fine - just put it into the package
> > itself and then load it either by relative paths from the package or via
> > pkg_resources. For instance, one could put paste.conf into
> > /usr/local/lib/python2.7/site-packages/keystone/paste.ini instead of
> > into etc.
> >
> > Which is my way of saying:
> >
> > - If it's a file that one expects the user to edit, having pip/setup.py
> > install it is a bad idea
> > - If it's a file that one does NOT expect the user to edit, put it into
> > the package.
> 
> I agree with Monty here. The best design we could have is to have
> software that work by default with 0 configuration file.
> Unfortunately, this is not the case for any OpenStack server, and no
> design path has been drawn to follow yet. I'd be glad to see some change
> in this area.
> 
> We're almost at this point in Telemetry projects, we just have to
> carry/ship policy.json (no way to override easily AFAIK) and Paste
> config (no better way to tweak middleware config so far).
> (and the oslo-config-generator configuration file, but that should be
> embeddable easily).

For policy there is now https://review.openstack.org/#/c/309152/ which
allows for registering rules in code. It is currently being incorporated
into Nova https://review.openstack.org/#/c/290155/.

> 
> We'll try to push further in that direction in the future. Let's hope
> everyone will follow. :)
> 
> -- 
> Julien Danjou
> ;; Free Software hacker
> ;; https://julien.danjou.info
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Question about redundant API samples tests for microversions

[Andrew Laski]

On Fri, Jun 17, 2016, at 04:16 PM, Matt Riedemann wrote:
> I was reviewing this today:
> 
> https://review.openstack.org/#/c/326940/
> 
> And I said to myself, 'self, do we really need to subclass the API 
> samples functional tests for this microversion given this change doesn't 
> modify the request/response body, it's only adding paging support?'.
> 
> https://review.openstack.org/#/c/326940/6/nova/tests/functional/api_sample_tests/test_hypervisors.py
> 
> The only change here is listing hypervisors, and being able to page on 
> those if the microversion is high enough. So the API samples don't 
> change at all, they are just running against a different microversion.
> 
> The same goes for the REST API unit tests really:
> 
> https://review.openstack.org/#/c/326940/6/nova/tests/unit/api/openstack/compute/test_hypervisors.py
> 
> I'm not sure if the test subclassing is just done like this for new 
> microversions because it's convenient or if it's because of regression 
> testing - knowing that we aren't changing a bunch of other REST methods 
> in the process, so the subclassed tests aren't testing anything 
> different from the microversion that came before them.
> 
> The thing I don't like about the test subclassing is all of the 
> redundant testing that goes on, and people might add tests to the parent 
> class not realizing it's subclassed and thus duplicating test cases with 
> no functional change.

I agree that the naive subclassing is wasteful. I would rather see tests
that purposely check the changes not just duplicate things at a
different microversion. If there's a concern about regressions I think a
better approach would be to have a check against 'latest' which would
catch accidental changes, not check unchanged request/reponses against
microversions where they aren't intended to change. For my own curiosity
I checked the timings and the duplicated tests took about 20 seconds to
run. That could quickly add up to the point where a significant amount
of time is spent on unnecessary testing.

> 
> Am I just having some Friday crazies? Ultimately this doesn't hurt 
> anything really but thought I'd ask.
> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] using ironic as a replacement for existing datacenter baremetal provisioning

[Andrew Laski]

On Tue, Jun 7, 2016, at 02:34 PM, Joshua Harlow wrote:
> Devananda van der Veen wrote:
> > On 06/07/2016 09:55 AM, Joshua Harlow wrote:
> >> Joshua Harlow wrote:
> >>> Clint Byrum wrote:
>  Excerpts from Joshua Harlow's message of 2016-06-07 08:46:28 -0700:
> > Clint Byrum wrote:
> >> Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +:
> >>> Hi ironic folks,
> >>> As I'm trying to explore how GoDaddy can use ironic I've created
> >>> the following in an attempt to document some of my concerns, and
> >>> I'm wondering if you folks could help myself identity ongoing work
> >>> to solve these (or alternatives?)
> >> Hi Kris. I've been using Ironic in various forms for a while, and I can
> >> answer a few of these things.
> >>
> >>> List of concerns with ironic:
> >>>
> >>> 1.)Nova<->  ironic interactions are generally seem terrible?
> >> I don't know if I'd call it terrible, but there's friction. Things that
> >> are unchangable on hardware are just software configs in vms (like mac
> >> addresses, overlays, etc), and things that make no sense in VMs are
> >> pretty standard on servers (trunked vlans, bonding, etc).
> >>
> >> One way we've gotten around it is by using Ironic standalone via
> >> Bifrost[1]. This deploys Ironic in wide open auth mode on 127.0.0.1,
> >> and includes playbooks to build config drives and deploy images in a
> >> fairly rudimentary way without Nova.
> >>
> >> I call this the "better than Cobbler" way of getting a toe into the
> >> Ironic waters.
> >>
> >> [1] https://github.com/openstack/bifrost
> > Out of curiosity, why ansible vs turning
> > https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py
> > (or something like it) into a tiny-wsgi-app (pick useful name here) that
> > has its own REST api (that looks pretty similar to the public functions
> > in that driver file)?
>  That's an interesting idea. I think a reason Bifrost doesn't just import
>  nova virt drivers is that they're likely _not_ a supported public API
>  (despite not having _'s at the front). Also, a lot of the reason Bifrost
>  exists is to enable users to get the benefits of all the baremetal
>  abstraction work done in Ironic without having to fully embrace all of
>  OpenStack's core. So while you could get a little bit of the stuff from
>  nova (like config drive building), you'd still need to handle network
>  address assignment, image management, etc. etc., and pretty soon you
>  start having to run a tiny glance and a tiny neutron. The Bifrost way
>  is the opposite: I just want a tiny Ironic, and _nothing_ else.
> 
> >>> Ya, I'm just thinking that at a certain point
> >
> > You've got two statements in here, which I'm going to reply to separately:
> >
> >> Oops forgot to fill this out, was just thinking that at a certain point it 
> >> might
> >> be easier to figure out how to extract that API (meh, if its public or 
> >> private)
> >
> > The nova-core team has repeatedly stated that they do not have plans to 
> > support
> > the nova virt driver API as a stable or externally-consumable python API.
> > Changing that would affect a lot more than Ironic (eg. defcore). A change 
> > like
> > that is not just about what is easier for developers, but also what is 
> > better
> > for the community.
> >
> 
> Right, I'm starting to come to the belief that what is better for the 
> community is to change this; because from what I can tell (from my view 
> of the world) that tying all the things to nova has really been 
> detrimental (to a degree) to the whole progression of the cloud as a
> whole.

When I have heard the statement made about the virt driver API it has
simply been that it is not stable and there are no plans at this point
to make it stable. My own opinion is that it is sometimes painful to use
from within Nova and I do not wish to expose that pain to others. There
are changes I would like to see made before it could be considered
externally consumable.


> 
> It's an opinionated thing to say, yes I understand that, but there 
> becomes a point where I feel we need to re-evaluate what people really 
> care about from openstack (because I start to believe that treating the 
> whole thing as a single product, well that went out the window a long 
> time ago, with the creation of the big-tent by the TC, with the creation 
> of mesos, k8s and others by other companies not in openstack...); and 
> really what's left after that is a bunch of services that to survive (as 
> a useful set of services) must accept that there is more than just 
> openstack in the wider world (ie, kubernetes, mesos, 
> the-next-best-thing...) and if we don't start embracing those other 
> communities (and no that doesn't mean be an `integration engine` on-top 
> or around them) then we are pretty much obsoleting ourselv

Re: [openstack-dev] [Nova] State machines in Nova

[Andrew Laski]

 
 
 
On Wed, Jun 1, 2016, at 06:06 AM, Timofei Durakov wrote:
> From my sight, I concerned proposed transition from option #1 to
> option #2.
> because it would be quite big change. So I wonder, has any
> component team
> implemented such transition. Open questions:
>  * upgrades story potential issues
>  * dealing with clients(?)
>  * promoting state machine from verification of states to conductor of
>the task(success stories)
 
I would also be interested in hearing post mortems from projects that
have done this.
 
It would be a big change to transition from #1 to #2 but I don't think
there's any less work involved to just do #2 first. Formalizing the
states we want and adding logic around that has to take place in either
option. I see option 1 as a small chunk of option 2, not an alternative.
 
 
> Timofey
>
> On Wed, Jun 1, 2016 at 12:51 PM, Miles Gould
>  wrote:
>> On 31/05/16 21:03, Timofei Durakov wrote:
>>> there is blueprint[1] that was approved during Liberty and
>>> resubmitted to Newton(with spec[2]). The idea is to define state
>>> machines for operations as live-migration, resize, etc. and to deal
>>> with them operation states.
>>
>> +1 to introducing an explicit state machine - IME they make complex
>> logic much easier to reason about. However, think carefully about how
>> you'll make changes to that state machine later. In Ironic, this is
>> an ongoing problem: every time we change the state machine, we have
>> to decide whether to lie to older clients (and if so, what lie to
>> tell them), or whether to present them with the truth (and if so, how
>> badly they'll break). AIUI this would be a much smaller problem if
>> we'd considered this possibility carefully at the beginning.
>>
>>  Miles
>>
>>
>> ___-
>> ___
>>  OpenStack Development Mailing List (not for usage questions)
>>  Unsubscribe: OpenStack-dev-
>>  requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

[Andrew Laski]

On Wed, Jun 1, 2016, at 05:51 AM, Miles Gould wrote:
> On 31/05/16 21:03, Timofei Durakov wrote:
> > there is blueprint[1] that was approved during Liberty and resubmitted
> > to Newton(with spec[2]).
> > The idea is to define state machines for operations as live-migration,
> > resize, etc. and to deal with them operation states.
> 
> +1 to introducing an explicit state machine - IME they make complex 
> logic much easier to reason about. However, think carefully about how 
> you'll make changes to that state machine later. In Ironic, this is an 
> ongoing problem: every time we change the state machine, we have to 
> decide whether to lie to older clients (and if so, what lie to tell 
> them), or whether to present them with the truth (and if so, how badly 
> they'll break). AIUI this would be a much smaller problem if we'd 
> considered this possibility carefully at the beginning.

This is a great point. I think most people have an implicit assumption
that the state machine will be exposed to end users via the API. I would
like to avoid that for exactly the reason you've mentioned. Of course
we'll want to expose something to users but whatever that is should be
loosely coupled with the internal states that actually drive the system. 


> 
> Miles
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

[Andrew Laski]

On Tue, May 31, 2016, at 04:26 PM, Joshua Harlow wrote:
> Timofei Durakov wrote:
> > Hi team,
> >
> > there is blueprint[1] that was approved during Liberty and resubmitted
> > to Newton(with spec[2]).
> > The idea is to define state machines for operations as live-migration,
> > resize, etc. and to deal with them operation states.
> > The spec PoC patches are overall good. At the same time I think is will
> > be good to get agreement on the usage of state-machines in Nova.
> > There are 2 options:
> >
> >   * implement proposed change and use state machines to deal with states
> > only
> 
> I think this is what could be called the ironic equivalent correct?
> 
> In ironic @ 
> https://github.com/openstack/ironic/blob/master/ironic/common/states.py 
> the state machine here is used to ensure proper states are transitioned 
> over and no invalid/unexpected state transitions happen. The code though 
> itself still runs in a implicit fashion and afaik only interacts with 
> the state machine as a side-effect of actions occurring (instead of the 
> reverse where the state machine itself is 'driving' those actions to 
> happen/to completion).

Yes. This exists in a limited form already in Nova for instances and
task_states.

> 
> >   o procs:
> >   + could be implemented/merged right now
> >   + cleans up states for migrations
> >   o cons:
> >   + state machine only deal with states, and it will be hard to
> > build on top of it task API, as bp [1] was designed for
> > another thing.
> >
> >   * use state machines in Task API(which I'm going to work on during
> > next release):
> 
> So this would be the second model described above, where the state 
> machine (or set of state machines) itself (together could be formed into 
> a action plan, or action workflow or ...) would be the 'entity' 
> realizing a given action and ensuring that it is performed until 
> completed (or tracking where it was paused and such); is that correct?
> 
> >   o procs:
> >   + Task API will orchestrate and deal with long running tasks
> >   + usage state-machines could help with actions
> > rollbacks/retries/etc.
> >   o cons:
> >   + big amount of work
> >   + requires time.
> >
> > I'd like to discuss these options in this thread.
> 
> It seems like one could progress from the first model to the second one, 
> although that kind of progression would still be large (because if my 
> understanding is correct the control of who runs what has to be given 
> over to something else in the second model, similar to the control a 
> taskflow engine or mistral engine has over what it runs); said control 
> means that certain programming models may not map so well (from what I 
> have seen).

I think working through this as a progression from the first model to
the second one would be the best plan. Start with formalizing the states
and their allowed transitions and add checking and error handling around
that. Then work towards handing off control to an engine that could
drive the operation.

> 
> >
> > Timofey
> >
> > [1] -
> > https://blueprints.launchpad.net/openstack/?searchtext=migration-state-machine
> > [2] - https://review.openstack.org/#/c/320849/
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][qa][ironic][nova] When Nova should mark instance as successfully deleted?

[Andrew Laski]

On Fri, May 27, 2016, at 11:25 AM, Matthew Treinish wrote:
> On Fri, May 27, 2016 at 05:52:51PM +0300, Vasyl Saienko wrote:
> > Lucas, Andrew
> > 
> > Thanks for fast response.
> > 
> > On Fri, May 27, 2016 at 4:53 PM, Andrew Laski  wrote:
> > 
> > >
> > >
> > > On Fri, May 27, 2016, at 09:25 AM, Lucas Alvares Gomes wrote:
> > > > Hi,
> > > >
> > > > Thanks for bringing this up Vasyl!
> > > >
> > > > > At the moment Nova with ironic virt_driver consider instance as
> > > deleted,
> > > > > while on Ironic side server goes to cleaning which can take a while. 
> > > > > As
> > > > > result current implementation of Nova tempest tests doesn't work for
> > > case
> > > > > when Ironic is enabled.
> > >
> > > What is the actual failure? Is it a capacity issue because nodes do not
> > > become available again quickly enough?
> > >
> > >
> > The actual failure is that temepest community doesn't want to accept 1
> > option.
> > https://review.openstack.org/315422/
> > And I'm not sure that it is the right way.
> 
> No Andrew is right, this is a resource limitation in the gate. The
> failures
> you're hitting are caused by resource constraints in the gate and not
> having
> enough available nodes to run all the tests because deleted nodes are
> still
> cleaning (or doing another operation) and aren't available to nova for
> booting
> another guest.
> 
> I -2d that patch because it's a workaround for the fundamental issue here
> and
> not actually an appropriate change for Tempest. What you've implemented
> in that
> patch is the equivalent of talking to libvirt or some other hypervisor
> directly
> to find out if something is actually deleted. It's a layer violation,
> there is
> never a reason that should be necessary especially in a test of the nova
> api.
> 
> > 
> > > >
> > > > > There are two possible options how to fix it:
> > > > >
> > > > >  Update Nova tempest test scenarios for Ironic case to wait when
> > > cleaning is
> > > > > finished and Ironic node goes to 'available' state.
> > > > >
> > > > > Mark instance as deleted in Nova only after cleaning is finished on
> > > Ironic
> > > > > side.
> > > > >
> > > > > I'm personally incline to 2 option. From user side successful instance
> > > > > termination means that no instance data is available any more, and
> > > nobody
> > > > > can access/restore that data. Current implementation breaks this rule.
> > > > > Instance is marked as successfully deleted while in fact it may be not
> > > > > cleaned, it may fail to clean and user will not know anything about 
> > > > > it.
> > > > >
> > 
> > >
> > > > I don't really like option #2, cleaning can take several hours
> > > > depending on the configuration of the node. I think that it would be a
> > > > really bad experience if the user of the cloud had to wait a really
> > > > long time before his resources are available again once he deletes an
> > > > instance. The idea of marking the instance as deleted in Nova quickly
> > > > is aligned with our idea of making bare metal deployments
> > > > look-and-feel like VMs for the end user. And also (one of) the
> > > > reason(s) why we do have a separated state in Ironic for DELETING and
> > > > CLEANING.
> > >
> > 
> > The resources will be available only if there are other available baremetal
> > nodes in the cloud.
> > User doesn't have ability to track for status of available resources
> > without admin access.
> > 
> > 
> > > I agree. From a user perspective once they've issued a delete their
> > > instance should be gone. Any delay in that actually happening is purely
> > > an internal implementation detail that they should not care about.
> > >
> 
> Delete is an async operation in Nova. There is never any immediacy here
> it
> always takes an indeterminate amount of time between it being issued by
> the user
> and the server actually going away. The disconnect here is that when
> running
> with the ironic driver the server disappears from Nova but the resources
> aren't
> freed back when that happens until the cleaning is done. I'm pr

Re: [openstack-dev] [tempest][qa][ironic][nova] When Nova should mark instance as successfully deleted?

[Andrew Laski]

On Fri, May 27, 2016, at 09:25 AM, Lucas Alvares Gomes wrote:
> Hi,
> 
> Thanks for bringing this up Vasyl!
> 
> > At the moment Nova with ironic virt_driver consider instance as deleted,
> > while on Ironic side server goes to cleaning which can take a while. As
> > result current implementation of Nova tempest tests doesn't work for case
> > when Ironic is enabled.

What is the actual failure? Is it a capacity issue because nodes do not
become available again quickly enough?

> >
> > There are two possible options how to fix it:
> >
> >  Update Nova tempest test scenarios for Ironic case to wait when cleaning is
> > finished and Ironic node goes to 'available' state.
> >
> > Mark instance as deleted in Nova only after cleaning is finished on Ironic
> > side.
> >
> > I'm personally incline to 2 option. From user side successful instance
> > termination means that no instance data is available any more, and nobody
> > can access/restore that data. Current implementation breaks this rule.
> > Instance is marked as successfully deleted while in fact it may be not
> > cleaned, it may fail to clean and user will not know anything about it.
> >
> 
> I don't really like option #2, cleaning can take several hours
> depending on the configuration of the node. I think that it would be a
> really bad experience if the user of the cloud had to wait a really
> long time before his resources are available again once he deletes an
> instance. The idea of marking the instance as deleted in Nova quickly
> is aligned with our idea of making bare metal deployments
> look-and-feel like VMs for the end user. And also (one of) the
> reason(s) why we do have a separated state in Ironic for DELETING and
> CLEANING.

I agree. From a user perspective once they've issued a delete their
instance should be gone. Any delay in that actually happening is purely
an internal implementation detail that they should not care about.

> 
> I think we should go with #1, but instead of erasing the whole disk
> for real maybe we should have a "fake" clean step that runs quickly
> for tests purposes only?
> 
> Cheers,
> Lucas
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] determining or clarifying a path for gabbi+nova

[Andrew Laski]

On Wed, May 25, 2016, at 11:13 AM, Chris Dent wrote:
> 
> Earlier this year I worked with jaypipes to compose a spec[1] for using
> gabbi[2] with nova. Summit rolled around and there were some legitimate
> concerns about the focus of the spec being geared towards replacing the
> api sample tests. I wasn't at summit ☹ but my understanding of the
> outcome of the discussion was (please correct me if I'm wrong):
> 
> * gabbi is not a straight replacement for the api-samples (notably
>it doesn't address the documentation functionality provided by
>api-samples)
> 
> * there are concerns, because of the style of response validation
>that gabbi does, that there could be a coverage gap[3] when a
>representation changes (in, for example, a microversion bump)
> 
> * we'll see how things go with the placement API work[4], which uses
>gabbi for TDD, and allow people to learn more about gabbi from
>that
> 
> Since that all seems to make sense, I've gone ahead and abandoned
> the review associated with the spec as overreaching for the time
> being.
> 
> I'd like, however, to replace it with a spec that is somewhat less
> reaching in its plans. Rather than replace api-samples with gabbi,
> augment existing tests of the API with gabbi-based tests. I think
> this is a useful endeavor that will find and fix inconsistencies but
> I'd like to get some feedback from people so I can formulate a spec
> that will actually be useful.
> 
> For reference, I started working on some integration of tempest and
> gabbi[5] (based on some work that Mehdi did), and in the first few
> minutes of writing tests found and reported bugs against nova and
> glance, some of which have even been fixed since then. Win! We like
> win.
> 
> The difficulty here, and the reason I'm writing this message, is
> simply this: The biggest benefit of gabbi is the actual writing and
> initial (not the repeated) running of the tests. You write tests, you
> find bugs and inconsistencies. The second biggest benefit is going
> back and being a human and reading the tests and being able to see
> what the API is doing, request and response in the same place. That's
> harder to write a spec about than "I want to add or change feature X".
> There's no feature here.

After reading this my first thought is that gabbi would handle what I'm
testing in
https://review.openstack.org/#/c/263927/33/nova/tests/functional/wsgi/test_servers.py,
or any of the other tests in that directory. Does that seem accurate?
And what would the advantage of gabbi be versus what I have currently
written?


> 
> I'm also aware that there is concern about adding yet another thing to
> understand in the codebase.
> 
> So what's a reasonable course of action here?
> 
> Thanks.
> 
> P.S: If any other project is curious about using gabbi, it is easier
> to use and set up than this discussion is probably making it sound
> and extremely capable. If you want to try it and need some help,
> just ask me: cdent on IRC.
> 
> [1] https://review.openstack.org/#/c/291352/
> 
> [2] https://gabbi.readthedocs.io/
> 
> [3] This would be expected: Gabbi considers its job to be testing
> the API layer, not the serializers and object that the API might be
> using (although it certainly can validate those things).
> 
> [4] https://review.openstack.org/#/c/293104/
> 
> [5] http://markmail.org/message/z6z6ego4wqdaelhq
> 
> -- 
> Chris Dent   (╯°□°)╯︵┻━┻http://anticdent.org/
> freenode: cdent tw: @anticdent
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Deprecated options in sample configs?

[Andrew Laski]

 
 
 
On Tue, May 17, 2016, at 03:00 PM, Dolph Mathews wrote:
> I think the metadata_manager is one of many terrible examples of
> deprecated configuration options. The documentation surrounding a
> deprecation should *always* tell you why something is being
> deprecated, and what you should be using instead to achieve the
> same, or better, result moving forward. But instead, we get "you
> don't need to see any documentation... these aren't the configs
> you're looking for."
 
In this case there is no replacement to point towards, Nova is just
removing a plug point. But it is fair to say that the reasoning behind
the removal should be captured in the comment there.
 
>
> If all our deprecated config options provided useful pointers in their
> descriptions, there would be tremendous value in retaining deprecated
> configuration options in sample config files -- in which case, I don't
> believe we would be having this conversation which questions their
> value in the first place.
 
If config sample files are being used as a living document then that
would be a reason to leave the deprecated options in there. In my
experience as a cloud deployer I never once used them in that manner so
it didn't occur to me that people might, hence my question to the list.
 
This may also indicate that people aren't checking release notes as we
hope they are. A release note is where I would expect to find this
information aggregated with all the other changes I should be aware of.
That seems easier to me than aggregating that data myself by checking
various sources.
 
Anyways, I have no strong cause for removing the deprecated options. I
just wondered if it was a low hanging fruit and thought I would ask.
 
 
 
>
> On Tue, May 17, 2016 at 1:49 PM Andrew Laski
>  wrote:
>>
>>
>>
>> On Tue, May 17, 2016, at 02:36 PM, Matt Fischer wrote:
>>> On Tue, May 17, 2016 at 12:25 PM, Andrew Laski 
>>> wrote:
>>>> I was in a discussion earlier about discouraging deployers from
>>>> using
>>>> deprecated options and the question came up about why we put
>>>> deprecated
>>>> options into the sample files generated in the various projects.
>>>> So, why
>>>> do we do that?
>>>>
>>>> I view the sample file as a reference to be used when setting up a
>>>> service for the first time, or when looking to configure
>>>> something for
>>>> the first time. In neither of those cases do I see a benefit to
>>>> advertising options that are marked deprecated.
>>>>
>>>> Is there some other case I'm not considering here? And how does
>>>> everyone
>>>> feel about modifying the sample file generation to exclude options
>>>> which
>>>> are marked with "deprecated_for_removal"?
>>>>
>>>
>>>
>>> Can you clarify what you mean by having them? The way they are now
>>> is great for deployers I think and people (like me) who work on
>>> things like puppet and need to update options sometimes. For
>>> example, I like this way, example from keystone:
>>>
>>> # Deprecated group/name - [DEFAULT]/log_config
>>> #log_config_append = 
>>>
>>> Are you proposing removing that top line?
>>
>> That is a different type of deprecation which I didn't do a great job
>> of distinguishing.
>>
>> There is deprecation of where a config option is defined, as in your
>> example. I am not proposing to touch that at all. That simply
>> indicates that a config option used to be in a different group or
>> used to be named something else. That's very useful.
>>
>> There is deprecation of a config option in the sense that it is going
>> away completely. An example would be:
>>
>> # DEPRECATED: OpenStack metadata service manager (string value)
>> # This option is deprecated for removal.
>> # Its value may be silently ignored in the future.
>> #metadata_manager = nova.api.manager.MetadataManager
>>
>> I'm wondering if anyone sees a benefit to including that in the
>> sample file when it is clearly not meant for use.
>>
>>
>>> ___-
>>> _
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-
>>> requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> ___-
>> __

Re: [openstack-dev] [all] Deprecated options in sample configs?

[Andrew Laski]

 
 
 
On Tue, May 17, 2016, at 02:36 PM, Matt Fischer wrote:
> On Tue, May 17, 2016 at 12:25 PM, Andrew Laski
>  wrote:
>> I was in a discussion earlier about discouraging deployers from using
>>  deprecated options and the question came up about why we put
>>  deprecated
>>  options into the sample files generated in the various projects.
>>  So, why
>>  do we do that?
>>
>>  I view the sample file as a reference to be used when setting up a
>>  service for the first time, or when looking to configure
>>  something for
>>  the first time. In neither of those cases do I see a benefit to
>>  advertising options that are marked deprecated.
>>
>>  Is there some other case I'm not considering here? And how does
>>  everyone
>>  feel about modifying the sample file generation to exclude
>>  options which
>>  are marked with "deprecated_for_removal"?
>>
>
>
> Can you clarify what you mean by having them? The way they are now is
> great for deployers I think and people (like me) who work on things
> like puppet and need to update options sometimes. For example, I like
> this way, example from keystone:
>
> # Deprecated group/name - [DEFAULT]/log_config
> #log_config_append = 
>
> Are you proposing removing that top line?
 
That is a different type of deprecation which I didn't do a great job of
distinguishing.
 
There is deprecation of where a config option is defined, as in your
example. I am not proposing to touch that at all. That simply indicates
that a config option used to be in a different group or used to be named
something else. That's very useful.
 
There is deprecation of a config option in the sense that it is going
away completely. An example would be:
 
# DEPRECATED: OpenStack metadata service manager (string value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
#metadata_manager = nova.api.manager.MetadataManager
 
I'm wondering if anyone sees a benefit to including that in the sample
file when it is clearly not meant for use.
 
 
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Deprecated options in sample configs?

[Andrew Laski]

I was in a discussion earlier about discouraging deployers from using
deprecated options and the question came up about why we put deprecated
options into the sample files generated in the various projects. So, why
do we do that?

I view the sample file as a reference to be used when setting up a
service for the first time, or when looking to configure something for
the first time. In neither of those cases do I see a benefit to
advertising options that are marked deprecated.

Is there some other case I'm not considering here? And how does everyone
feel about modifying the sample file generation to exclude options which
are marked with "deprecated_for_removal"?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed Database

[Andrew Laski]

On Tue, May 3, 2016, at 08:05 PM, Mark Doffman wrote:
> This thread has been a depressing read.
> 
> I understand that the content is supposed to be distributed databases 
> but for me it has become an inquisition of cellsV2.
> 
> Our question has clearly become "Should we continue efforts on 
> cellsV2?", which I will address head-on.
> 
> We shouldn't be afraid to abandon CellsV2. If there are designs that are 
> proven to be a better solution then our current momentum shouldn't keep 
> us from an abrupt change. As someone who is working on this I have an 
> attachment to the current design, but Its important for me to keep an 
> open mind.
> 
> Here are my *main* reasons for continuing work on CellsV2.
> 
> 1. It provides a proven solution to an immediate message queue problem.
> 
> Yes CellsV2 is different to CellsV1, but the previous solution showed 
> that application-level sharding of the message queue can work. CellsV2 
> provides this solution with a (moderately) easy upgrade path for 
> existing deployments. These deployments may not be comfortable with 
> changing MQ technologies or may already be using CellsV1. Application 
> level sharding of the message queue is not pretty, but will work.
> 
> 2. The 'complexity' of CellsV2 is vastly overstated.
> 
> Sure there is a-lot of *work* to do for cellsv2, but this doesn't imply 
> increased complexity: any refactoring requires work. CellsV1 added 
> complexity to our codebase, Cellsv2 does not. In-fact by clearly 
> separating data that is 'owned'by the different services we have I 
> believe that we are improving the modularity and encapsulation present 
> in Nova.
> 
> 3. CellsV2 does not prohibit *ANY* of the alternative scaling methods
> mentioned in this thread.
> 
> Really, it doesn't. Both message queue and database switching are 
> completely optional. Both in the sense of running a single cell, and 
> even when running multiple cells. If anything, the ability to run 
> separate message queues and database connections could give us the 
> ability to trial these alternative technologies within a real, running, 
> cloud.
> 
> Just imagine the ability to set up a cell in your existing cloud that 
> runs 0mq rather than rabbit. How about a NewSQL database integrated in 
> to an existing cloud? Both of these things may (With some work) be
> possible.
> 
> 
> 
> I could go on, but I won't. These are my main reasons and I'll stick to 
> them.
> 
> Its difficult to be proven wrong, but sometimes necessary to get the 
> best product that we can. I don't think that the existence of 
> alternative message queue and database options is enough to stop cellsV2 
> work now. A proven solution, that meets the upgrade constraints that we 
> have in Nova, would be a good reason to do so. We should of-course 
> explore other options, nothing we are doing prevents that. When they 
> work out, I'll be super excited.

Thank you for writing this. You have eloquently described the situation
and I completely agree.


> 
> Thanks
> 
> Mark
> 
> On 4/29/16 12:53 AM, Clint Byrum wrote:
> > Excerpts from Mike Bayer's message of 2016-04-28 22:16:54 -0500:
> >>
> >> On 04/28/2016 08:25 PM, Edward Leafe wrote:
> >>
> >>> Your own tests showed that a single RDBMS instance doesn’t even break a 
> >>> sweat
> >>> under your test loads. I don’t see why we need to shard it in the first
> >>> place, especially if in doing so we add another layer of complexity and
> >>> another dependency in order to compensate for that choice. Cells are a 
> >>> useful
> >>> concept, but this proposed implementation is adding way too much 
> >>> complexity
> >>> and debt to make it worthwhile.
> >>
> >> now that is a question I have also.  Horizontal sharding is usually for
> >> the case where you need to store say, 10B rows, and you'd like to split
> >> it up among different silos.  Nothing that I've seen about Nova suggests
> >> this is a system with any large data requirements, or even medium size
> >> data (a few million rows in relational databases is nothing).I
> >> didn't have the impression that this was the rationale behind Cells, it
> >> seems like this is more of some kind of logical separation of some kind
> >> that somehow suits some environments (but I don't know how).
> >> Certainly, if you're proposing a single large namespace of data across a
> >> partition of nonrelational databases, and then the data size itself is
> >> not that large, as long as "a single namespace" is appropriate then
> >> there's no reason to break out of more than one MySQL database.  There's
> >> not much reason to transparently shard unless you are concerned about
> >> adding limitless storage capacity.   The Cells sharding seems to be
> >> intentionally explicit and non-transparent.
> >>
> >
> > There's a bit more to it than the number of rows. There's also a desire
> > to limit failure domains. IMO, that is entirely unfounded, as I've run
> > thousands of servers that depended on a single pair

Re: [openstack-dev] [nova] Distributed Database

[Andrew Laski]

On Mon, May 2, 2016, at 01:13 PM, Edward Leafe wrote:
> On May 2, 2016, at 10:51 AM, Mike Bayer  wrote:
> 
> >> Concretely, we think that there are three possible approaches:
> >> 1) We can use the SQLAlchemy API as the common denominator between a 
> >> relational and non-relational implementation of the db.api component. 
> >> These two implementation could continue to converge by sharing a large 
> >> amount of code.
> >> 2) We create a new non-relational implementation (from scratch) of the 
> >> db.api component. It would require probably more work.
> >> 3) We are also studying a last alternative: writing a SQLAlchemy 
> >> engine that targets NewSQL databases (scalability + ACID):
> >>  - https://github.com/cockroachdb/cockroach
> >>  - https://github.com/pingcap/tidb
> > 
> > Going with a NewSQL backend is by far the best approach here.   That way, 
> > very little needs to be reinvented and the application's approach to data 
> > doesn't need to dramatically change.
> 
> I’m glad that Matthieu responded, but I did want to emphasize one thing:
> of *course* this isn’t an ideal approach, but it *is* a practical one.
> The biggest problem in any change like this isn’t getting it to work, or
> to perform better, or anything else except being able to make the change
> while disrupting as little of the existing code as possible. Taking an
> approach that would be more efficient would be a non-starter since it
> wouldn’t provide a clean upgrade path for existing deployments.

I would like to point out that this same logic applies to the current
cellsv2 effort. It is a very practical set of changes which allows Nova
to move forward with only minor effort on the part of deployers. And it
moves towards a model that is already used and well understood by large
deployers of Nova while also learning from the shortcomings of the
previous architecture. In short, much of this is already battle tested
and proven.

If we started Nova from scratch, I hear golang is lovely for this sort
of thing, would we do things differently? Probably. However that's not
the position we're in. And we're able to make measurable progress with
cellsv2 at the moment and have a pretty clear idea of the end state. I
can recall conversations about NoSQL as far back as the San Diego
summit, which was my first so I can't say they didn't happen previously,
and this is the first time I've seen any measurable progress on moving
forward with it. But where it would go is not at all clear.

I also want to point out that what was being solved with ROME and what
cellsv2 is solving are two very different things. I saw the talk and was
very impressed, but it was looking to improve upon db access times in a
very specific deployment type. And I didn't get the sense that the point
being made was that ROME/redis was the best solution generally, but for
very geographically distributed controllers with a shared database it
performed much better than an active/active Galera cluster with a large
number of nodes.


> 
> By getting this working without ripping out all of the data models that
> currently exist is an amazing feat. And if by doing so it shows that a
> distributed database is indeed possible, it’s done more than anything
> else that has ever been discussed in the past few years. 
> 
> 
> -- Ed Leafe
> 
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Austin summit cells v2 session recap

[Andrew Laski]

Thanks for the summary, this is great. Comments inline.


On Mon, May 2, 2016, at 09:32 PM, Matt Riedemann wrote:
> Andrew Laski led a double session for cells v2 on Wednesday afternoon. 
> The full session etherpad is here [1].
> 
> Andrew started with an overview of what's done and what's in progress. 
> Note that some of the background on cells, what's been completed for 
> cells v2 and what's being worked on is also in a summit video from a 
> user conference talk that Andrew gave [2].
> 
> We agreed to add the MQ switching [3] to get_cell_client and see what, 
> if anything, breaks.
> 
> DB migrations
> -
> 
> We had a quick rundown on the database tables slated for migration to 
> the API database. Notable items for the DB table migrations:
> 
> * Aggregates and quotas will be migrated, there are specs up for both of 
> these from Mark Doffman.
> * The nova-network related tables won't be migrated since we've 
> deprecated nova-network.
> * The agent_builds table won't be migrated. We plan on deprecating this 
> API since it's only used by the XenAPI virt driver and it sounds like 
> Rackspace doesn't even use/enable it.
> * We have to figure out what to do about the certificates table. The 
> only thing using it is the os-certificates REST API and nova-cert 
> service, and nothing in tree is using either of those now. The problem 
> is, the ec2api repo on GitHub is using the nova-cert rpc api directly 
> for s3 image download. So we need to figure out if we can move that into 
> the ec2api repo and drop it from Nova or find some other solution.
> * keypairs will be migrated to the API DB. There was a TODO about 
> needing to store the keypair type in the instance. I honestly can't 
> remember exactly what that was for now, I'm hoping Andrew remembers.

The metadata api exposes the keypair type but that information is not
passed down during the boot request. Currently the metadata service is
pulling the keypair from the db on each access, and for cellsv1 making
an RPC request to the parent cell for that data. To avoid requiring the
metadata service to query the nova_api database we should pass down
keypair information and persist it with the instance, perhaps in
instance_extra, so that lookups can be done locally to the cell.


> * We agreed to move instance_groups and instance_group_policy to the API 
> DB, but there is a TODO to sort out if instance_group_members should be 
> in the API DB.
> 
> nova-network
> 
> 
> For nova-network we agreed that we'll fail hard if someone tries to add 
> a second cell to a cells v2 deployment and they aren't using Neutron.
> 
> Testing
> ---
> 
> Chuck Carmack is working on some test plans for cells v2. There would be 
> a multi-node/cell job where one node is running the API and cell0 and 
> another is running a regular nova cell. There would also be migration 
> testing as part of grenade.
> 
> Documentation
> -
> 
> We discussed what needs to be documented and where it should live.
> 
> Since all deployments will at least be a cell of one, setting that up 
> will be in the install guide in docs.o.o. A multi-cell deployment would 
> be documented in the admin guide.
> 
> Anything related to the call path flow for different requests would live 
> in the nova developer documentation (devref).
> 
> Pagination
> --
> 
> This took a significant portion of the second cells v2 session and is 
> one of the more complicated problems to sort out. There are problems 
> with listing all instances across all cells especially when we support 
> sorting. And we really have a latent bug in the API since we never 
> restricted the list of valid sort keys for listing instances, so you can 
> literally sort on anything in the instances table in the DB.
> 
> There were some ideas about how to handle this:
> 
> 1. Don't support sorting in the API if you have multiple cells. Leave it 
> up to the caller to sort the results on their own. Obviously this isn't 
> a great solution for existing users that rely on this in the API.
> 
> 2. Each cell sorts the results individually, and the API merge sorts the 
> results from the cells. There is still overhead here.
> 
> 3. Don't split the database, or use a distributed database like Redis. 
> Since this wasn't brought up in person in the session, or on Friday, it 
> wasn't discussed. There is another thread about this though [4].
> 
> 4. Use the OpenStack Searchlight project for doing massive queries like 
> this. This would be optional for a cell of one but recommended/required 
> for anyone running multiple cells. The downside to this is i

Re: [openstack-dev] [nova] Distributed Database

[Andrew Laski]

On Fri, Apr 22, 2016, at 04:27 PM, Ed Leafe wrote:
> OK, so I know that Friday afternoons are usually the worst times to
> write a blog post and start an email discussion, and that the Friday
> immediately before a Summit is the absolute worst, but I did it anyway.
> 
> http://blog.leafe.com/index.php/2016/04/22/distributed_data_nova/
> 
> Summary: we are creating way too much complexity by trying to make Nova
> handle things that are best handled by a distributed database. The
> recent split of the Nova DB into an API database and separate cell
> databases is the glaring example of going down the wrong road.
> 
> Anyway, read it on your flight (or, in my case, drive) to Austin, and
> feel free to pull me aside to explain just how wrong I am. ;-)

I agree with a lot of what Monty wrote in his response. And agree that
given a greenfield there are much better approaches that could be taken
rather than partitioning the database.

However I do want to point out that cells v2 is not just about dealing
with scale in the database. The message queue is another consideration,
and as far as I know there is not an analog to the "distributed
database" option available for the persistence layer.

Additionally with v1 we found that deployers have enjoyed being able to
group their hardware with cells. Baremetal goes in this cell, SSD filled
computes over here, and spinning disks over there. And beyond that
there's the ability to create a cell, fill it with hardware, and then
test it without plugging it up to the production API. Cells provides an
entry point for poking at things that isn't available without it.

I don't want to get too sidetracked on talking about cells. I just
wanted to point out that cells v2 did not come to fruition due to a fear
of distributed databases.

> 
> 
> -- 
> 
> -- Ed Leafe
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Nova quota statistics counting issue

[Andrew Laski]

 
 
 
On Wed, Apr 13, 2016, at 12:27 PM, Dmitry Stepanenko wrote:
> Hi Team,
> I worked on nova quota statistics issue
> (https://bugs.launchpad.net/nova/+bug/1284424) happenning when nova-*
> processes are restarted during removing instances and was able to
> reproduce it. For repro I used devstack and started nova-api and nova-
> compute in separate screen windows. For killing them I used ctrl+c. As
> I found this issue happened if nova-* processes are killed after
> instance was deleted but right before quota commit procedure finishes.
> We discussed these results with Markus Zoeller and decided that even
> though killing nova processes is a bit exotic event, this still
> should be fixed because quotas counting affects billing and very
> important for us.
 
+1. This is very important to get right. And while killing Nova
processes is exotic during normal operation it could happen for upgrades
and that should not cause quota issues.
 
> So, we need to introduce some mechanism that will prevent us from
> reaching inconsistent states in terms of quotas. In other words, this
> mechanism should work in such a way that both instance create/remove
> operation and quota usage recount operation happened or not happened
> together.
 
There's been some discussion around this, and there are other ML threads
somewhat discussing it in the context of moving quota enforcement into a
centralized service/library. There are a couple of approaches that could
be taken for tackling quotas, but a larger issue is that we have no good
way of knowing if some change helps the situation. What we need before
making any changes  is a functional test that reproduces the issue.
 
Once that is in place I would love to see the removal of the
quota_usages table and reservations and have quota be based on actual
usage represented in the instances table. But there are a lot of other
viewpoints and I think work in this area is going to have to start
making small incremental improvements.
 
 
> Any ideas how to do that properly?
> Kind regards,
> Dmitry
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposing Andrey Kurilin for python-novaclient core

[Andrew Laski]

+1

On Wed, Apr 13, 2016, at 01:53 PM, Matt Riedemann wrote:
> I'd like to propose that we make Andrey Kurilin core on
> python-novaclient.
> 
> He's been doing a lot of the maintenance the last several months and a 
> lot of times is the first to jump on any major issue, does a lot of the
> microversion work, and is also working on cleaning up docs and helping 
> me with planning releases.
> 
> His work is here [1].
> 
> Review stats for the last 4 months (although he's been involved in the 
> project longer than that) [2].
> 
> Unless there is disagreement I plan to make Andrey core by the end of 
> the week.
> 
> [1] 
> https://review.openstack.org/#/q/owner:akurilin%2540mirantis.com+project:openstack/python-novaclient
> [2] http://stackalytics.com/report/contribution/python-novaclient/120
> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cross-project] [all] Quotas and the need for reservation

[Andrew Laski]

On Tue, Apr 5, 2016, at 09:57 AM, Ryan McNair wrote:
> >It is believed that reservation help to to reserve a set of resources
> >beforehand and hence eventually preventing any other upcoming request
> >(serial or parallel) to exceed quota if because of original request the
> >project might have reached the quota limits.
> >
> >Questions :-
> >1. Does reservation in its current state as used by Nova, Cinder, Neutron
> >help to solve the above problem ?
> 
> In Cinder the reservations are useful for grouping quota
> for a single request, and if the request ends up failing
> the reservation gets rolled back. The reservations also
> rollback automatically if not committed within a certain
> time. We also use reservations with Cinder nested quotas
> to group a usage request that may propagate up to a parent
> project in order to manage commit/rollback of the request
> as a single unit.
> 
> >
> >2. Is it consistent, reliable ?  Even with reservation can we run into
> >in-consistent behaviour ?
> 
> Others can probably answer this better, but I have not
> seen the reservations be a major issue. In general with
> quotas we're not doing the check and set atomically which
> can get us in an inconsistent state with quota-update,
> but that's unrelated to the reservations.
> 
> >
> >3. Do we really need it ?
> >
> 
> Seems like we need *some* way of keeping track of usage
> reserved during a particular request and a way to easily
> roll that back at a later time. I'm open to alternatives
> to reservations, just wondering what the big downside of
> the current reservation system is.

Jay goes into it a little bit in his response to another quota thread
http://lists.openstack.org/pipermail/openstack-dev/2016-March/090560.html
and I share his thoughts here.

With a reservation system you're introducing eventual consistency into
the system rather than being strict because reservations are not tied to
a concrete thing. You can't do a point in time check of whether the
reserved resources are going to eventually be used if something happens
like a service restart and a request is lost. You have to have no
activity for the duration of the expiration time to let things settle
before getting a real view of quota usages.

Instead if you tie quota usage to the resource records then you can
always get a view of what's actually in use.

One thing that should probably be clarified in all of these discussion
is what exactly is the quota on. I see two answers: the quota is against
the actual resource usage, or the quota is against the records tracking
usage. Since we currently track quotas with a reservation system I think
it's fair to say that we're not actually tracking against resource like
disk/RAM/CPU being in use. I would consider the resource to be in use as
soon as there's a db record for something like an instance or volume
which indicates that there will be consumption of those resources. What
that means in practice is that quotas can be committed right away and
there doesn't seem to be any need for a reservation system.



> 
> - Ryan McNair (mc_nair)
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] API priorities in Newton

[Andrew Laski]

On Wed, Mar 30, 2016, at 04:47 PM, Adam Young wrote:
> On 03/30/2016 04:16 PM, Andrew Laski wrote:
> >
> > On Wed, Mar 30, 2016, at 03:54 PM, Matt Riedemann wrote:
> >>
> >> On 3/30/2016 2:42 PM, Andrew Laski wrote:
> >>>
> >>> On Wed, Mar 30, 2016, at 03:26 PM, Sean Dague wrote:
> >>>> During the Nova API meeting we had some conversations about priorities,
> >>>> but this feels like the thing where a mailing list conversation is more
> >>>> inclusive to get agreement on things. I think we need to remain focused
> >>>> on what API related work will have the highest impact on our users.
> >>>> (some brain storming was here -
> >>>> https://etherpad.openstack.org/p/newton-nova-api-ideas). Here is a
> >>>> completely straw man proposal on priorities for the Newton cycle.
> >>>>
> >>>> * Top Priority Items *
> >>>>
> >>>> 1. API Reference docs in RST which include microversions (drivers: me,
> >>>> auggy, annegentle) -
> >>>> https://blueprints.launchpad.net/nova/+spec/api-ref-in-rst
> >>>> 2. Discoverable Policy (drivers: laski, claudio) -
> >> Selfishly I'd like Laski to be as focused on cells v2 as possible, but
> >> he does have a spec up related to this.
> > At the midcycle I agreed to look into the oslo.policy work involved with
> > this because I have some experience there. I wasn't planning to be much
> > involved beyond that, and Claudiu has a spec up for the API side of it.
> > But in my mind there's a chain backwards from capabilities API to
> > discoverable policy and I want to move the oslo.policy work ahead
> > quickly if I can to unblock that.
> 
> There is a CLI that does something like what you want already:
> 
> https://review.openstack.org/#/c/170978/
> 
> You basically want a server based version of that that returns all the 
> "true" values.

Exactly.

The shortcoming of the CLI is that not all policies are guaranteed to be
defined in a policy.json file. It's entirely possible for there to be a
policy check with no definition anywhere which will just fall back to a
defined default rule. A big part of my policy
proposal(https://review.openstack.org/#/c/290155/) is to require
policies to be registered in code like configuration options are. This
allows for an exhaustive check against all used policies. And will allow
for policy file generation from services.


> 
> >
> >
> >>>> https://review.openstack.org/#/c/289405/
> >>>> 3. ?? (TBD)
> >>>>
> >>>> I think realistically 3 priority items is about what we can sustain, and
> >>>> I'd like to keep it there. Item #3 has a couple of options.
> >> Agree to keep the priority list as small as possible, because this is
> >> just a part of our overall backlog of priorities.
> >>
> >>>> * Lower Priority Background Work *
> >>>>
> >>>> - POC of Gabbi for additional API validation
> >> I'm assuming cdent would be driving this, and he's also working on the
> >> resource providers stuff for the scheduler, but might be a decent side
> >> project for him to stay sane.
> >>
> >>>> - Microversion Testing in Tempest (underway)
> >> How much coverage do we have today? This could be like novaclient where
> >> people just start hacking on adding tests for each microversion
> >> (assuming gmann would be working on this).
> >>
> >>>> - Some of the API WG recommendations
> >>>>
> >>>> * Things we shouldn't do this cycle *
> >>>>
> >>>> - Tasks API - not because it's not a good idea, but because I think
> >>>> until we get ~ 3 core team members agreed that it's their number #1 item
> >>>> for the cycle, it's just not going to get enough energy to go somewhere
> >>>> useful. There are some other things on deck that we just need to clear
> >>>> first.
> >>> Agreed. I would love to drive this forward but there are just too many
> >>> other areas to focus on right now.
> >> +1
> >>
> >>>
> >>>> - API wg changes for error codes - we should fix that eventually, but
> >>>> that should come as a single microversion to minimize churn. That's
> >>>> coordination we don't really have the bandwidth for this cycle.
> >> +1
> >>
> >>>> * Things we need to deci

Re: [openstack-dev] [nova] API priorities in Newton

[Andrew Laski]

On Wed, Mar 30, 2016, at 03:54 PM, Matt Riedemann wrote:
> 
> 
> On 3/30/2016 2:42 PM, Andrew Laski wrote:
> >
> >
> > On Wed, Mar 30, 2016, at 03:26 PM, Sean Dague wrote:
> >> During the Nova API meeting we had some conversations about priorities,
> >> but this feels like the thing where a mailing list conversation is more
> >> inclusive to get agreement on things. I think we need to remain focused
> >> on what API related work will have the highest impact on our users.
> >> (some brain storming was here -
> >> https://etherpad.openstack.org/p/newton-nova-api-ideas). Here is a
> >> completely straw man proposal on priorities for the Newton cycle.
> >>
> >> * Top Priority Items *
> >>
> >> 1. API Reference docs in RST which include microversions (drivers: me,
> >> auggy, annegentle) -
> >> https://blueprints.launchpad.net/nova/+spec/api-ref-in-rst
> >> 2. Discoverable Policy (drivers: laski, claudio) -
> 
> Selfishly I'd like Laski to be as focused on cells v2 as possible, but 
> he does have a spec up related to this.

At the midcycle I agreed to look into the oslo.policy work involved with
this because I have some experience there. I wasn't planning to be much
involved beyond that, and Claudiu has a spec up for the API side of it.
But in my mind there's a chain backwards from capabilities API to
discoverable policy and I want to move the oslo.policy work ahead
quickly if I can to unblock that.


> 
> >> https://review.openstack.org/#/c/289405/
> >> 3. ?? (TBD)
> >>
> >> I think realistically 3 priority items is about what we can sustain, and
> >> I'd like to keep it there. Item #3 has a couple of options.
> 
> Agree to keep the priority list as small as possible, because this is 
> just a part of our overall backlog of priorities.
> 
> >>
> >> * Lower Priority Background Work *
> >>
> >> - POC of Gabbi for additional API validation
> 
> I'm assuming cdent would be driving this, and he's also working on the 
> resource providers stuff for the scheduler, but might be a decent side 
> project for him to stay sane.
> 
> >> - Microversion Testing in Tempest (underway)
> 
> How much coverage do we have today? This could be like novaclient where 
> people just start hacking on adding tests for each microversion 
> (assuming gmann would be working on this).
> 
> >> - Some of the API WG recommendations
> >>
> >> * Things we shouldn't do this cycle *
> >>
> >> - Tasks API - not because it's not a good idea, but because I think
> >> until we get ~ 3 core team members agreed that it's their number #1 item
> >> for the cycle, it's just not going to get enough energy to go somewhere
> >> useful. There are some other things on deck that we just need to clear
> >> first.
> >
> > Agreed. I would love to drive this forward but there are just too many
> > other areas to focus on right now.
> 
> +1
> 
> >
> >
> >> - API wg changes for error codes - we should fix that eventually, but
> >> that should come as a single microversion to minimize churn. That's
> >> coordination we don't really have the bandwidth for this cycle.
> 
> +1
> 
> >>
> >> * Things we need to decide this cycle *
> >>
> >> - When are we deleting the legacy v2 code base in tree?
> 
> Do you have some high-level milestone thoughts here? I thought there was 
> talk about not even thinking about this until Barcelona?
> 
> >>
> >> * Final priority item *
> >>
> >> For the #3 priority item one of the things that came up today was the
> >> structured errors spec by the API working group. That would be really
> >> nice... but in some ways really does need the entire new API reference
> >> docs in place. And maybe is better in O.
> >>
> >> One other issue that we've been blocking on for a while has been
> >> Capabilities discovery. Some API proposed adds like live resize have
> >> been conceptually blocked behind this one. Once upon a time there was a
> >> theory that JSON Home was a thing, and would slice our bread and julien
> >> our fries, and solve all this. But it's a big thing to get right, and
> >> JSON Home has an unclear future. And, we could server our users pretty
> >> well with a much simpler take on capabilities. For instance
> >>
> >>   GET /servers/{id}/capabilities
> >>
> >> {
> >>  "capabilitie

Re: [openstack-dev] [nova] API priorities in Newton

[Andrew Laski]

On Wed, Mar 30, 2016, at 03:26 PM, Sean Dague wrote:
> During the Nova API meeting we had some conversations about priorities,
> but this feels like the thing where a mailing list conversation is more
> inclusive to get agreement on things. I think we need to remain focused
> on what API related work will have the highest impact on our users.
> (some brain storming was here -
> https://etherpad.openstack.org/p/newton-nova-api-ideas). Here is a
> completely straw man proposal on priorities for the Newton cycle.
> 
> * Top Priority Items *
> 
> 1. API Reference docs in RST which include microversions (drivers: me,
> auggy, annegentle) -
> https://blueprints.launchpad.net/nova/+spec/api-ref-in-rst
> 2. Discoverable Policy (drivers: laski, claudio) -
> https://review.openstack.org/#/c/289405/
> 3. ?? (TBD)
> 
> I think realistically 3 priority items is about what we can sustain, and
> I'd like to keep it there. Item #3 has a couple of options.
> 
> * Lower Priority Background Work *
> 
> - POC of Gabbi for additional API validation
> - Microversion Testing in Tempest (underway)
> - Some of the API WG recommendations
> 
> * Things we shouldn't do this cycle *
> 
> - Tasks API - not because it's not a good idea, but because I think
> until we get ~ 3 core team members agreed that it's their number #1 item
> for the cycle, it's just not going to get enough energy to go somewhere
> useful. There are some other things on deck that we just need to clear
> first.

Agreed. I would love to drive this forward but there are just too many
other areas to focus on right now.


> - API wg changes for error codes - we should fix that eventually, but
> that should come as a single microversion to minimize churn. That's
> coordination we don't really have the bandwidth for this cycle.
> 
> * Things we need to decide this cycle *
> 
> - When are we deleting the legacy v2 code base in tree?
> 
> * Final priority item *
> 
> For the #3 priority item one of the things that came up today was the
> structured errors spec by the API working group. That would be really
> nice... but in some ways really does need the entire new API reference
> docs in place. And maybe is better in O.
> 
> One other issue that we've been blocking on for a while has been
> Capabilities discovery. Some API proposed adds like live resize have
> been conceptually blocked behind this one. Once upon a time there was a
> theory that JSON Home was a thing, and would slice our bread and julien
> our fries, and solve all this. But it's a big thing to get right, and
> JSON Home has an unclear future. And, we could server our users pretty
> well with a much simpler take on capabilities. For instance
> 
>  GET /servers/{id}/capabilities
> 
> {
> "capabilities" : {
> "resize": True,
> "live-resize": True,
> "live-migrate": False
> ...
>  }
> }
> 
> Effectively an actions map for servers. Lots of details would have to be
> sorted out on this one, clearly needs a spec, however I think that this
> would help unstick some other things people would like in Nova, without
> making our interop story terrible. This would need a driver for this
> effort.

I think this ties directly into the discoverable policy item above. I
may be misunderstanding this proposal but I would expect that it has
some link with what a user is allowed to do. Without some improvements
to the policy handling within Nova this is not currently possible.


> 
> Every thing here is up for discussion. This is a summary of some of what
> was in the meeting, plus some of my own thoughts. Please chime in on any
> of this. It would be good to be of general agreement pre summit, so we
> could focus conversation there more on the hows for getting things done.
> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Andrew Laski]

On Tue, Mar 22, 2016, at 07:00 AM, Doug Hellmann wrote:
> 
> 
> > On Mar 21, 2016, at 9:43 PM, Adam Young  wrote:
> > 
> > I had a good discussion with the Nova folks in IRC today.
> > 
> > My goal was to understand what could talk to what, and the short according 
> > to dansmith
> > 
> > " any node in nova land has to be able to talk to the queue for any other 
> > one for the most part: compute->compute, compute->conductor, 
> > conductor->compute, api->everything. There might be a few exceptions, but 
> > not worth it, IMHO, in the current architecture."
> > 
> > Longer conversation is here:
> > http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27
> > 
> > Right now, the message queue is a nightmare.  All sorts of sensitive 
> > information flows over the message queue: Tokens (including admin) are the 
> > most obvious.  Every piece of audit data. All notifications and all control 
> > messages.
> > 
> > Before we continue down the path of "anything can talk to anything" can we 
> > please map out what needs to talk to what, and why?  Many of the use cases 
> > seem to be based on something that should be kicked off by the conductor, 
> > such as "migrate, resize, live-migrate" and it sounds like there are plans 
> > to make that happen.
> > 
> > So, let's assume we can get to the point where, if node 1 needs to talk to 
> > node 2, it will do so only via the conductor.  With that in place, we can 
> > put an access control rule in place:

If you specifically mean compute nodes by "node 1" and "node 2" that is
something that's being worked towards. If you mean "node" more generally
that's not something that is planned.

> 
> Shouldn't we be trying to remove central bottlenecks by decentralizing
> communications where we can?

I think that's a good goal to continue having. Some deployers have setup
firewalls between compute nodes, or between compute nodes and the
database, so we use the conductor to facilitate communications between
those nodes. But in general we don't want to send all communications
through the conductor. 

> 
> Doug
> 
> 
> > 
> > 1.  Compute nodes can only read from the queue 
> > compute.-novacompute-.localdomain
> > 2.  Compute nodes can only write to response queues in the RPC vhost
> > 3.  Compute nodes can only write to notification queus in the notification 
> > host.
> > 
> > I know that with AMQP, we should be able to identify the writer of a 
> > message.  This means that each compute node should have its own user.  I 
> > have identified how to do that for Rabbit and QPid.  I assume for 0mq is 
> > would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather 
> > the 0mq maintainers chime in here.
> > 
> > I think it is safe (and sane) to have the same use on the compute node 
> > communicate with  Neutron, Nova, and Ceilometer.  This will avoid a false 
> > sense of security: if one is compromised, they are all going to be 
> > compromised.  Plan accordingly.
> > 
> > Beyond that, we should have message broker users for each of the components 
> > that is a client of the broker.
> > 
> > Applications that run on top of the cloud, and that do not get presence on 
> > the compute nodes, should have their own VHost.  I see Sahara on my Tripleo 
> > deploy, but I assume there are others.  Either they completely get their 
> > own vhost, or the apps should share one separate from the RPC/Notification 
> > vhosts we currently have.  Even Heat might fall into this category.
> > 
> > Note that those application users can be allowed to read from the 
> > notification queues if necessary.  They just should not be using the same 
> > vhost for their own traffic.
> > 
> > Please tell me if/where I am blindingly wrong in my analysis.
> > 
> > 
> > 
> > 
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] revert new gerrit

[Andrew Laski]

 
 
 
On Fri, Mar 18, 2016, at 10:31 AM, Andrey Kurilin wrote:
> Hi all!
> I want to start this thread because I'm tired. I spent a lot of time, but I 
> can't review as easy as it was with old interface. New Gerrit is awful. Here 
> are several issues:
>
> * It is not possible to review patches at mobile phone. "New" "modern"
>   theme is not adopted for small screens.
> * Leaving comments is a hard task. Position of page can jump anytime.
> * It is impossible to turn off hot-keys. position of page changed->I
>   don't see that comment pop-up is closed->continue type several letters-
>   >make unexpected things(open edit mode, modify something, save,
>   exit...)
> * patch-dependency tree is not user-friendly
> * summary table doesn't include status of patch(I need list to the end
>   of a page to know if patch is merged or not)
> * there is no button "Comment"/"Reply" at the end of page(after all
>   comments).
> * it is impossible to turn off "new" search mechanism
>
> Does it possible to return old, classic theme? It was a good time when
> we have old and new themes together...
 
I spent a little bit of time investigating the possibility of a chrome
extension to turn off the keyboard shortcuts and search mechanism a
little while ago. I eventually gave up but what I learned is that those
changes are apparently related to the inline edit ability that was
added. The edit window only loads in what is currently visible so
regular browser search would not work.
 
I've adapted to the new interface and really like some of the new
capabilities it provides, but having the page jump around while I'm
commenting has been a huge annoyance.
 
 
>
> --
> Best regards,
> Andrey Kurilin.
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Standardized executable return codes/logging conventions?

[Andrew Laski]

On Tue, Mar 8, 2016, at 06:16 AM, Finucane, Stephen wrote:
> I'm working on a fix [1] for bug #1538227, "Failed `nova-manage db
> sync` returns exitcode of 0" [2]. Having done this, I've noted that
> there seems to be no consistent policy around (a) what return codes to
> use (if any), and (b) whether to use logging or printing. Are there
> any guidelines on this in Nova (or any OpenStack project, for that
> matter) and, if not, do we need some? This is important for me as I'm
> adding a new return code to said executable and I've been asked for
> provide some rationale for my choice of code [3]
> 
> FWIW, I think the returns codes don't matter so much so long as they
> always return non-zero for failed/error cases and don't change (without
> good reason) once they do so. I also think output should be done via
> print, per the recommendations of the Python documentation [4], but
> some consitency in output would be appreciated.

I commented on the review as well, but after thinking on this some more
I'm inclined to have it return an exitcode of 1 for a generic failure
and we could later get into splitting the return code based on exception
type or some other distinct property of failures.

> 
> Cheers,
> Stephen
> 
> [1] https://review.openstack.org/#/c/289308/2
> [2] https://bugs.launchpad.net/nova/+bug/1538227
> [3] https://review.openstack.org/#/c/289308/2/nova/cmd/manage.py
> [4] https://docs.python.org/3.3/howto/logging.html#when-to-use-logging
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules

[Andrew Laski]

On Wed, Mar 2, 2016, at 02:36 PM, Gregory Haynes wrote:
> Clearly, some operators and users disagree with the opinion that 'by
> default security groups should closed off' given that we have several
> large public providers who have changed these defaults (despite there
> being no documented way to do so), and we have users in this thread
> expressing that opinion. Given that, I am not sure there is any value
> behind us expressing we have different opinions on what defaults
> should be (let alone enforcing them by not allowing them to be
> configured) unless there are some technical reasons beyond 'this is
> not what my policy is, what my customers wants', etc. I also
> understand the goal of trying to make clouds more similar for better
> interoperability (and I think that is extremely important), but the
> reality is we have created a situation where clouds are already not
> identical here in an even worse, undocumented way because we are
> enforcing a certain set of opinions here.
>
> To me this is an extremely clear indication that at a minimum the
> defaults should be configurable since discussion around them seems to
> devolve into different opinions on security policies, and there is no
> way we should be in the business of dictating that.
>

+1. While I happen to agree with closed by default there are many others
who feel differently, and there are cloud deployment scenarios where it
may not be the reasonable default.

It seems to me that visibility should be the primary focus. Make it easy
for users to know what they're getting, and make it clear that it's
something they should check rather than assume it's set a certain way.


> Cheers, Greg
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Cells meeting cancelled next week

[Andrew Laski]

Since we'll be past FF by then work in progress will be slowing down. We
will still meet occasionally to discuss specs or prepare for the summit,
but not next week.

The next meeting will be March 16th at 1700 UTC.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova hooks - document & test or deprecate?

[Andrew Laski]

On Mon, Feb 29, 2016, at 01:18 PM, Dan Smith wrote:
> > Forgive my ignorance or for playing devil's advocate, but wouldn't the
> > main difference between notifications and hooks be that notifications
> > are asynchronous and hooks aren't?
> 
> The main difference is that notifications are external and intended to
> be stable (especially with the versioned notifications effort). The
> hooks are internal and depend wholly on internal data structures.
> 
> > In the case of how Rdo was using it,
> > they are adding things to the injected_files list before the instance is
> > created in the compute API.  You couldn't do that with notifications as
> > far as I know.
> 
> Nope, definitely not, but I see that as a good thing. Injecting files
> like that is likely to be very fragile and I think mostly regarded as
> substantially less desirable than the alternatives, regardless of how it
> happens.
> 
> I think that Laski's point was that the most useful and least dangerous
> thing that hooks can be used for is the use case that is much better
> served by notifications.

Yep. My experience with them was things like updating an external cache
on create/delete or calling out to a DNS provider to remove a reverse
DNS entry on instance delete. Things that could easily be handled with
notifications, and use cases that I think we should continue to support
by improving notifications if necessary.


> 
> So, if file injection (and any other internals-mangling that other
> people may use them for) is not a reason to keep hooks, and if
> notifications are the proper way to trigger on things happening, then
> there's no reason to keep hooks.
> 
> --Dan
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova hooks - document & test or deprecate?

[Andrew Laski]

On Mon, Feb 29, 2016, at 12:53 PM, Rob Crittenden wrote:
> Andrew Laski wrote:
> > 
> > 
> > On Mon, Feb 29, 2016, at 12:12 PM, Dan Smith wrote:
> >>> In our continued quest on being more explicit about plug points it feels
> >>> like we should other document the interface (which means creating
> >>> stability on the hook parameters) or we should deprecate this construct
> >>> as part of a bygone era.
> >>>
> >>> I lean on deprecation because it feels like a thing we don't really want
> >>> to support going forward, but I can go either way.
> >>
> >> Deprecate and remove, please. We've been removing these sorts of things
> >> over time, and nova hooks have been ignored in that process. But really,
> >> making them more rigid is going to get in the way over time, trying to
> >> continue to honor an interface that codifies internals at a certain
> >> point in time, and leaving them as-is will just continue to generate
> >> issues like the quoted bug.
> >>
> >> I don't "lean" on deprecation, I feel strongly that these should go away.
> > 
> > I've worked on a deployment that uses them heavily and would be impacted
> > by their removal. They are a very convenient place to put code that
> > should run based on Nova events but I have yet to see a usage that
> > couldn't have been implemented by having a service listen to
> > notifications and run that same code. However there is no service that
> > does this. So the only argument I can see for keeping them is that it's
> > more convenient to put that code into Nova rather than implement
> > something that listens for notifications. And that's not a convincing
> > argument to me.
> 
> The ability to inject files on a per-VM basis is one thing that would be
> missing. Right now decisions can be made inside the hook using
> attributes of the VM to decide whether and what to inject, including
> files generated inside the hook itself.
> 
> I won't argue that the API is nice, it isn't, but I'd prefer a replace
> then deprecate model instead.
> 
> I'm not entirely sure that the notifications contain all the same
> information as is available now in the hooks.

This is something we can and should address if there is useful
information missing from notifications.

> 
> rob
> 
> > So I agree with moving forward on deprecation and think that
> > notifications provide a suitable replacement for the functionality
> > provided.
> > 
> > 
> >>
> >> --Dan
> >>
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova hooks - document & test or deprecate?

[Andrew Laski]

On Mon, Feb 29, 2016, at 12:12 PM, Dan Smith wrote:
> > In our continued quest on being more explicit about plug points it feels
> > like we should other document the interface (which means creating
> > stability on the hook parameters) or we should deprecate this construct
> > as part of a bygone era.
> > 
> > I lean on deprecation because it feels like a thing we don't really want
> > to support going forward, but I can go either way.
> 
> Deprecate and remove, please. We've been removing these sorts of things
> over time, and nova hooks have been ignored in that process. But really,
> making them more rigid is going to get in the way over time, trying to
> continue to honor an interface that codifies internals at a certain
> point in time, and leaving them as-is will just continue to generate
> issues like the quoted bug.
> 
> I don't "lean" on deprecation, I feel strongly that these should go away.

I've worked on a deployment that uses them heavily and would be impacted
by their removal. They are a very convenient place to put code that
should run based on Nova events but I have yet to see a usage that
couldn't have been implemented by having a service listen to
notifications and run that same code. However there is no service that
does this. So the only argument I can see for keeping them is that it's
more convenient to put that code into Nova rather than implement
something that listens for notifications. And that's not a convincing
argument to me.

So I agree with moving forward on deprecation and think that
notifications provide a suitable replacement for the functionality
provided.


> 
> --Dan
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Andrew Laski]

On Wed, Feb 24, 2016, at 07:48 AM, Sean Dague wrote:
> We have a specific bug around aggregrate metadata setting in Nova which
> exposes a larger issue with our mysql schema.
> https://bugs.launchpad.net/nova/+bug/1538011
> 
> On mysql the following will explode with a 500:
> 
> > nova aggregate-create agg1
> > nova aggregate-set-metadata agg1 abc=1
> > nova aggregate-set-metadata agg1 ABC=2
> 
> mysql (by default) treats abc == ABC. However the python code does not.
> 
> We have a couple of options:
> 
> 1) make the API explicitly case fold
> 
> 2) update the mysql DB to use latin_bin collation for these columns
> 
> 3) make this a 400 error because duplicates were found
> 
> 
> Options 1 & 2 make all OpenStack environments consistent regardless of
> backend.
> 
> Option 2 is potentially expensive TABLE alter.
> 
> Option 3 gets rid of the 500 error, however at the risk that the
> behavior for this API is different depending on DB backend. Which is
> less than ideal.
> 
> 
> My preference is slightly towards #1. It's taken a long time for someone
> to report this issue, so I think it's an edge case, and people weren't
> think about this being case sensitive. It has the risk of impacting
> someone on an odd db platform that has been using that feature.

#1 is my preference as well. Nova should be the only consumer of this
metadata and it does not need case sensitivity for this.

> 
> There are going to be a few other APIs to clean up in a similar way. I
> don't think this comes in under a microversion because of how deep in
> the db api layer this is, and it's just not viable to keep both paths.
> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Mon, Feb 22, 2016, at 02:42 PM, Matt Riedemann wrote:
> 
> 
> On 2/22/2016 5:56 AM, Sean Dague wrote:
> > On 02/19/2016 12:49 PM, John Garbutt wrote:
> > 
> >>
> >> Consider a user that uses these four clouds:
> >> * nova-network flat DHCP
> >> * nova-network VLAN manager
> >> * neutron with a single provider network setup
> >> * neutron where user needs to create their own network
> >>
> >> For the first three, the user specifies no network, and they just get
> >> a single NIC with some semi-sensible IP address, likely with a gateway
> >> to the internet.
> >>
> >> For the last one, the user ends up with a network with zero NICs. If
> >> they then go and configure a network in neutron (and they can now use
> >> the new easy one shot give-me-a-network CLI), they start to get VMs
> >> just like they would have with nova-network VLAN manager.
> >>
> >> We all agree the status quo is broken. For me, this is a bug in the
> >> API where we need to fix the consistency. Because its a change in the
> >> behaviour, it needs to be gated by a micro version.
> >>
> >> Now, if we step back and created this again, I would agree that
> >> --nic=auto is a good idea, so its explicit. However, all our users are
> >> used to automatic being the default, all be it a very patchy default.
> >> So I think the best evolution here is to fix the inconsistency by
> >> making a VM with no network being the explicit option (--no-nic or
> >> something?), and failing the build if we are unable to get a nic using
> >> an "automatic guess" route. So now the default is more consistent, and
> >> those that what a VM with no NIC have a way to get their special case
> >> sorted.
> >>
> >> I think this means I like "option 2" in the summary mail on the ops list.
> >
> > Thinking through this over the weekend.
> >
> >  From the API I think I agree with Laski now. An API shouldn't doesn't
> > typically need default behavior, it's ok to make folks be explicit. So
> > making nic a required parameter is fine.
> >
> > "nic": "auto"
> > "nic": "none"
> > "nic": "$name"
> >
> > nic is now jsonschema enforced, 400 if not provided.
> >
> > that being said... I think the behavior of CLI tools should default to
> > nic auto being implied. The user experience there is different. You use
> > cli tools for one off boots of things, so should be as easy as possible.
> >
> > I think this is one of the places where the UX needs of the API and the
> > CLI are definitely different.
> >
> > -Sean
> >
> 
> Is nic only required when using neutron? Or as part of the microversion 
> are we also going to enforce this for nova-network, because if so, that 
> seems like a step backward. But if we don't enforce that check for both 
> neutron and nova-network, then we have differences in the API again.

I think it makes sense to require it in both cases and keep users
blissfully unaware of which networking service is in use.

> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

[Andrew Laski]

On Mon, Feb 22, 2016, at 12:15 PM, Mike Bayer wrote:
> 
> 
> On 02/22/2016 11:30 AM, Chris Friesen wrote:
> > On 02/22/2016 11:17 AM, Jay Pipes wrote:
> >> On 02/22/2016 10:43 AM, Chris Friesen wrote:
> >>> Hi all,
> >>>
> >>> We've recently run into some interesting behaviour that I thought I
> >>> should bring up to see if we want to do anything about it.
> >>>
> >>> Basically the problem seems to be that nova-compute is doing disk I/O
> >>> from the main thread, and if it blocks then it can block all of
> >>> nova-compute (since all eventlets will be blocked).  Examples that we've
> >>> found include glance image download, file renaming, instance directory
> >>> creation, opening the instance xml file, etc.  We've seen nova-compute
> >>> block for upwards of 50 seconds.
> >>>
> >>> Now the specific case where we hit this is not a production
> >>> environment.  It's only got one spinning disk shared by all the guests,
> >>> the guests were hammering on the disk pretty hard, the IO scheduler for
> >>> the instance disk was CFQ which seems to be buggy in our kernel.
> >>>
> >>> But the fact remains that nova-compute is doing disk I/O from the main
> >>> thread, and if the guests push that disk hard enough then nova-compute
> >>> is going to suffer.
> >>>
> >>> Given the above...would it make sense to use eventlet.tpool or similar
> >>> to perform all disk access in a separate OS thread?  There'd likely be a
> >>> bit of a performance hit, but at least it would isolate the main thread
> >>> from IO blocking.
> >>
> >> This is probably a good idea, but will require quite a bit of code
> >> change. I
> >> think in the past we've taken the expedient route of just exec'ing
> >> problematic
> >> code in a greenthread using utils.spawn().
> >
> > I'm not an expert on eventlet, but from what I've seen this isn't
> > sufficient to deal with disk access in a robust way.
> >
> > It's my understanding that utils.spawn() will result in the code running
> > in the same OS thread, but in a separate eventlet greenthread.  If that
> > code tries to access the disk via a potentially-blocking call the
> > eventlet subsystem will not jump to another greenthread.  Because of
> > this it can potentially block the whole OS thread (and thus all other
> > greenthreads running in that OS thread).
> 
> not sure what utils.spawn() does but if it is in fact an "exec" (or if 
> Jay is suggesting that an exec() be used within) then the code would be 
> in a different process entirely, and communicating with it becomes an 
> issue of pipe IO over unix sockets which IIRC can do non blocking.

utils.spawn() is just a wrapper around eventlet.spawn(), mostly there to
be stubbed out in testing.


> 
> 
> >
> > I think we need to eventlet.tpool for disk IO (or else fork a whole
> > separate process).  Basically we need to ensure that the main OS thread
> > never issues a potentially-blocking syscall.
> 
> tpool would probably be easier (and more performant because no socket 
> needed).
> 
> 
> >
> > Chris
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Mon, Feb 22, 2016, at 07:07 AM, Chris Dent wrote:
> On Mon, 22 Feb 2016, Sean Dague wrote:
> 
> > that being said... I think the behavior of CLI tools should default to
> > nic auto being implied. The user experience there is different. You use
> > cli tools for one off boots of things, so should be as easy as possible.
> 
> Thanks Sean, throughout this entire conversation I've been confused
> by the apparent overlap or conflation of the API's interface and the
> command line client's interface.
> 
> I agree that the API should be explicit and that the command line
> should have sane defaults that don't violate the principle of least
> surprise.
> 
> +1

I completely agree with this.

> 
> -- 
> Chris Dent   (╯°□°)╯︵┻━┻http://anticdent.org/
> freenode: cdent tw: @anticdent
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 19, 2016, at 11:14 AM, Sean Dague wrote:
> On 02/19/2016 09:30 AM, Andrew Laski wrote:
> > 
> > 
> > On Thu, Feb 18, 2016, at 05:34 PM, melanie witt wrote:
> >> On Feb 12, 2016, at 14:49, Jay Pipes  wrote:
> >>
> >>> This would be my preference as well, even though it's technically a 
> >>> backwards-incompatible API change.
> >>>
> >>> The idea behind get-me-a-network was specifically to remove the current 
> >>> required complexity of the nova boot command with regards to networking 
> >>> options and allow a return to the nova-net model where an admin could 
> >>> auto-create a bunch of unassigned networks and the first time a user 
> >>> booted an instance and did not specify any network configuration (the 
> >>> default, sane behaviour in nova-net), one of those unassigned networks 
> >>> would be grabbed for the troject, I mean prenant, sorry.
> >>>
> >>> So yeah, the "opt-in to having no networking at all with a 
> >>> --no-networking or --no-nics option" would be my preference.
> >>
> >> +1 to this, especially opting in to have no network at all. It seems most
> >> friendly to me to have the network allocation automatically happen if
> >> nothing special is specified.
> >>
> >> This is something where it seems like we need a "reset" to a default
> >> behavior that is user-friendly. And microversions is the way we have to
> >> "fix" an undesirable current default behavior.
> > 
> > The question I would still like to see addressed is why do we need to
> > have a default behavior here? The get-me-a-network effort is motivated
> > by the current complexity of setting up a network for an instance
> > between Nova and Neutron and wants to get back to a simpler time of
> > being able to just boot an instance and get a network. But it still
> > isn't clear to me why requiring something like "--nic auto" wouldn't
> > work here, and eliminate the confusion of changing a default behavior.
> 
> The point was the default behavior was a major concern to people. It's
> not like this was always the behavior. If you were (or are) on nova net,
> you don't need that option at all.

Which is why I would prefer to shy away from default behaviors.

> 
> The major reason we implemented API microversions was so that we could
> make the base API experience better for people, some day. One day, we'll
> have an API we love, hopefully. Doing so means that we do need to make
> changes to defaults. Deprecate some weird and unmaintained bits.
> 
> The principle of least surprise to me is that you don't need that
> attribute at all. Do the right thing with the least amount of work.
> Instead of making the majority of clients and users do extra work
> because once upon a time when we brought in neutron a thing happen.

The principal of least surprise to me is that a user explicitly asks for
something rather than relying on a default that changes based on network
service and/or microversion. This is the only area in the API where
something did, and would, happen without explicitly being requested by a
user. I just don't understand why it's special compared to
flavor/image/volume which we require to be explicit. But I think we just
need to agree to disagree here.

> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Thu, Feb 18, 2016, at 05:34 PM, melanie witt wrote:
> On Feb 12, 2016, at 14:49, Jay Pipes  wrote:
> 
> > This would be my preference as well, even though it's technically a 
> > backwards-incompatible API change.
> > 
> > The idea behind get-me-a-network was specifically to remove the current 
> > required complexity of the nova boot command with regards to networking 
> > options and allow a return to the nova-net model where an admin could 
> > auto-create a bunch of unassigned networks and the first time a user booted 
> > an instance and did not specify any network configuration (the default, 
> > sane behaviour in nova-net), one of those unassigned networks would be 
> > grabbed for the troject, I mean prenant, sorry.
> > 
> > So yeah, the "opt-in to having no networking at all with a --no-networking 
> > or --no-nics option" would be my preference.
> 
> +1 to this, especially opting in to have no network at all. It seems most
> friendly to me to have the network allocation automatically happen if
> nothing special is specified.
> 
> This is something where it seems like we need a "reset" to a default
> behavior that is user-friendly. And microversions is the way we have to
> "fix" an undesirable current default behavior.

The question I would still like to see addressed is why do we need to
have a default behavior here? The get-me-a-network effort is motivated
by the current complexity of setting up a network for an instance
between Nova and Neutron and wants to get back to a simpler time of
being able to just boot an instance and get a network. But it still
isn't clear to me why requiring something like "--nic auto" wouldn't
work here, and eliminate the confusion of changing a default behavior.



> 
> While I get that a backward-incompatible change may appear to "sneak in"
> for a user specifying a later microversion to get an unrelated feature,
> it seems reasonable to me that a user specifying a microversion would
> consult the documentation for the version delta to get a clear picture of
> what to expect once they specify the new version. This of course hinges
> on users knowing how microversions work and being familiar with
> consulting documentation when changing versions. I hope that is the case
> and I hope this change will come with a very clear and concise release
> note with a link to [1].

I do hope that users read the documentation and aren't caught unaware if
we make a change like this. But we can make it even easier for them to
know that something has changed if instead of changing default behavior
we make that behavior explicit.


> 
> -melanie
> 
> [1]
> http://docs.openstack.org/developer/nova/api_microversion_history.html
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Should we signal backwards incompatible changes in microversions?

[Andrew Laski]

On Tue, Feb 16, 2016, at 07:54 AM, Alex Xu wrote:
>
>
> 2016-02-16 19:53 GMT+08:00 Sean Dague :
>> On 02/12/2016 03:55 PM, Andrew Laski wrote:
>>
> Starting a new thread to continue a thought that came up in
>>
> http://lists.openstack.org/pipermail/openstack-dev/2016-February/086457.html.
>>
> The Nova API microversion framework allows for backwards
> compatible and
>>
> backwards incompatible changes but there is no way to programmatically
>>
> distinguish the two. This means that as a user of the API I need to
>>
> understand every change between the version I'm using now and a new
>>
> version I would like to move to in case an intermediate version
> changes
>>
> default behaviors or removes something I'm currently using.
>>
>
>>
> I would suggest that a more user friendly approach would be to
>>
> distinguish the two types of changes. Perhaps something like
> 2.x.y where
>>
> x is bumped for a backwards incompatible change and y is still
>>
> monotonically increasing regardless of bumps to x. So if the current
>>
> version is 2.2.7 a new backwards compatible change would bump to 2.2.8
>>
> or a new backwards incompatible change would bump to 2.3.8. As a user
>>
> this would allow me to fairly freely bump the version I'm consuming
>>
> until x changes at which point I need to take more care in moving to a
>>
> new version.
>>
>
>>
> Just wanted to throw the idea out to get some feedback. Or
> perhaps this
>>
> was already discussed and dismissed when microversions were
> added and I
>>
> just missed it.
>>
>> Please no.
>>
>>
We specifically stated many times that microversions aren't semver. Each
>>
version is just that.
>>
>>
Semver only makes sense when you are always talking to one installation,
>>
and the version numbers can only increase. When your code retargets to
>>
multiple installations version numbers can very easily go backwards. So
>>
unless a change in compatible forward and backwards, it's a breaking
>>
change for someone.
>
> indeed, learned this point.

Fair enough, I wasn't thinking a lot about moving between installations
just that we've hidden information within one installation.

Since any change except one that is backwards and forwards compatible is
a breaking change for users of multiple clouds what is essentially being
said is that we have a new API with every microversion. Given that I
wonder if we shouldn't make a stronger statement that the API differs,
as in why even have a 2. prefix which implies that 2.x has some relation
to 2.x+1 when it doesn't.

It was mentioned elsewhere in the thread that we have a hard time
knowing what's going to end up being compatible or not before it's
released. This seems like something we should be able to determine and
indicate somehow, even just through docs, otherwise we're passing that
burden on to users to determine for themselves.

I very much like that microversions have enabled development to move
forward on the API without the mess of extensions that we had
previously. I fear that we have no real measurement of the cost of
consuming the API under this new scheme. It's easy enough to think that
users will just read the docs and carefully consider every version
increment that they want to consume but when they've been on version 2.7
for a while and a new thing comes out in 2.35 that they want they need
to fully digest the implications of all 27 intervening versions purely
through docs and with the understanding that literally almost anything
about the semantics can have changed. So while I love the freedom that
it provides to developers I think it would be useful to have a small set
of constraints in place that helps users. Of course all of my ideas have
been duds so far and perhaps that's because I'm imagining future
scenarios that won't come to pass or that we don't care about. But
something has me concerned and I can't quite get my finger on it.


>
>>
>>
-Sean
>>
>>
--
>> 
Sean Dague
>> http://dague.net
>>  
>> 
__
>> 
OpenStack Development Mailing List (not for usage questions)
>> 
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 12, 2016, at 03:51 PM, Ed Leafe wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> On 02/12/2016 02:41 PM, Andrew Laski wrote:
> 
> >> I think if the point of the experience for this API is to be
> >> working out
> >>> of the box. So I very much like the idea of a defaults change
> >>> here to the thing we want to be easy. And an explicit option to
> >>> disable it if you want to do something more complicated.
> 
> > I think this creates a potential for confusing behavior for users. 
> > They're happily booting instances with no network on 2.1, as silly
> > as that might be, and then decide "ooh I'd like to use fancy new
> > flag foo which is available in 2.35". So they add the flag to their
> > request and set the version to 2.35 and suddenly multiple things
> > change about their boot process because they missed that 2.24(or
> > whatever) changed a default behavior. If this fits within the scope
> > of microversions then users will need to expect that, but it's
> > something that would be likely to trip me up as a user of the API.
> 
> I agree - that's always been the trade-off with microversions. You
> never get surprises, but you can't move to a new feature in 2.X
> without also having to get everything that was also introduced in
> 2.X-1 and before. The benefit, of course, is that the user will have
> changed something explicitly before getting the new behavior, and at
> least has a chance of figuring it out.

I completely agree with the idea that the form of a request/response
could change in any number of backwards incompatible ways due to a
microversion. That's easily discoverable through a published schema.
What I'm struggling with is the idea that we would be comfortable
changing the meaning of something, or a default behavior, with a
microversion. There's no discoverability for that sort of change except
to try it and see what happens or read the docs. But the potential for
surprise is high and there's no immediate feedback on misusing the new
API like there is if the form changes and JSONSchema validation kicks it
back.

> 
> - -- 
> 
> - -- Ed Leafe
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2
> Comment: GPGTools - https://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iQIcBAEBCgAGBQJWvkXeAAoJEKMgtcocwZqLdEwP/R36392zeHL55LR19ewoSg8/
> U9MJEmZo2RiWaJBqWlRsBF5DSNzi7oNzhot8bOcY+aO7XQAs2kfG1QF9YMj/YfTw
> iqsCtKNfdJZR1lNtq7u/TodkkFLP7gO8Q36efOYvAMIIlZlOoMAyvLWRxDGTGN+t
> ahgnw2z6oQDpARb6Yx7NFjogjccTENdkuDNyLy/hmuUpvLyvhUDQC1EouVNHPglA
> Sb8tQYSsdKDHrggs8b3XuJZjJXYvn0M4Knnw3i/0DAVoZamVfsnHJ1EWRfOh7hq3
> +C+MJfzfyz5K46ikvjpuSGPPZ8rPPLR1gaih/W2fmXdvKG7NSK3sIUcgJZ7lm4rh
> VpVDCRWi9rlPuJa4JIKlZ8h6NNMHwiEq8Ea+dP7lHnx0qp8EIxkPDBdU6sCmeUGM
> tqBeHjUU7f8/fbZkOdorn1NAEZfXcRew3/BeFFxrmu6X8Z2XHHXMBBtlmehEoDHO
> 6/BzZH3I/5VPcvFZQfsYYivBj3vtmB8cVLbUNpD3xBLyJKVFEwfmGkkYQTlL0KDx
> B+bqNDw2pK72/qN39rjmdZY/cZ4vGfBGu2CzJxbX+Zn2E8Mgg5rAuARG0OCNg9ll
> uuVBy37vbPNrNV9UZSkSjmRma/l8kl1IzBbszH0ENbH/ov3ngKB0xWiLc1pBZKC9
> GcPgzIoclwLrVooRqOSf
> =Dqga
> -END PGP SIGNATURE-
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 12, 2016, at 05:29 PM, Doug Wiegley wrote:
> 
> > On Feb 12, 2016, at 3:17 PM, Andrew Laski  wrote:
> > 
> > 
> > 
> > On Fri, Feb 12, 2016, at 04:53 PM, Doug Wiegley wrote:
> >> 
> >>> On Feb 12, 2016, at 2:15 PM, Andrew Laski  wrote:
> >>> 
> >>> 
> >>> 
> >>> On Fri, Feb 12, 2016, at 04:03 PM, Doug Wiegley wrote:
> >>>> Correct me if I’m wrong, but with nova-net, ‘nova boot’ automatically
> >>>> gives it one nic with an IP, right?
> >>>> 
> >>>> So how is this changing that behavior? Making it harder to use, for the
> >>>> sake of preserving a really unusual corner case (no net with neutron),
> >>>> seems a much worse user experience here.
> >>> 
> >>> I was just going off of the behavior Matt described, that it was
> >>> possible to boot with no network by leaving that out of the request. If
> >>> the behavior already differs based on what networking backend is in use
> >>> then we've put users in a bad spot, and IMO furthers my case for having
> >>> explicit parameters in the request. I'm really not seeing how "--nic
> >>> auto|none" is any harder than leaving all network related parameters off
> >>> of the boot request, and it avoids potential confusion as to what will
> >>> happen.
> >> 
> >> It hurts discoverability, and “expectedness”. If I’m new to openstack,
> >> having it default boot unusable just means the first time I use ’nova
> >> boot’, I’ll end up with a useless VM. People don’t read docs first, it
> >> should “just work” as far as that’s sane. And OpenStack has a LOT of
> >> these little annoyances for the sake of strict correctness while
> >> optimizing for an unusual or rare case.
> >> 
> >> The original stated goal of this simpler neutron api was to get back to
> >> the simpler nova boot. I’d like to see that happen.
> > 
> > I'm not suggesting that the default boot be unusable. I'm saying that
> > just like it's required to pass in a flavor and image/volume to boot an
> > instance why not require the user to specify something about the
> > network. And that network request can be as simple as specifying "auto"
> > or "none". That seems to meet all of the requirements without the
> > confusion of needing to guess what the default behavior will be when
> > it's left off because it can apparently mean different things based on
> > which backed is in use. For users that don't read the docs they'll get
> > an immediate failure response indicating that they need to specify
> > something about the network versus the current and proposed state where
> > it's not clear what will happen unless they've read the docs on
> > microversions and understand which network service is being used.
> 
> I understand what you’re saying, and we’re almost down to style here. We
> have the previous nova-net ‘nova boot’ behavior, plus I think a VM with a
> network is kinda nonsensical, so adding that hoop for the default case
> doesn’t make sense to me. I’m not sure we’ll convince each other here,
> and that’s ok, I’ve said my peace.  :-)

Me too :)

Thanks for the discussion.


> 
> Thanks,
> doug
> 
> > 
> >> 
> >> Thanks,
> >> doug
> >> 
> >>> 
> >>>> 
> >>>> Thanks,
> >>>> doug
> >>>> 
> >>>> 
> >>>>> On Feb 12, 2016, at 1:51 PM, Ed Leafe  wrote:
> >>>>> 
> >>>>> -BEGIN PGP SIGNED MESSAGE-
> >>>>> Hash: SHA512
> >>>>> 
> >>>>> On 02/12/2016 02:41 PM, Andrew Laski wrote:
> >>>>> 
> >>>>>>> I think if the point of the experience for this API is to be
> >>>>>>> working out
> >>>>>>>> of the box. So I very much like the idea of a defaults change
> >>>>>>>> here to the thing we want to be easy. And an explicit option to
> >>>>>>>> disable it if you want to do something more complicated.
> >>>>> 
> >>>>>> I think this creates a potential for confusing behavior for users. 
> >>>>>> They're happily booting instances with no network on 2.1, as silly
> >>>>>> as that might be, and then decide "ooh I'd like to use fancy new
> >>>>&g

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 12, 2016, at 04:53 PM, Doug Wiegley wrote:
> 
> > On Feb 12, 2016, at 2:15 PM, Andrew Laski  wrote:
> > 
> > 
> > 
> > On Fri, Feb 12, 2016, at 04:03 PM, Doug Wiegley wrote:
> >> Correct me if I’m wrong, but with nova-net, ‘nova boot’ automatically
> >> gives it one nic with an IP, right?
> >> 
> >> So how is this changing that behavior? Making it harder to use, for the
> >> sake of preserving a really unusual corner case (no net with neutron),
> >> seems a much worse user experience here.
> > 
> > I was just going off of the behavior Matt described, that it was
> > possible to boot with no network by leaving that out of the request. If
> > the behavior already differs based on what networking backend is in use
> > then we've put users in a bad spot, and IMO furthers my case for having
> > explicit parameters in the request. I'm really not seeing how "--nic
> > auto|none" is any harder than leaving all network related parameters off
> > of the boot request, and it avoids potential confusion as to what will
> > happen.
> 
> It hurts discoverability, and “expectedness”. If I’m new to openstack,
> having it default boot unusable just means the first time I use ’nova
> boot’, I’ll end up with a useless VM. People don’t read docs first, it
> should “just work” as far as that’s sane. And OpenStack has a LOT of
> these little annoyances for the sake of strict correctness while
> optimizing for an unusual or rare case.
> 
> The original stated goal of this simpler neutron api was to get back to
> the simpler nova boot. I’d like to see that happen.

I'm not suggesting that the default boot be unusable. I'm saying that
just like it's required to pass in a flavor and image/volume to boot an
instance why not require the user to specify something about the
network. And that network request can be as simple as specifying "auto"
or "none". That seems to meet all of the requirements without the
confusion of needing to guess what the default behavior will be when
it's left off because it can apparently mean different things based on
which backed is in use. For users that don't read the docs they'll get
an immediate failure response indicating that they need to specify
something about the network versus the current and proposed state where
it's not clear what will happen unless they've read the docs on
microversions and understand which network service is being used.

> 
> Thanks,
> doug
> 
> > 
> >> 
> >> Thanks,
> >> doug
> >> 
> >> 
> >>> On Feb 12, 2016, at 1:51 PM, Ed Leafe  wrote:
> >>> 
> >>> -BEGIN PGP SIGNED MESSAGE-
> >>> Hash: SHA512
> >>> 
> >>> On 02/12/2016 02:41 PM, Andrew Laski wrote:
> >>> 
> >>>>> I think if the point of the experience for this API is to be
> >>>>> working out
> >>>>>> of the box. So I very much like the idea of a defaults change
> >>>>>> here to the thing we want to be easy. And an explicit option to
> >>>>>> disable it if you want to do something more complicated.
> >>> 
> >>>> I think this creates a potential for confusing behavior for users. 
> >>>> They're happily booting instances with no network on 2.1, as silly
> >>>> as that might be, and then decide "ooh I'd like to use fancy new
> >>>> flag foo which is available in 2.35". So they add the flag to their
> >>>> request and set the version to 2.35 and suddenly multiple things
> >>>> change about their boot process because they missed that 2.24(or
> >>>> whatever) changed a default behavior. If this fits within the scope
> >>>> of microversions then users will need to expect that, but it's
> >>>> something that would be likely to trip me up as a user of the API.
> >>> 
> >>> I agree - that's always been the trade-off with microversions. You
> >>> never get surprises, but you can't move to a new feature in 2.X
> >>> without also having to get everything that was also introduced in
> >>> 2.X-1 and before. The benefit, of course, is that the user will have
> >>> changed something explicitly before getting the new behavior, and at
> >>> least has a chance of figuring it out.
> >>> 
> >>> - -- 
> >>> 
> >>> - -- Ed Leafe
> >>> -BEGIN PGP SIGNATURE-
> >>> Version: GnuPG v2

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 12, 2016, at 04:03 PM, Doug Wiegley wrote:
> Correct me if I’m wrong, but with nova-net, ‘nova boot’ automatically
> gives it one nic with an IP, right?
> 
> So how is this changing that behavior? Making it harder to use, for the
> sake of preserving a really unusual corner case (no net with neutron),
> seems a much worse user experience here.

I was just going off of the behavior Matt described, that it was
possible to boot with no network by leaving that out of the request. If
the behavior already differs based on what networking backend is in use
then we've put users in a bad spot, and IMO furthers my case for having
explicit parameters in the request. I'm really not seeing how "--nic
auto|none" is any harder than leaving all network related parameters off
of the boot request, and it avoids potential confusion as to what will
happen.

> 
> Thanks,
> doug
> 
> 
> > On Feb 12, 2016, at 1:51 PM, Ed Leafe  wrote:
> > 
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA512
> > 
> > On 02/12/2016 02:41 PM, Andrew Laski wrote:
> > 
> >>> I think if the point of the experience for this API is to be
> >>> working out
> >>>> of the box. So I very much like the idea of a defaults change
> >>>> here to the thing we want to be easy. And an explicit option to
> >>>> disable it if you want to do something more complicated.
> > 
> >> I think this creates a potential for confusing behavior for users. 
> >> They're happily booting instances with no network on 2.1, as silly
> >> as that might be, and then decide "ooh I'd like to use fancy new
> >> flag foo which is available in 2.35". So they add the flag to their
> >> request and set the version to 2.35 and suddenly multiple things
> >> change about their boot process because they missed that 2.24(or
> >> whatever) changed a default behavior. If this fits within the scope
> >> of microversions then users will need to expect that, but it's
> >> something that would be likely to trip me up as a user of the API.
> > 
> > I agree - that's always been the trade-off with microversions. You
> > never get surprises, but you can't move to a new feature in 2.X
> > without also having to get everything that was also introduced in
> > 2.X-1 and before. The benefit, of course, is that the user will have
> > changed something explicitly before getting the new behavior, and at
> > least has a chance of figuring it out.
> > 
> > - -- 
> > 
> > - -- Ed Leafe
> > -BEGIN PGP SIGNATURE-
> > Version: GnuPG v2
> > Comment: GPGTools - https://gpgtools.org
> > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> > 
> > iQIcBAEBCgAGBQJWvkXeAAoJEKMgtcocwZqLdEwP/R36392zeHL55LR19ewoSg8/
> > U9MJEmZo2RiWaJBqWlRsBF5DSNzi7oNzhot8bOcY+aO7XQAs2kfG1QF9YMj/YfTw
> > iqsCtKNfdJZR1lNtq7u/TodkkFLP7gO8Q36efOYvAMIIlZlOoMAyvLWRxDGTGN+t
> > ahgnw2z6oQDpARb6Yx7NFjogjccTENdkuDNyLy/hmuUpvLyvhUDQC1EouVNHPglA
> > Sb8tQYSsdKDHrggs8b3XuJZjJXYvn0M4Knnw3i/0DAVoZamVfsnHJ1EWRfOh7hq3
> > +C+MJfzfyz5K46ikvjpuSGPPZ8rPPLR1gaih/W2fmXdvKG7NSK3sIUcgJZ7lm4rh
> > VpVDCRWi9rlPuJa4JIKlZ8h6NNMHwiEq8Ea+dP7lHnx0qp8EIxkPDBdU6sCmeUGM
> > tqBeHjUU7f8/fbZkOdorn1NAEZfXcRew3/BeFFxrmu6X8Z2XHHXMBBtlmehEoDHO
> > 6/BzZH3I/5VPcvFZQfsYYivBj3vtmB8cVLbUNpD3xBLyJKVFEwfmGkkYQTlL0KDx
> > B+bqNDw2pK72/qN39rjmdZY/cZ4vGfBGu2CzJxbX+Zn2E8Mgg5rAuARG0OCNg9ll
> > uuVBy37vbPNrNV9UZSkSjmRma/l8kl1IzBbszH0ENbH/ov3ngKB0xWiLc1pBZKC9
> > GcPgzIoclwLrVooRqOSf
> > =Dqga
> > -END PGP SIGNATURE-
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Should we signal backwards incompatible changes in microversions?

[Andrew Laski]

Starting a new thread to continue a thought that came up in
http://lists.openstack.org/pipermail/openstack-dev/2016-February/086457.html.
The Nova API microversion framework allows for backwards compatible and
backwards incompatible changes but there is no way to programmatically
distinguish the two. This means that as a user of the API I need to
understand every change between the version I'm using now and a new
version I would like to move to in case an intermediate version changes
default behaviors or removes something I'm currently using.

I would suggest that a more user friendly approach would be to
distinguish the two types of changes. Perhaps something like 2.x.y where
x is bumped for a backwards incompatible change and y is still
monotonically increasing regardless of bumps to x. So if the current
version is 2.2.7 a new backwards compatible change would bump to 2.2.8
or a new backwards incompatible change would bump to 2.3.8. As a user
this would allow me to fairly freely bump the version I'm consuming
until x changes at which point I need to take more care in moving to a
new version.

Just wanted to throw the idea out to get some feedback. Or perhaps this
was already discussed and dismissed when microversions were added and I
just missed it.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 12, 2016, at 03:11 PM, Sean Dague wrote:
> On 02/12/2016 02:19 PM, Andrew Laski wrote:
> > 
> > 
> > On Fri, Feb 12, 2016, at 01:45 PM, John Garbutt wrote:
> >> On 12 February 2016 at 18:17, Andrew Laski  wrote:
> >>>
> >>>
> >>> On Fri, Feb 12, 2016, at 12:15 PM, Matt Riedemann wrote:
> >>>> Forgive me for thinking out loud, but I'm trying to sort out how nova
> >>>> would use a microversion in the nova API for the get-me-a-network
> >>>> feature recently added to neutron [1] and planned to be leveraged in
> >>>> nova (there isn't a spec yet for nova, I'm trying to sort this out for a
> >>>> draft).
> >>>>
> >>>> Originally I was thinking that a network is required for nova boot, so
> >>>> we'd simply check for a microversion and allow not specifying a network,
> >>>> easy peasy.
> >>>>
> >>>> Turns out you can boot an instance in nova (with neutron as the network
> >>>> backend) without a network. All you get is a measly debug log message in
> >>>> the compute logs [2]. That's kind of useless though and seems silly.
> >>>>
> >>>> I haven't tested this out yet to confirm, but I suspect that if you
> >>>> create a nova instance w/o a network, you can latter try to attach a
> >>>> network using the os-attach-interfaces API as long as you either provide
> >>>> a network ID *or* there is a public shared network or the tenant has a
> >>>> network at that point (nova looks those up if a specific network ID
> >>>> isn't provided).
> >>>>
> >>>> The high-level plan for get-me-a-network in nova was simply going to be
> >>>> if the user tries to boot an instance and doesn't provide a network, and
> >>>> there isn't a tenant network or public shared network to default to,
> >>>> then nova would call neutron's new auto-allocated-topology API to get a
> >>>> network. This, however, is a behavior change.
> >>>>
> >>>> So I guess the question now is how do we handle that behavior change in
> >>>> the nova API?
> >>>>
> >>>> We could add an auto-create-net boolean to the boot server request which
> >>>> would only be available in a microversion, then we could check that
> >>>> boolean in the compute API when we're doing network validation.
> >>>>
> >>>
> >>> I think a flag like this is the right approach. If it's currently valid
> >>> to boot an instance without a network than there needs to be something
> >>> to distinguish a request that wants a network created vs. a request that
> >>> doesn't want a network.
> >>>
> >>> This is still hugely useful if all that's required from a user is to
> >>> indicate that they would like a network, they still don't need to
> >>> understand/provide details of the network.
> >>
> >> I was thinking a sort of opposite. Would this work?
> >>
> >> We add a new micro-version that does this:
> >> * nothing specified: do the best we can to get a port created
> >> (get-me-a-network, etc,), or fail if not possible
> >> * --no-nics option (or similar) that says "please don't give me any nics"
> >>
> >> This means folks that don't want a network, reliably have a way to do
> >> that. For everyone else, we do the same thing when using either
> >> neutron or nova-network VLAN manager.
> > 
> > I think this pushes our microversions meaning a bit further than
> > intended. I don't think the API should flip behaviors simply based on a
> > microversion.
> > 
> > What about requiring nic information with the microversion? Make users
> > indicate explicitly if they want a network or not and avoid changing a
> > default behavior.
> 
> I think changing default behavior like that is totally within bounds,
> and was part of the original design point of microversions (and why you
> have to opt into them). So people that don't want a network that go past
> that boundary know to start saying "hands-off".
> 
> I think if the point of the experience for this API is to be working out
> of the box. So I very much like the idea of a defaults change here to
> the thing we want to be easy. And an explicit option to disable it if
> you want to 

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 12, 2016, at 01:45 PM, John Garbutt wrote:
> On 12 February 2016 at 18:17, Andrew Laski  wrote:
> >
> >
> > On Fri, Feb 12, 2016, at 12:15 PM, Matt Riedemann wrote:
> >> Forgive me for thinking out loud, but I'm trying to sort out how nova
> >> would use a microversion in the nova API for the get-me-a-network
> >> feature recently added to neutron [1] and planned to be leveraged in
> >> nova (there isn't a spec yet for nova, I'm trying to sort this out for a
> >> draft).
> >>
> >> Originally I was thinking that a network is required for nova boot, so
> >> we'd simply check for a microversion and allow not specifying a network,
> >> easy peasy.
> >>
> >> Turns out you can boot an instance in nova (with neutron as the network
> >> backend) without a network. All you get is a measly debug log message in
> >> the compute logs [2]. That's kind of useless though and seems silly.
> >>
> >> I haven't tested this out yet to confirm, but I suspect that if you
> >> create a nova instance w/o a network, you can latter try to attach a
> >> network using the os-attach-interfaces API as long as you either provide
> >> a network ID *or* there is a public shared network or the tenant has a
> >> network at that point (nova looks those up if a specific network ID
> >> isn't provided).
> >>
> >> The high-level plan for get-me-a-network in nova was simply going to be
> >> if the user tries to boot an instance and doesn't provide a network, and
> >> there isn't a tenant network or public shared network to default to,
> >> then nova would call neutron's new auto-allocated-topology API to get a
> >> network. This, however, is a behavior change.
> >>
> >> So I guess the question now is how do we handle that behavior change in
> >> the nova API?
> >>
> >> We could add an auto-create-net boolean to the boot server request which
> >> would only be available in a microversion, then we could check that
> >> boolean in the compute API when we're doing network validation.
> >>
> >
> > I think a flag like this is the right approach. If it's currently valid
> > to boot an instance without a network than there needs to be something
> > to distinguish a request that wants a network created vs. a request that
> > doesn't want a network.
> >
> > This is still hugely useful if all that's required from a user is to
> > indicate that they would like a network, they still don't need to
> > understand/provide details of the network.
> 
> I was thinking a sort of opposite. Would this work?
> 
> We add a new micro-version that does this:
> * nothing specified: do the best we can to get a port created
> (get-me-a-network, etc,), or fail if not possible
> * --no-nics option (or similar) that says "please don't give me any nics"
> 
> This means folks that don't want a network, reliably have a way to do
> that. For everyone else, we do the same thing when using either
> neutron or nova-network VLAN manager.

I think this pushes our microversions meaning a bit further than
intended. I don't think the API should flip behaviors simply based on a
microversion.

What about requiring nic information with the microversion? Make users
indicate explicitly if they want a network or not and avoid changing a
default behavior.


> 
> Thanks,
> johnthetubaguy
> 
> PS
> I think we should focus on the horizon experience, CLI experience, and
> API experience separately, for a moment, to make sure each of those
> cases actually works out OK.
> 
> >> Today if you don't specify a network and don't have a network available,
> >> then the validation in the API is basically just quota checking that you
> >> can get at least one port in your tenant [3]. With a flag on a
> >> microversion, we could also validate some other things about
> >> auto-creating a network (if we know that's going to be the case once we
> >> hit the compute).
> >>
> >> Anyway, this is mostly me getting thoughts out of my head before the
> >> weekend so I don't forget it and am looking for other ideas here or
> >> things I might be missing.
> >>
> >> [1] https://blueprints.launchpad.net/neutron/+spec/get-me-a-network
> >> [2]
> >> https://github.com/openstack/nova/blob/30ba0c5eb19a9c9628957ac8e617ae78c0c1fa84/nova/network/neutronv2/api.py#L594-L595
> >> [3]
> >> https://github.com/openstack/nova/bl

Re: [openstack-dev] [nova][neutron] How would nova microversion get-me-a-network in the API?

[Andrew Laski]

On Fri, Feb 12, 2016, at 12:15 PM, Matt Riedemann wrote:
> Forgive me for thinking out loud, but I'm trying to sort out how nova 
> would use a microversion in the nova API for the get-me-a-network 
> feature recently added to neutron [1] and planned to be leveraged in 
> nova (there isn't a spec yet for nova, I'm trying to sort this out for a 
> draft).
> 
> Originally I was thinking that a network is required for nova boot, so 
> we'd simply check for a microversion and allow not specifying a network, 
> easy peasy.
> 
> Turns out you can boot an instance in nova (with neutron as the network 
> backend) without a network. All you get is a measly debug log message in 
> the compute logs [2]. That's kind of useless though and seems silly.
> 
> I haven't tested this out yet to confirm, but I suspect that if you 
> create a nova instance w/o a network, you can latter try to attach a 
> network using the os-attach-interfaces API as long as you either provide 
> a network ID *or* there is a public shared network or the tenant has a 
> network at that point (nova looks those up if a specific network ID 
> isn't provided).
> 
> The high-level plan for get-me-a-network in nova was simply going to be 
> if the user tries to boot an instance and doesn't provide a network, and 
> there isn't a tenant network or public shared network to default to, 
> then nova would call neutron's new auto-allocated-topology API to get a 
> network. This, however, is a behavior change.
> 
> So I guess the question now is how do we handle that behavior change in 
> the nova API?
> 
> We could add an auto-create-net boolean to the boot server request which 
> would only be available in a microversion, then we could check that 
> boolean in the compute API when we're doing network validation.
> 

I think a flag like this is the right approach. If it's currently valid
to boot an instance without a network than there needs to be something
to distinguish a request that wants a network created vs. a request that
doesn't want a network.

This is still hugely useful if all that's required from a user is to
indicate that they would like a network, they still don't need to
understand/provide details of the network.



> Today if you don't specify a network and don't have a network available, 
> then the validation in the API is basically just quota checking that you 
> can get at least one port in your tenant [3]. With a flag on a 
> microversion, we could also validate some other things about 
> auto-creating a network (if we know that's going to be the case once we 
> hit the compute).
> 
> Anyway, this is mostly me getting thoughts out of my head before the 
> weekend so I don't forget it and am looking for other ideas here or 
> things I might be missing.
> 
> [1] https://blueprints.launchpad.net/neutron/+spec/get-me-a-network
> [2] 
> https://github.com/openstack/nova/blob/30ba0c5eb19a9c9628957ac8e617ae78c0c1fa84/nova/network/neutronv2/api.py#L594-L595
> [3] 
> https://github.com/openstack/nova/blob/30ba0c5eb19a9c9628957ac8e617ae78c0c1fa84/nova/network/neutronv2/api.py#L1107
> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][glance][cinder][neutron]How to make use of x-openstack-request-id

[Andrew Laski]

On Wed, Jan 27, 2016, at 05:47 AM, Kuvaja, Erno wrote:
> > -Original Message-
> > From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> > Sent: Wednesday, January 27, 2016 9:56 AM
> > To: openstack-dev@lists.openstack.org
> > Subject: Re: [openstack-dev] [nova][glance][cinder][neutron]How to make
> > use of x-openstack-request-id
> > 
> > 
> > 
> > On 1/27/2016 9:40 AM, Tan, Lin wrote:
> > > Thank you so much. Eron. This really helps me a lot!!
> > >
> > > Tan
> > >
> > > *From:*Kuvaja, Erno [mailto:kuv...@hpe.com]
> > > *Sent:* Tuesday, January 26, 2016 8:34 PM
> > > *To:* OpenStack Development Mailing List (not for usage questions)
> > > *Subject:* Re: [openstack-dev] [nova][glance][cinder][neutron]How to
> > > make use of x-openstack-request-id
> > >
> > > Hi Tan,
> > >
> > > While the cross project spec was discussed Glance already had
> > > implementation of request ids in place. At the time of the Glance
> > > implementation we assumed that one request id is desired through the
> > > chain of services and we implemented the req id to be accepted as part
> > > of the request. This was mainly driven to have same request id through
> > > the chain between glance-api and glance-registry but as the same code
> > > was used in both api and registry services we got this functionality
> > > across glance.
> > >
> > > The cross project discussion turned this approach down and decided
> > > that only new req id will be returned. We did not want to utilize 2
> > > different code bases to handle req ids in glance-api and
> > > glance-registry, nor we wanted to remove the functionality to allow
> > > the req ids being passed to the service as that was already merged to
> > > our API. Thus is requests are passed without req id defined to the
> > > services they behave (apart from nova having different header name)
> > > same way, but with glance the request maker has the liberty to specify
> > > request id they want to use (within configured length limits).
> > >
> > > Hopefully that clarifies it for you.
> > >
> > > -Erno
> > >
> > > *From:*Tan, Lin [mailto:lin@intel.com]
> > > *Sent:* 26 January 2016 01:26
> > > *To:* OpenStack Development Mailing List (not for usage questions)
> > > *Subject:* Re: [openstack-dev] [nova][glance][cinder][neutron]How to
> > > make use of x-openstack-request-id
> > >
> > > Thanks Kebane, I test glance/neutron/keystone with
> > > ``x-openstack-request-id`` and find something interesting.
> > >
> > > I am able to pass ``x-openstack-request-id``  to glance and it will
> > > use the UUID as its request-id. But it failed with neutron and keystone.
> > >
> > > Here is my test:
> > >
> > > http://paste.openstack.org/show/484644/
> > >
> > > It looks like because keystone and neutron are using
> > > oslo_middleware:RequestId.factory and in this part:
> > >
> > >
> > https://github.com/openstack/oslo.middleware/blob/master/oslo_middlew
> > a
> > > re/request_id.py#L35
> > >
> > > It will always generate an UUID and append to response as
> > > ``x-openstack-request-id`` header.
> > >
> > > My question is should we accept an external passed request-id as the
> > > project's own request-id or having its unique request-id?
> > >
> > > In other words, which one is correct way, glance or neutron/keystone?
> > > There must be something wrong with one of them.
> > >
> > > Thanks
> > >
> > > B.R
> > >
> > > Tan
> > >
> > > *From:*Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com]
> > > *Sent:* Wednesday, December 2, 2015 2:24 PM
> > > *To:* OpenStack Development Mailing List
> > > (openstack-dev@lists.openstack.org
> > > )
> > > *Subject:* Re: [openstack-dev] [nova][glance][cinder][neutron]How to
> > > make use of x-openstack-request-id
> > >
> > > Hi Tan,
> > >
> > > Most of the OpenStack RESTful API returns `X-Openstack-Request-Id` in
> > > the API response header but thisrequest id isnotavailable to the
> > > callerfromthe python client.
> > >
> > > When you use --debug option from command from the command prompt
> > using
> > > client, you can see `X-Openstack-Request-Id`on the console but it is
> > > not logged anywhere.
> > >
> > > Currently a cross-project specs [1] is submitted and approved for
> > > returning X-Openstack-Request-Id to the caller and the implementation
> > > for the same is in progress.
> > >
> > > Please go through the specs for detail information which will help you
> > > to understand more about request-ids and current work about the same.
> > >
> > > Please feel free to revert back anytime for your doubts.
> > >
> > > [1]
> > > https://github.com/openstack/openstack-
> > specs/blob/master/specs/return-
> > > request-id.rst
> > >
> > > Thanks,
> > >
> > > Abhishek Kekane
> > >
> > > Hi guys
> > >
> > >  I recently play around with 'x-openstack-request-id' header
> > > but have a dump question about how it works. At beginning, I thought
> > > an action across different services should use a same reques