date:20160108

Hi, all

Now I'm working on the migrations list/show API.
The new api defines  migrations is a sub-collection of an instance.
GET   v2.1/tenant_id/servers/server-id/migrations/migration-id

I need to support  the new cell API to get a migration of a specified
instance.
get_migration_by_instance_and_id


I read up the nova cell doc.
http://docs.openstack.org/developer/nova/cells.html

It tells us that, the migrations table will be API-level.
The instances table is Cell-level.
So does that means that we can do live migration cross cells?


And also this will affect my code.
If the migrations is API-level, so I do not need a cell name to get the
migrations info from DB.

Or I need to get the cell name by the specified instance, and then get
the migrations
info by _TargetedMessage.


BR
ShaoHe Feng
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ec2-api] Removing Nova's In tree ec2 API code in favor of ec2-api project


Davanum Srinivas wrote:

Dear Nova cores,

Sorry this did not make into the Nova weekly meeting agenda yesterday.

Is it time for dropping the EC2/ObjectStore REST API Since we've told
folks since Kilo that we will be removing this code and we dropped the
entries api-paste.ini in Liberty?
https://review.openstack.org/#/c/263368/


Could we have a quick status update on the ec2-api project ? It has not 
applied to become an OpenStack project yet...


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][stackalytics] some projects not visible in stackalytics now


joehuang wrote:

Only api-wg, openstack-user-stories, opos-tools-contrib,
opos-tools-generic, opos-tools-monitoring are listed in
http://stackalytics.com/?project_type=openstack-others

Many OpenStack other projects are disappeared.


We should take the opportunity to rename those from "openstack-others" 
to "Others", since there is nothing like an OpenStack "other project" in 
our governance. A project is either in the big tent and an OpenStack 
project, or it's not an OpenStack project.


All the projects you are mentioning are hosted on the OpenStack 
infrastructure (in the space formally known as Stackforge) but are not 
official OpenStack projects.


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][stackalytics] some projects not visible in stackalytics now


Thierry Carrez wrote:

[...]
All the projects you are mentioning are hosted on the OpenStack
infrastructure (in the space formally known as Stackforge) but are not
official OpenStack projects.


I meant "formerly" :)

--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum] Temporarily remove swarm func test from gate

2016-01-08 Thread Egor Guz

Hongbin,

I belive most failures are related to containers tests. Maybe we should comment 
only them out and keep Swarm cluster provisioning.
Thoughts?

—
Egor

On Jan 8, 2016, at 06:37, Hongbin Lu 
> wrote:

Done: https://review.openstack.org/#/c/264998/

Best regards,
Hongbin

-Original Message-
From: Adrian Otto [mailto:adrian.o...@rackspace.com]
Sent: January-07-16 10:19 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [magnum] Temporarily remove swarm func test from 
gate

Hongbin,

I’m not aware of any viable options besides using a nonvoting gate job. Are 
there other alternatives to consider? If not, let’s proceed with that approach.

Adrian

On Jan 7, 2016, at 3:34 PM, Hongbin Lu 
> wrote:

Clark,

That is true. The check pipeline must pass in order to enter the gate pipeline. 
Here is the problem we are facing. A patch that was able to pass the check 
pipeline is blocked in gate pipeline, due to the instability of the test. The 
removal of unstable test from gate pipeline aims to unblock the patches that 
already passed the check.

An alternative is to remove the unstable test from check pipeline as well or 
mark it as non-voting test. If that is what the team prefers, I will adjust the 
review accordingly.

Best regards,
Honbgin

-Original Message-
From: Clark Boylan [mailto:cboy...@sapwetik.org]
Sent: January-07-16 6:04 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [magnum] Temporarily remove swarm func
test from gate

On Thu, Jan 7, 2016, at 02:59 PM, Hongbin Lu wrote:
Hi folks,

It looks the swarm func test is currently unstable, which negatively
impacts the patch submission workflow. I proposed to remove it from
Jenkins gate (but keep it in Jenkins check), until it becomes stable.
Please find the details in the review
(https://review.openstack.org/#/c/264998/) and let me know if you
have any concern.

Removing it from gate but not from check doesn't necessarily help much because 
you can only enter the gate pipeline once the change has a +1 from Jenkins. 
Jenkins applies the +1 after check tests pass.

Clark

__
 OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
 OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-08 Thread Ben Meyer

On 01/08/2016 12:56 PM, Jim Rollenhagen wrote:
> On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote:
>> Jim Rollenhagen wrote:
>>> [...]
>>> Here's the catch - mimic is built on twisted. I know twisted was
>>> previously removed from OpenStack (or at least people said "pls no", I
>>> don't know the full history). We didn't intend to stealth-introduce
>>> twisted back into g-r, but it was pointed out to me that it may appear
>>> this way, so here I am letting everyone know. lifeless pointed out that
>>> when tests are failing, people may end up digging into mimic or twisted
>>> code, which most people in this community aren't familiar with AFAIK,
>>> which is a valid point though I hope it isn't required often.
>> A bit of history with Twisted.
>>
>> Back in 2010 we decided we could not afford asking OpenStack developers to
>> be familiar with multiple service architecture frameworks, and eventlet was
>> chosen as the simplest framework to learn and debug. The best reference I
>> found on this is still visible in the wiki:
>>
>> https://wiki.openstack.org/wiki/UnifiedServiceArchitecture
>>
>>> So, the primary question here is: do folks have a problem with adding
>>> twisted here? We're holding off on Ironic changes that depend on this
>>> until this discussion has happened, but aren't reverting the g-r change
>>> until we decide one way or another.
>> The only friction I see is how many developers would be expected to need to
>> learn Twisted in order to complete their jobs. My understanding is that
>> Twisted expertise could be needed to debug python-ironicclient functional
>> tests, which makes the cost relatively limited. So if Mimic brings in a
>> clear and significant benefit, I don't think its Twisted dependence should
>> play that much against it.
>>
>> However, I agree with Sean and Jay that the benefit is unclear -- the few
>> features that Mimic brings seem to be outweighed by the increased risk of
>> introducing a delta between the implementation and the mock. If the main
>> benefit is that it's used in other Rackspace projects for testing (like Ben
>> said), I'm not sure that makes a compelling argument for the rest of the
>> community...
> No, that is not the main benefit, at all. Ben isn't involved in Ironic
> and until now has had nothing to do with this work to add mimic here, so
> I'm not sure where he got that impression from, or why he's speaking on
> our behalf as to the goals here.
I never claimed involvement in Ironic and from the outset noted my
status on this list. I've only commented in so far as trying to show the
value of mimic and similar tools.

I'm sorry if there was any confusion in that with respect to ironic.
That was certainly not my intent.

> As pointed out before, the risk of a delta between the mocks in mimic
> and reality is identical to the risk of a delta between the mocks in a
> client's unit tests and reality, so I don't see a particular downside
> there.
Agreed.

Ben

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-08 Thread gord chung




On 08/01/16 04:49 AM, Julien Danjou wrote:

On Thu, Jan 07 2016, Jay Pipes wrote:


OK, I just watched that. Sorry, still don't see the value that Mimic provides
over unit testing the client interfaces and mocking out the HTTP payloads so
you have strict control over the expectations.

The problem that Glyph noted in the video about the unit test that mocked out
os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
find it odd that that example was used in discussing the value of Mimic. Bad
mocks are just that: bad mocks. The same false positive failure could easily be
introduced with a typo in the "metadata injection" that Mimic does to inject
failures into the system.

Mimic isn't testing anything on the server side at all so I'm not sure why
folks call it "integration testing". It isn't testing the integration of
anything at all. All it enables is multi-language client library testing, and
see my response to Ben, the surface area it introduces for bugs in the test
platform itself in my opinion outweigh the multi-language value it might have.

I completely agree with what Jay points out here.

To illustrate with what we do in the Telemetry team: in order test
gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
client against that freshly installed server. It works very well, and
we're sure we test against real data. That's what I would call
integration testing.
We could also easily test against older version of the Gnocchi server by
pip install-ing an older version if we wanted.

This has way more value than mocking the whole HTTP server and crossing
fingers we did a good job mimicking it. Oh, and it's also actually less
work. :)

¹  https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh



this happens with aodhclient as well. the one caveat i would mention is 
that it doesn't really allow for test isolation which requires you to 
put a bit more thought into your tests.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Adding "IPv6" bracket handling utilities in openstacklib.

2016-01-08 Thread Cody Herriges

Sofer Athlan-Guyot wrote:
> Hi all,
> 
> I've got an input from a fellow worker[1].  Basically, transparently
> transforming user data in puppet is opening a big can of worms.
> 
> He came up with a rather contrived example, but it's definitively worth
> discussing it.
> 
> So in the vncproxy example, if the user does this:
> 
>$vncproxy_host => 'ff02::1:ff00:1:80',
>$vncproxy_port => '80'
> 
> thinking that vncproxy_host is set to host 'ff02::1:ff00:1' with port
> ":80" (he forgot to add brackets) then the code will transform that to a
> valid uri '[ff02::1:ff00:1:80]:80'
> 
> Without the code it would give this 'ff02::1:ff00:1:80:80' which would
> fail as it lacks the brackets and is an invalid uri.
> 
> There is no way to make the difference between "wrong ipv6 + port" and
> "valid ipv6", so mangling user input can lead to unexpected result.
> 
> I'm going to put the patches on WIP, as maybe, this may not be a good
> idea to have user input transformed at all in puppet as all corner cases
> cannot be detected.
> 
> The trade-off, of course, is user convenience.
> 
> So what do you think, parse and transform user input or not ?
> 
> [1] thanks Lukas
> 

Since I was already skeptical and personally shy away from introducing
implicit behavior, then you bring up a concrete example of why it might
be a bad idea...pretty solidly against the transformation now.

To just provided more feedback to the user we could pass the
vncproxy_host value through a couple regexes and report a warning,
maybe?  If not regex match IPv4 and not regex match brackets at
beginning and end then tell them it is not valid.

-- 
Cody



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] Seeking FFE for "Support single disk image OVA/OVF package upload"

On 08/01/16 17:03 +, stuart.mcla...@hp.com wrote:

Hello Glance Team!
Hope you had a wonderful vacation and wishing you health and
happiness for 2016.

Would very much appreciate your considering https://review.openstack.org/194868
for a feature freeze exception.

I believe the spec is pretty solid, and we can deliver on the implementation by
M-2. But were unable to get enough core

Votes during the holiday season.

Regards

Malini

-- next part --
An HTML attachment was scrubbed...
URL:

I downloaded the patch which implements the spec
(https://review.openstack.org/#/c/214810):

I can make this REST API call to perform OVA import:

http://paste.openstack.org/show/483332/

(it exercises the new OVA extract code path).

There's a parallel effort in the project to provide 'official' (Defcore)
APIs for image upload/conversion. What will be the advantage of having
two different REST API calls (a 'tasks' based one and a DefCore one)
for importing OVAs?

As you mentioned above, the team is working on refactoring the image
import process for Glance. The solution has different requirements and
dependencies. One of those dependencies is making the existing task
API admin only to then be able to deprecate it in the future.

This has been discussed several times at the summit, in meetings and,
to make sure it's clear to everyone (apparently it isn't) it's also
been made of the spec of the refactor you mentioned[0] (see work
items).

[0]
https://specs.openstack.org/openstack/glance-specs/specs/mitaka/approved/image-import/image-import-refactor.html#work-items

It seems to be possible to successfully make the above API call whether
you're admin or not and whether the server has the patch applied or
not. Is that expected?

You'll be able to run that request until this[0] is done. In addition
to this work, there's also the requirement to make the task executable
only by admins. This has been explicitly mentioned in the OVF spec and
we need to test/enforce that the code respects this.

Note that we're evaluating an exception for the *spec* and not the
code. Therefore, using the current version of the code as a reference
rather than what's in the spec is probably not ideal.

One final note that I'd like to make. The *task class/implementation*
has nothing to do with the API. It can be functionally tested without
API calls and it can be disabled. The fact that you can run it using
the old task API doesn't mean that you should or that it's what we're
recomending. The old task API is taking its first step down the
deprecation path and it'll take some time for that to happen. This, I
insist, does not mean the team is encouraging such API.

The OVF work was delayed in Kilo. We also blocked in Liberty because
we knew the upload path needed to be refectored. In Mitaka we blocked
it until the very end of the spec review process because we wanted to
make sure it wouldn't interfeer with the priorities of the cycle. Now
that we know that, I can't hardly think of a reason to not let this
through. One motivation is that I don't think it'll confuse folks as
we're clearly saying (in code, communications and whatnot) that the
old task API should go away.

Sure, some ppl don't listen and the world isn't perfect but there are
trade offs and those are the ones we're evaluating.

[0] https://bugs.launchpad.net/glance/+bug/1527716

Cheers,
Flavio

-Stuart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
@flaper87
Flavio Percoco

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [API] [Glance] New v2 Image Import Specification Discussion

2016-01-08 Thread Ian Cordasco

-Original Message-
From: Ian Cordasco 
Reply: OpenStack Development Mailing List (not for usage questions) 

Date: January 8, 2016 at 11:16:35
To: OpenStack Development Mailing List (not for usage questions) 

Subject:  [openstack-dev] [API] [Glance] New v2 Image Import Specification  
Discussion

[snip]  
> So what if we modeled this a little
>  
> 1. The user creates an image record (image-create)
> 2. The user uses the link in the response to upload the image data (we don't 
> care whether  
> it is direct or not because the user can follow it without thinking about it)
> 3. The user sends a request to process the image without having to specify 
> what to translate  
> the image to because the operator has chosen a preferred image type (and 
> maybe has other  
> acceptable types that mean the image will not need to be converted if it's 
> already of that  
> type).
>  
> This adds an extra step and it could remove the need for an "uploading" 
> status.
>  
> This also means that the user does not need to be cognizant of the method 
> that they're using  
> to create an image (glance-direct, swift-local, foo-bar-bogus-whatever, etc.) 
> and  
> that they do not need to think too hard about the target format for that 
> cloud or pre-process  
> any extra information or worry whether that cloud needs some location URL for 
> an indirect  
> upload. Glance can manage that complexity for the user.
>  
> Now let's talk about step 2 briefly. If the operator wants all image bytes to 
> go to Glance  
> directly, this can be the default behaviour. In that case, the URL would be 
> "https:///v2/images//files;  
> and the user would do it like the default PUT behaviour that we have now. If 
> it's an indirect  
> upload, then the user would make another PUT request with that URL and their 
> authentication  
> credentials. If the user is out of storage quota than that service will tell 
> the user.  
>  
> I think in this way, we can provide the user some information about their 
> uploaded data  
> as well as well as letting admins handle quota. This still doesn't discuss 
> giving admin  
> more visibility (in a direct upload case) about quota of uploaded but not yet 
> processed  
> data. That's an important question to have, but I don't have an easy answer 
> for it though.  
> I hope someone else does.

So one edge case/snag in this workflow is the following:

Once an image is in the "active" state, what happens to the URL that is 
returned as part of the image representation for where to upload data?

I don't think we want to have an attribute that disappears once an image 
becomes active. I think it would make sense to return `null` in that case, but 
I'd like to hear other people's thoughts.

Also, to prevent questions, I do envision that before an image is processed 
that the user could upload several times (overwriting each time) the image data 
to the same resource. Hopefully that makes sense to everyone. (This is 
identical to what's being proposed in the current spec.)

--  
Ian Cordasco
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Glance] Nova + Glance_v2 = Love

2016-01-08 Thread Sam Matzek

On Fri, Jan 8, 2016 at 11:54 AM, Sam Matzek  wrote:
> On Fri, Jan 8, 2016 at 8:31 AM, Flavio Percoco  wrote:
>> On 29/12/15 07:41 -0600, Sam Matzek wrote:
>>>
>>> On Thu, Dec 24, 2015 at 7:49 AM, Mikhail Fedosin 
>>> wrote:

 Hello, it's another topic about glance v2 adoption in Nova, but it's
 different from the others. I want to declare that there is a set of
 commits,
 that make Nova version agnostic and allow to work with both glance apis.
 The
 idea of the solution is to determine the current api version in the
 beginning and make appropriate requests after.
 (https://review.openstack.org/#/c/228578/,
 https://review.openstack.org/#/c/238309/,
 https://review.openstack.org/#/c/259097/)

 Indeed, it requires some additional (and painful) work, but now all
 tempest
 tests pass in Jenkins.

 Note: this thread is not about xenplugin, there is another topic, called
 'Xenplugin + Glance_v2 = Hate'

 Here the main issues we faced and how we've solved them:

 1. "changes-since" filter for image-list is not supported in v2 api.
 Steve
 Lewis made a great job and implemented a set of filters with comparison
 operators https://review.openstack.org/#/c/197388/. Filtering by
 'changes-since' is completely similar to 'gte:updated_at'.

 2. Filtering by custom properties must have prefix 'property-'. In v2
 it's
 not required.

 3. V2 states that all custom properties must be image attributes, but
 Nova
 assumes that they are included in 'properties' dictionary. It's fixed
 with
 glanceclient's method 'is_base_property(prop_name)', that returns False
 in
 case of custom property.

 4. is_public=True/False is visibility="public"/"private" respectively.

 5. Deleting custom image properties in Nova is performed with
 'purge_props'
 flag. If it's set to True, then all prop names, that are not included in
 updated data, will be removed. In case of v2 we have to explicitly
 specify
 prop names in the list param 'remove_props'. To implement this behaviour,
 if
 'purge_props' is set, we make additional 'get' request to determine the
 existing properties and put in 'remove_prop' list only those, that are
 not
 in updated_data.

 6. My favourite:
 There is an ability to activate an empty image by just providing 'size =
 0':
 https://review.openstack.org/#/c/9715/, in this case image will be a
 collection of metadata. Glance v2 doesn't support this "feature" and
 that's
 why we have to implement a very dirty workaround:
 * v2 requires, that disk_format and container-format must be set
 before
 the activation. if these params are not provided to 'create' method then
 we
 hardcode it to 'qcow2' and 'bare'.
 * we call 'upload' method with empty data (data = '') to activate
 image.
 Me (and the rest glance team) think that this image activation with
 zero-size is inconsistent and we won't implement it in v2. So, I'm going
 to
 ask if Nova really needs this feature and maybe it's possible to make it
 work without it.
>>>
>>>
>>> Nova uses this functionality when it creates snapshots of volume
>>> backed instances, that is, instances that only have Cinder volumes
>>> attached and do not have an ephemeral disk.
>>> In this case Nova API creates Cinder snapshots for the Cinder volumes
>>> and builds the block_device_mapping property with block devices that
>>> reference the Cinder snapshots.  Nova activates this image with size=0
>>> because this image does not have a disk and simply refers to the
>>> collection of Cinder snapshots that collectively comprise the binary
>>> image.  Callers of Glance outside of Nova may also use the APIs to
>>> create "block device mapping" images as well that contain references
>>> to Cinder volumes to attach, blank volumes to create, snapshots to
>>> create volumes from, etc during the server creation.  Not having the
>>> ability to create these images with V2 is a loss of function.
>>
>>
>> I disagree. Being able to activate empty images breaks the consistency
>> of Glances API and I don't think it should be considered a feature but
>> a bug. An active image is an image that can be used to *boot* a VM. If
>> the image is empty, you simply can't do that.
>>
>> Note that pointing to a block device makes the image "non-empty"
>> already. There's a spec merged[0] and code written[1] to allow for
>> using cinder as backend for Glance.
>>
>> If there's a missing functionality for the mapping users require, I
>> believe activating empty images is not the solution for it.
>>
>> [0] https://review.openstack.org/183363
>> [1] https://review.openstack.org/166414
>>
>>> The callers could supply 1 byte of junk data, like a space character,
>>> but that would

Re: [openstack-dev] [Nova][Glance] Nova + Glance_v2 = Love

2016-01-08 Thread Sam Matzek

On Fri, Jan 8, 2016 at 8:31 AM, Flavio Percoco  wrote:
> On 29/12/15 07:41 -0600, Sam Matzek wrote:
>>
>> On Thu, Dec 24, 2015 at 7:49 AM, Mikhail Fedosin 
>> wrote:
>>>
>>> Hello, it's another topic about glance v2 adoption in Nova, but it's
>>> different from the others. I want to declare that there is a set of
>>> commits,
>>> that make Nova version agnostic and allow to work with both glance apis.
>>> The
>>> idea of the solution is to determine the current api version in the
>>> beginning and make appropriate requests after.
>>> (https://review.openstack.org/#/c/228578/,
>>> https://review.openstack.org/#/c/238309/,
>>> https://review.openstack.org/#/c/259097/)
>>>
>>> Indeed, it requires some additional (and painful) work, but now all
>>> tempest
>>> tests pass in Jenkins.
>>>
>>> Note: this thread is not about xenplugin, there is another topic, called
>>> 'Xenplugin + Glance_v2 = Hate'
>>>
>>> Here the main issues we faced and how we've solved them:
>>>
>>> 1. "changes-since" filter for image-list is not supported in v2 api.
>>> Steve
>>> Lewis made a great job and implemented a set of filters with comparison
>>> operators https://review.openstack.org/#/c/197388/. Filtering by
>>> 'changes-since' is completely similar to 'gte:updated_at'.
>>>
>>> 2. Filtering by custom properties must have prefix 'property-'. In v2
>>> it's
>>> not required.
>>>
>>> 3. V2 states that all custom properties must be image attributes, but
>>> Nova
>>> assumes that they are included in 'properties' dictionary. It's fixed
>>> with
>>> glanceclient's method 'is_base_property(prop_name)', that returns False
>>> in
>>> case of custom property.
>>>
>>> 4. is_public=True/False is visibility="public"/"private" respectively.
>>>
>>> 5. Deleting custom image properties in Nova is performed with
>>> 'purge_props'
>>> flag. If it's set to True, then all prop names, that are not included in
>>> updated data, will be removed. In case of v2 we have to explicitly
>>> specify
>>> prop names in the list param 'remove_props'. To implement this behaviour,
>>> if
>>> 'purge_props' is set, we make additional 'get' request to determine the
>>> existing properties and put in 'remove_prop' list only those, that are
>>> not
>>> in updated_data.
>>>
>>> 6. My favourite:
>>> There is an ability to activate an empty image by just providing 'size =
>>> 0':
>>> https://review.openstack.org/#/c/9715/, in this case image will be a
>>> collection of metadata. Glance v2 doesn't support this "feature" and
>>> that's
>>> why we have to implement a very dirty workaround:
>>> * v2 requires, that disk_format and container-format must be set
>>> before
>>> the activation. if these params are not provided to 'create' method then
>>> we
>>> hardcode it to 'qcow2' and 'bare'.
>>> * we call 'upload' method with empty data (data = '') to activate
>>> image.
>>> Me (and the rest glance team) think that this image activation with
>>> zero-size is inconsistent and we won't implement it in v2. So, I'm going
>>> to
>>> ask if Nova really needs this feature and maybe it's possible to make it
>>> work without it.
>>
>>
>> Nova uses this functionality when it creates snapshots of volume
>> backed instances, that is, instances that only have Cinder volumes
>> attached and do not have an ephemeral disk.
>> In this case Nova API creates Cinder snapshots for the Cinder volumes
>> and builds the block_device_mapping property with block devices that
>> reference the Cinder snapshots.  Nova activates this image with size=0
>> because this image does not have a disk and simply refers to the
>> collection of Cinder snapshots that collectively comprise the binary
>> image.  Callers of Glance outside of Nova may also use the APIs to
>> create "block device mapping" images as well that contain references
>> to Cinder volumes to attach, blank volumes to create, snapshots to
>> create volumes from, etc during the server creation.  Not having the
>> ability to create these images with V2 is a loss of function.
>
>
> I disagree. Being able to activate empty images breaks the consistency
> of Glances API and I don't think it should be considered a feature but
> a bug. An active image is an image that can be used to *boot* a VM. If
> the image is empty, you simply can't do that.
>
> Note that pointing to a block device makes the image "non-empty"
> already. There's a spec merged[0] and code written[1] to allow for
> using cinder as backend for Glance.
>
> If there's a missing functionality for the mapping users require, I
> believe activating empty images is not the solution for it.
>
> [0] https://review.openstack.org/183363
> [1] https://review.openstack.org/166414
>
>> The callers could supply 1 byte of junk data, like a space character,
>> but that would result in a junk image being stored in Glance's default
>> backing store for the image.  It would also give the impression that a
>> real disk image exists for the image in a backing

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote:
> Jim Rollenhagen wrote:
> >[...]
> >Here's the catch - mimic is built on twisted. I know twisted was
> >previously removed from OpenStack (or at least people said "pls no", I
> >don't know the full history). We didn't intend to stealth-introduce
> >twisted back into g-r, but it was pointed out to me that it may appear
> >this way, so here I am letting everyone know. lifeless pointed out that
> >when tests are failing, people may end up digging into mimic or twisted
> >code, which most people in this community aren't familiar with AFAIK,
> >which is a valid point though I hope it isn't required often.
> 
> A bit of history with Twisted.
> 
> Back in 2010 we decided we could not afford asking OpenStack developers to
> be familiar with multiple service architecture frameworks, and eventlet was
> chosen as the simplest framework to learn and debug. The best reference I
> found on this is still visible in the wiki:
> 
> https://wiki.openstack.org/wiki/UnifiedServiceArchitecture
> 
> >So, the primary question here is: do folks have a problem with adding
> >twisted here? We're holding off on Ironic changes that depend on this
> >until this discussion has happened, but aren't reverting the g-r change
> >until we decide one way or another.
> 
> The only friction I see is how many developers would be expected to need to
> learn Twisted in order to complete their jobs. My understanding is that
> Twisted expertise could be needed to debug python-ironicclient functional
> tests, which makes the cost relatively limited. So if Mimic brings in a
> clear and significant benefit, I don't think its Twisted dependence should
> play that much against it.
> 
> However, I agree with Sean and Jay that the benefit is unclear -- the few
> features that Mimic brings seem to be outweighed by the increased risk of
> introducing a delta between the implementation and the mock. If the main
> benefit is that it's used in other Rackspace projects for testing (like Ben
> said), I'm not sure that makes a compelling argument for the rest of the
> community...

No, that is not the main benefit, at all. Ben isn't involved in Ironic
and until now has had nothing to do with this work to add mimic here, so
I'm not sure where he got that impression from, or why he's speaking on
our behalf as to the goals here.

As pointed out before, the risk of a delta between the mocks in mimic
and reality is identical to the risk of a delta between the mocks in a
client's unit tests and reality, so I don't see a particular downside
there.

Again, I think "benefit is unclear" isn't a valid reason to block this,
so unless someone posts a revert we're going to move forward with this.

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposal for new metrics-threshold-filter

2016-01-08 Thread SURO


Hi John,
Thanks for the reply!

Originally, I had uploaded the code for review on 2nd Dec [1].
As I did not receive any review/feedback on the same, I thought having a 
blue-print/spec may help people review the same. So I had put the 
bp/spec up for review on 14th Dec [2].


As it is a plugin-based light-weight feature[3], it may not necessarily 
require the bp/spec and if we can complete the review, we can get the 
code in before the feature freeze.


I would request you and the community to consider the code for merge ...

[1] - https://review.openstack.org/#/c/254423/
[2] - https://review.openstack.org/#/c/257596/
[3] 
-http://docs.openstack.org/developer/nova/process.html#how-do-i-get-my-code-merged 



Regards,
SURO
irc//freenode: suro-patz

On 1/8/16 4:47 AM, John Garbutt wrote:

On 7 January 2016 at 18:49, SURO  wrote:

Hi all,

I have proposed a nova-spec[1] for a new scheduling filter based on
metrics thresholds. This is slightly different than weighted metrics filter.
The rationale and use-case is explained in detail in the spec[1].

The implementation is also ready for review[2]. The effort is tracked with
a blueprint[3].

I request the community to review them and provide valuable feedback.


[1] - https://review.openstack.org/#/c/257596/
[2] - https://review.openstack.org/#/c/254423/
[3] - https://blueprints.launchpad.net/nova/+spec/metrics-threshold-filter

This is very related to the ideas I have written up here:
https://review.openstack.org/#/c/256323/

Please note we have not been accepting new specs for the Mitaka
release, mostly because we have feature freeze in a few weeks.
For more information please see:
http://docs.openstack.org/releases/schedules/mitaka.html#m-nova-bp-freeze
http://docs.openstack.org/developer/nova/process.html#how-do-i-get-my-code-merged

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][bugs] help needed: volunteers for bug skimming (1 week)

2016-01-08 Thread Augustina Ragwitz

I signed up for the week before the Nova Midcycle meeting. Even though 
I'm still new, I feel capable of at least getting the tags mostly right. 
When I'm not quite sure which tag it would fall under, I search for 
similar bugs and see how they were tagged.


I did have one clarifying question, what exactly are the expectations of 
the bug skimming duty? I assumed it was just tagging bugs to get them 
into the appropriate queues for triaging. Do we also need to confirm the 
bugs once they've been tagged or does that fall under "triage" and not 
"skimming"?


Thanks!
Augustina

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

On Fri, Jan 08, 2016 at 03:39:51PM -0500, Jay Pipes wrote:
> On 01/08/2016 02:52 PM, Jim Rollenhagen wrote:
> >On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> >>On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> >snippity snip snip
> >
> >>>We haven't made it a dep for anything yet, only added to g-r.
> >>
> >>According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> >>makes functionally any difference, though (pun intended).
> >
> >Both. https://review.openstack.org/#/c/220268/
> >
> >This thread was originally about twisted, which is added to u-c with the
> >introduction of mimic.
> 
> Got it, thanks.
> 
> >>>However, now that you mention that, a really ambitious goal would be to
> >>>add a rabbit interface to mimic, and functionally test the API server
> >>>(that it sends the right messages, etc). Another would be to mimic
> >>>(Neutron, Glance) and test Ironic by itself.
> >>
> >>So you would reimplement AMQP communication protocols using an in-memory
> >>data store for queues. Sounds like an even greater surface area for bugs to
> >>be introduced.
> >>
> >>>Last, I frankly don't understand why there's
> >>>such heavy opposition to the ironic team using an additional tool for
> >>>testing.
> >>
> >>Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> >>though.
> >>
> >>a) Because it fractures the testing and QA processes used by upstream
> >>contributors that work on OpenStack projects by requiring them to learn
> >>another system -- and one that potentially would require them to understand
> >>a whole new surface area for potential bugs
> >
> >I don't think there's a large risk of needing to dig deep into mimic,
> >and especially twisted. If this does prove to be a problem, I'm happy to
> >remove it. However, we can't even explore what it would be like, or how
> >hard we would depend on mimic, without mimic being in g-r.
> 
> Sure, fair enough.
> 
> >>b) Because it represents yet another RAX-driven divergence in the QA space.
> >>CloudCafe took essentially all of the RAX folks that were at one point
> >>working on Tempest and upstream QA and siloed them into a totally different
> >>organization, in true RAX fashion. Instead of pulling the OpenStack QA
> >>community along together, RAX QA continues to just do its own thing and
> >>there's still bitterness on the tips of tongues.
> >
> >So, this isn't trying to replace anything. This is adding a different
> >way to run functional tests, that is *much* faster than standing up a
> >full ironic environment. This is helpful for developers that want to
> >quickly run tests before posting them to gerrit, people that need to
> >test in constrained environments, etc.
> 
> I recognize it is much faster than standing up a real environment. And I
> recognize that running faster client tests is a useful thing -- as long as
> we can be confident that what is tested does not suffer from some of the
> issues I identified earlier (syncing with real API and introduction of
> greater surface area for bugs in the test platform itself).
> 
> >I'm 100% against doing things like Rackspace did with tempest and
> >cloudcafe, and I wouldn't be supporting this effort if I felt it was
> >similar. Here's how this went:
> >
> >* Lekha started working on OnMetal QA, with a goal of doing some amount
> >   of upstream work as well.
> >
> >* She's previously worked on projects (like autoscale) that interact
> >   with OpenStack APIs, and seeing the need to test without a full nova
> >   environment, built mimic.
> >
> >* In talking with some of the other Ironic folks working on QA (from
> >   Intel, IBM, more), she presented mimic as something that may be useful
> >   for testing the client (and more). They (and I) agreed it was a neat
> >   idea worth trying.
> >
> >* Jay offered to help with the global-requirements patch as it's
> >   something he's done before, and did the review grinding here.
> >
> >* It finally landed, and lifeless asked me to bring up the Twisted
> >   conversation on the list. Note that this is not the "is mimic
> >   useful" conversation we're having now. Nobody remotely voiced
> >   concerns about using a new test tool until this thread happened.
> >
> >Please do let me know if any of that seems nefarious; I hope it doesn't.
> 
> No, nothing nefarious there. Sorry for letting my personal frustrations
> bubble over into this.
> 
> I am not blocking anything from going forward and I definitely am not asking
> for a revert of any g-r patch. Nor am I trying to obstruct you in your
> governance of Ironic.
> 
> I was just raising my concerns as an OpenStack citizen and getting my
> opinion out on paper.

As you should; thanks for doing that. I'm eager to see how this goes. :)

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [Fuel] Openstack version changes affects Fuel-CI

2016-01-08 Thread Sergii Golovatiuk

Hi,

We finished the maintenance a while ago. Please rebase your patches to
respin CI. Thank you everyone for your patience...

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Fri, Jan 8, 2016 at 3:59 PM, Sergii Golovatiuk 
wrote:

> Hi crew,
>
> Igor and I are about to merge
>
> https://review.openstack.org/#/c/238846/
> https://review.openstack.org/#/c/235254/
>
> These patches are destructive to our CI. They require to upload a new ISO
> to CI slaves. However, the process takes 2-3 hours to roll our a new ISO.
> Please do not submit patches to gerrit within next 3 hours for
> 'fuel-library' master branch as CI will fail. CI for all submitted patches
> during this period of time must be retriggered.
>
> I will send another announce when CI is unblocked.
>
> --
> Best regards,
> Sergii Golovatiuk,
> Skype #golserge
> IRC #holser
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] Nominating Mariam John to Trove Core

2016-01-08 Thread Vyvial, Craig

Looks like we have a consensus from the team and I’d like to welcome Mariam 
John to the Trove Core Team!

I look forward to continuing to work with Mariam!

Thanks,
-Craig

On Jan 7, 2016, at 12:21 PM, Nikhil Manchanda 
> wrote:

+1 from me as well.
Couldn't agree more!

Thanks,
-Nikhil

On Wed, Jan 6, 2016 at 12:03 PM, Peter Stachowski 
> wrote:
I agree!

+1

Peter

-Original Message-
From: Vyvial, Craig [mailto:craig.vyv...@hpe.com]
Sent: January-06-16 2:40 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] Nominating Mariam John to Trove Core

Thanks for the nomination Amrith and I think Mariam will be a great addition to 
the core team.

+1

-Craig

On Jan 6, 2016, at 12:39 PM, Amrith Kumar 
>>
 wrote:

Members of the Trove team,

I would like to nominate Mariam John (johnma on IRC) to the Trove core review 
team.

Mariam has been an active member of the Trove team for some time now. She added 
support for IBM DB2 in Trove and provided elements for building Trove guest 
images. She also implemented code that provided CouchDB support in Trove. She 
has been an active reviewer and I have found her review comments to be 
insightful and useful.

You can review her activity report at

http://stackalytics.com/report/users/mariamj

Members of the Trove core team, please reply to this message with your feedback 
to this proposed change.

Thanks,

-amrith

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

Excerpts from Jim Rollenhagen's message of 2016-01-07 11:09:32 -0800:
> Hi all,
> 
> A change to global-requirements[1] introduces mimic, which is an http
> server that can mock various APIs, including nova and ironic, including
> control of error codes and timeouts. The ironic team plans to use this
> for testing python-ironicclient without standing up a full ironic
> environment.
> 
> Here's the catch - mimic is built on twisted. I know twisted was
> previously removed from OpenStack (or at least people said "pls no", I
> don't know the full history). We didn't intend to stealth-introduce
> twisted back into g-r, but it was pointed out to me that it may appear
> this way, so here I am letting everyone know. lifeless pointed out that
> when tests are failing, people may end up digging into mimic or twisted
> code, which most people in this community aren't familiar with AFAIK,
> which is a valid point though I hope it isn't required often.
> 
> So, the primary question here is: do folks have a problem with adding
> twisted here? We're holding off on Ironic changes that depend on this
> until this discussion has happened, but aren't reverting the g-r change
> until we decide one way or another.
> 
> // jim
> 
> [1] https://review.openstack.org/#/c/220268/
> 

I looked through the Ironic specs for anything mentioning "mimic" (using
the search box) and didn't find anything. Is there a plan or something
describing what's happening that I can read as background for this?

Doug


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
snippity snip snip

> >We haven't made it a dep for anything yet, only added to g-r.
> 
> According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> makes functionally any difference, though (pun intended).

Both. https://review.openstack.org/#/c/220268/

This thread was originally about twisted, which is added to u-c with the
introduction of mimic.

> >However, now that you mention that, a really ambitious goal would be to
> >add a rabbit interface to mimic, and functionally test the API server
> >(that it sends the right messages, etc). Another would be to mimic
> >(Neutron, Glance) and test Ironic by itself.
> 
> So you would reimplement AMQP communication protocols using an in-memory
> data store for queues. Sounds like an even greater surface area for bugs to
> be introduced.
> 
> >Last, I frankly don't understand why there's
> >such heavy opposition to the ironic team using an additional tool for
> >testing.
> 
> Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> though.
> 
> a) Because it fractures the testing and QA processes used by upstream
> contributors that work on OpenStack projects by requiring them to learn
> another system -- and one that potentially would require them to understand
> a whole new surface area for potential bugs

I don't think there's a large risk of needing to dig deep into mimic,
and especially twisted. If this does prove to be a problem, I'm happy to
remove it. However, we can't even explore what it would be like, or how
hard we would depend on mimic, without mimic being in g-r.

> b) Because it represents yet another RAX-driven divergence in the QA space.
> CloudCafe took essentially all of the RAX folks that were at one point
> working on Tempest and upstream QA and siloed them into a totally different
> organization, in true RAX fashion. Instead of pulling the OpenStack QA
> community along together, RAX QA continues to just do its own thing and
> there's still bitterness on the tips of tongues.

So, this isn't trying to replace anything. This is adding a different
way to run functional tests, that is *much* faster than standing up a
full ironic environment. This is helpful for developers that want to
quickly run tests before posting them to gerrit, people that need to
test in constrained environments, etc.

I'm 100% against doing things like Rackspace did with tempest and
cloudcafe, and I wouldn't be supporting this effort if I felt it was
similar. Here's how this went:

* Lekha started working on OnMetal QA, with a goal of doing some amount
  of upstream work as well.

* She's previously worked on projects (like autoscale) that interact
  with OpenStack APIs, and seeing the need to test without a full nova
  environment, built mimic.

* In talking with some of the other Ironic folks working on QA (from
  Intel, IBM, more), she presented mimic as something that may be useful
  for testing the client (and more). They (and I) agreed it was a neat
  idea worth trying.

* Jay offered to help with the global-requirements patch as it's
  something he's done before, and did the review grinding here.

* It finally landed, and lifeless asked me to bring up the Twisted
  conversation on the list. Note that this is not the "is mimic
  useful" conversation we're having now. Nobody remotely voiced
  concerns about using a new test tool until this thread happened.

Please do let me know if any of that seems nefarious; I hope it doesn't.

> 
> 
> 
> >There was *more* than sufficient time for "I don't think this is useful"
> >to be posted on the review.
> 
> Sorry, this was the first I'd heard of it all.
> 
> > If anyone has a hard technical reason why
> >mimic should not be used to test an OpenStack project, I'm happy to
> >listen. Otherwise I'd like to work on actually getting things done.
> 
> I've listed my hard technical reason: it introduces, IMHO, too large of a
> surface area for bugs to creep in to the testing process versus the benefit
> it provides.

Again, I don't think that's any larger of a surface area than mocks in client
unit tests being incorrect, and we really won't know whether it causes
problems in reality.

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][bugs] help needed: volunteers for bug skimming (1 week)

2016-01-08 Thread Anthony Chow

Cool.

And how or where did you sign up for this skimming duty?

thanks and have a nice weekend,

anthony.

On Fri, Jan 8, 2016 at 12:01 PM, Augustina Ragwitz 
wrote:

> On 2016-01-08 11:27, Anthony Chow wrote:
>
>> There is an URL for the duty:
>>
>> https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty
>> [3]
>>
>
> Right, I read the wiki and it wasn't clear to me. Here's what the wiki
> says:
>
> "Since Nova is such an active project with a high number of incoming bug
> reports, we have a separate process for categorizing bugs and distributing
> the load of triage to a larger group of people."
>
> and
>
> "To help make sure that the triage queue stays under control, the
> following table lists the people that have committed to regularly triaging
> bugs for a given tag."
>
> The term "skimming" implies something done quickly at a surface level
> (like skimming off the fat or skimming a book). So I'm just asking, as part
> of "skimming" duty, are we also confirming the bugs, (confirming looks like
> it falls under the "Triaging" step), or is "skimming" just about tagging
> and getting the bugs into the triaging queues?
>
> Thanks!
>
> Augustina
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][bugs] help needed: volunteers for bug skimming (1 week)

2016-01-08 Thread Augustina Ragwitz

I think I just answered my own question ;) I looked at the blog post 
Markus provided in another part of this thread and it looks like 
"Skimming Duty" means both tag the bugs and the triaging step of 
confirming them such that they should no longer be in "New" status after 
you've touched them. I was confused because when I was looking at bugs 
in Launchpad, many had been tagged but left as "New" thus my need for 
clarification. Markus' blog post [1] is more clear than than the wiki on 
the triage expectations.


Also, Anthony, you can follow the instructions in Markus' original post. 
Just edit the wiki page and add your name to the table. The best irc 
channel for nova issues is #openstack-nova. You might want to see the 
Nova How to Get Involved document [2].


[1] 
http://markuszoeller.github.io/posts/openstack-bugs/#status-and-contributor-responsibility

[2] http://docs.openstack.org/developer/nova/how_to_get_involved.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][bugs] help needed: volunteers for bug skimming (1 week)

2016-01-08 Thread Anthony Chow

thanks so much. :)

On Fri, Jan 8, 2016 at 12:33 PM, Augustina Ragwitz 
wrote:

> I think I just answered my own question ;) I looked at the blog post
> Markus provided in another part of this thread and it looks like "Skimming
> Duty" means both tag the bugs and the triaging step of confirming them such
> that they should no longer be in "New" status after you've touched them. I
> was confused because when I was looking at bugs in Launchpad, many had been
> tagged but left as "New" thus my need for clarification. Markus' blog post
> [1] is more clear than than the wiki on the triage expectations.
>
> Also, Anthony, you can follow the instructions in Markus' original post.
> Just edit the wiki page and add your name to the table. The best irc
> channel for nova issues is #openstack-nova. You might want to see the Nova
> How to Get Involved document [2].
>
> [1]
> http://markuszoeller.github.io/posts/openstack-bugs/#status-and-contributor-responsibility
> [2] http://docs.openstack.org/developer/nova/how_to_get_involved.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-01-08 Thread Mike Bayer

On 01/08/2016 04:44 AM, Radomir Dopieralski wrote:
> On 01/07/2016 05:55 PM, Mike Bayer wrote:
> 
>> but also even if you're under something like
>> mod_wsgi, you can spawn a child process or worker thread regardless.
>> You always have a Python interpreter running and all the things it can
>> do.
> 
> Actually you can't, reliably. Or, more precisely, you really shouldn't.
> Most web servers out there expect to do their own process/thread
> management and get really embarrassed if you do something like this,
> resulting in weird stuff happening.

I have to disagree with this as an across-the-board rule, partially
because my own work in building an enhanced database connection
management system is probably going to require that a background thread
be running in order to reap stale database connections.   Web servers
certainly do their own process/thread management, but a thoughtfully
organized background thread in conjunction with a supporting HTTP
service allows this to be feasible.   In the case of mod_wsgi,
particularly when using mod_wsgi in daemon mode, spawning of threads,
processes and in some scenarios even wholly separate applications are
supported use cases.

In mod_wsgi daemon mode (which is its recommended mode of use [1]), the
Python interpreter is not in-process with Apache in any case, and if you
set your own thread to be a "daemon", it won't block the process from
exiting.   I have successfully used this technique (again, carefully and
thoughtfully) to achieve asynchronous workers within Apache mod_wsgi
daemon-mode processes, without negative consequences.

Graham Dumpleton's own mod_wsgi documentation illustrates how to run a
background thread on development servers in mod_wsgi daemon mode in
order to achieve code reloading [2], and he also has produced a tool [3]
that uses a background thread in order to provide a debugging shell to a
running WSGI application which can work in either embedded or daemon
mode.

In [4], he illustrates using the WSGIImportScript mod_wsgi directive
under mod_wsgi daemon mode to actually spawn a whole docker container
when an Apache mod_wsgi process starts up; this isn't something I'd want
to do myself, but this is the author of mod_wsgi illustrating even
something as heavy as spinning up a whole docker instance under mod_wsgi
which then runs it's own WSGI process as a supported technique.

It is certainly reasonable that not all web application containers would
be effective with apps that include custom background threads or
processes (even though IMO any system that's running a Python
interpreter shouldn't have any issues with a limited number of
well-behaved daemon-mode threads), but at least in the case of mod_wsgi,
this is supported; that gives Openstack's HTTP-related applications with
carefully/thoughtfully organized background threads at least one
industry-standard alternative besides being forever welded to its
current homegrown WSGI server implementation.

[1] http://lanyrd.com/2013/pycon/scdyzk/

[2] https://code.google.com/p/modwsgi/wiki/ReloadingSourceCode

[3] https://github.com/GrahamDumpleton/ispyd

[4]
http://blog.dscpl.com.au/2015/07/using-apache-to-start-and-manage-docker.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [celery][taskflow] Reg. celery and task-flow


So actually they are quite different, (although similar at some level),

Given that celery isn't really a replacement for taskflow although one 
could say, from what I've heard from others, that taskflow is a 
super-set of what celery is so taskflow likely can replace parts of 
celery (but not vice-versa).


Feel free to jump on #openstack-state-management IRC channel if u want 
to chat in person more about why (it gets into details that might just 
be easier to explain in person).


ESWAR RAO wrote:

Hi All,

Please let me know whether celery is replacement for taskflow.

As per my understanding, task-flow can break jobs into tasks and execute
them.

 From celery wiki, it also does almost similar behaviour.

I guess in most of openstack components taskflow is widely used.
Any places where its being replaced with celery ??

Celery: https://wiki.openstack.org/wiki/Celery
Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
TaskFlow: https://wiki.openstack.org/wiki/TaskFlow

Thanks
Eswar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Heads up for decomposed plugin break

2016-01-08 Thread Sean M. Collins

On Fri, Jan 08, 2016 at 07:50:47AM PST, Chris Dent wrote:
> On Fri, 8 Jan 2016, Gary Kotton wrote:
> 
> >The commit https://github.com/openstack/neutron/commit/5d53dfb8d64186-
> >b5b1d2f356fbff8f222e15d1b2 may break the decomposed plugins that make
> >use of the method _get_tenant_id_for_create
> 
> Just out of curiosity, is it not standard practice that a plugin
> shouldn't use a private method?

+1 - hopefully decomposed plugins will audit their code and look for
other calls to private methods. 

-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][bugs] help needed: volunteers for bug skimming (1 week)

2016-01-08 Thread Augustina Ragwitz


On 2016-01-08 11:27, Anthony Chow wrote:

There is an URL for the duty:

https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty
[3]


Right, I read the wiki and it wasn't clear to me. Here's what the wiki 
says:


"Since Nova is such an active project with a high number of incoming bug 
reports, we have a separate process for categorizing bugs and 
distributing the load of triage to a larger group of people."


and

"To help make sure that the triage queue stays under control, the 
following table lists the people that have committed to regularly 
triaging bugs for a given tag."


The term "skimming" implies something done quickly at a surface level 
(like skimming off the fat or skimming a book). So I'm just asking, as 
part of "skimming" duty, are we also confirming the bugs, (confirming 
looks like it falls under the "Triaging" step), or is "skimming" just 
about tagging and getting the bugs into the triaging queues?


Thanks!

Augustina



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800:
> On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> snippity snip snip
> 
> > >We haven't made it a dep for anything yet, only added to g-r.
> > 
> > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> > makes functionally any difference, though (pun intended).
> 
> Both. https://review.openstack.org/#/c/220268/
> 
> This thread was originally about twisted, which is added to u-c with the
> introduction of mimic.
> 
> > >However, now that you mention that, a really ambitious goal would be to
> > >add a rabbit interface to mimic, and functionally test the API server
> > >(that it sends the right messages, etc). Another would be to mimic
> > >(Neutron, Glance) and test Ironic by itself.
> > 
> > So you would reimplement AMQP communication protocols using an in-memory
> > data store for queues. Sounds like an even greater surface area for bugs to
> > be introduced.
> > 
> > >Last, I frankly don't understand why there's
> > >such heavy opposition to the ironic team using an additional tool for
> > >testing.
> > 
> > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> > though.
> > 
> > a) Because it fractures the testing and QA processes used by upstream
> > contributors that work on OpenStack projects by requiring them to learn
> > another system -- and one that potentially would require them to understand
> > a whole new surface area for potential bugs
> 
> I don't think there's a large risk of needing to dig deep into mimic,
> and especially twisted. If this does prove to be a problem, I'm happy to
> remove it. However, we can't even explore what it would be like, or how
> hard we would depend on mimic, without mimic being in g-r.
> 
> > b) Because it represents yet another RAX-driven divergence in the QA space.
> > CloudCafe took essentially all of the RAX folks that were at one point
> > working on Tempest and upstream QA and siloed them into a totally different
> > organization, in true RAX fashion. Instead of pulling the OpenStack QA
> > community along together, RAX QA continues to just do its own thing and
> > there's still bitterness on the tips of tongues.
> 
> So, this isn't trying to replace anything. This is adding a different
> way to run functional tests, that is *much* faster than standing up a
> full ironic environment. This is helpful for developers that want to
> quickly run tests before posting them to gerrit, people that need to
> test in constrained environments, etc.

So it won't be used in the gate, just on local developer systems?

Doug

> 
> I'm 100% against doing things like Rackspace did with tempest and
> cloudcafe, and I wouldn't be supporting this effort if I felt it was
> similar. Here's how this went:
> 
> * Lekha started working on OnMetal QA, with a goal of doing some amount
>   of upstream work as well.
> 
> * She's previously worked on projects (like autoscale) that interact
>   with OpenStack APIs, and seeing the need to test without a full nova
>   environment, built mimic.
> 
> * In talking with some of the other Ironic folks working on QA (from
>   Intel, IBM, more), she presented mimic as something that may be useful
>   for testing the client (and more). They (and I) agreed it was a neat
>   idea worth trying.
> 
> * Jay offered to help with the global-requirements patch as it's
>   something he's done before, and did the review grinding here.
> 
> * It finally landed, and lifeless asked me to bring up the Twisted
>   conversation on the list. Note that this is not the "is mimic
>   useful" conversation we're having now. Nobody remotely voiced
>   concerns about using a new test tool until this thread happened.
> 
> Please do let me know if any of that seems nefarious; I hope it doesn't.
> 
> > 
> > 
> > 
> > >There was *more* than sufficient time for "I don't think this is useful"
> > >to be posted on the review.
> > 
> > Sorry, this was the first I'd heard of it all.
> > 
> > > If anyone has a hard technical reason why
> > >mimic should not be used to test an OpenStack project, I'm happy to
> > >listen. Otherwise I'd like to work on actually getting things done.
> > 
> > I've listed my hard technical reason: it introduces, IMHO, too large of a
> > surface area for bugs to creep in to the testing process versus the benefit
> > it provides.
> 
> Again, I don't think that's any larger of a surface area than mocks in client
> unit tests being incorrect, and we really won't know whether it causes
> problems in reality.
> 
> // jim
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

On Fri, Jan 08, 2016 at 03:13:15PM -0500, Doug Hellmann wrote:
> Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800:
> > On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> > > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> > snippity snip snip
> > 
> > > >We haven't made it a dep for anything yet, only added to g-r.
> > > 
> > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> > > makes functionally any difference, though (pun intended).
> > 
> > Both. https://review.openstack.org/#/c/220268/
> > 
> > This thread was originally about twisted, which is added to u-c with the
> > introduction of mimic.
> > 
> > > >However, now that you mention that, a really ambitious goal would be to
> > > >add a rabbit interface to mimic, and functionally test the API server
> > > >(that it sends the right messages, etc). Another would be to mimic
> > > >(Neutron, Glance) and test Ironic by itself.
> > > 
> > > So you would reimplement AMQP communication protocols using an in-memory
> > > data store for queues. Sounds like an even greater surface area for bugs 
> > > to
> > > be introduced.
> > > 
> > > >Last, I frankly don't understand why there's
> > > >such heavy opposition to the ironic team using an additional tool for
> > > >testing.
> > > 
> > > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> > > though.
> > > 
> > > a) Because it fractures the testing and QA processes used by upstream
> > > contributors that work on OpenStack projects by requiring them to learn
> > > another system -- and one that potentially would require them to 
> > > understand
> > > a whole new surface area for potential bugs
> > 
> > I don't think there's a large risk of needing to dig deep into mimic,
> > and especially twisted. If this does prove to be a problem, I'm happy to
> > remove it. However, we can't even explore what it would be like, or how
> > hard we would depend on mimic, without mimic being in g-r.
> > 
> > > b) Because it represents yet another RAX-driven divergence in the QA 
> > > space.
> > > CloudCafe took essentially all of the RAX folks that were at one point
> > > working on Tempest and upstream QA and siloed them into a totally 
> > > different
> > > organization, in true RAX fashion. Instead of pulling the OpenStack QA
> > > community along together, RAX QA continues to just do its own thing and
> > > there's still bitterness on the tips of tongues.
> > 
> > So, this isn't trying to replace anything. This is adding a different
> > way to run functional tests, that is *much* faster than standing up a
> > full ironic environment. This is helpful for developers that want to
> > quickly run tests before posting them to gerrit, people that need to
> > test in constrained environments, etc.
> 
> So it won't be used in the gate, just on local developer systems?

Initially, yes. I'd like to also add it to the gate at some point to
make sure we don't break those interactions, though I could deal with a
non-voting job just in case.

// jim

> 
> Doug
> 
> > 
> > I'm 100% against doing things like Rackspace did with tempest and
> > cloudcafe, and I wouldn't be supporting this effort if I felt it was
> > similar. Here's how this went:
> > 
> > * Lekha started working on OnMetal QA, with a goal of doing some amount
> >   of upstream work as well.
> > 
> > * She's previously worked on projects (like autoscale) that interact
> >   with OpenStack APIs, and seeing the need to test without a full nova
> >   environment, built mimic.
> > 
> > * In talking with some of the other Ironic folks working on QA (from
> >   Intel, IBM, more), she presented mimic as something that may be useful
> >   for testing the client (and more). They (and I) agreed it was a neat
> >   idea worth trying.
> > 
> > * Jay offered to help with the global-requirements patch as it's
> >   something he's done before, and did the review grinding here.
> > 
> > * It finally landed, and lifeless asked me to bring up the Twisted
> >   conversation on the list. Note that this is not the "is mimic
> >   useful" conversation we're having now. Nobody remotely voiced
> >   concerns about using a new test tool until this thread happened.
> > 
> > Please do let me know if any of that seems nefarious; I hope it doesn't.
> > 
> > > 
> > > 
> > > 
> > > >There was *more* than sufficient time for "I don't think this is useful"
> > > >to be posted on the review.
> > > 
> > > Sorry, this was the first I'd heard of it all.
> > > 
> > > > If anyone has a hard technical reason why
> > > >mimic should not be used to test an OpenStack project, I'm happy to
> > > >listen. Otherwise I'd like to work on actually getting things done.
> > > 
> > > I've listed my hard technical reason: it introduces, IMHO, too large of a
> > > surface area for bugs to creep in to the testing process versus the 
> > > benefit
> > > it provides.
> > 
> > Again, I don't think that's any larger of a surface area than mocks in 
>

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

Excerpts from Jim Rollenhagen's message of 2016-01-08 12:30:33 -0800:
> On Fri, Jan 08, 2016 at 03:13:15PM -0500, Doug Hellmann wrote:
> > Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800:
> > > On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> > > > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> > > snippity snip snip
> > > 
> > > > >We haven't made it a dep for anything yet, only added to g-r.
> > > > 
> > > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> > > > makes functionally any difference, though (pun intended).
> > > 
> > > Both. https://review.openstack.org/#/c/220268/
> > > 
> > > This thread was originally about twisted, which is added to u-c with the
> > > introduction of mimic.
> > > 
> > > > >However, now that you mention that, a really ambitious goal would be to
> > > > >add a rabbit interface to mimic, and functionally test the API server
> > > > >(that it sends the right messages, etc). Another would be to mimic
> > > > >(Neutron, Glance) and test Ironic by itself.
> > > > 
> > > > So you would reimplement AMQP communication protocols using an in-memory
> > > > data store for queues. Sounds like an even greater surface area for 
> > > > bugs to
> > > > be introduced.
> > > > 
> > > > >Last, I frankly don't understand why there's
> > > > >such heavy opposition to the ironic team using an additional tool for
> > > > >testing.
> > > > 
> > > > Since you asked, I'll be blunt. This isn't a personal attack on you, 
> > > > Jim,
> > > > though.
> > > > 
> > > > a) Because it fractures the testing and QA processes used by upstream
> > > > contributors that work on OpenStack projects by requiring them to learn
> > > > another system -- and one that potentially would require them to 
> > > > understand
> > > > a whole new surface area for potential bugs
> > > 
> > > I don't think there's a large risk of needing to dig deep into mimic,
> > > and especially twisted. If this does prove to be a problem, I'm happy to
> > > remove it. However, we can't even explore what it would be like, or how
> > > hard we would depend on mimic, without mimic being in g-r.
> > > 
> > > > b) Because it represents yet another RAX-driven divergence in the QA 
> > > > space.
> > > > CloudCafe took essentially all of the RAX folks that were at one point
> > > > working on Tempest and upstream QA and siloed them into a totally 
> > > > different
> > > > organization, in true RAX fashion. Instead of pulling the OpenStack QA
> > > > community along together, RAX QA continues to just do its own thing and
> > > > there's still bitterness on the tips of tongues.
> > > 
> > > So, this isn't trying to replace anything. This is adding a different
> > > way to run functional tests, that is *much* faster than standing up a
> > > full ironic environment. This is helpful for developers that want to
> > > quickly run tests before posting them to gerrit, people that need to
> > > test in constrained environments, etc.
> > 
> > So it won't be used in the gate, just on local developer systems?
> 
> Initially, yes. I'd like to also add it to the gate at some point to
> make sure we don't break those interactions, though I could deal with a
> non-voting job just in case.

OK, if that's the case then although I share Jay's concerns about bug
surface area and keeping the APIs in sync, I think the risk is low
enough that it would only annoy developers and not actually allow bugs
to slip through testing. So, I'm OK with continuing with the experiment
in Ironic.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-08 Thread Jay Pipes


On 01/08/2016 02:52 PM, Jim Rollenhagen wrote:

On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:

On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:

snippity snip snip


We haven't made it a dep for anything yet, only added to g-r.


According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
makes functionally any difference, though (pun intended).


Both. https://review.openstack.org/#/c/220268/

This thread was originally about twisted, which is added to u-c with the
introduction of mimic.


Got it, thanks.


However, now that you mention that, a really ambitious goal would be to
add a rabbit interface to mimic, and functionally test the API server
(that it sends the right messages, etc). Another would be to mimic
(Neutron, Glance) and test Ironic by itself.


So you would reimplement AMQP communication protocols using an in-memory
data store for queues. Sounds like an even greater surface area for bugs to
be introduced.


Last, I frankly don't understand why there's
such heavy opposition to the ironic team using an additional tool for
testing.


Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
though.

a) Because it fractures the testing and QA processes used by upstream
contributors that work on OpenStack projects by requiring them to learn
another system -- and one that potentially would require them to understand
a whole new surface area for potential bugs


I don't think there's a large risk of needing to dig deep into mimic,
and especially twisted. If this does prove to be a problem, I'm happy to
remove it. However, we can't even explore what it would be like, or how
hard we would depend on mimic, without mimic being in g-r.


Sure, fair enough.


b) Because it represents yet another RAX-driven divergence in the QA space.
CloudCafe took essentially all of the RAX folks that were at one point
working on Tempest and upstream QA and siloed them into a totally different
organization, in true RAX fashion. Instead of pulling the OpenStack QA
community along together, RAX QA continues to just do its own thing and
there's still bitterness on the tips of tongues.


So, this isn't trying to replace anything. This is adding a different
way to run functional tests, that is *much* faster than standing up a
full ironic environment. This is helpful for developers that want to
quickly run tests before posting them to gerrit, people that need to
test in constrained environments, etc.


I recognize it is much faster than standing up a real environment. And I 
recognize that running faster client tests is a useful thing -- as long 
as we can be confident that what is tested does not suffer from some of 
the issues I identified earlier (syncing with real API and introduction 
of greater surface area for bugs in the test platform itself).



I'm 100% against doing things like Rackspace did with tempest and
cloudcafe, and I wouldn't be supporting this effort if I felt it was
similar. Here's how this went:

* Lekha started working on OnMetal QA, with a goal of doing some amount
   of upstream work as well.

* She's previously worked on projects (like autoscale) that interact
   with OpenStack APIs, and seeing the need to test without a full nova
   environment, built mimic.

* In talking with some of the other Ironic folks working on QA (from
   Intel, IBM, more), she presented mimic as something that may be useful
   for testing the client (and more). They (and I) agreed it was a neat
   idea worth trying.

* Jay offered to help with the global-requirements patch as it's
   something he's done before, and did the review grinding here.

* It finally landed, and lifeless asked me to bring up the Twisted
   conversation on the list. Note that this is not the "is mimic
   useful" conversation we're having now. Nobody remotely voiced
   concerns about using a new test tool until this thread happened.

Please do let me know if any of that seems nefarious; I hope it doesn't.


No, nothing nefarious there. Sorry for letting my personal frustrations 
bubble over into this.


I am not blocking anything from going forward and I definitely am not 
asking for a revert of any g-r patch. Nor am I trying to obstruct you in 
your governance of Ironic.


I was just raising my concerns as an OpenStack citizen and getting my 
opinion out on paper.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [openstack-ansible] Tracking jenkins jobs

2016-01-08 Thread Major Hayden

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hey folks,

I've been chasing down some annoying gate failures in kilo and I put together a 
script to populate a spreadsheet[1] with information about our gate jobs.  
Within a few seconds of a gate job completing, you'll see a new line pop on the 
spreadsheet.

I'm only collecting data from the gate-openstack-ansible-dsvm-commit at the 
moment, but other jobs could be added if needed.

Let me know if this is useful or if it should be expanded to include something 
else.  Of course, if it's totally useless, let me know about that too. ;)

[1] 
https://docs.google.com/spreadsheets/d/1YZC6ng-AIHqbHHHeGPC2mar_JPYunvFm4BzqfAEOYLI/edit#gid=0

- --
Major Hayden
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWkCTcAAoJEHNwUeDBAR+xyKgP+wc4EC74SNjkz5wcwjJjR67L
KfA3y719XXVLmuYyB2PllDHC9cDYTxVJFM57/tR0xM4O5ubHm3ywjDD0G0iFQZWl
amVzkXto0BgaAe7w2esHOrTnP2K3x06y2i1tCj5zDGpC+b3RRBrh28Fjj8f43JpX
zg7ThrdichVE7VENEYWP3p83bq30ur/d3+moKwoVtZ280PCU13Zs7kVReI4DHaMk
15AAEq+akHXOuiAn6wDYLyWOVsMSb+boP3plHByRggYH89JrnhFOrq8qrhS0je1B
Glptieb0MYtmkDOyRwhmoUoKB74zp/mSPyS5uOh+vq6Ah85Ex5GYuTGrgCWG6en+
0X3dH6Jaw0kTf8SMgxOkgeMSBglNpp56jzJtlM4+YtT40eF8PR6DQdFS38evG8O7
36BAEm+R0+KoQGiFVuiBttXeZs5JWR4Ee3T70xOXbYy5vYpZ5t6GfsDDBY2l6eY0
/s1jbwKq/6FRtkekcTenvDeFffpqCI8K7gPt86mLF2P+XTxeCT/JGQt2rIUiZzGD
Xs1XlUgMsS5Ghf7P3FtVz3zFWxmeDjkWIS5Qlm5qkn/5Ct+hlfQ6fVgAFFQKDmnp
tAp9+SvrDJi1QyQ/7NyNSiZvQT5l5WieCWppEaZKk2HNjsDiMaS5a18c/ZsHwh/D
ZVek5thkvlWCCiE5qD3T
=a+Gr
-END PGP SIGNATURE-

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-08 Thread Davanum Srinivas

yes Jay, i have -W on the bot proposed merge, pending this discussion:
https://review.openstack.org/#/c/263592/

-- Dims

On Fri, Jan 8, 2016 at 2:08 PM, Jay Pipes  wrote:
> On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
>>
>> On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:
>>>
>>> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:

 On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
>
> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
>>
>> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
>>>
>>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:

 Hi all,

 A change to global-requirements[1] introduces mimic, which is an
 http
 server that can mock various APIs, including nova and ironic,
 including
 control of error codes and timeouts. The ironic team plans to use
 this
 for testing python-ironicclient without standing up a full ironic
 environment.

 Here's the catch - mimic is built on twisted. I know twisted was
 previously removed from OpenStack (or at least people said "pls no",
 I
 don't know the full history). We didn't intend to stealth-introduce
 twisted back into g-r, but it was pointed out to me that it may
 appear
 this way, so here I am letting everyone know. lifeless pointed out
 that
 when tests are failing, people may end up digging into mimic or
 twisted
 code, which most people in this community aren't familiar with
 AFAIK,
 which is a valid point though I hope it isn't required often.

 So, the primary question here is: do folks have a problem with
 adding
 twisted here? We're holding off on Ironic changes that depend on
 this
 until this discussion has happened, but aren't reverting the g-r
 change
 until we decide one way or another.

 // jim

 [1] https://review.openstack.org/#/c/220268/
>>>
>>>
>>> What is the advantage of running another server like this over using
>>> requests-mock (which is used by other OpenStack projects for testing
>>> today)? The only difference here seems to be that you actually
>>> execute
>>> requests code in one case and not in the other.
>>>
>>> Requests-mock debugging when things go wrong seems a bit simpler.
>>>
>>> This is less about twisted and more about trying to not introduce yet
>>> another way to mock code in the tree that people need to understand.
>>>
>>> -Sean
>>
>>
>> We'd be using this for functional tests, not unit, where we can't
>> really
>> inject mocks. The idea is that we could run a full functional suite
>> against either mimic or a full ironic environment, just by changing a
>> test setting.
>
>
> I don't really see the point of a separate project like Mimic that has
> a
> whole bunch of reimplementations (mocked out) of all sorts of OpenStack
> (and
> RAX-specific) API services. It's just a great way to introduce a larger
> surface area for bugs to creep in -- since you have to keep the Mimic
> interfaces up to date with the real interfaces. Better to keep
> something
> like this -- if it is TRULY needed -- in-tree with the API service
> itself,
> so that the chances of divergence are reduced. This is similar to the
> fakevirt driver in Nova. It's in tree for good reason: when someone
> changes
> the virt driver interface, the fakevirt driver goes boom and needs to
> be
> changed in a corresponding fashion in the same patch.

 I tend to think our REST APIs are much more stable than internal python
 APIs, and so we can manage the divergence much easier.

 Also, mimic can simulate:

 * old versions (less needed with all the microversion stuff)
 * old bugs that were shipped
 * misconfigurations
 * networking errors
 * the passage of time (including timeouts)

 We probably don't want to keep a catalog of old bugs and
 misconfigurations in
 tree forever.

> What value does a functional test against an HTTP API service that does
> nothing (other than introduce greater surface area for bugs) actually
> offer
> over unit tests anyway?

 Testing the full stack of the client instead of mocking the bottom
 half of requests is a big one.

 Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
 https://www.youtube.com/watch?v=HKUUQme3dwA
>>>
>>>
>>> OK, I just watched that. Sorry, still don't see the value that Mimic
>>> provides over unit testing the client interfaces and mocking out the HTTP
>>> payloads so you have strict control over the expectations.
>>>
>>> The problem that Glyph noted

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

On Fri, Jan 08, 2016 at 02:39:52PM -0500, Davanum Srinivas wrote:
> yes Jay, i have -W on the bot proposed merge, pending this discussion:
> https://review.openstack.org/#/c/263592/

Note that this doesn't actually block mimic from being in g-r, or being
used. It's already in g-r. You'd need to revert
https://review.openstack.org/#/c/220268/ .

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

Excerpts from Jim Rollenhagen's message of 2016-01-08 09:56:46 -0800:
> On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote:
> > Jim Rollenhagen wrote:
> > >[...]
> > >Here's the catch - mimic is built on twisted. I know twisted was
> > >previously removed from OpenStack (or at least people said "pls no", I
> > >don't know the full history). We didn't intend to stealth-introduce
> > >twisted back into g-r, but it was pointed out to me that it may appear
> > >this way, so here I am letting everyone know. lifeless pointed out that
> > >when tests are failing, people may end up digging into mimic or twisted
> > >code, which most people in this community aren't familiar with AFAIK,
> > >which is a valid point though I hope it isn't required often.
> > 
> > A bit of history with Twisted.
> > 
> > Back in 2010 we decided we could not afford asking OpenStack developers to
> > be familiar with multiple service architecture frameworks, and eventlet was
> > chosen as the simplest framework to learn and debug. The best reference I
> > found on this is still visible in the wiki:
> > 
> > https://wiki.openstack.org/wiki/UnifiedServiceArchitecture
> > 
> > >So, the primary question here is: do folks have a problem with adding
> > >twisted here? We're holding off on Ironic changes that depend on this
> > >until this discussion has happened, but aren't reverting the g-r change
> > >until we decide one way or another.
> > 
> > The only friction I see is how many developers would be expected to need to
> > learn Twisted in order to complete their jobs. My understanding is that
> > Twisted expertise could be needed to debug python-ironicclient functional
> > tests, which makes the cost relatively limited. So if Mimic brings in a
> > clear and significant benefit, I don't think its Twisted dependence should
> > play that much against it.
> > 
> > However, I agree with Sean and Jay that the benefit is unclear -- the few
> > features that Mimic brings seem to be outweighed by the increased risk of
> > introducing a delta between the implementation and the mock. If the main
> > benefit is that it's used in other Rackspace projects for testing (like Ben
> > said), I'm not sure that makes a compelling argument for the rest of the
> > community...
> 
> No, that is not the main benefit, at all. Ben isn't involved in Ironic
> and until now has had nothing to do with this work to add mimic here, so
> I'm not sure where he got that impression from, or why he's speaking on
> our behalf as to the goals here.
> 
> As pointed out before, the risk of a delta between the mocks in mimic
> and reality is identical to the risk of a delta between the mocks in a
> client's unit tests and reality, so I don't see a particular downside
> there.
> 
> Again, I think "benefit is unclear" isn't a valid reason to block this,
> so unless someone posts a revert we're going to move forward with this.

I have not made a decision myself about whether it's right to go
ahead or not, but if we need to have a revert in place to continue
the conversation, I'll do that.  Please see
https://review.openstack.org/#/c/265416/ as the patch to revert
adding mimic.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

I also updated https://wiki.openstack.org/wiki/DistributedTaskManagement 
to denote that said wiki is no longer active (it was an attempt to back 
a taskflow engine[1] with celery); although if u are interested in 
continuing down this path feel free.


Hopefully that clears up some 'confusion' around that wiki.

[1] http://docs.openstack.org/developer/taskflow/engines.html

Joshua Harlow wrote:

So actually they are quite different, (although similar at some level),

Given that celery isn't really a replacement for taskflow although one
could say, from what I've heard from others, that taskflow is a
super-set of what celery is so taskflow likely can replace parts of
celery (but not vice-versa).

Feel free to jump on #openstack-state-management IRC channel if u want
to chat in person more about why (it gets into details that might just
be easier to explain in person).

ESWAR RAO wrote:

Hi All,

Please let me know whether celery is replacement for taskflow.

As per my understanding, task-flow can break jobs into tasks and execute
them.

From celery wiki, it also does almost similar behaviour.

I guess in most of openstack components taskflow is widely used.
Any places where its being replaced with celery ??

Celery: https://wiki.openstack.org/wiki/Celery
Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
TaskFlow: https://wiki.openstack.org/wiki/TaskFlow

Thanks
Eswar

__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow


ESWAR RAO wrote:

Hi Joshua,

Thanks for the response.

As you mentioned, I went through below which has taskflow as celery
front end:

https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine

https://blueprints.launchpad.net/taskflow/+spec/distributed-celery


I think the idea way-back-when was celery would be a backend, not a 
front-end, although my memory might be fuzzy in this area (since it was 
a few years ago).





I guess if we have a job = task1 + task2 ; if we execute them through a
taskflow parallel pattern, they will be executed in parallel.

Just curious to know that since this threading is built on
eventlets/green threads/python threads they will be affected by GIL and
we may not utilize multi-core capability of our systems.


So that assumes u are using a taskflow engine which is powered by 
threads (or greenthreads), there are two types that do not use threads 
but actually run out of process:


http://docs.openstack.org/developer/taskflow/engines.html#types

1. http://docs.openstack.org/developer/taskflow/workers.html (runs out 
of process, across different machines, using kombu for triggering 
start/status... of tasks on other machines)


2. A parallel engine that uses a process pool executor 
(https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor) 
will then execute tasks using out of process child-processes.


So both of these don't have the same GIL issue (although greenthreads by 
there very nature are not affected by the GIL either).


The example (aptly named hello world) shows these different types 
(although it doesn't show the worker based one, since that requires more 
things to setup) @ 
https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82


-Josh



It seems in celery, we can have these tasks run as multiple processes so
that they are not affected by GIL.


Right, the above starts to show why taskflow is really a super-set of 
celery (for some combination/interpretation of what taskflow or celery do).




Please correct me if my understanding is wrong ??

Thanks
Eswar


On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow > wrote:

I also updated
https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
that said wiki is no longer active (it was an attempt to back a
taskflow engine[1] with celery); although if u are interested in
continuing down this path feel free.

Hopefully that clears up some 'confusion' around that wiki.

[1] http://docs.openstack.org/developer/taskflow/engines.html


Joshua Harlow wrote:

So actually they are quite different, (although similar at some
level),

Given that celery isn't really a replacement for taskflow
although one
could say, from what I've heard from others, that taskflow is a
super-set of what celery is so taskflow likely can replace parts of
celery (but not vice-versa).

Feel free to jump on #openstack-state-management IRC channel if
u want
to chat in person more about why (it gets into details that
might just
be easier to explain in person).

ESWAR RAO wrote:

Hi All,

Please let me know whether celery is replacement for taskflow.

As per my understanding, task-flow can break jobs into tasks
and execute
them.

 From celery wiki, it also does almost similar behaviour.

I guess in most of openstack components taskflow is widely used.
Any places where its being replaced with celery ??

Celery: https://wiki.openstack.org/wiki/Celery
Distributed:
https://wiki.openstack.org/wiki/DistributedTaskManagement
TaskFlow: https://wiki.openstack.org/wiki/TaskFlow

Thanks
Eswar


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org

Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org

Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [doc] [api] Vision and changes for OpenStack API Guides

2016-01-08 Thread Anne Gentle

On Fri, Jan 8, 2016 at 3:39 PM, Anne Gentle 
wrote:

> Hi all,
>
> With milestone 2 coming next week I want to have a chat about API guides,
> API reference information, and the http://developer.openstack.org site.
> We have a new tool, fairy-slipper [1], yep you read that right, that can
> migrate files from WADL to Swagger as well as serve up reference info. We
> also have new build jobs that can build API guides right from your projects
> repo to developer.openstack.org.
>
> There's a lot going on, so I've got an agenda item for next week to hold a
> cross-project meeting so that you can send a team representative to get the
> scoop on API guides and bring it back to your teams. I've fielded great
> questions from a few teams, and want to ensure everyone's in the loop.
>
> A pair of specs set the stage for this work, feel free to review these in
> advance of the meeting and come with questions. [2] [3]
>
> Join me next week at the first pop-up Cross Project meeting, Tuesday at
> 1700, in #openstack-meeting-cp. Feel free to add to the agenda at
>

Woops, agenda here:

https://wiki.openstack.org/wiki/Meetings/CrossProjectMeeting#Proposed_agenda



>
> Thanks,
> Anne
>
>
> 1. https://github.com/openstack/fairy-slipper
> 2. mitaka spec:
> http://specs.openstack.org/openstack/docs-specs/specs/mitaka/app-guides-mitaka-vision.html
> 3. liberty spec:
> http://specs.openstack.org/openstack/docs-specs/specs/liberty/api-site.html
>
> --
> Anne Gentle
> Rackspace
> Principal Engineer
> www.justwriteclick.com
>



-- 
Anne Gentle
Rackspace
Principal Engineer
www.justwriteclick.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

2016-01-08 Thread ESWAR RAO

Hi Joshua,

Thanks for the response.

As you mentioned, I went through below which has taskflow as celery front
end:

 https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine

 https://blueprints.launchpad.net/taskflow/+spec/distributed-celery

I guess if we have a job = task1 + task2 ; if we execute them through a
taskflow parallel pattern, they will be executed in parallel.

Just curious to know that since this threading is built on eventlets/green
threads/python threads they will be affected by GIL and we may not utilize
multi-core capability of our systems.

It seems in celery, we can have these tasks run as multiple processes so
that they are not affected by GIL.

Please correct me if my understanding is wrong ??

Thanks
Eswar


On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow  wrote:

> I also updated https://wiki.openstack.org/wiki/DistributedTaskManagement
> to denote that said wiki is no longer active (it was an attempt to back a
> taskflow engine[1] with celery); although if u are interested in continuing
> down this path feel free.
>
> Hopefully that clears up some 'confusion' around that wiki.
>
> [1] http://docs.openstack.org/developer/taskflow/engines.html
>
>
> Joshua Harlow wrote:
>
>> So actually they are quite different, (although similar at some level),
>>
>> Given that celery isn't really a replacement for taskflow although one
>> could say, from what I've heard from others, that taskflow is a
>> super-set of what celery is so taskflow likely can replace parts of
>> celery (but not vice-versa).
>>
>> Feel free to jump on #openstack-state-management IRC channel if u want
>> to chat in person more about why (it gets into details that might just
>> be easier to explain in person).
>>
>> ESWAR RAO wrote:
>>
>>> Hi All,
>>>
>>> Please let me know whether celery is replacement for taskflow.
>>>
>>> As per my understanding, task-flow can break jobs into tasks and execute
>>> them.
>>>
>>> From celery wiki, it also does almost similar behaviour.
>>>
>>> I guess in most of openstack components taskflow is widely used.
>>> Any places where its being replaced with celery ??
>>>
>>> Celery: https://wiki.openstack.org/wiki/Celery
>>> Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
>>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>>
>>> Thanks
>>> Eswar
>>>
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> ___
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openst...@lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openst...@lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] OpenStack Developer Mailing List Digest January 2-8

2016-01-08 Thread Mike Perez

Permalink: 
http://www.openstack.org/blog/2016/01/openstack-developer-mailing-list-digest-20160108/

SuccessBot Says
===
* russellb: Ported OVS to Python 3.
* notmyname: Removed the beta tag on the swift erasure code docs.
* AJaeger: OpenStack configuration reference has been migrated from DocBook XML
  to RST [1].
* notmyname: This season’s Outreachy intern had her first patch merged.
* odyssey4me: OpenStack-Ansible 12.0.3 tagged.
* jordanP: My 3rd party CI correctly warned me that a patch would break my
  driver.
* asselin: OpenStack Third Party CI documentation published [2].
* Tell us yours via IRC with a message “#success [insert success]”.
* More: https://wiki.openstack.org/wiki/Successes
 
Slightly Longer Delays for Gate CI Testing
==
* When: January 31, 2016
* Why: Due to the sunsetting of a public cloud [3] used by the OpenStack Infra
  team, we will have less resources available.
* Resolving: Jeremy Stanley says that the Infra team is working on bringing in
  new providers to absorb the impact.
 
Release Countdown For Week R-12, Jan 11-25
==
* Focus: Second milestone is coming up. Finish up major features or reevaluate
  if things really need to land in this cycle.
* Actions:
  - Identify work that needs to be completed before the M-2 tag.
  - Release liaisons responsibilities are being updated. Provide comments for
questions or concerns [4].
* Important Dates:
  - Mitaka 2: Jan 19-21
  - Release schedule [5].
* Full thread: 
http://lists.openstack.org/pipermail/openstack-dev/2016-January/083507.html
 
Re-introduce Twisted to global-requirements
===
* A change in global-requirements [6] introduces mimic, an http server that can
  mock various APIs.
* Mimic depends on twisted, which in the past was removed in favor of Eventlet
  to avoid developers having to know multiple frameworks [7].
* Jim Rollenhagen explains Ironic’s need for Mimic:
  - To do functional tests, not unit tests which they could do with
requests-mock.
  - To bring up a less expensive Ironic environment for doing functional tests.
* Jay Pipes notes:
  - This is another way to introduce a larger surface area of bugs to creep in
since you have to keep Mimic up-to-date with the real interfaces.
  - There’s not a clear value in the advantage of functional tests of a client
that calls a fake HTTP API versus unit tests of a client that simply uses
requests-mock.
* Full thread: 
http://lists.openstack.org/pipermail/openstack-dev/2016-January/083510.html
 

[1] - http://docs.openstack.org/draft/config-reference/
[2] - http://docs.openstack.org/infra/openstackci/
[3] - 
http://lists.openstack.org/pipermail/openstack-infra/2015-October/003309.html
[4] - https://review.openstack.org/#/c/262003/
[5] - http://docs.openstack.org/releases/schedules/mitaka.html
[6] - https://review.openstack.org/#/c/220268/
[7] - https://wiki.openstack.org/wiki/UnifiedServiceArchitecture

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [doc] [api] Vision and changes for OpenStack API Guides

2016-01-08 Thread michael mccune


On 01/08/2016 04:39 PM, Anne Gentle wrote:

Join me next week at the first pop-up Cross Project meeting, Tuesday at
1700, in #openstack-meeting-cp. Feel free to add to the agenda at


i'll definitely be there Anne, i'm very curious about this topic but 
sadly have not had the extra cycles to commit to it.


regards,
mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [doc] [api] Vision and changes for OpenStack API Guides

2016-01-08 Thread Anne Gentle

Hi all,

With milestone 2 coming next week I want to have a chat about API guides,
API reference information, and the http://developer.openstack.org site. We
have a new tool, fairy-slipper [1], yep you read that right, that can
migrate files from WADL to Swagger as well as serve up reference info. We
also have new build jobs that can build API guides right from your projects
repo to developer.openstack.org.

There's a lot going on, so I've got an agenda item for next week to hold a
cross-project meeting so that you can send a team representative to get the
scoop on API guides and bring it back to your teams. I've fielded great
questions from a few teams, and want to ensure everyone's in the loop.

A pair of specs set the stage for this work, feel free to review these in
advance of the meeting and come with questions. [2] [3]

Join me next week at the first pop-up Cross Project meeting, Tuesday at
1700, in #openstack-meeting-cp. Feel free to add to the agenda at

Thanks,
Anne

1. https://github.com/openstack/fairy-slipper
2. mitaka spec:
http://specs.openstack.org/openstack/docs-specs/specs/mitaka/app-guides-mitaka-vision.html
3. liberty spec:
http://specs.openstack.org/openstack/docs-specs/specs/liberty/api-site.html

--
Anne Gentle
Rackspace
Principal Engineer
www.justwriteclick.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [celery][taskflow] Reg. celery and task-flow

2016-01-08 Thread Greg Hill

I'm opinionated because I work with/on Taskflow and had mostly bad
experiences with Celery, but here's my $0.02.  It's possible that the
codebase I inherited just made bad use of Celery or things have improved a
lot in the last 18 months, but all I can speak from is my own experience.

Taskflow and Celery at a high level can be used for the same thing, but if
anything Taskflow would be considered a replacement for Celery, IMO.  We
migrated a codebase from Celery to Taskflow because Taskflow gives you a
lot more capabilities related to managing the whole process rather than
just a bunch of individual async tasks.  Some still swear by Celery
because they like that they can just throw some decorators on existing
methods and execute them asynchronously, and if that's all you need,
Taskflow is probably overkill.  For us, we wanted a lot more control over
the whole flow, including how to handle task failures, being able to run
many things in parallel and others in serial, etc. Just conceptually
thinking of the process as a series of tasks in a flow makes it a lot
easier to reason about what is going on than having to trace throw a bunch
of asynchronous functions. We also liked the resiliency guarantees that
Taskflow's job board and conductor/worker engine provided more than what
Celery had at the time (it didn't seem to guarantee that tasks were
executed, and we had fairly often where rabbitmq connections would time
out and our process would just not continue, no log, no error, just the
next task was never executed).

Greg

On 1/8/16, 3:19 PM, "Joshua Harlow"  wrote:

>So actually they are quite different, (although similar at some level),
>
>Given that celery isn't really a replacement for taskflow although one
>could say, from what I've heard from others, that taskflow is a
>super-set of what celery is so taskflow likely can replace parts of
>celery (but not vice-versa).
>
>Feel free to jump on #openstack-state-management IRC channel if u want
>to chat in person more about why (it gets into details that might just
>be easier to explain in person).
>
>ESWAR RAO wrote:
>> Hi All,
>>
>> Please let me know whether celery is replacement for taskflow.
>>
>> As per my understanding, task-flow can break jobs into tasks and execute
>> them.
>>
>>  From celery wiki, it also does almost similar behaviour.
>>
>> I guess in most of openstack components taskflow is widely used.
>> Any places where its being replaced with celery ??
>>
>> Celery: https://wiki.openstack.org/wiki/Celery
>> Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>
>> Thanks
>> Eswar
>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][upgrade] Grenade multinode partial upgrade

2016-01-08 Thread Sean M. Collins

Just a quick update where we are:

Increasing the verbosity of the SSH session into the instance that is
created during the cinder portion is showing that we are actually
connecting to the instance successfully. We get the dropbear SSH banner,
but then the instance hangs. Eventually SSH terminates the connection, 5
minutes later.

http://logs.openstack.org/35/187235/12/experimental/gate-grenade-dsvm-neutron-multinode/984e651/logs/grenade.sh.txt.gz#_2016-01-08_20_13_40_040


-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][osdk] PrettyTable needs a home in OpenStack


I'm fine with #2 or #1

The oslo automaton lib also uses pretty table to generate state-machine 
tables (a useful feature to have, but not a necessity).


https://github.com/openstack/automaton/blob/master/requirements.txt#L14

So I'd def help keep prettytable going, of course another option is to 
move to https://pypi.python.org/pypi/tabulate (which does seem active 
and/or maintained); tabulate provides pretty much the same thing 
(actually more table formats @ 
https://pypi.python.org/pypi/tabulate#table-format ) than prettytable 
and the api is pretty much the same (or nearly).


So that's another way to handle this (just to move off prettytable 
entirely).


-Josh

Davanum Srinivas wrote:

#2 please :)

Thanks,
Dims

On Fri, Jan 8, 2016 at 8:06 AM, Flavio Percoco  wrote:

Greetings,

As some of you know already, google code is going to be shutdown. Some
projects we're using are hosted and, unfortunately, some of them are
unmaintained and perhaps going away.

One of these projects is PrettyTable. This point was raised by Erno in
this patch[0] from jd__. PrettyTable is not just being used in several
openstack specific projects but it's also a transitive dependency for
all client libraries using cliff.

With all that in mind, I've contacted the author of the library and
asked him if it'd be ok for us (OpenStack) to adopt this library. The
author accepted and granted me access to the project on pypi.

I'm saying all the above because we now need to find a home for it in
OpenStack.

I've identified 2 possible places:

1) Oslo, as we maintaing cross-project libraries and some of them are
not in the oslo namespace

2) OpenStack Client team as they maintain cliff already and it'd
perhaps make more sense to have this library there.

One thing to note is that this library has been quite stable, which
means it won't, hopefully, add too much work to the team.

Thoughts?

[0] https://review.openstack.org/#/c/234340/

--
@flaper87
Flavio Percoco

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [telemetry] mitaka virtual meetup follow-up

2016-01-08 Thread gord chung


hi folks,

just to recap the meeting item[1], we won't have a meetup this cycle as 
we all have our tasks and there doesn't seem to be any items that are 
controversial. we've agree that going forward, if something does require 
quicker feedback than ML/irc, we'll set up a conference and promote it 
to list before hand.


for those looking for help, whether development or operations, feel free 
to reach out on this list or irc.


have a great weekend.

[1] 
http://eavesdrop.openstack.org/meetings/telemetry/2016/telemetry.2016-01-07-15.00.log.html


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] Nominating Mariam John to Trove Core

2016-01-08 Thread Mariam John

Thank you for all the support. I am humbled to be part of a truly great
team and I look forward to working with all of you and continuing to
contribute to OpenStack Trove.

Thank you.

Mariam.




From:   "Vyvial, Craig" 
To: "OpenStack Development Mailing List (not for usage questions)"

Date:   01/08/2016 03:30 PM
Subject:Re: [openstack-dev] [trove] Nominating Mariam John to Trove
Core



Looks like we have a consensus from the team and I’d like to welcome Mariam
John to the Trove Core Team!

I look forward to continuing to work with Mariam!

Thanks,
-Craig


On Jan 7, 2016, at 12:21 PM, Nikhil Manchanda > wrote:

+1 from me as well.
Couldn't agree more!

Thanks,
-Nikhil

On Wed, Jan 6, 2016 at 12:03 PM, Peter Stachowski > wrote:
I agree!

+1

Peter

-Original Message-
From: Vyvial, Craig [mailto:craig.vyv...@hpe.com<
mailto:craig.vyv...@hpe.com>]
Sent: January-06-16 2:40 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] Nominating Mariam John to Trove Core

Thanks for the nomination Amrith and I think Mariam will be a great
addition to the core team.

+1

-Craig

On Jan 6, 2016, at 12:39 PM, Amrith Kumar >> wrote:

Members of the Trove team,

I would like to nominate Mariam John (johnma on IRC) to the Trove core
review team.

Mariam has been an active member of the Trove team for some time now. She
added support for IBM DB2 in Trove and provided elements for building Trove
guest images. She also implemented code that provided CouchDB support in
Trove. She has been an active reviewer and I have found her review comments
to be insightful and useful.

You can review her activity report at

http://stackalytics.com/report/users/mariamj

Members of the Trove core team, please reply to this message with your
feedback to this proposed change.

Thanks,

-amrith

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org<
mailto:openstack-dev-requ...@lists.openstack.org><
mailto:openstack-dev-requ...@lists.openstack.org<
mailto:openstack-dev-requ...@lists.openstack.org>>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<
http://openstack-dev-requ...@lists.openstack.org/?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<
http://openstack-dev-requ...@lists.openstack.org/?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org<
mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Deprecating the live_migration_flag and block_migration_flag config options

2016-01-08 Thread Mark McLoughlin

On Fri, 2016-01-08 at 14:11 +, Daniel P. Berrange wrote:
> On Thu, Jan 07, 2016 at 09:07:00PM +, Mark McLoughlin wrote:
> > On Thu, 2016-01-07 at 12:23 +0100, Sahid Orentino Ferdjaoui wrote:
> > > On Mon, Jan 04, 2016 at 09:12:06PM +, Mark McLoughlin wrote:
> > > > Hi
> > > > 
> > > > commit 8ecf93e[1] got me thinking - the live_migration_flag config
> > > > option unnecessarily allows operators choose arbitrary behavior of the
> > > > migrateToURI() libvirt call, to the extent that we allow the operator
> > > > to configure a behavior that can result in data loss[1].
> > > > 
> > > > I see that danpb recently said something similar:
> > > > 
> > > >   https://review.openstack.org/171098
> > > > 
> > > >   "Honestly, I wish we'd just kill off  'live_migration_flag' and
> > > >   'block_migration_flag' as config options. We really should not be
> > > >   exposing low level libvirt API flags as admin tunable settings.
> > > > 
> > > >   Nova should really be in charge of picking the correct set of flags
> > > >   for the current libvirt version, and the operation it needs to
> > > >   perform. We might need to add other more sensible config options in
> > > >   their place [..]"
> > > 
> > > Nova should really handle internal flags and this serie is running in
> > > the right way.
> > > 
> > > > ...
> > > 
> > > >   4) Add a new config option for tunneled versus native:
> > > > 
> > > >    [libvirt]
> > > >    live_migration_tunneled = true
> > > > 
> > > >  This enables the use of the VIR_MIGRATE_TUNNELLED flag. We have 
> > > >      historically defaulted to tunneled mode because it requires the 
> > > >      least configuration and is currently the only way to have a 
> > > >      secure migration channel.
> > > > 
> > > >      danpb's quote above continues with:
> > > > 
> > > >        "perhaps a "live_migration_secure_channel" to indicate that 
> > > >         migration must use encryption, which would imply use of 
> > > >         TUNNELLED flag"
> > > > 
> > > >      So we need to discuss whether the config option should express the
> > > >      choice of tunneled vs native, or whether it should express another
> > > >      choice which implies tunneled vs native.
> > > > 
> > > >        https://review.openstack.org/263434
> > > 
> > > We probably have to consider that operator does not know much about
> > > internal libvirt flags, so options we are exposing for him should
> > > reflect benefice of using them. I commented on your review we should
> > > at least explain benefice of using this option whatever the name is.
> > 
> > As predicted, plenty of discussion on this point in the review :)
> > 
> > You're right that we don't give the operator any guidance in the help
> > message about how to choose true or false for this:
> > 
> >   Whether to use tunneled migration, where migration data is 
> >   transported over the libvirtd connection. If True,
> >   we use the VIR_MIGRATE_TUNNELLED migration flag
> > 
> > libvirt's own docs on this are here:
> > 
> >   https://libvirt.org/migration.html#transport
> > 
> > which emphasizes:
> > 
> >   - the data copies involved in tunneling
> >   - the extra configuration steps required for native
> >   - the encryption support you get when tunneling
> > 
> > The discussions I've seen on this topic wrt Nova have revolved around:
> > 
> >   - that tunneling allows for an encrypted transport[1]
> >   - that qemu's NBD based drive-mirror block migration isn't supported
> >     using tunneled mode, and that danpb is working on fixing this
> >     limitation in libvirt
> >   - "selective" block migration[2] won't work with the fallback qemu
> >     block migration support, and so won't currently work in tunneled
> >     mode
> 
> I'm not working on fixing it, but IIRC some other dev had proposed
> patches.
> 
> > 
> > So, the advise to operators would be:
> > 
> >   - You may want to choose tunneled=False for improved block migration 
> >     capabilities, but this limitation will go away in future.
> >   - You may want to choose tunneled=False if you wish to trade and
> >     encrypted transport for a (potentially negligible) performance
> >     improvement.
> > 
> > Does that make sense?
> > 
> > As for how to name the option, and as I said in the review, I think it
> > makes sense to be straightforward here and make it clearly about
> > choosing to disable libvirt's tunneled transport.
> > 
> > If we name it any other way, I think our explanation for operators will
> > immediately jump to explaining (a) that it influences the TUNNELLED
> > flag, and (b) the differences between the tunneled and native
> > transports. So, if we're going to have to talk about tunneled versus
> > native, why obscure that detail?
> 
> Ultimately we need to recognise that libvirt's tunnelled mode was
> added as a hack, to work around fact that QEMU lacked any kind of
> native security capabilities & didn't appear likely to ever get
> them at that time.  As well

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-08 Thread Jay Pipes


On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:

On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:

On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:

On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:

Hi all,

A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.

Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.

So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.

// jim

[1] https://review.openstack.org/#/c/220268/


What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.

Requests-mock debugging when things go wrong seems a bit simpler.

This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.

-Sean


We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.


I don't really see the point of a separate project like Mimic that has a
whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
RAX-specific) API services. It's just a great way to introduce a larger
surface area for bugs to creep in -- since you have to keep the Mimic
interfaces up to date with the real interfaces. Better to keep something
like this -- if it is TRULY needed -- in-tree with the API service itself,
so that the chances of divergence are reduced. This is similar to the
fakevirt driver in Nova. It's in tree for good reason: when someone changes
the virt driver interface, the fakevirt driver goes boom and needs to be
changed in a corresponding fashion in the same patch.


I tend to think our REST APIs are much more stable than internal python
APIs, and so we can manage the divergence much easier.

Also, mimic can simulate:

* old versions (less needed with all the microversion stuff)
* old bugs that were shipped
* misconfigurations
* networking errors
* the passage of time (including timeouts)

We probably don't want to keep a catalog of old bugs and misconfigurations in
tree forever.


What value does a functional test against an HTTP API service that does
nothing (other than introduce greater surface area for bugs) actually offer
over unit tests anyway?


Testing the full stack of the client instead of mocking the bottom
half of requests is a big one.

Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
https://www.youtube.com/watch?v=HKUUQme3dwA


OK, I just watched that. Sorry, still don't see the value that Mimic
provides over unit testing the client interfaces and mocking out the HTTP
payloads so you have strict control over the expectations.

The problem that Glyph noted in the video about the unit test that mocked
out os.chdir to improperly return a value isn't solved whatsoever by Mimic,
so I find it odd that that example was used in discussing the value of
Mimic. Bad mocks are just that: bad mocks. The same false positive failure
could easily be introduced with a typo in the "metadata injection" that
Mimic does to inject failures into the system.

Mimic isn't testing anything on the server side at all so I'm not sure why
folks call it "integration testing". It isn't testing the integration of
anything at all. All it enables is multi-language client library testing,
and see my response to Ben, the surface area it introduces for bugs in the
test platform itself in my opinion outweigh the multi-language value it
might have.


FWIW, I've only been calling it functional testing. I also don't care
about the multi-language parts here. The sole purpose we have for mimic
(so far) is to functionally test python-ironicclient without mocking the
world in

Re: [openstack-dev] [nova][bugs] help needed: volunteers for bug skimming (1 week)

2016-01-08 Thread Anthony Chow

There is an URL for the duty:

https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty

BTW: how or where do you sign up?

Have a nice weekend and happy bug hunting.

anthony.

On Fri, Jan 8, 2016 at 10:50 AM, Augustina Ragwitz 
wrote:

> I signed up for the week before the Nova Midcycle meeting. Even though I'm
> still new, I feel capable of at least getting the tags mostly right. When
> I'm not quite sure which tag it would fall under, I search for similar bugs
> and see how they were tagged.
>
> I did have one clarifying question, what exactly are the expectations of
> the bug skimming duty? I assumed it was just tagging bugs to get them into
> the appropriate queues for triaging. Do we also need to confirm the bugs
> once they've been tagged or does that fall under "triage" and not
> "skimming"?
>
> Thanks!
> Augustina
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-01-08 Thread Chris Friesen

On 01/07/2016 06:55 PM, Mike Bayer wrote:

On 01/07/2016 11:02 AM, Sean Dague wrote:

On 01/07/2016 09:56 AM, Brant Knudson wrote:

On Thu, Jan 7, 2016 at 6:39 AM, Clayton O'Neill > wrote:

 On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka
 > wrote:
 >
 > Linux gurus please correct me here, but my understanding is that Linux
 > kernel queues up to $backlog number of connections *per socket*. In
 > our case child processes inherited the FD of the socket, so they will
 > accept() connections from the same queue in the kernel, i.e. the
 > backlog value is for *all* child processes, not *per* process.

 Yes, it will be shared across all children.

 >
 > In each child process eventlet WSGI server calls accept() in a loop to
 > get a client socket from the kernel and then puts into a greenlet from
 > a pool for processing:

 It’s worse than that.  What I’ve seen (via strace) is that eventlet
 actually
 converts socket into a non-blocking socket, then converts that
 accept() into a
 epoll()/accept() pair in every child.  Then when a connection comes
 in, every
 child process wakes up out of poll and races to try to accept on the the
 non-blocking socket, and all but one of them fails.

 This means that every time there is a request, every child process
 is woken
 up, scheduled on CPU and then put back to sleep.  This is one of the
 reasons we’re (slowly) moving to uWSGI.

I just want to note that I've got a change proposed to devstack that
adds a config option to run keystone in uwsgi (rather than under
eventlet or in apache httpd mod_wsgi), see
https://review.openstack.org/#/c/257571/ . It's specific to keystone
since I didn't think other projects were moving away from eventlet, too.

I feel like this is a confused point that keeps being brought up.

The preferred long term direction of all API services is to be deployed
on a real web server platform. It's a natural fit for those services as
they are accepting HTTP requests and doing things with them.

Most OpenStack projects have worker services beyond just an HTTP server.
(Keystone is one of the very few exceptions here). Nova has nearly a
dozen of these worker services. These don't naturally fit as wsgi apps,
they are more traditional daemons, which accept requests over the
network, but also have periodic jobs internally and self initiate
actions. They are not just call / response. There is no long term
direction for these to move off of eventlet.

This is totally speaking as an outsider without taking into account all
the history of these decisions, but the notion of "Python + we're a
daemon" == "we must use eventlet" seems a little bit rigid.  Also, the
notion of "we have background tasks" == "we cant run in a web server",
also not clear.  If a service intends to serve HTTP requests, that
portion of that service should be deployed in a web server; if the
system has other "background tasks", ideally those are in a separate
daemon altogether, but also even if you're under something like
mod_wsgi, you can spawn a child process or worker thread regardless.
You always have a Python interpreter running and all the things it can do.

In the case of nova at least most (all?) of these separate worker services do 
not process HTTP requests, but rather RPC requests.

It might make sense for nova-api to run under a web server even if 
nova-compute/nova-conductor/nova-scheduler/etc don't.

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-01-08 Thread Chris Friesen


On 01/07/2016 05:44 PM, Mike Bayer wrote:



On 01/07/2016 07:39 AM, Clayton O'Neill wrote:

On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka  wrote:


Linux gurus please correct me here, but my understanding is that Linux
kernel queues up to $backlog number of connections *per socket*. In
our case child processes inherited the FD of the socket, so they will
accept() connections from the same queue in the kernel, i.e. the
backlog value is for *all* child processes, not *per* process.



Yes, it will be shared across all children.



In each child process eventlet WSGI server calls accept() in a loop to
get a client socket from the kernel and then puts into a greenlet from
a pool for processing:


It’s worse than that.  What I’ve seen (via strace) is that eventlet actually
converts socket into a non-blocking socket, then converts that accept() into a
epoll()/accept() pair in every child.  Then when a connection comes in, every
child process wakes up out of poll and races to try to accept on the the
non-blocking socket, and all but one of them fails.


is that eventlet-specific or would we see the same thing in gevent ?


If you've got multiple processes all doing select()/poll()/epoll()/etc on a 
single socket that has become readable, you're going to run into this sort of 
thundering herd problem unless you have a separate mechanism to serialize things.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Adding "IPv6" bracket handling utilities in openstacklib.

2016-01-08 Thread Sofer Athlan-Guyot

Cody Herriges  writes:

Sorry I didn't see you reply before.  Comments below.

> Sofer Athlan-Guyot wrote:
>> Hi,
>> 
>> There are a few places where I would like to be able to check for IPv6
>> address and add bracket to the parameters.  I think that would be a nice
>> addition to the puppet-openstacklib/lib/puppet/parser.
>> 
>> Here the interface I have in mind with the puppet-nova module:
>> 
>> class nova::vncproxy::common (
>>   $vncproxy_host = undef,
>>   $vncproxy_protocol = undef,
>>   $vncproxy_port = undef,
>>   $vncproxy_path = undef,
>> ) {
>> 
>>   include ::nova::deps
>> 
>>   $vncproxy_host_real = pick(
>> ipv6_add_bracket_maybe($vncproxy_host,
>>$::nova::compute::vncproxy_host,
>>$::nova::vncproxy::host,
>>false)
>> 
>> 
>> This would returns an array with the host decorated with "[]" if the
>> value is an IPv6 address.  Ideally the function could take only one
>> value and return it or take an array and return an array for seamless
>> integration in the code.
>> 
>> WDYT?
>> 
>
> I see this and it looks like that only only reason this is a problem is
> because we've broken up all the pieces of data needed to generate a URI
> so it becomes inappropriate to decorate the vncproxy_host variable's
> value with "[]" because it lacks the port appended to the end.  What are
> the ramifications of simply switching to a "$vnc_uri" variable much the
> same that has happened with identity_uri and auth_uri, e.g.
> https://review.openstack.org/262799. If one has to simply define the
> entire URI, they'll be able to properly decorate the IPv6 address.

Yes, that could be something to consider as well.  The difficulties with
this approach, that I see are that it's not easy on the user (change of
interface) and must rely on a deprecation period (code to maintain).

Adding a function on the other hand is transparent for the user and it
may be useful in other part of the code.

As for this solution (adding a function) I had to lower my expectation
to meet puppet < 4.1 reality.  There is no splat operator, and no way to
chain functions as I wanted at the beginning.  What I did is a simpler
function that take only one argument, the stuff to maybe transform.  The
example above adjusted:


  $vncproxy_host_real = ipv6_add_bracket_maybe(pick(
  $vncproxy_host,
  $::nova::compute::vncproxy_host,
  $::nova::vncproxy::host,
false))

Please let me know what you think.
-- 
Sofer Athlan-Guyot

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] kilo and liberty is blocked on bug 1532048

2016-01-08 Thread Jordan Pittier

On Fri, Jan 8, 2016 at 6:34 AM, Ian Cordasco 
wrote:

> -Original Message-
> From: Matt Riedemann 
> Reply: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> Date: January 7, 2016 at 19:11:10
> To: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> Subject:  [openstack-dev] [stable] kilo and liberty is blocked on bug
> 1532048
>
> > django_compressor 2.0 was released today which drops support for
> > django<1.8 which is what stable/kilo g-r has django capped at, so
> > horizon fails to install in kilo which breaks kilo jobs and grenade jobs
> > in liberty.
> >
> > The cap in stable/kilo g-r is here:
> >
> > https://review.openstack.org/#/c/265025/
> >
> > I'll babysit the reqs sync to horizon in kilo tonight which should then
> > free things up again.
>
> Thanks for babysitting those Matt!
>
Yes, thanks a lot !
It's bad when you wake up in the morning and you see that your CI is broken
but it's really good to see a fix has already been submitted.

>
> --
> Ian Cordasco
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [doc] DocImpact vs. reno

2016-01-08 Thread Tom Fifield


On 08/01/16 21:15, Sean Dague wrote:

On 01/07/2016 06:21 PM, Lana Brindley wrote:



On 7 Jan 2016, at 2:09 AM, Sean Dague  wrote:

On 01/06/2016 09:02 AM, Jeremy Stanley wrote:

On 2016-01-06 07:52:48 -0500 (-0500), Sean Dague wrote:
[...]

I think auto openning against a project, and shuffling it to
manuals manually (with details added by humans) would be fine.

It's not clear to me why a new job was required for that.


The new check job was simply a requirement of the Docs team, since
they were having trouble triaging auto-generated bugs they were
receiving and wanted to enforce the inclusion of some expository
metadata.


Sure, but if that triage is put back on the Nova team, that seems fine.


So you’re thinking we should make all docimpact bugs go to the project bug 
queue? Even for defcore?


Yes, because then it would be the responsibility of the project team to
ensure there is enough info before passing it onto the docs team.




It also doesn't make sense to me there would be a DocImpact change that
wouldn't also have a reno section. The reason someone thinks that a
change needs reflection in the manual is that it adds/removes/changes a
feature that would also show up in release notes. Perhaps my imagination
isn't sufficient to come up with a scenario where DocImpact is valid,
but we wouldn't have content in one of those sections.


I can think of plenty. What about where a command is changed slightly? Or an 
output is formatted differently? Or where flags have been removed, or default 
values changed, or ….


Nearly all of those changes have been triggering release notes at this
point. They are all changes the user needs to adapt to because they
potentially impact compatibility.


Would love it to be the case, but I don't think that's correct. Or if 
it's supposed to be correct, it hasn't been well communicated :)


Few random reviews from the DocImpact queue that didn't have relnotes:

https://review.openstack.org/#/c/180202/
https://review.openstack.org/#/c/249814/
https://review.openstack.org/#/c/250818/
https://review.openstack.org/#/c/230983/

Didn't really look closely into these - would encourage someone with a 
bit more time to do so, but the fact that these were so trivial to eke 
out means that "nearly all" is almost certainly a bad assumption.





Bugs and relnotes are two very different things.

L

Lana Brindley
writer:speaker:blogger
http://lanabrindley.com





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] Nominating Mariam John to Trove Core

2016-01-08 Thread Amrith Kumar

Congratulations Mariam, I look forward to continuing to work with you on Trove.

-amrith 

> -Original Message-
> From: Vyvial, Craig [mailto:craig.vyv...@hpe.com]
> Sent: Friday, January 08, 2016 4:27 PM
> To: OpenStack Development Mailing List (not for usage questions)
> 
> Subject: Re: [openstack-dev] [trove] Nominating Mariam John to Trove Core
> 
> Looks like we have a consensus from the team and I’d like to welcome
> Mariam John to the Trove Core Team!
> 
> I look forward to continuing to work with Mariam!
> 
> Thanks,
> -Craig
> 
> 
> On Jan 7, 2016, at 12:21 PM, Nikhil Manchanda
> > wrote:
> 
> +1 from me as well.
> Couldn't agree more!
> 
> Thanks,
> -Nikhil
> 
> On Wed, Jan 6, 2016 at 12:03 PM, Peter Stachowski
> > wrote:
> I agree!
> 
> +1
> 
> Peter
> 
> -Original Message-
> From: Vyvial, Craig
> [mailto:craig.vyv...@hpe.com]
> Sent: January-06-16 2:40 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [trove] Nominating Mariam John to Trove Core
> 
> Thanks for the nomination Amrith and I think Mariam will be a great addition
> to the core team.
> 
> +1
> 
> -Craig
> 
> On Jan 6, 2016, at 12:39 PM, Amrith Kumar
>  m>> wrote:
> 
> Members of the Trove team,
> 
> I would like to nominate Mariam John (johnma on IRC) to the Trove core
> review team.
> 
> Mariam has been an active member of the Trove team for some time now.
> She added support for IBM DB2 in Trove and provided elements for building
> Trove guest images. She also implemented code that provided CouchDB
> support in Trove. She has been an active reviewer and I have found her
> review comments to be insightful and useful.
> 
> You can review her activity report at
> 
> http://stackalytics.com/report/users/mariamj
> 
> Members of the Trove core team, please reply to this message with your
> feedback to this proposed change.
> 
> Thanks,
> 
> -amrith
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org requ...@lists.openstack.org> requ...@lists.openstack.org requ...@lists.openstack.org>>?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe requ...@lists.openstack.org/?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe requ...@lists.openstack.org/?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org requ...@lists.openstack.org>?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

2016-01-08 Thread ESWAR RAO

Thanks Joshua for the clear explanation.

On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow  wrote:

> ESWAR RAO wrote:
>
>> Hi Joshua,
>>
>> Thanks for the response.
>>
>> As you mentioned, I went through below which has taskflow as celery
>> front end:
>>
>> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
>>
>> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery
>>
>
> I think the idea way-back-when was celery would be a backend, not a
> front-end, although my memory might be fuzzy in this area (since it was a
> few years ago).
>
>
>>
>> I guess if we have a job = task1 + task2 ; if we execute them through a
>> taskflow parallel pattern, they will be executed in parallel.
>>
>> Just curious to know that since this threading is built on
>> eventlets/green threads/python threads they will be affected by GIL and
>> we may not utilize multi-core capability of our systems.
>>
>
> So that assumes u are using a taskflow engine which is powered by threads
> (or greenthreads), there are two types that do not use threads but actually
> run out of process:
>
> http://docs.openstack.org/developer/taskflow/engines.html#types
>
> 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of
> process, across different machines, using kombu for triggering
> start/status... of tasks on other machines)
>
> 2. A parallel engine that uses a process pool executor (
> https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor)
> will then execute tasks using out of process child-processes.
>
> So both of these don't have the same GIL issue (although greenthreads by
> there very nature are not affected by the GIL either).
>
> The example (aptly named hello world) shows these different types
> (although it doesn't show the worker based one, since that requires more
> things to setup) @
> https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82
>
> -Josh
>
>
>> It seems in celery, we can have these tasks run as multiple processes so
>> that they are not affected by GIL.
>>
>
> Right, the above starts to show why taskflow is really a super-set of
> celery (for some combination/interpretation of what taskflow or celery do).
>
>
>> Please correct me if my understanding is wrong ??
>>
>> Thanks
>> Eswar
>>
>>
>> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow > > wrote:
>>
>> I also updated
>> https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
>> that said wiki is no longer active (it was an attempt to back a
>> taskflow engine[1] with celery); although if u are interested in
>> continuing down this path feel free.
>>
>> Hopefully that clears up some 'confusion' around that wiki.
>>
>> [1] http://docs.openstack.org/developer/taskflow/engines.html
>>
>>
>> Joshua Harlow wrote:
>>
>> So actually they are quite different, (although similar at some
>> level),
>>
>> Given that celery isn't really a replacement for taskflow
>> although one
>> could say, from what I've heard from others, that taskflow is a
>> super-set of what celery is so taskflow likely can replace parts
>> of
>> celery (but not vice-versa).
>>
>> Feel free to jump on #openstack-state-management IRC channel if
>> u want
>> to chat in person more about why (it gets into details that
>> might just
>> be easier to explain in person).
>>
>> ESWAR RAO wrote:
>>
>> Hi All,
>>
>> Please let me know whether celery is replacement for taskflow.
>>
>> As per my understanding, task-flow can break jobs into tasks
>> and execute
>> them.
>>
>>  From celery wiki, it also does almost similar behaviour.
>>
>> I guess in most of openstack components taskflow is widely
>> used.
>> Any places where its being replaced with celery ??
>>
>> Celery: https://wiki.openstack.org/wiki/Celery
>> Distributed:
>> https://wiki.openstack.org/wiki/DistributedTaskManagement
>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>
>> Thanks
>> Eswar
>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> 
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> ___
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openst...@lists.openstack.org
>> 
>>

Re: [openstack-dev] [nova][bugs] help needed: volunteers for bug skimming (1 week)

2016-01-08 Thread Markus Zoeller

Anthony Chow  wrote on 01/07/2016 07:02:04 PM:

> From: Anthony Chow 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: 01/07/2016 07:04 PM
> Subject: Re: [openstack-dev] [nova][bugs] help needed: volunteers for 
> bug skimming (1 week)
> 
> Sounds like the volunteer has to be working with Nova for sometime so 
> as to know how to properly tag the bug otherwise I am interested.
> 
> anthony.
> 

The more you know about Nova the easier is the bug triage. 
The more bug triage you do, the more you know about Nova.
My understanding of the current process is summarized in [1].

To have some numbers, there are 2-5 new bugs each day, which
takes me in sum 0.5-1.5 hours to triage per day (I'm with Nova
for 3 cycles).

[1] http://markuszoeller.github.io/posts/openstack-bugs/

Regards, Markus Zoeller (markus_z)

> 
> On Thu, Jan 7, 2016 at 3:06 AM, Markus Zoeller  
wrote:
> The bug triage is an important first step to improve the quality
> of Nova. Every code contributor should be aware of it and can
> contribute without any special permissions [1]. We need more
> volunteers to do this. If you want to be one of them for 1 week,
> raise your hand in this thread or add your name in the wiki [2].
> At best we will have at least 1 person until today's Nova meeting
> at 21:00 UTC [3]. When you volunteer, I will be available in IRC
> for requests, too.
> 
> Regards, Markus Zoeller (markus_z)
> 
> References:
> [1]
> 
https://wiki.openstack.org/wikiBugTriage#Task_1:_Confirm_new_bugs_.28anyone.29

> [2]
> https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty
> [3] https://wiki.openstack.org/wiki/Meetings/Nova
> 
> 
> 
__
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
__
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-01-08 Thread Radomir Dopieralski


On 01/07/2016 05:55 PM, Mike Bayer wrote:


but also even if you're under something like
mod_wsgi, you can spawn a child process or worker thread regardless.
You always have a Python interpreter running and all the things it can do.


Actually you can't, reliably. Or, more precisely, you really shouldn't. 
Most web servers out there expect to do their own process/thread 
management and get really embarrassed if you do something like this,

resulting in weird stuff happening.

--
Radomir Dopieralski

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-08 Thread Julien Danjou

On Thu, Jan 07 2016, Jay Pipes wrote:

> OK, I just watched that. Sorry, still don't see the value that Mimic provides
> over unit testing the client interfaces and mocking out the HTTP payloads so
> you have strict control over the expectations.
>
> The problem that Glyph noted in the video about the unit test that mocked out
> os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
> find it odd that that example was used in discussing the value of Mimic. Bad
> mocks are just that: bad mocks. The same false positive failure could easily 
> be
> introduced with a typo in the "metadata injection" that Mimic does to inject
> failures into the system.
>
> Mimic isn't testing anything on the server side at all so I'm not sure why
> folks call it "integration testing". It isn't testing the integration of
> anything at all. All it enables is multi-language client library testing, and
> see my response to Ben, the surface area it introduces for bugs in the test
> platform itself in my opinion outweigh the multi-language value it might have.

I completely agree with what Jay points out here.

To illustrate with what we do in the Telemetry team: in order test
gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
client against that freshly installed server. It works very well, and
we're sure we test against real data. That's what I would call
integration testing.
We could also easily test against older version of the Gnocchi server by
pip install-ing an older version if we wanted.

This has way more value than mocking the whole HTTP server and crossing
fingers we did a good job mimicking it. Oh, and it's also actually less
work. :)

¹  https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements


Jim Rollenhagen wrote:

[...]
Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.


A bit of history with Twisted.

Back in 2010 we decided we could not afford asking OpenStack developers 
to be familiar with multiple service architecture frameworks, and 
eventlet was chosen as the simplest framework to learn and debug. The 
best reference I found on this is still visible in the wiki:


https://wiki.openstack.org/wiki/UnifiedServiceArchitecture


So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.


The only friction I see is how many developers would be expected to need 
to learn Twisted in order to complete their jobs. My understanding is 
that Twisted expertise could be needed to debug python-ironicclient 
functional tests, which makes the cost relatively limited. So if Mimic 
brings in a clear and significant benefit, I don't think its Twisted 
dependence should play that much against it.


However, I agree with Sean and Jay that the benefit is unclear -- the 
few features that Mimic brings seem to be outweighed by the increased 
risk of introducing a delta between the implementation and the mock. If 
the main benefit is that it's used in other Rackspace projects for 
testing (like Ben said), I'm not sure that makes a compelling argument 
for the rest of the community...


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [chef] deprecation and removal of old branches

2016-01-08 Thread Jan Klare

Hi,

Good point, i will talk to infra and figure out how to properly push these eol 
tags to the last commits of the branches before we remove them. Since i only 
got positive feedback on this, i will also go ahead and ask them to help me 
with the cleanup.


Cheers,
Jan

> On 07 Jan 2016, at 07:42, zhiwei  wrote:
> 
> Sounds good, but I think it's better to create some tags just like other 
> OpenStack projects.
> 
> On Thu, Jan 7, 2016 at 9:05 AM, JJ Asghar  > wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> On 1/6/16 10:22 AM, Samuel Cassiba wrote:
> > I'm +1 on this idea. Looking at other OpenStack projects (Nova, Neutron,
> > etc.) on GitHub, they only have stable/liberty and stable/kilo
> 
> Sounds good to me too.
> 
> 
> - --
> Best Regards,
> JJ Asghar
> c: 512.619.0722  t: @jjasghar irc: j^2
> -BEGIN PGP SIGNATURE-
> Version: GnuPG/MacGPG2 v2
> Comment: GPGTools - https://gpgtools.org 
> 
> iQIcBAEBCgAGBQJWjbnBAAoJEDZbxzMH0+jTbTcP/i/RbxfmQ2T3wasWUKLup9Y3
> bbsiUcfocw40ZT2Y/i/fLiH+mGfIRcvXTLMkYdwji7jhoes2Vt3l+S/CmDR9kb59
> hqxiIft/6kfW3JLjtjkAZtZ5xwa+QSN8DY4aZnnVU9mjdCgDH0DZRuwOZcERZH0Q
> 6gQeIFp0udGVLfDaF32mEdKRRbSlRil6reciDg6W7g+/1lCys8H7ig0/5rqQxE0o
> GmKxyGOlEnpPjpVQz1JeBadktHNjRPOJ1ihiHmC8XusMdg0FDppiJUmLd8gmLFNY
> hRpD2QFHjzlsy+7u5chsiN+ctjawuEZsHCeDD3UDAiSQjptHQy8jw7gl+V4nQry6
> SSpRYkdfV8k5Pf3nsJ8Yyt0SqYggnqjok7DQ9VxEiLdMyQbG+DE7CQI6w/TxmniC
> oiScygu8dCh+KUZ7OXUSBWquHWln6eQlWgJNqv/o4UxZJl9NLxkhDU7OgvMvvCjl
> I1Gx2BdNIcknZhG3KfzUfj1j97whVJkJyHvWMem+pj2BXFY3eIcAkHKx9nHO+cNy
> hBFWoCUXGJHxsUMHhDH7FhbuvQR8qCHVaRvmMaKMyO+6S+5R1Q/L4dYvxZjHNScn
> 3JcWaC5+k91Acj6DYrv4hxszUBtM+KpyG/13Z9ed+qwd6QxH8beKD82kES1ZEl+P
> crT1uNDOT9ToRLWRHEwd
> =2adb
> -END PGP SIGNATURE-
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

2016-01-08 Thread Chris Friesen


On 01/07/2016 09:49 AM, Roman Podoliaka wrote:

Actually we already do that in the parent process. The parent process:

1) starts and creates a socket

2) binds the socket and calls listen() on it passing the backlog value
(http://linux.die.net/man/2/listen)

3) passes the socket to the eventlet WSGI server
(https://github.com/openstack/oslo.service/blob/master/oslo_service/wsgi.py#L177-L192)

4) forks $*_workers times (child processes inherit all open file
descriptors including the socket one)

5) child processes call accept() in a loop

Linux gurus please correct me here, but my understanding is that Linux
kernel queues up to $backlog number of connections *per socket*. In
our case child processes inherited the FD of the socket, so they will
accept() connections from the same queue in the kernel, i.e. the
backlog value is for *all* child processes, not *per* process.


I believe this is correct, the limit is on the (shared) socket and not the 
individual processes.


Also, an interesting point from the listen man page above:

"If the backlog argument is greater than the value in 
/proc/sys/net/core/somaxconn, then it is silently truncated to that value; the 
default value in this file is 128."


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ec2-api] Removing Nova's In tree ec2 API code in favor of ec2-api project

2016-01-08 Thread Alexandre Levine


Thierry,

The ec2-api project is in gating and totally functional. We'll apply for 
it to become OpenStack project very shortly. Next week in fact.


Best regards,
  Alex Levine


On 1/8/16 9:47 AM, Thierry Carrez wrote:

Davanum Srinivas wrote:

Dear Nova cores,

Sorry this did not make into the Nova weekly meeting agenda yesterday.

Is it time for dropping the EC2/ObjectStore REST API Since we've told
folks since Kilo that we will be removing this code and we dropped the
entries api-paste.ini in Liberty?
https://review.openstack.org/#/c/263368/


Could we have a quick status update on the ec2-api project ? It has 
not applied to become an OpenStack project yet...





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] config options: IRC meeting at Jan. 11th

2016-01-08 Thread Markus Zoeller

First things first, I'd like to thank those people who are contributing
to this task very much! It's good to see the progress we already made
and that the knowledge of this area is now spread amongst more people.

The second thing I'd like to talk about is the next steps.
It would be great if we could have a short realtime conversation in IRC
to:
* make clear that the help text changes are necessary for a patch series
* clarify open questions from your side
* discuss the impact of [1] which should be the last piece in place 
  which reduces the merge conflicts we faced to a very minimum.

Let's use the channel #openstack-meeting-3 at coming Monday, 
January the 11th, at 15:00 UTC for that.

References:
[1] "single point of entry for sample config generation"
https://review.openstack.org/#/c/260015

Regards, Markus Zoeller (markus_z)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] Seeking FFE for "Add CIM namespace metadata definitions"


O 04/01/16 20:46 +, Bhandaru, Malini K wrote:

Hello Glance Team!



   Hope you had a wonderful vacation and wishing you health and
happiness for 2016.

Would very much appreciate your considering https://review.openstack.org/259694
for a feature freeze exception.

Thank you to Travis Tripp for chiming in. Searchlight APIs will provide elastic
search capability, and users such as Horizon can leverage these.

Supporting CIM namespace thus requires just importing the namespace tags and we
already have a PoC implementation by Lin, which is how

we were able to generate the graphics in Horizon that were attached to the
spec. Would appreciate core votes to approve this spec.


Hi Malini,

Given the short notice on the spec and the fact that not many members
of the Glance team had a chance to read it before the spec freeze, I
think it'll be hard to justify a spec freeze exception for this one.

Hope you understand,
Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] [sahara] Adding support for HBase in Trove

2016-01-08 Thread Amrith Kumar

As Kevin suggests, I'm adding [sahara] to the subject line. 

Others in sahara who now see this thread, apologies for sending you a delayed 
invitation to the party. There's still lots of food and beer so come on in!

-amrith

> -Original Message-
> From: Fox, Kevin M [mailto:kevin@pnnl.gov]
> Sent: Thursday, January 07, 2016 7:32 PM
> To: OpenStack Development Mailing List (not for usage questions)
> 
> Subject: Re: [openstack-dev] [trove] Adding support for HBase in Trove
> 
> While I applaud raising the issue on the mailing list to get more folks to 
> weigh
> in, I think part of the problem maybe the lack of a [sahara] tag on the 
> subject.
> The thread is still tagged to be a Trove centric conversation. All respondents
> please consider adding [sahara] to the subject.
> 
> Thanks,
> Kevin
> 
> From: Amrith Kumar [amr...@tesora.com]
> Sent: Thursday, January 07, 2016 1:59 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [trove] Adding support for HBase in Trove
> 
> > -Original Message-
> > From: michael mccune [mailto:m...@redhat.com]
> > Sent: Thursday, January 07, 2016 3:12 PM
> > To: openstack-dev@lists.openstack.org
> > Subject: Re: [openstack-dev] [trove] Adding support for HBase in Trove
> >
> > On 01/07/2016 11:59 AM, Amrith Kumar wrote:
> > >  From the things that you and Pete (Peter MacKinnon) are saying, I
> > > don't
> > understand why there is an objection to accepting the currently
> > proposed implementation which is clearly for single node deployments?
> > Both Standalone and Pseudo-Distributed are by definition, explicitly,
> > necessarily, absolutely, positively, definitely single node. I can't
> > be more explicit about that. That's all that is being proposed at this
> > time. See more comments below.
> >
> > i didn't think i explicitly objected to the spec, if it seems that way
> > then i apologize. after reading the spec and the comments, it seemed
> > that there was some question about engagement with the sahara team. i
> > wanted to help bring some light to the issues surrounding deploying
> > hbase and thought it would be good to participate in the discussion.
> 
> You are correct Michael. There was a suggestion that we should engage with
> the Sahara team (in the Trove team meeting yesterday) and that is what
> prompted this email thread. So I appreciate your participation as one who is a
> member of the Sahara team.
> 
> >
> > > Further, the current proposal also chooses an implementation
> > > strategy that
> > makes it much easier to handle fully-distributed in a different way in
> > the future. Consider this, Trove could equally well have dealt with
> > HBase using a single datastore for all operating modes. In the current
> > implementation, one would create a HBase standalone instance using a
> command that included:
> > >
> > > --datastore hbase-standalone
> > >
> > > And a pseudo-distributed instance by including
> > >
> > > --datastore hbase-pseudo-distributed.
> > >
> >
> > and this delineation sounds reasonable to me
> >
> > > Trove could equally well function by having a single datastore
> > > (hbase) but
> > this would make hbase-fully-distributed harder to do in a different
> > way in the future. I consciously eschewed that path, for this very
> > specific reason; it would limit choice in the future.
> >
> > agreed
> >
> > > Now, the implementation behind hbase-fully-distributed could be a
> > custom Trove guest agent that could (if we decided to go that route)
> > interact with Sahara. However, an alternative implementation of
> > hbase-fully- distributed could orchestrate everything natively in
> > Trove. There is much flexibility in the current proposal, and I submit
> > to you that this is being lost in your reading of the specification
> > and the current implementation as proposed.
> >
> > i don't think your characterization of my reading comprehension is fair.
> > as i stated earlier, i wanted to participate in the discussion
> > surrounding deploying a technology that sahara currently deploys.
> > fwiw, i agree with what you are saying here, but i also think it is
> > axiomatic, the trove team can choose whichever path it would like for
> implementation.
> >
> > >> i think this sounds reasonable, as long as we are limiting it to
> > >> standalone mode. if the deployments start to take on a larger scope
> > >> i agree it would be useful to leverage sahara for provisioning and 
> > >> scaling.
> > >
> > > Why only standalone? The current proposal explicitly covers only
> > standalone and pseudo-distributed which are both valid strictly (add
> > other adjectives here to taste) single node topologies and the
> > currently submitted specification specifically carves out
> > fully-distributed operation as requiring further thought and contemplation.
> >
> > i think starting with standalone mode (and not

Re: [openstack-dev] [oslo][osdk] PrettyTable needs a home in OpenStack

2016-01-08 Thread Major Hayden

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 01/08/2016 07:06 AM, Flavio Percoco wrote:
> I'm saying all the above because we now need to find a home for it in
> OpenStack.
> 
> I've identified 2 possible places:
> 
> 1) Oslo, as we maintaing cross-project libraries and some of them are
> not in the oslo namespace
> 
> 2) OpenStack Client team as they maintain cliff already and it'd
> perhaps make more sense to have this library there.

#2 makes the most sense to me.  Thanks for taking action to keep PrettyTable 
alive! :)

- --
Major Hayden
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWj7taAAoJEHNwUeDBAR+xBPkP/2HPChBdroQH/bW9l1LfEvxR
KzkJa8GeMMjs5nhc55AttR6H8lOyp+f2sj0+3lcSV/9AmWdK0+TN+KcZTHhWHYqM
ONMVN6ii4tAvwN48lJALHxr02D+iPEH6HWw3iH5sMfnoQfoDNSQM6m42XWtHP0GR
cGlYr3M2lPXJbpEiDqJHLWBWWbHeuy5wCyZpcJY4GNNZUcv9Xm+XT3s+bE27tFhm
mVYZRgBbxfHHhNaaEkI5e9n6DSmFc5ScJU94O3lSNPP439pDxHHVVODwDnJXhKHx
dAAE7wxFpJEYrYKFJNDK8g48qKKXevqrUVDhnAKW8ivsDENDYhwdqJ3U0p0i6cCj
qeScjSqCwzjbVxibj89YtcosstkDZgyASIfp99MRQ9/TxsO9vqa9wi8O/WQsljbY
y0WwFRXQkpcndE6Ia/t1uu7EaXmUbtPsVldMKwdUpelr/b/R0F/T6Z503rG83Dy1
FLvVxqDk3Leb4VV/H6zNNqKm9mcw1hB7JfDcEDRR7HarEA/p8HZezlCXg9Fa0w7x
ZngtPO9/23g35Bk62XqU01yy7d6OmBpCsGMXcjsmyWojhF7VIjZCvWlY2BK5+tUQ
IkdMgXXS6AIyXBef89sRxZ/48Nw6ZyYkepiKUoSVztvQzqNOOhKHBFd+9visC9De
+tjioMLJ3V2QO8qszGrs
=wUJt
-END PGP SIGNATURE-

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Glance] Nova + Glance_v2 = Love


On 24/12/15 16:49 +0300, Mikhail Fedosin wrote:

Hello, it's another topic about glance v2 adoption in Nova, but it's different
from the others. I want to declare that there is a set of commits, that make
Nova version agnostic and allow to work with both glance apis. The idea of the
solution is to determine the current api version in the beginning and make
appropriate requests after.
(https://review.openstack.org/#/c/228578/, https://review.openstack.org/#/c/
238309/, https://review.openstack.org/#/c/259097/)

Indeed, it requires some additional (and painful) work, but now all tempest
tests pass in Jenkins.

Note: this thread is not about xenplugin, there is another topic, called
'Xenplugin + Glance_v2 = Hate'

Here the main issues we faced and how we've solved them:

1. "changes-since" filter for image-list is not supported in v2 api. Steve
Lewis made a great job and implemented a set of filters with comparison
operators https://review.openstack.org/#/c/197388/. Filtering by
'changes-since' is completely similar to 'gte:updated_at'.

2. Filtering by custom properties must have prefix 'property-'. In v2 it's not
required.

3. V2 states that all custom properties must be image attributes, but Nova
assumes that they are included in 'properties' dictionary. It's fixed with
glanceclient's method 'is_base_property(prop_name)', that returns False in case
of custom property.

4. is_public=True/False is visibility="public"/"private" respectively.

5. Deleting custom image properties in Nova is performed with 'purge_props'
flag. If it's set to True, then all prop names, that are not included in
updated data, will be removed. In case of v2 we have to explicitly specify prop
names in the list param 'remove_props'. To implement this behaviour, if
'purge_props' is set, we make additional 'get' request to determine the
existing properties and put in 'remove_prop' list only those, that are not in
updated_data.

6. My favourite:
There is an ability to activate an empty image by just providing 'size = 0':
https://review.openstack.org/#/c/9715/, in this case image will be a collection
of metadata. Glance v2 doesn't support this "feature" and that's why we have to
implement a very dirty workaround:
    * v2 requires, that disk_format and container-format must be set before the
activation. if these params are not provided to 'create' method then we
hardcode it to 'qcow2' and 'bare'.
    * we call 'upload' method with empty data (data = '') to activate image.
Me (and the rest glance team) think that this image activation with zero-size
is inconsistent and we won't implement it in v2. So, I'm going to ask if Nova
really needs this feature and maybe it's possible to make it work without it.

In conclusion, I want to congratulate you with this huge progress and say there
is a big chance that we can deprecate glance v1 in Mitaka cycle.


Thanks for all your efforts here. Lets work closely with the Nova core
team to help this patches move forward. There has been feedback
already, which is great.

Flavio



Hooray!
And Merry Christmas!
   
--
Best regards,
Mikhail Fedosin



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Glance] Nova + Glance_v2 = Love


On 29/12/15 07:41 -0600, Sam Matzek wrote:

On Thu, Dec 24, 2015 at 7:49 AM, Mikhail Fedosin  wrote:

Hello, it's another topic about glance v2 adoption in Nova, but it's
different from the others. I want to declare that there is a set of commits,
that make Nova version agnostic and allow to work with both glance apis. The
idea of the solution is to determine the current api version in the
beginning and make appropriate requests after.
(https://review.openstack.org/#/c/228578/,
https://review.openstack.org/#/c/238309/,
https://review.openstack.org/#/c/259097/)

Indeed, it requires some additional (and painful) work, but now all tempest
tests pass in Jenkins.

Note: this thread is not about xenplugin, there is another topic, called
'Xenplugin + Glance_v2 = Hate'

Here the main issues we faced and how we've solved them:

1. "changes-since" filter for image-list is not supported in v2 api. Steve
Lewis made a great job and implemented a set of filters with comparison
operators https://review.openstack.org/#/c/197388/. Filtering by
'changes-since' is completely similar to 'gte:updated_at'.

2. Filtering by custom properties must have prefix 'property-'. In v2 it's
not required.

3. V2 states that all custom properties must be image attributes, but Nova
assumes that they are included in 'properties' dictionary. It's fixed with
glanceclient's method 'is_base_property(prop_name)', that returns False in
case of custom property.

4. is_public=True/False is visibility="public"/"private" respectively.

5. Deleting custom image properties in Nova is performed with 'purge_props'
flag. If it's set to True, then all prop names, that are not included in
updated data, will be removed. In case of v2 we have to explicitly specify
prop names in the list param 'remove_props'. To implement this behaviour, if
'purge_props' is set, we make additional 'get' request to determine the
existing properties and put in 'remove_prop' list only those, that are not
in updated_data.

6. My favourite:
There is an ability to activate an empty image by just providing 'size = 0':
https://review.openstack.org/#/c/9715/, in this case image will be a
collection of metadata. Glance v2 doesn't support this "feature" and that's
why we have to implement a very dirty workaround:
* v2 requires, that disk_format and container-format must be set before
the activation. if these params are not provided to 'create' method then we
hardcode it to 'qcow2' and 'bare'.
* we call 'upload' method with empty data (data = '') to activate image.
Me (and the rest glance team) think that this image activation with
zero-size is inconsistent and we won't implement it in v2. So, I'm going to
ask if Nova really needs this feature and maybe it's possible to make it
work without it.


Nova uses this functionality when it creates snapshots of volume
backed instances, that is, instances that only have Cinder volumes
attached and do not have an ephemeral disk.
In this case Nova API creates Cinder snapshots for the Cinder volumes
and builds the block_device_mapping property with block devices that
reference the Cinder snapshots.  Nova activates this image with size=0
because this image does not have a disk and simply refers to the
collection of Cinder snapshots that collectively comprise the binary
image.  Callers of Glance outside of Nova may also use the APIs to
create "block device mapping" images as well that contain references
to Cinder volumes to attach, blank volumes to create, snapshots to
create volumes from, etc during the server creation.  Not having the
ability to create these images with V2 is a loss of function.


I disagree. Being able to activate empty images breaks the consistency
of Glances API and I don't think it should be considered a feature but
a bug. An active image is an image that can be used to *boot* a VM. If
the image is empty, you simply can't do that.

Note that pointing to a block device makes the image "non-empty"
already. There's a spec merged[0] and code written[1] to allow for
using cinder as backend for Glance.

If there's a missing functionality for the mapping users require, I
believe activating empty images is not the solution for it.

[0] https://review.openstack.org/183363
[1] https://review.openstack.org/166414


The callers could supply 1 byte of junk data, like a space character,
but that would result in a junk image being stored in Glance's default
backing store for the image.  It would also give the impression that a
real disk image exists for the image in a backing store which could be
fetched, which is incorrect.

If I remember correctly Glance V2 lets you transition an image from
queued to active state with size = 0 if the image is using an external
backing store such as http.  The store is then called to fetch the
size.  Would it be possible to allow Glance to consider images with
size = 0 and the block_device_mapping property to be "externally
sourced" and allow the transition?


In v2 you can

Re: [openstack-dev] [magnum] Temporarily remove swarm func test from gate

2016-01-08 Thread Hongbin Lu

Done: https://review.openstack.org/#/c/264998/

Best regards,
Hongbin

-Original Message-
From: Adrian Otto [mailto:adrian.o...@rackspace.com] 
Sent: January-07-16 10:19 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [magnum] Temporarily remove swarm func test from 
gate

Hongbin,

I’m not aware of any viable options besides using a nonvoting gate job. Are 
there other alternatives to consider? If not, let’s proceed with that approach.

Adrian

> On Jan 7, 2016, at 3:34 PM, Hongbin Lu  wrote:
> 
> Clark,
> 
> That is true. The check pipeline must pass in order to enter the gate 
> pipeline. Here is the problem we are facing. A patch that was able to pass 
> the check pipeline is blocked in gate pipeline, due to the instability of the 
> test. The removal of unstable test from gate pipeline aims to unblock the 
> patches that already passed the check.
> 
> An alternative is to remove the unstable test from check pipeline as well or 
> mark it as non-voting test. If that is what the team prefers, I will adjust 
> the review accordingly.
> 
> Best regards,
> Honbgin
> 
> -Original Message-
> From: Clark Boylan [mailto:cboy...@sapwetik.org]
> Sent: January-07-16 6:04 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [magnum] Temporarily remove swarm func 
> test from gate
> 
> On Thu, Jan 7, 2016, at 02:59 PM, Hongbin Lu wrote:
>> Hi folks,
>> 
>> It looks the swarm func test is currently unstable, which negatively 
>> impacts the patch submission workflow. I proposed to remove it from 
>> Jenkins gate (but keep it in Jenkins check), until it becomes stable.
>> Please find the details in the review
>> (https://review.openstack.org/#/c/264998/) and let me know if you 
>> have any concern.
>> 
> Removing it from gate but not from check doesn't necessarily help much 
> because you can only enter the gate pipeline once the change has a +1 from 
> Jenkins. Jenkins applies the +1 after check tests pass.
> 
> Clark
> 
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova][cells] Should flavors in the API DB for cells v2 be soft-deletable?

2016-01-08 Thread Belmiro Moreira

IMHO I think it's a great way to fix the URI problem.
+1

Belmiro

On Fri, Jan 8, 2016 at 3:23 PM, Sylvain Bauza  wrote:

>
>
> Le 08/01/2016 15:10, Andrew Laski a écrit :
>
>> On 01/08/16 at 12:43pm, John Garbutt wrote:
>>
>>> On 7 January 2016 at 19:59, Matt Riedemann 
>>> wrote:
>>>
 There is a cells v2 change up for review [1] which creates the flavor
 tables
 in the API DB.

 I noted that those table definitions include the soft-delete columns
 (deleted and deleted_at), which at the YVR summit and in other threads
 [2]
 we've said we're not doing anymore for new tables.

 The point raised to keep them soft-deletable is that the flavor API
 allows
 showing soft-deleted flavors if you know the id [3]. And you can get the
 flavor id for an instance (we always store the flavor info that was
 used to
 boot the instance).

 The question is, can we break that wrinkle in the API by not allow
 soft-deleting the flavor in the API DB?

>>>
>>> On balance, I think this is OK.
>>>
>>
>> There's an alternate approach to this that we can take which I'll outline
>> below.  But it should meet the needs of anyone currently looking up deleted
>> flavors.
>>
>>
>>
>>> Note that in the normal nova DB, if the admin archives/purges the
 instance_types table, the wrinkle is already broken because the
 soft-deleted
 flavor is now hard-deleted, but that's maybe not a great justification
 for
 consciously removing this support in the API DB.

>>>
>>> This is what won me over. All be it, reluctantly.
>>>
>>
>> I think this is good justification for removing the ability because
>> currently it is essentially undefined behavior. Requesting a deleted flavor
>> may or may not work and either response is correct.
>>
>>
>>> If we made the flavor soft-deletable in the API DB, one issue is we don't
 have an in-tree entrypoint for cleaning this up (there are no
 archive/purge
 CLIs for the API DB). We could always add that, but it's not there right
 now.

 Another thing that came up in the cells meeting this week is that if we
 didn't make the flavor soft-deletable, we could still show the flavor
 information for a given instance via the server GET API. However, that
 would
 be a microversion change to show the full flavor information for the
 server
 rather than just the flavor id.

>>>
>>> This is really only possible because we now store flavor info inside
>>> every instance object.
>>> I think before that, deleting the flavor would make some instance
>>> operations fail.
>>>
>>
>> Because we now store flavor info with each instance it opens up the
>> following possibility:
>>
>> Currently the flavor portion of an instance show request looks like
>> "flavor": {"id": "8", "links": [{"href": "https://example.com/flavors/8;,
>> "rel": "bookmark"}]}
>>
>> If that were instead changed to return 
>> "https://example.com/servers//flavor"
>> as the link and we exposed the flavor information stored with the instance
>> on that endpoint then we no longer need to expose deleted flavors.
>>
>> So for an instance booted with flavor 8 https://example.com/flavors/8
>> would be equivalent to https://example.com/servers//flavor except
>> that the latter would be available even if flavor 8 were deleted.
>>
>> Does that seem like an acceptable path for users?
>>
>>
> Yes, that's something we discussed on IRC yesterday which I think would be
> very good. It means a microversion but it means that operators would be
> very happy because if a flavor is removed, the flavor URI would still be
> working.
>
> +1000 to that.
>
> -Sylvain
>
>
>
>
>>> Thoughts? I'm cross-posting this to -dev and the -operators list to see
 what
 kind of user impact there would be if we didn't soft-delete flavors in
 the
 API DB (so you couldn't look up deleted flavors in the API).

>>>
>>> In balance, I think we should not allow soft_delete of flavors in the
>>> API DB.
>>>
>>> In a related note, and I thinking about a backlog spec looking at a
>>> flavor lifecycle. Thinking about early release, to production, then
>>> phasing out of flavors. I don't think soft delete is needed for that.
>>>
>>> Thanks,
>>> johnthetubaguy
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>

[openstack-dev] [Neutron] Heads up for decomposed plugin break

2016-01-08 Thread Gary Kotton

Hi,
The commit 
https://github.com/openstack/neutron/commit/5d53dfb8d64186b5b1d2f356fbff8f222e15d1b2
 may break the decomposed plugins that make use of the method 
_get_tenant_id_for_create

It would have been nice if there was a deprecation warning.

Options:

  1.  Decomposed plugins fix this
  2.  Revert the patch and find a solution that does not break the plugins

If we do go for #1 then we can start to stop enforcing the usage of the 
deprecation warnings...

Thanks
Gary
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][osdk] PrettyTable needs a home in OpenStack


Greetings,

As some of you know already, google code is going to be shutdown. Some
projects we're using are hosted and, unfortunately, some of them are
unmaintained and perhaps going away.

One of these projects is PrettyTable. This point was raised by Erno in
this patch[0] from jd__. PrettyTable is not just being used in several
openstack specific projects but it's also a transitive dependency for
all client libraries using cliff.

With all that in mind, I've contacted the author of the library and
asked him if it'd be ok for us (OpenStack) to adopt this library. The
author accepted and granted me access to the project on pypi.

I'm saying all the above because we now need to find a home for it in
OpenStack.

I've identified 2 possible places:

1) Oslo, as we maintaing cross-project libraries and some of them are
not in the oslo namespace

2) OpenStack Client team as they maintain cliff already and it'd
perhaps make more sense to have this library there.

One thing to note is that this library has been quite stable, which
means it won't, hopefully, add too much work to the team.

Thoughts?

[0] https://review.openstack.org/#/c/234340/

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][osdk] PrettyTable needs a home in OpenStack

2016-01-08 Thread Davanum Srinivas

#2 please :)

Thanks,
Dims

On Fri, Jan 8, 2016 at 8:06 AM, Flavio Percoco  wrote:
> Greetings,
>
> As some of you know already, google code is going to be shutdown. Some
> projects we're using are hosted and, unfortunately, some of them are
> unmaintained and perhaps going away.
>
> One of these projects is PrettyTable. This point was raised by Erno in
> this patch[0] from jd__. PrettyTable is not just being used in several
> openstack specific projects but it's also a transitive dependency for
> all client libraries using cliff.
>
> With all that in mind, I've contacted the author of the library and
> asked him if it'd be ok for us (OpenStack) to adopt this library. The
> author accepted and granted me access to the project on pypi.
>
> I'm saying all the above because we now need to find a home for it in
> OpenStack.
>
> I've identified 2 possible places:
>
> 1) Oslo, as we maintaing cross-project libraries and some of them are
> not in the oslo namespace
>
> 2) OpenStack Client team as they maintain cliff already and it'd
> perhaps make more sense to have this library there.
>
> One thing to note is that this library has been quite stable, which
> means it won't, hopefully, add too much work to the team.
>
> Thoughts?
>
> [0] https://review.openstack.org/#/c/234340/
>
> --
> @flaper87
> Flavio Percoco
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Openstack version changes affects Fuel-CI

2016-01-08 Thread Sergii Golovatiuk

Hi crew,

Igor and I are about to merge

https://review.openstack.org/#/c/238846/
https://review.openstack.org/#/c/235254/

These patches are destructive to our CI. They require to upload a new ISO
to CI slaves. However, the process takes 2-3 hours to roll our a new ISO.
Please do not submit patches to gerrit within next 3 hours for
'fuel-library' master branch as CI will fail. CI for all submitted patches
during this period of time must be retriggered.

I will send another announce when CI is unblocked.

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova][cells] Should flavors in the API DB for cells v2 be soft-deletable?

On 7 January 2016 at 19:59, Matt Riedemann  wrote:
> There is a cells v2 change up for review [1] which creates the flavor tables
> in the API DB.
>
> I noted that those table definitions include the soft-delete columns
> (deleted and deleted_at), which at the YVR summit and in other threads [2]
> we've said we're not doing anymore for new tables.
>
> The point raised to keep them soft-deletable is that the flavor API allows
> showing soft-deleted flavors if you know the id [3]. And you can get the
> flavor id for an instance (we always store the flavor info that was used to
> boot the instance).
>
> The question is, can we break that wrinkle in the API by not allow
> soft-deleting the flavor in the API DB?

On balance, I think this is OK.

> Note that in the normal nova DB, if the admin archives/purges the
> instance_types table, the wrinkle is already broken because the soft-deleted
> flavor is now hard-deleted, but that's maybe not a great justification for
> consciously removing this support in the API DB.

This is what won me over. All be it, reluctantly.

> If we made the flavor soft-deletable in the API DB, one issue is we don't
> have an in-tree entrypoint for cleaning this up (there are no archive/purge
> CLIs for the API DB). We could always add that, but it's not there right
> now.
>
> Another thing that came up in the cells meeting this week is that if we
> didn't make the flavor soft-deletable, we could still show the flavor
> information for a given instance via the server GET API. However, that would
> be a microversion change to show the full flavor information for the server
> rather than just the flavor id.

This is really only possible because we now store flavor info inside
every instance object.
I think before that, deleting the flavor would make some instance
operations fail.

> Thoughts? I'm cross-posting this to -dev and the -operators list to see what
> kind of user impact there would be if we didn't soft-delete flavors in the
> API DB (so you couldn't look up deleted flavors in the API).

In balance, I think we should not allow soft_delete of flavors in the API DB.

In a related note, and I thinking about a backlog spec looking at a
flavor lifecycle. Thinking about early release, to production, then
phasing out of flavors. I don't think soft delete is needed for that.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [doc] DocImpact vs. reno

2016-01-08 Thread Sean Dague

On 01/07/2016 06:21 PM, Lana Brindley wrote:
> 
>> On 7 Jan 2016, at 2:09 AM, Sean Dague  wrote:
>>
>> On 01/06/2016 09:02 AM, Jeremy Stanley wrote:
>>> On 2016-01-06 07:52:48 -0500 (-0500), Sean Dague wrote:
>>> [...]
 I think auto openning against a project, and shuffling it to
 manuals manually (with details added by humans) would be fine.

 It's not clear to me why a new job was required for that.
>>>
>>> The new check job was simply a requirement of the Docs team, since
>>> they were having trouble triaging auto-generated bugs they were
>>> receiving and wanted to enforce the inclusion of some expository
>>> metadata.
>>
>> Sure, but if that triage is put back on the Nova team, that seems fine.
> 
> So you’re thinking we should make all docimpact bugs go to the project bug 
> queue? Even for defcore?

Yes, because then it would be the responsibility of the project team to
ensure there is enough info before passing it onto the docs team.
> 
>>
>> It also doesn't make sense to me there would be a DocImpact change that
>> wouldn't also have a reno section. The reason someone thinks that a
>> change needs reflection in the manual is that it adds/removes/changes a
>> feature that would also show up in release notes. Perhaps my imagination
>> isn't sufficient to come up with a scenario where DocImpact is valid,
>> but we wouldn't have content in one of those sections.
> 
> I can think of plenty. What about where a command is changed slightly? Or an 
> output is formatted differently? Or where flags have been removed, or default 
> values changed, or ….

Nearly all of those changes have been triggering release notes at this
point. They are all changes the user needs to adapt to because they
potentially impact compatibility.

> 
> Bugs and relnotes are two very different things.
> 
> L
> 
> Lana Brindley
> writer:speaker:blogger
> http://lanabrindley.com
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Glance] Nova + Glance_v2 = Love

2016-01-08 Thread Daniel P. Berrange

On Fri, Jan 08, 2016 at 10:01:45AM -0430, Flavio Percoco wrote:
> On 29/12/15 07:41 -0600, Sam Matzek wrote:
> >On Thu, Dec 24, 2015 at 7:49 AM, Mikhail Fedosin  
> >wrote:
> >>Hello, it's another topic about glance v2 adoption in Nova, but it's
> >>different from the others. I want to declare that there is a set of commits,
> >>that make Nova version agnostic and allow to work with both glance apis. The
> >>idea of the solution is to determine the current api version in the
> >>beginning and make appropriate requests after.
> >>(https://review.openstack.org/#/c/228578/,
> >>https://review.openstack.org/#/c/238309/,
> >>https://review.openstack.org/#/c/259097/)
> >>
> >>Indeed, it requires some additional (and painful) work, but now all tempest
> >>tests pass in Jenkins.
> >>
> >>Note: this thread is not about xenplugin, there is another topic, called
> >>'Xenplugin + Glance_v2 = Hate'
> >>
> >>Here the main issues we faced and how we've solved them:
> >>
> >>1. "changes-since" filter for image-list is not supported in v2 api. Steve
> >>Lewis made a great job and implemented a set of filters with comparison
> >>operators https://review.openstack.org/#/c/197388/. Filtering by
> >>'changes-since' is completely similar to 'gte:updated_at'.
> >>
> >>2. Filtering by custom properties must have prefix 'property-'. In v2 it's
> >>not required.
> >>
> >>3. V2 states that all custom properties must be image attributes, but Nova
> >>assumes that they are included in 'properties' dictionary. It's fixed with
> >>glanceclient's method 'is_base_property(prop_name)', that returns False in
> >>case of custom property.
> >>
> >>4. is_public=True/False is visibility="public"/"private" respectively.
> >>
> >>5. Deleting custom image properties in Nova is performed with 'purge_props'
> >>flag. If it's set to True, then all prop names, that are not included in
> >>updated data, will be removed. In case of v2 we have to explicitly specify
> >>prop names in the list param 'remove_props'. To implement this behaviour, if
> >>'purge_props' is set, we make additional 'get' request to determine the
> >>existing properties and put in 'remove_prop' list only those, that are not
> >>in updated_data.
> >>
> >>6. My favourite:
> >>There is an ability to activate an empty image by just providing 'size = 0':
> >>https://review.openstack.org/#/c/9715/, in this case image will be a
> >>collection of metadata. Glance v2 doesn't support this "feature" and that's
> >>why we have to implement a very dirty workaround:
> >>* v2 requires, that disk_format and container-format must be set before
> >>the activation. if these params are not provided to 'create' method then we
> >>hardcode it to 'qcow2' and 'bare'.
> >>* we call 'upload' method with empty data (data = '') to activate image.
> >>Me (and the rest glance team) think that this image activation with
> >>zero-size is inconsistent and we won't implement it in v2. So, I'm going to
> >>ask if Nova really needs this feature and maybe it's possible to make it
> >>work without it.
> >
> >Nova uses this functionality when it creates snapshots of volume
> >backed instances, that is, instances that only have Cinder volumes
> >attached and do not have an ephemeral disk.
> >In this case Nova API creates Cinder snapshots for the Cinder volumes
> >and builds the block_device_mapping property with block devices that
> >reference the Cinder snapshots.  Nova activates this image with size=0
> >because this image does not have a disk and simply refers to the
> >collection of Cinder snapshots that collectively comprise the binary
> >image.  Callers of Glance outside of Nova may also use the APIs to
> >create "block device mapping" images as well that contain references
> >to Cinder volumes to attach, blank volumes to create, snapshots to
> >create volumes from, etc during the server creation.  Not having the
> >ability to create these images with V2 is a loss of function.
> 
> I disagree. Being able to activate empty images breaks the consistency
> of Glances API and I don't think it should be considered a feature but
> a bug. An active image is an image that can be used to *boot* a VM. If
> the image is empty, you simply can't do that.

NB if an empty image is associated with a kernel/initrd image then it
could conceptually still be bootable, as the VM could run entirely from
content contained in the initrd, or the initrd might have activated
some network accessed root device. Whether this works in practice
with glance empty images though I don't know. Just from a conceptual
POV, images don't need to contain any content at all.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

Re: [openstack-dev] [nova][cell] Can nova do live migration cross cells?

Thank you very much. .

And Andrew Laski  has answer my second question.

2016-01-08 20:33 GMT+08:00 John Garbutt :

> On 8 January 2016 at 08:17, 少合冯  wrote:
> > Hi, all
> >
> > Now I'm working on the migrations list/show API.
> > The new api defines  migrations is a sub-collection of an instance.
> > GET   v2.1/tenant_id/servers/server-id/migrations/migration-id
> >
> > I need to support  the new cell API to get a migration of a specified
> > instance.
> > get_migration_by_instance_and_id
> >
> > I read up the nova cell doc.
> > http://docs.openstack.org/developer/nova/cells.html
> >
> > It tells us that, the migrations table will be API-level.
> > The instances table is Cell-level.
> > So does that means that we can do live migration cross cells?
>
> In sort, you can't currently live-migrate between cells.
>
> So there is the current cells v1. When using cells v1 you will never
> be able to move between cells. In addition, when using nova-network,
> network assumptions generally mean IPs can't move between cells.
>
> In cells v2, we were talking about ways that could happen. Its still
> hard, because you would need to copy the instance record between one
> cell database to another, with the API understanding how the move is
> going. Complexity best avoided, if possible.
>
> > And also this will affect my code.
> > If the migrations is API-level, so I do not need a cell name to get the
> > migrations info from DB.
> >
> > Or I need to get the cell name by the specified instance, and then get
> the
> > migrations info by _TargetedMessage.
>
> Afraid I don't have enough context from the rest of your email to
> answer these questions.
>
> Thanks,
> John
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Adding "IPv6" bracket handling utilities in openstacklib.

2016-01-08 Thread Sofer Athlan-Guyot

Hi all,

I've got an input from a fellow worker[1].  Basically, transparently
transforming user data in puppet is opening a big can of worms.

He came up with a rather contrived example, but it's definitively worth
discussing it.

So in the vncproxy example, if the user does this:

   $vncproxy_host => 'ff02::1:ff00:1:80',
   $vncproxy_port => '80'

thinking that vncproxy_host is set to host 'ff02::1:ff00:1' with port
":80" (he forgot to add brackets) then the code will transform that to a
valid uri '[ff02::1:ff00:1:80]:80'

Without the code it would give this 'ff02::1:ff00:1:80:80' which would
fail as it lacks the brackets and is an invalid uri.

There is no way to make the difference between "wrong ipv6 + port" and
"valid ipv6", so mangling user input can lead to unexpected result.

I'm going to put the patches on WIP, as maybe, this may not be a good
idea to have user input transformed at all in puppet as all corner cases
cannot be detected.

The trade-off, of course, is user convenience.

So what do you think, parse and transform user input or not ?

[1] thanks Lukas

Sofer Athlan-Guyot  writes:

> Cody Herriges  writes:
>
> Sorry I didn't see you reply before.  Comments below.
>
>> Sofer Athlan-Guyot wrote:
>>> Hi,
>>> 
>>> There are a few places where I would like to be able to check for IPv6
>>> address and add bracket to the parameters.  I think that would be a nice
>>> addition to the puppet-openstacklib/lib/puppet/parser.
>>> 
>>> Here the interface I have in mind with the puppet-nova module:
>>> 
>>> class nova::vncproxy::common (
>>>   $vncproxy_host = undef,
>>>   $vncproxy_protocol = undef,
>>>   $vncproxy_port = undef,
>>>   $vncproxy_path = undef,
>>> ) {
>>> 
>>>   include ::nova::deps
>>> 
>>>   $vncproxy_host_real = pick(
>>> ipv6_add_bracket_maybe($vncproxy_host,
>>>$::nova::compute::vncproxy_host,
>>>$::nova::vncproxy::host,
>>>false)
>>> 
>>> 
>>> This would returns an array with the host decorated with "[]" if the
>>> value is an IPv6 address.  Ideally the function could take only one
>>> value and return it or take an array and return an array for seamless
>>> integration in the code.
>>> 
>>> WDYT?
>>> 
>>
>> I see this and it looks like that only only reason this is a problem is
>> because we've broken up all the pieces of data needed to generate a URI
>> so it becomes inappropriate to decorate the vncproxy_host variable's
>> value with "[]" because it lacks the port appended to the end.  What are
>> the ramifications of simply switching to a "$vnc_uri" variable much the
>> same that has happened with identity_uri and auth_uri, e.g.
>> https://review.openstack.org/262799. If one has to simply define the
>> entire URI, they'll be able to properly decorate the IPv6 address.
>
> Yes, that could be something to consider as well.  The difficulties with
> this approach, that I see are that it's not easy on the user (change of
> interface) and must rely on a deprecation period (code to maintain).
>
> Adding a function on the other hand is transparent for the user and it
> may be useful in other part of the code.
>
> As for this solution (adding a function) I had to lower my expectation
> to meet puppet < 4.1 reality.  There is no splat operator, and no way to
> chain functions as I wanted at the beginning.  What I did is a simpler
> function that take only one argument, the stuff to maybe transform.  The
> example above adjusted:
>
>
>   $vncproxy_host_real = ipv6_add_bracket_maybe(pick(
>   $vncproxy_host,
>   $::nova::compute::vncproxy_host,
>   $::nova::vncproxy::host,
> false))
>
> Please let me know what you think.

-- 
Sofer Athlan-Guyot

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cell] Can nova do live migration cross cells?

On 8 January 2016 at 08:17, 少合冯  wrote:
> Hi, all
>
> Now I'm working on the migrations list/show API.
> The new api defines  migrations is a sub-collection of an instance.
> GET   v2.1/tenant_id/servers/server-id/migrations/migration-id
>
> I need to support  the new cell API to get a migration of a specified
> instance.
> get_migration_by_instance_and_id
>
> I read up the nova cell doc.
> http://docs.openstack.org/developer/nova/cells.html
>
> It tells us that, the migrations table will be API-level.
> The instances table is Cell-level.
> So does that means that we can do live migration cross cells?

In sort, you can't currently live-migrate between cells.

So there is the current cells v1. When using cells v1 you will never
be able to move between cells. In addition, when using nova-network,
network assumptions generally mean IPs can't move between cells.

In cells v2, we were talking about ways that could happen. Its still
hard, because you would need to copy the instance record between one
cell database to another, with the API understanding how the move is
going. Complexity best avoided, if possible.

> And also this will affect my code.
> If the migrations is API-level, so I do not need a cell name to get the
> migrations info from DB.
>
> Or I need to get the cell name by the specified instance, and then get the
> migrations info by _TargetedMessage.

Afraid I don't have enough context from the rest of your email to
answer these questions.

Thanks,
John

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Trove][Outreachy] Add configuration groups for CouchDB

2016-01-08 Thread Victoria Martínez de la Cruz

Having passed more than a week an this task not being claimed by anybody,
Sonali will take this task.

The spec is already up for review in
https://review.openstack.org/#/c/263980/.

Thanks!

Victoria

2016-01-04 9:32 GMT-03:00 Victoria Martínez de la Cruz <
victo...@vmartinezdelacruz.com>:

> Retaking this after the holidays.
>
> If someone is working on this, please let us know. Otherwise, Sonali will
> submit the spec for review by the end of this week.
>
> Best,
>
> Victoria
>
> 2015-12-22 13:05 GMT-03:00 Amrith Kumar :
>
>> Victoria,
>>
>>
>>
>> As several people are now on holiday (and likely through the end of the
>> year), you may not receive an answer either positive or otherwise before
>> the New Year.
>>
>>
>>
>> Please take that into consideration in your project choice.
>>
>>
>>
>> -amrith
>>
>>
>>
>> *From:* Victoria Martínez de la Cruz [mailto:
>> victo...@vmartinezdelacruz.com]
>> *Sent:* Tuesday, December 22, 2015 9:21 AM
>> *To:* OpenStack Development Mailing List <
>> openstack-dev@lists.openstack.org>
>> *Subject:* [openstack-dev] [Trove][Outreachy] Add configuration groups
>> for CouchDB
>>
>>
>>
>> Hi all,
>>
>>
>>
>> For those who are not aware, we have 7 Outreachy interns [0] working on
>> different projects on OpenStack since last Dec 7th.
>>
>>
>>
>> Sonali (sonali5) will be working with us on Trove and we have been trying
>> to locate a task for her to work on during her internship.
>>
>>
>>
>> Before the internship was announced, one month before Tokyo summit, I
>> proposed "Adding CRUD for CouchDB" as her internship idea [1], but it was
>> not notified as it should so other contributors took started working on
>> this task.
>>
>>
>>
>> To avoid this to happen again, I decided to send this email and ask if
>> someone if working on "Add configuration groups for CouchDB".
>>
>>
>>
>> If its ok for Trove team, Sonali will be working on that.
>>
>>
>>
>> Please, let me know if this is not the case and we will look for a
>> different task for her.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Victoria
>>
>>
>>
>> [0] https://www.gnome.org/outreachy/
>>
>> [1] https://wiki.openstack.org/wiki/Internship_ideas
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] centos6.5 libvirt 1.2.14 virsh hang

2016-01-08 Thread 王李明

 

hi all：

 my environment is centos6.5 

libvirt version is 1.2.14-1

qemu version is 1.7.0-1

 

I use openstack create a  windows guest 

about two days later I run virsh list but the the process is hang 

virsh list can not return any thing 

it hang  

 

 

then I run  /etc/init.d/libvirtd restart

it show libivrtd stop faild and start failed

 

and the libvirtd process status is running and cpu is always more than 100%

 

i have used the gdb and find the gdb can not print any process stack 

 

 

I run /etc/init.d/libvirtd status it show libvired dead but subsys locked 

 

who can help me ?

thans

 

 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Glance] Xenplugin + Glance_v2 = Hate

On 4 January 2016 at 12:55, Sean Dague  wrote:
> On 12/24/2015 08:51 AM, Mikhail Fedosin wrote:
>> Hello! As you may know there is a big initiative to adopt glance v2 api
>> in Nova and the important part is making related changes in xenglugin.
>> Unfortunately xenplugin doesn't use neither nova.image api nor
>> glanceclient. Instead of this it has own http client implementation and
>> bunch of hardcoded 'v1' urls (example,
>> https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance#L130).
>> It leads to the fact, that it will be really hard to switch it on v2 api
>> right.
>>
>> Personally I see 2 solutions:
>> 1. Make xenplugin to adopt nova.image api, which will make it
>> version-agnostic. Here it's not easy to implement and we won't be able
>> to keep backward compatibility with the existing lowlevel code.
>> 2. Also hardcode v2 urls on par with v1 and do the same thing as in
>> nova.image - to determine current glance api version before request and
>> then use appropriate urls in methods.
>>
>> IMHO, the second solution is more preferable, because I understand how
>> to do it and the v1/v2 compatibility will be 100%. It guarantees that we
>> won't break any of existing deployments and it will allow to merge
>> glance_v2 code in nova.image much quicker.
>>
>> All opinions and advice will be very helpful. Thanks in advance!
>
> So it is a little weird, but it's not as terrible as it could be.
>
> The glance plugin is effectively an RPC interface from Nova proper. I
> just had to bump it to make glance pass around urls instead of tuples -
> https://review.openstack.org/#/c/254785/6/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance
>
> I would just add another set of calls in that glance plugin that use v2
> instead of v1. Then let nova decide it's going to call v2 vs. v1 from
> it's side.

+1
Thats a great summary of the approach we need here.

We should try get some XenServer using folks to help out with this one.
Bob, lets bring this up at the next Xen meeting?

> As a bonus, having *some* testing added to this path would be great as
> well. The xenserver glance plugins have no tests right now, and doing
> something like v2 vs. v1 would be good to get something basic tested there.

I have prototype unit tests for the plugins, but I have not had time
to finish it.
There are some ideas in here on how it could work:
https://review.openstack.org/#/c/128886

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [celery][taskflow] Reg. celery and task-flow

2016-01-08 Thread ESWAR RAO

Hi All,

Please let me know whether celery is replacement for taskflow.

As per my understanding, task-flow can break jobs into tasks and execute
them.

>From celery wiki, it also does almost similar behaviour.

I guess in most of openstack components taskflow is widely used.
Any places where its being replaced with celery ??

Celery: https://wiki.openstack.org/wiki/Celery
Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
TaskFlow: https://wiki.openstack.org/wiki/TaskFlow

Thanks
Eswar
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][stackalytics] some projects not visible in stackalytics now

2016-01-08 Thread Zhipeng Huang

Hi Thierry,

Does this mean big tent projects = TC approved ? My understanding was that
for those project migrated from stackforge to OpenStack infra now, is
OpenStack big tent projects but not TC-approved OpenStack projects.

On Fri, Jan 8, 2016 at 4:57 PM, Thierry Carrez 
wrote:

> Thierry Carrez wrote:
>
>> [...]
>> All the projects you are mentioning are hosted on the OpenStack
>> infrastructure (in the space formally known as Stackforge) but are not
>> official OpenStack projects.
>>
>
> I meant "formerly" :)
>
>
> --
> Thierry Carrez (ttx)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

-- 
Zhipeng (Howard) Huang

Standard Engineer
IT Standard & Patent/IT Prooduct Line
Huawei Technologies Co,. Ltd
Email: huangzhip...@huawei.com
Office: Huawei Industrial Base, Longgang, Shenzhen

(Previous)
Research Assistant
Mobile Ad-Hoc Network Lab, Calit2
University of California, Irvine
Email: zhipe...@uci.edu
Office: Calit2 Building Room 2402

OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RDO, packstack, Keypair creation is failing


Hey Thales,

This is not a support mailing list but a development one. To have
support on RDO specific matters, I recommend you to join the RDO
mailing list[0]. You can also join the openstack one[1].

[0] http://www.redhat.com/mailman/listinfo/rdo-list
[1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Cheers,
Flavio

On 07/01/16 23:37 +, Thales wrote:

Hello,

I'm trying to learn OpenStack via RDO, so I downloaded packstack.  I went to
this link to start on RDO
https://www.rdoproject.org/install/quickstart/

I set up a fixed IP in CentOS 7.CentOS 7 is a guest OS.   I'm using
Virtualbox as my virtual machine, and Windows 10 is my host OS.  I'm using a
Bridged adpater for Virtual Box.

I downloaded and installed RDO via packstack.

I then went here:
https://www.rdoproject.org/install/running-an-instance/


I went there to learn how to run an instance, and on step three on that page,
"Create or Import a Pair", I  received an error.The error occurred when I
tried to "create" the key.  It error was "Keypair data is invalid: failed to
generate fingerprint (HTTP 400)". I also tried to import a key, by
generating one on the command line.  However, in both cases it failed.  I got
some guidance from this video
,https://www.youtube.com/watch?v=GYTctLuPbOs, but that didn't do the trick for
me, either.

 I have been looking around for similar problems on the web, and I found a
couple, but none of the solutions worked for me.

 Does anyone have any idea what his could be?   I'm really just learning now,
so I'm sure it's got to be something rudimentary.

 Thanks for any help!

 ...John


 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cell] Can nova do live migration cross cells?

2016-01-08 21:53 GMT+08:00 Andrew Laski :

> On 01/08/16 at 12:33pm, John Garbutt wrote:
>
>> On 8 January 2016 at 08:17, 少合冯  wrote:
>>
>>> Hi, all
>>>
>>> Now I'm working on the migrations list/show API.
>>> The new api defines  migrations is a sub-collection of an instance.
>>> GET   v2.1/tenant_id/servers/server-id/migrations/migration-id
>>>
>>> I need to support  the new cell API to get a migration of a specified
>>> instance.
>>> get_migration_by_instance_and_id
>>>
>>> I read up the nova cell doc.
>>> http://docs.openstack.org/developer/nova/cells.html
>>>
>>> It tells us that, the migrations table will be API-level.
>>> The instances table is Cell-level.
>>> So does that means that we can do live migration cross cells?
>>>
>>
>> In sort, you can't currently live-migrate between cells.
>>
>> So there is the current cells v1. When using cells v1 you will never
>> be able to move between cells. In addition, when using nova-network,
>> network assumptions generally mean IPs can't move between cells.
>>
>
> Assuming a global network the limitation is just within Nova.  There is no
> mechanism for moving instance data from one cell database to another.  And
> it's not something that will be added since v1 is in a freeze.
>
>
>> In cells v2, we were talking about ways that could happen. Its still
>> hard, because you would need to copy the instance record between one
>> cell database to another, with the API understanding how the move is
>> going. Complexity best avoided, if possible.
>>
>
> Cells v2 makes some architecture choices that will make it easier to
> accomplish this.  But it's not necessary to have for cells v2 so it's not
> likely to be in place initially.
>
>
>> And also this will affect my code.
>>> If the migrations is API-level, so I do not need a cell name to get the
>>> migrations info from DB.
>>>
>>> Or I need to get the cell name by the specified instance, and then get
>>> the
>>> migrations info by _TargetedMessage.
>>>
>>
>> Afraid I don't have enough context from the rest of your email to
>> answer these questions.
>>
>
> I'm not entirely clear on the question either, but if the migration is in
> the api level database then you do not need a cell name to get the
> migration from the db.  The cell name is only used to route a request to a
> cell database.
>
> Thanks very much.
Sorry for my question is not clear.
But you  have  gave me the answer.

>
>> Thanks,
>> John
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cell] Can nova do live migration cross cells?

still a question in line.
Thanks.

2016-01-08 21:53 GMT+08:00 Andrew Laski :

> On 01/08/16 at 12:33pm, John Garbutt wrote:
>
>> On 8 January 2016 at 08:17, 少合冯  wrote:
>>
>>> Hi, all
>>>
>>> Now I'm working on the migrations list/show API.
>>> The new api defines  migrations is a sub-collection of an instance.
>>> GET   v2.1/tenant_id/servers/server-id/migrations/migration-id
>>>
>>> I need to support  the new cell API to get a migration of a specified
>>> instance.
>>> get_migration_by_instance_and_id
>>>
>>> I read up the nova cell doc.
>>> http://docs.openstack.org/developer/nova/cells.html
>>>
>>> It tells us that, the migrations table will be API-level.
>>> The instances table is Cell-level.
>>> So does that means that we can do live migration cross cells?
>>>
>>
>> In sort, you can't currently live-migrate between cells.
>>
>> So there is the current cells v1. When using cells v1 you will never
>> be able to move between cells. In addition, when using nova-network,
>> network assumptions generally mean IPs can't move between cells.
>>
>
> Assuming a global network the limitation is just within Nova.  There is no
> mechanism for moving instance data from one cell database to another.  And
> it's not something that will be added since v1 is in a freeze.
>
>
>> In cells v2, we were talking about ways that could happen. Its still
>> hard, because you would need to copy the instance record between one
>> cell database to another, with the API understanding how the move is
>> going. Complexity best avoided, if possible.
>>
>
> Cells v2 makes some architecture choices that will make it easier to
> accomplish this.  But it's not necessary to have for cells v2 so it's not
> likely to be in place initially.
>
>
>> And also this will affect my code.
>>> If the migrations is API-level, so I do not need a cell name to get the
>>> migrations info from DB.
>>>
>>> Or I need to get the cell name by the specified instance, and then get
>>> the
>>> migrations info by _TargetedMessage.
>>>
>>
>> Afraid I don't have enough context from the rest of your email to
>> answer these questions.
>>
>
> I'm not entirely clear on the question either, but if the migration is in
> the api level database then you do not need a cell name to get the
> migration from the db.  The cell name is only used to route a request to a
> cell database.
>

still puzzle the current nova code.
https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1668
https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1330

why get_migrations in the above link still need to route a request  to a
cell database, or broadcast to all cell database?


>
>> Thanks,
>> John
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ec2-api] Removing Nova's In tree ec2 API code in favor of ec2-api project

2016-01-08 Thread Tim Bell

> -Original Message-
> From: Davanum Srinivas [mailto:dava...@gmail.com]
> Sent: 07 January 2016 23:04
> To: OpenStack Development Mailing List (not for usage questions)
> 
> Subject: [openstack-dev] [nova][ec2-api] Removing Nova's In tree ec2 API
> code in favor of ec2-api project
> 
> Dear Nova cores,
> 
> Sorry this did not make into the Nova weekly meeting agenda yesterday.
> 
> Is it time for dropping the EC2/ObjectStore REST API Since we've told
folks
> since Kilo that we will be removing this code and we dropped the entries
api-
> paste.ini in Liberty?
> https://review.openstack.org/#/c/263368/
> 

For those who would like to test, Marcos has been working on completing the
other necessary parts for EC2 support which were previously available with
Nova:

- RDO RPM packages with the CentOS cloud SIG
- Puppet modules 

We're due to start testing once these are upstream.

Tim

> Thanks,
> Dims
> 
> 
> --
> Davanum Srinivas :: https://twitter.com/dims
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ec2-api] Removing Nova's In tree ec2 API code in favor of ec2-api project


Alexandre Levine wrote:

The ec2-api project is in gating and totally functional. We'll apply for
it to become OpenStack project very shortly. Next week in fact.


Great to hear! Thanks Alex.

--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposal for new metrics-threshold-filter

On 7 January 2016 at 18:49, SURO  wrote:
>
>> Hi all,
>>
>> I have proposed a nova-spec[1] for a new scheduling filter based on
>> metrics thresholds. This is slightly different than weighted metrics filter.
>> The rationale and use-case is explained in detail in the spec[1].
>>
>> The implementation is also ready for review[2]. The effort is tracked with
>> a blueprint[3].
>>
>> I request the community to review them and provide valuable feedback.
>>
>>
>> [1] - https://review.openstack.org/#/c/257596/
>> [2] - https://review.openstack.org/#/c/254423/
>> [3] - https://blueprints.launchpad.net/nova/+spec/metrics-threshold-filter

This is very related to the ideas I have written up here:
https://review.openstack.org/#/c/256323/

Please note we have not been accepting new specs for the Mitaka
release, mostly because we have feature freeze in a few weeks.
For more information please see:
http://docs.openstack.org/releases/schedules/mitaka.html#m-nova-bp-freeze
http://docs.openstack.org/developer/nova/process.html#how-do-i-get-my-code-merged

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cell] Can nova do live migration cross cells?

2016-01-08 Thread Andrew Laski


On 01/08/16 at 12:33pm, John Garbutt wrote:

On 8 January 2016 at 08:17, 少合冯  wrote:

Hi, all

Now I'm working on the migrations list/show API.
The new api defines  migrations is a sub-collection of an instance.
GET   v2.1/tenant_id/servers/server-id/migrations/migration-id

I need to support  the new cell API to get a migration of a specified
instance.
get_migration_by_instance_and_id

I read up the nova cell doc.
http://docs.openstack.org/developer/nova/cells.html

It tells us that, the migrations table will be API-level.
The instances table is Cell-level.
So does that means that we can do live migration cross cells?


In sort, you can't currently live-migrate between cells.

So there is the current cells v1. When using cells v1 you will never
be able to move between cells. In addition, when using nova-network,
network assumptions generally mean IPs can't move between cells.


Assuming a global network the limitation is just within Nova.  There is 
no mechanism for moving instance data from one cell database to another.  
And it's not something that will be added since v1 is in a freeze.




In cells v2, we were talking about ways that could happen. Its still
hard, because you would need to copy the instance record between one
cell database to another, with the API understanding how the move is
going. Complexity best avoided, if possible.


Cells v2 makes some architecture choices that will make it easier to 
accomplish this.  But it's not necessary to have for cells v2 so it's 
not likely to be in place initially.





And also this will affect my code.
If the migrations is API-level, so I do not need a cell name to get the
migrations info from DB.

Or I need to get the cell name by the specified instance, and then get the
migrations info by _TargetedMessage.


Afraid I don't have enough context from the rest of your email to
answer these questions.


I'm not entirely clear on the question either, but if the migration is 
in the api level database then you do not need a cell name to get the 
migration from the db.  The cell name is only used to route a request to 
a cell database.




Thanks,
John

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova][cells] Should flavors in the API DB for cells v2 be soft-deletable?

2016-01-08 Thread Andrew Laski


On 01/08/16 at 12:43pm, John Garbutt wrote:

On 7 January 2016 at 19:59, Matt Riedemann  wrote:

There is a cells v2 change up for review [1] which creates the flavor tables
in the API DB.

I noted that those table definitions include the soft-delete columns
(deleted and deleted_at), which at the YVR summit and in other threads [2]
we've said we're not doing anymore for new tables.

The point raised to keep them soft-deletable is that the flavor API allows
showing soft-deleted flavors if you know the id [3]. And you can get the
flavor id for an instance (we always store the flavor info that was used to
boot the instance).

The question is, can we break that wrinkle in the API by not allow
soft-deleting the flavor in the API DB?


On balance, I think this is OK.


There's an alternate approach to this that we can take which I'll 
outline below.  But it should meet the needs of anyone currently looking 
up deleted flavors.






Note that in the normal nova DB, if the admin archives/purges the
instance_types table, the wrinkle is already broken because the soft-deleted
flavor is now hard-deleted, but that's maybe not a great justification for
consciously removing this support in the API DB.


This is what won me over. All be it, reluctantly.


I think this is good justification for removing the ability because 
currently it is essentially undefined behavior.  Requesting a deleted 
flavor may or may not work and either response is correct.





If we made the flavor soft-deletable in the API DB, one issue is we don't
have an in-tree entrypoint for cleaning this up (there are no archive/purge
CLIs for the API DB). We could always add that, but it's not there right
now.

Another thing that came up in the cells meeting this week is that if we
didn't make the flavor soft-deletable, we could still show the flavor
information for a given instance via the server GET API. However, that would
be a microversion change to show the full flavor information for the server
rather than just the flavor id.


This is really only possible because we now store flavor info inside
every instance object.
I think before that, deleting the flavor would make some instance
operations fail.


Because we now store flavor info with each instance it opens up the 
following possibility:


Currently the flavor portion of an instance show request looks like 
"flavor": {"id": "8", "links": [{"href": 
"https://example.com/flavors/8;, "rel": "bookmark"}]}


If that were instead changed to return 
"https://example.com/servers//flavor" as the link and we exposed the 
flavor information stored with the instance on that endpoint then we no 
longer need to expose deleted flavors.


So for an instance booted with flavor 8 https://example.com/flavors/8 
would be equivalent to https://example.com/servers//flavor except 
that the latter would be available even if flavor 8 were deleted.


Does that seem like an acceptable path for users?




Thoughts? I'm cross-posting this to -dev and the -operators list to see what
kind of user impact there would be if we didn't soft-delete flavors in the
API DB (so you couldn't look up deleted flavors in the API).


In balance, I think we should not allow soft_delete of flavors in the API DB.

In a related note, and I thinking about a backlog spec looking at a
flavor lifecycle. Thinking about early release, to production, then
phasing out of flavors. I don't think soft delete is needed for that.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Deprecating the live_migration_flag and block_migration_flag config options

2016-01-08 Thread Daniel P. Berrange

On Thu, Jan 07, 2016 at 09:07:00PM +, Mark McLoughlin wrote:
> On Thu, 2016-01-07 at 12:23 +0100, Sahid Orentino Ferdjaoui wrote:
> > On Mon, Jan 04, 2016 at 09:12:06PM +, Mark McLoughlin wrote:
> > > Hi
> > > 
> > > commit 8ecf93e[1] got me thinking - the live_migration_flag config
> > > option unnecessarily allows operators choose arbitrary behavior of the
> > > migrateToURI() libvirt call, to the extent that we allow the operator
> > > to configure a behavior that can result in data loss[1].
> > > 
> > > I see that danpb recently said something similar:
> > > 
> > >   https://review.openstack.org/171098
> > > 
> > >   "Honestly, I wish we'd just kill off  'live_migration_flag' and
> > >   'block_migration_flag' as config options. We really should not be
> > >   exposing low level libvirt API flags as admin tunable settings.
> > > 
> > >   Nova should really be in charge of picking the correct set of flags
> > >   for the current libvirt version, and the operation it needs to
> > >   perform. We might need to add other more sensible config options in
> > >   their place [..]"
> > 
> > Nova should really handle internal flags and this serie is running in
> > the right way.
> > 
> > > ...
> > 
> > >   4) Add a new config option for tunneled versus native:
> > > 
> > >    [libvirt]
> > >    live_migration_tunneled = true
> > > 
> > >  This enables the use of the VIR_MIGRATE_TUNNELLED flag. We have 
> > >      historically defaulted to tunneled mode because it requires the 
> > >      least configuration and is currently the only way to have a 
> > >      secure migration channel.
> > > 
> > >      danpb's quote above continues with:
> > > 
> > >        "perhaps a "live_migration_secure_channel" to indicate that 
> > >         migration must use encryption, which would imply use of 
> > >         TUNNELLED flag"
> > > 
> > >      So we need to discuss whether the config option should express the
> > >      choice of tunneled vs native, or whether it should express another
> > >      choice which implies tunneled vs native.
> > > 
> > >        https://review.openstack.org/263434
> > 
> > We probably have to consider that operator does not know much about
> > internal libvirt flags, so options we are exposing for him should
> > reflect benefice of using them. I commented on your review we should
> > at least explain benefice of using this option whatever the name is.
> 
> As predicted, plenty of discussion on this point in the review :)
> 
> You're right that we don't give the operator any guidance in the help
> message about how to choose true or false for this:
> 
>   Whether to use tunneled migration, where migration data is 
>   transported over the libvirtd connection. If True,
>   we use the VIR_MIGRATE_TUNNELLED migration flag
> 
> libvirt's own docs on this are here:
> 
>   https://libvirt.org/migration.html#transport
> 
> which emphasizes:
> 
>   - the data copies involved in tunneling
>   - the extra configuration steps required for native
>   - the encryption support you get when tunneling
> 
> The discussions I've seen on this topic wrt Nova have revolved around:
> 
>   - that tunneling allows for an encrypted transport[1]
>   - that qemu's NBD based drive-mirror block migration isn't supported
>     using tunneled mode, and that danpb is working on fixing this
>     limitation in libvirt
>   - "selective" block migration[2] won't work with the fallback qemu
>     block migration support, and so won't currently work in tunneled
>     mode

I'm not working on fixing it, but IIRC some other dev had proposed
patches.

> 
> So, the advise to operators would be:
> 
>   - You may want to choose tunneled=False for improved block migration 
>     capabilities, but this limitation will go away in future.
>   - You may want to choose tunneled=False if you wish to trade and
>     encrypted transport for a (potentially negligible) performance
>     improvement.
> 
> Does that make sense?
> 
> As for how to name the option, and as I said in the review, I think it
> makes sense to be straightforward here and make it clearly about
> choosing to disable libvirt's tunneled transport.
> 
> If we name it any other way, I think our explanation for operators will
> immediately jump to explaining (a) that it influences the TUNNELLED
> flag, and (b) the differences between the tunneled and native
> transports. So, if we're going to have to talk about tunneled versus
> native, why obscure that detail?

Ultimately we need to recognise that libvirt's tunnelled mode was
added as a hack, to work around fact that QEMU lacked any kind of
native security capabilities & didn't appear likely to ever get
them at that time.  As well as not working with modern NBD based
block device encryption, it really sucks for performance because
it introduces many extra data copies. So it is going to be quite
poor for large VMs with heavy rate of data dirtying.

The only long term relative "benefit" of tunnelled mode is that

Re: [openstack-dev] [Openstack-operators] [nova][cells] Should flavors in the API DB for cells v2 be soft-deletable?

2016-01-08 Thread Sylvain Bauza




Le 08/01/2016 15:10, Andrew Laski a écrit :

On 01/08/16 at 12:43pm, John Garbutt wrote:
On 7 January 2016 at 19:59, Matt Riedemann 
 wrote:
There is a cells v2 change up for review [1] which creates the 
flavor tables

in the API DB.

I noted that those table definitions include the soft-delete columns
(deleted and deleted_at), which at the YVR summit and in other 
threads [2]

we've said we're not doing anymore for new tables.

The point raised to keep them soft-deletable is that the flavor API 
allows
showing soft-deleted flavors if you know the id [3]. And you can get 
the
flavor id for an instance (we always store the flavor info that was 
used to

boot the instance).

The question is, can we break that wrinkle in the API by not allow
soft-deleting the flavor in the API DB?


On balance, I think this is OK.


There's an alternate approach to this that we can take which I'll 
outline below.  But it should meet the needs of anyone currently 
looking up deleted flavors.






Note that in the normal nova DB, if the admin archives/purges the
instance_types table, the wrinkle is already broken because the 
soft-deleted
flavor is now hard-deleted, but that's maybe not a great 
justification for

consciously removing this support in the API DB.


This is what won me over. All be it, reluctantly.


I think this is good justification for removing the ability because 
currently it is essentially undefined behavior. Requesting a deleted 
flavor may or may not work and either response is correct.




If we made the flavor soft-deletable in the API DB, one issue is we 
don't
have an in-tree entrypoint for cleaning this up (there are no 
archive/purge
CLIs for the API DB). We could always add that, but it's not there 
right

now.

Another thing that came up in the cells meeting this week is that if we
didn't make the flavor soft-deletable, we could still show the flavor
information for a given instance via the server GET API. However, 
that would
be a microversion change to show the full flavor information for the 
server

rather than just the flavor id.


This is really only possible because we now store flavor info inside
every instance object.
I think before that, deleting the flavor would make some instance
operations fail.


Because we now store flavor info with each instance it opens up the 
following possibility:


Currently the flavor portion of an instance show request looks like 
"flavor": {"id": "8", "links": [{"href": 
"https://example.com/flavors/8;, "rel": "bookmark"}]}


If that were instead changed to return 
"https://example.com/servers//flavor" as the link and we exposed 
the flavor information stored with the instance on that endpoint then 
we no longer need to expose deleted flavors.


So for an instance booted with flavor 8 https://example.com/flavors/8 
would be equivalent to https://example.com/servers//flavor 
except that the latter would be available even if flavor 8 were deleted.


Does that seem like an acceptable path for users?



Yes, that's something we discussed on IRC yesterday which I think would 
be very good. It means a microversion but it means that operators would 
be very happy because if a flavor is removed, the flavor URI would still 
be working.


+1000 to that.

-Sylvain





Thoughts? I'm cross-posting this to -dev and the -operators list to 
see what
kind of user impact there would be if we didn't soft-delete flavors 
in the

API DB (so you couldn't look up deleted flavors in the API).


In balance, I think we should not allow soft_delete of flavors in the 
API DB.


In a related note, and I thinking about a backlog spec looking at a
flavor lifecycle. Thinking about early release, to production, then
phasing out of flavors. I don't think soft delete is needed for that.

Thanks,
johnthetubaguy

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Heads up for decomposed plugin break

2016-01-08 Thread Henry Gessau

Gary Kotton  wrote:
> commit 
> https://github.com/openstack/neutron/commit/5d53dfb8d64186b5b1d2f356fbff8f222e15d1b2
>  may
> break the decomposed plugins that make use of the method 
> _get_tenant_id_for_create

Note that this is a private method. Plugins should avoid using private
methods. Unfortunately during the decomposition efforts not enough effort was
expended on scrubbing private methods from the plugins. This should be
something that plugin maintainers continue to work on and keep an eye out for.

> It would have been nice if there was a deprecation warning.

I don't think we should have deprecation warnings for private properties.

> Options:
> 
>  1. Decomposed plugins fix this
>  2. Revert the patch and find a solution that does not break the plugins
> 
> If we do go for #1 then we can start to stop enforcing the usage of the
> deprecation warnings…

We will continue to use deprecation warnings for changes to non-private
methods affecting plugins.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][osdk] PrettyTable needs a home in OpenStack