date:20131119

On Mon, Nov 18, 2013 at 07:10:53PM -0500, Krishna Raman wrote:
> We should probably meet on IRC or conf. call and discuss the suggested 
> architecture, open questions and REST API.
> 
> I have set up a doodle poll at http://doodle.com/w7y5qcdvq9i36757 to gather a 
> times when we can meet.
> Please sign up or let me know if none of the time slots works for you.

The time slots are pretty west coast centric there. Only the 8am/9am slots
work for people in Europe really.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Reg : Security groups implementation using openflows in quantum ovs plugin

2013-11-19 Thread Kanthi P

Hi All,

Thanks for the response!
Amir,Mike: Is your implementation being done according to ML2 plugin

Regards,
Kanthi


On Tue, Nov 19, 2013 at 1:43 AM, Mike Wilson  wrote:

> Hi Kanthi,
>
> Just to reiterate what Kyle said, we do have an internal implementation
> using flows that looks very similar to security groups. Jun Park was the
> guy that wrote this and is looking to get it upstreamed. I think he'll be
> back in the office late next week. I'll point him to this thread when he's
> back.
>
> -Mike
>
>
> On Mon, Nov 18, 2013 at 3:39 PM, Kyle Mestery (kmestery) <
> kmest...@cisco.com> wrote:
>
>> On Nov 18, 2013, at 4:26 PM, Kanthi P  wrote:
>> > Hi All,
>> >
>> > We are planning to implement quantum security groups using openflows
>> for ovs plugin instead of iptables which is the case now.
>> >
>> > Doing so we can avoid the extra linux bridge which is connected between
>> the vnet device and the ovs bridge, which is given as a work around since
>> ovs bridge is not compatible with iptables.
>> >
>> > We are planning to create a blueprint and work on it. Could you please
>> share your views on this
>> >
>> Hi Kanthi:
>>
>> Overall, this idea is interesting and removing those extra bridges would
>> certainly be nice. Some people at Bluehost gave a talk at the Summit [1] in
>> which they explained they have done something similar, you may want to
>> reach out to them since they have code for this internally already.
>>
>> The OVS plugin is in feature freeze during Icehouse, and will be
>> deprecated in favor of ML2 [2] at the end of Icehouse. I would advise you
>> to retarget your work at ML2 when running with the OVS agent instead. The
>> Neutron team will not accept new features into the OVS plugin anymore.
>>
>> Thanks,
>> Kyle
>>
>> [1]
>> http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/towards-truly-open-and-commoditized-software-defined-networks-in-openstack
>> [2] https://wiki.openstack.org/wiki/Neutron/ML2
>>
>> > Thanks,
>> > Kanthi
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Thierry Carrez

Russell Bryant wrote:
> My view of the outcome of the session was not "it *will* be a new
> service".  Instead, it was, "we *think* it should be a new service, but
> let's do some more investigation to decide for sure".
> 
> The action item from the session was to go off and come up with a
> proposal for what a new service would look like.  In particular, we
> needed a proposal for what the API would look like.  With that in hand,
> we need to come back and ask the question again of whether a new service
> is the right answer.

+1

I can see how a separate service can be a good way to avoid making Nova
support container-specific use cases and make it even bigger than it is.
However if you end up duplicating most of the code, I'm not sure there
would be any net gain.

I'm not talking about the base service infrastructure and the scheduler
(which are well-known duplication already) but more around specific nova
features (metadata/config drive, networking, image handling, etc.) that
would apply to VMs and containers alike.

So it would be great to come out of this first round of analysis with a
plan detailing how different (and how similar) from nova that new
service would be, and how much code duplication is to be expected.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

On Tue, Nov 19, 2013 at 10:57:51AM +0100, Thierry Carrez wrote:
> Russell Bryant wrote:
> > My view of the outcome of the session was not "it *will* be a new
> > service".  Instead, it was, "we *think* it should be a new service, but
> > let's do some more investigation to decide for sure".
> > 
> > The action item from the session was to go off and come up with a
> > proposal for what a new service would look like.  In particular, we
> > needed a proposal for what the API would look like.  With that in hand,
> > we need to come back and ask the question again of whether a new service
> > is the right answer.
> 
> +1
> 
> I can see how a separate service can be a good way to avoid making Nova
> support container-specific use cases and make it even bigger than it is.
> However if you end up duplicating most of the code, I'm not sure there
> would be any net gain.
> 
> I'm not talking about the base service infrastructure and the scheduler
> (which are well-known duplication already) but more around specific nova
> features (metadata/config drive, networking, image handling, etc.) that
> would apply to VMs and containers alike.
> 
> So it would be great to come out of this first round of analysis with a
> plan detailing how different (and how similar) from nova that new
> service would be, and how much code duplication is to be expected.

Yep, I'm far from convinced that having a separate service for
containers is going to be a net gain for the project as a whole.
It seems to me that we're likely to have an API and codebase that
is 95% the same in both cases here, with most differences just
being about the way things are initially booted/provisioned.

While containers certainly have some unique attributes, I believe
the level of differentiation is overblown. I also think the difference
is not about  containers vs VMs, but rather about OS virtualization vs
application virtualization.

It is entirely practical to do application virtualization using
either containers or VMs, as demonstrated by libvirt-sandbox which
can run individual app processes inside KVM without needing a full
OS intsall for it. There are notable security advantages to using
KVM for application virtualization since it avoids the shared kernel
single point of failure/compromise.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Static IPAddress configuration in nova.conf file

2013-11-19 Thread Balaji Patnala

Hi,

Nova-compute in Compute nodes send fanout_cast to the scheduler in
Controlle Node once every 60 seconds.  Configuration file Nova.conf in
Compute Node has to be configured with Management Network IP address and
there is no provision to configure Data Network IP address in the
configuration file. But if there is any change in the IPAddress for these
Management Network Interface and Data Network Interface, then we have to
configure them  manually in the configuration file of compute node.

We would like to create BP to address this issue of static configuration of
IPAddress for Management Network Interface and Data Network Interface of
Compute Node by providing the *interface names *in the nova.conf file.

So that any change in the ipaddress for these interfaces will be updated
dynamically in the fanout_cast  message to the Controller and update the DB.

We came to know that the current deployments are using some scripts to
handle this static ipaddress configuration in nova.conf file.

Any comments/suggestions will be useful.

Regards,

Balaji.P
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][heat][[keystone] RFC: introducing "request identification"

2013-11-19 Thread haruka tanizawa

Hi stackers!!

I'd like to ask for your opinions about my idea of identifying request.

Challenges
==

We have no way to know the final result of an API request.
Indeed we can continuously get the status of allocated resources,
but this is just resource status, not request status.

It doesn't matter so much for manual operations.
But it does for automated clients like heat.
We need request-oriented-status and it should be disclosed to clients.

Literally, we need to address two items for it.
 1. how to identify request from clients
 2. how to disclose status of request to clients

Note that this email includes only 1 for initiating the discussion.
Also, bp:instance-tasks-api[0] should include both two items above.

Proposal: introducing "request identification"
=

I'd like to introduce "request identification", which is disclosed to
clients.
There are two characteristics:

 - "request identification" is unique ID for each request
   so that we can identify tell a specific request from others.
 - "request identification" is available for clients
   so that we can enable any after-request-operations like check, retry[1]
or cancel[2].
   (of course we need to add more logic for check/retry/cancel,
but I'm pretty sure that indentifying request is the starting point for
these features)

In my understandings, main objections should be 'who should generate and
maintain such IDs?'.

How to implement "request identification"
=

There are several options at least. (See also recent discussion at
openstack-dev[3])

1. Enable user to provide his/her own "request identification" within API
request.
   This should be the simplest way. But providing id from outside looks
hard to be accepted.
   For example, Idempotency for OpenStack API[4].
   Or instance-tasks-api enable to user to put his/her own "request
identification".

2. Use correlation_id in oslo as "request identification".
   correlation_id looks similar concept of "request indentification",
   but correlation_id in nova was already rejected[5].

3. Enable keystone to generate "request identification" (we can call it
'request-token', for example).
   Before sending actual API request to nova-api, client sends a request to
keystone to get 'request-token'.
   Then client sends an actual API request with 'request-token'.
   Nova-api will consult it to keystone whether it was really generated.
   Sounds like a auth-token generated by keystone, differences are:
 [lifecycle] auth-token is used for multiple API requests before it
expires,
'request-token' is used for only single API request.
 [reusing] if the same 'request-token' was specified twice or more
times,
nova-api simply returns 20x (works like client token in AWS[6]).
Keystone needs to maintain 'request-tokens' until they expire.
   For backward compatibility, actual API request without 'request-token'
should work as before.
   We can consider several options for uniqueness of 'request-token':
 uuid, any string with uniqueness per tennant, etc.

IMO, since adding new implementation to Keystone is a little bit hard work,
so implement of 1 is reasonable for me, just idea.

Any comments will be appreciated.

Sincerely, Haruka Tanizawa

[0] https://blueprints.launchpad.net/nova/+spec/instance-tasks-api
[1] https://wiki.openstack.org/wiki/Support-retry-with-idempotency
[2] https://blueprints.launchpad.net/nova/+spec/cancel-swap-volume
[3]
http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg09023.html
[4] https://blueprints.launchpad.net/nova/+spec/idempotentcy-client-token
[5] https://review.openstack.org/#/c/29480/
[6]
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [savanna] bug day - Nov 19

2013-11-19 Thread Sergey Lukjanov

Reminder: today is the bug triage day on Savanna project.

I’ve prepared the wiki page for it - 
https://wiki.openstack.org/wiki/Savanna/BugTriage (based on 
https://wiki.openstack.org/wiki/BugTriage).

See you in #savanna channel.

Thank you.

Sincerely yours,
Sergey Lukjanov
Savanna Technical Lead
Mirantis Inc.

On Nov 15, 2013, at 11:34 AM, Sergey Lukjanov  wrote:

> Hi team,
> 
> we have some unmanaged bugs in LP for Savanna project and, so, we’ll have a 
> bug day to triage/cleanup them at November, 19 (Tuesday) as we discussed on 
> the last irc team meeting.
> 
> I’ll send a reminder at the start of the day.
> 
> Thanks.
> 
> Sincerely yours,
> Sergey Lukjanov
> Savanna Technical Lead
> Mirantis Inc.
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] TaskFlow 0.1 integration

2013-11-19 Thread Ivan Melnikov

On 19.11.2013 10:38, Kekane, Abhishek wrote:
> Hi All,
> 
> Greetings!!!

Hi there!

And thanks for your interest in cinder and taskflow!

> We are in process of implementing the TaskFlow 0.1 in Cinder for "copy
> volume to image" and "delete volume".
> 
> I have added two blueprints for the same.
> 1. 
> https://blueprints.launchpad.net/cinder/+spec/copy-volume-to-image-task-flow
> 2. https://blueprints.launchpad.net/cinder/+spec/delete-volume-task-flow
> 
> I would like to know if any other developers/teams are working or
> planning to work on any cinder api apart from above two api's.
> 
> Your help is appreciated.

Anastasia Karpinska works on updating existing flows to use released
TaskFlow 0.1.1 instead of internal copy:

https://review.openstack.org/53922

It was blocked because taskflow was not in openstack/requirements, but
now we're there, and Anastasia promised to finish the work and submit
updated changeset for review in couple of days.

There are also two changesets that convert cinder APIs to use TaskFlow:
- https://review.openstack.org/53480 for create_backup by Victor
  Rodionov
- https://review.openstack.org/55134 for create_snapshot by Stanislav
  Kudriashev

As far as I know both Stanislav and Victor suspended their work unitil
Anastasia's change lands.

-- 
WBR,
Ivan A. Melnikov

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Exposing service versions through APIs (was: [heat] Build Info Blueprint)

2013-11-19 Thread Zane Bitter


On 18/11/13 23:45, Anderson Mesquita wrote:

Hello fellows,

As suggested on our meeting last Wednesday (2013-11-13)
,
we're trying to get the build_info discussion going.

The idea is that Heat should provide an endpoint to query its current
build information so that support users, bug reporters, and possibly
regular users can quickly retrieve this information. A more detailed
description is available on the blueprint
,
specification , or the
latest patch  for even more
awesomeness!


Thanks for starting this discussion :)

I've changed the subject line to open this up to the whole community, 
because I think we should be pursuing a unified approach to this across 
all OpenStack projects - there's nothing particularly special about Heat 
that would make it the only place to address this.


For those who have not been following, the FAQ linked above at 
https://etherpad.openstack.org/p/heat-build-info actually does a pretty 
good job of answering my questions.


For the record, I still find the answer to #7 unconvincing (random 
polling is your plan to find the rogue out-of-date server on your 
network?). Also, I have a better idea for #8, #10 is not quite as simple 
as it seems for boring (and mostly Heat-specific) technical reasons, and 
this thread is here to address #11.


cheers,
Zane.


Some of us have already expressed interest in this feature, with some
minor fine tunes missing, like what would be a reasonable default value
for this information.

Any suggestions or concerns will be greatly appreciated so that we can
move this forward and make it available for everybody as soon as
possible! =D

Kind regards,


Anderson and Richard pairing



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] V3 API blueprints

2013-11-19 Thread Christopher Yeoh

Hi,

Some people have expressed interest in helping out with the V3 API related
work during Icehouse. The following are the relevant blueprints based on
ongoing work from Havana and discussions during the summit sessions:

Ongoing blueprints

https://blueprints.launchpad.net/nova/+spec/nova-v3-api (capstone blueprint)
https://blueprints.launchpad.net/nova/+spec/v3-api-cleanup-misc
https://blueprints.launchpad.net/nova/+spec/v3-api-specification
https://blueprints.launchpad.net/python-novaclient/+spec/v3-api
https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests
https://blueprints.launchpad.net/nova/+spec/nova-api-validation-fw

New in Icehouse

https://blueprints.launchpad.net/nova/+spec/api-v3-remove-disk-config
https://blueprints.launchpad.net/nova/+spec/v3-api-core
https://blueprints.launchpad.net/nova/+spec/v3-api-admin-actions-split
https://blueprints.launchpad.net/nova/+spec/v3-api-remove-nova-network
https://blueprints.launchpad.net/nova/+spec/v3-api-remove-extensions
https://blueprints.launchpad.net/nova/+spec/v3-api-pecan
https://blueprints.launchpad.net/nova/+spec/v3-api-policy
https://blueprints.launchpad.net/nova/+spec/v3-api-core-policy

There'll be a few people working on these so if you decide to work on
something please add a relevant entry in the work item section before
starting in order to avoid duplication of effort. Please ping me if you
think anything needs further explanation
or you want a bit of a hand getting started on something.

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] V3 API Review Checklist and style guide

2013-11-19 Thread Christopher Yeoh

Hi,

I've updated the Nova review check list with some details for reviewing V3
API changesets and started a bit of a style guide for the API.

Checklist:
https://wiki.openstack.org/w/index.php?title=ReviewChecklist#Nova_Review_Checklist

Style Guide: https://wiki.openstack.org/wiki/Nova/APIStyleGuide

Both (especially the latter) could do with a bit more work, but it's a
start :-)

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Isaku Yamahata

On Mon, Nov 18, 2013 at 03:55:49PM -0500,
Robert Kukura  wrote:

> On 11/18/2013 03:25 PM, Edgar Magana wrote:
> > Developers,
> > 
> > This topic has been discussed before but I do not remember if we have a
> > good solution or not.
> 
> The ML2 plugin addresses this by calling each MechanismDriver twice. The
> create_network_precommit() method is called as part of the DB
> transaction, and the create_network_postcommit() method is called after
> the transaction has been committed. Interactions with devices or
> controllers are done in the postcommit methods. If the postcommit method
> raises an exception, the plugin deletes that partially-created resource
> and returns the exception to the client. You might consider a similar
> approach in your plugin.

Splitting works into two phase, pre/post, is good approach.
But there still remains race window.
Once the transaction is committed, the result is visible to outside.
So the concurrent request to same resource will be racy.
There is a window after pre_xxx_yyy before post_xxx_yyy() where
other requests can be handled. 

The state machine needs to be enhanced, I think. (plugins need modification)
For example, adding more states like pending_{create, delete, update}.
Also we would like to consider serializing between operation of ports
and subnets. or between operation of subnets and network depending on
performance requirement.
(Or carefully audit complex status change. i.e.
changing port during subnet/network update/deletion.)

I think it would be useful to establish reference locking policy
for ML2 plugin for SDN controllers.
Thoughts or comments? If this is considered useful and acceptable,
I'm willing to help.

thanks,
Isaku Yamahata

> -Bob
> 
> > Basically, if concurrent API calls are sent to Neutron, all of them are
> > sent to the plug-in level where two actions have to be made:
> > 
> > 1. DB transaction ? No just for data persistence but also to collect the
> > information needed for the next action
> > 2. Plug-in back-end implementation ? In our case is a call to the python
> > library than consequentially calls PLUMgrid REST GW (soon SAL)
> > 
> > For instance:
> > 
> > def create_port(self, context, port):
> > with context.session.begin(subtransactions=True):
> > # Plugin DB - Port Create and Return port
> > port_db = super(NeutronPluginPLUMgridV2,
> > self).create_port(context,
> >port)
> > device_id = port_db["device_id"]
> > if port_db["device_owner"] == "network:router_gateway":
> > router_db = self._get_router(context, device_id)
> > else:
> > router_db = None
> > try:
> > LOG.debug(_("PLUMgrid Library: create_port() called"))
> > # Back-end implementation
> > self._plumlib.create_port(port_db, router_db)
> > except Exception:
> > …
> > 
> > The way we have implemented at the plugin-level in Havana (even in
> > Grizzly) is that both action are wrapped in the same "transaction" which
> > automatically rolls back any operation done to its original state
> > protecting mostly the DB of having any inconsistency state or left over
> > data if the back-end part fails.=.
> > The problem that we are experiencing is when concurrent calls to the
> > same API are sent, the number of operation at the plug-in back-end are
> > long enough to make the next concurrent API call to get stuck at the DB
> > transaction level, which creates a hung state for the Neutron Server to
> > the point that all concurrent API calls will fail.
> > 
> > This can be fixed if we include some "locking" system such as calling:
> > 
> > from neutron.common import utile
> > …
> > 
> > @utils.synchronized('any-name', external=True)
> > def create_port(self, context, port):
> > …
> > 
> > Obviously, this will create a serialization of all concurrent calls
> > which will ends up in having a really bad performance. Does anyone has a
> > better solution?
> > 
> > Thanks,
> > 
> > Edgar
> > 
> > 
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Isaku Yamahata 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] - Vendor specific erros

2013-11-19 Thread Avishay Balderman

Hi Salvatore
I think you are mixing between the state machine (ACTIVE,PENDEING_XYZ,etc)  and 
the status description
All I want to do is to write a vendor specific error message when the state is 
ERROR.
I DO NOT want to touch the state machine.

See: https://bugs.launchpad.net/neutron/+bug/1248423

Thanks

Avishay

From: Salvatore Orlando [mailto:sorla...@nicira.com]
Sent: Thursday, November 14, 2013 1:15 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron] - Vendor specific erros

In general, an error state make sense.
I think you might want to send more details about how this state plugs into the 
load balancer state machine, but I reckon it is a generally non-recoverable 
state which could be reached by any other state; in that case it would be a 
generic enough case which might be supported by all drivers.

It is good to point out that driver-specific state transitions however, in my 
opinion, are to avoid; application using the Neutron API will become 
non-portable, or at least users of the Neutron API would need to be aware that 
an entity might have a different state machine from driver to driver, which I 
reckon would be bad enough for a developer to decide to switch over to 
Cloudstack or AWS APIs!

Salvatore

PS: On the last point I am obviously joking, but not so much.


On 12 November 2013 08:00, Avishay Balderman 
mailto:avish...@radware.com>> wrote:

Hi
Some of the DB entities in the LBaaS domain inherit from 
HasStatusDescription
With this we can set the entity status (ACTIVE, PENDING_CREATE,etc) and a 
description for the status.
There are flows in the Radware LBaaS driver that the  driver needs to set the 
entity status to ERROR and it is able to set the description of the error -  
the description is Radware specific.
My question is:  Does it make sense to do that?
After all the tenant is aware to the fact that he works against Radware load 
balancer -  the tenant selected Radware as the lbaas provider in the UI.
Any reason not to do that?

This is a generic issue/question and does not relate  to a specific plugin or 
driver.

Thanks

Avishay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Blueprint: standard specification of guest CPU topology

For attention of maintainers of Nova virt drivers

A while back there was a bug requesting the ability to set the CPU
topology (sockets/cores/threads) for guests explicitly

   https://bugs.launchpad.net/nova/+bug/1199019

I countered that setting explicit topology doesn't play well with
booting images with a variety of flavours with differing vCPU counts.

This led to the following change which used an image property to
express maximum constraints on CPU topology (max-sockets/max-cores/
max-threads) which the libvirt driver will use to figure out the
actual topology (sockets/cores/threads)

  https://review.openstack.org/#/c/56510/

I believe this is a prime example of something we must co-ordinate
across virt drivers to maximise happiness of our users.

There's a blueprint but I find the description rather hard to
follow

  https://blueprints.launchpad.net/nova/+spec/support-libvirt-vcpu-topology

So I've created a standalone wiki page which I hope describes the
idea more clearly

  https://wiki.openstack.org/wiki/VirtDriverGuestCPUTopology

Launchpad doesn't let me link the URL to the blueprint since I'm not
the blurprint creator :-(

Anyway this mail is to solicit input on the proposed standard way to
express this which is hypervisor portable and the addition of some
shared code for doing the calculations which virt driver impls can
just all into rather than re-inventing

I'm looking for buy-in to the idea from the maintainers of each
virt driver that this conceptual approach works for them, before
we go merging anything with the specific impl for libvirt.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Core pinning

2013-11-19 Thread Tuomas Paappanen


Hi Roman,

I haven't but I will write a blueprint for the core pinning part.
I considered vcpu element usage as well but in that case you can not set 
e.g. vcpu-0 to run on pcpu-0. Vcpus and emulator are sharing all pcpus 
defined in cpuset so I decided to use cputune element.


Are you using extra specs for carrying cpuset attributes in your 
implementation?


Br,Tuomas

On 18.11.2013 17:14, Roman Verchikov wrote:

Tuomas,

Have you published your code/blueprints anywhere? Looks like we’re working on 
the same stuff. I have implemented almost the same feature set (haven’t 
published anything yet because of this thread), except for the scheduler part. 
The main goal is to be able to pin VCPUs in NUMA environment.

Have you considered adding placement and cpuset attributes to  element? 
For example:


Thanks,
Roman

On Nov 13, 2013, at 14:46, Tuomas Paappanen  wrote:


Hi all,

I would like to hear your thoughts about core pinning in Openstack. Currently 
nova(with qemu-kvm) supports usage of cpu set of PCPUs what can be used by 
instances. I didn't find blueprint, but I think this feature is for isolate 
cpus used by host from cpus used by instances(VCPUs).

But, from performance point of view it is better to exclusively dedicate PCPUs 
for VCPUs and emulator. In some cases you may want to guarantee that only one 
instance(and its VCPUs) is using certain PCPUs.  By using core pinning you can 
optimize instance performance based on e.g. cache sharing, NUMA topology, 
interrupt handling, pci pass through(SR-IOV) in multi socket hosts etc.

We have already implemented feature like this(PoC with limitations) to Nova 
Grizzly version and would like to hear your opinion about it.

The current implementation consists of three main parts:
- Definition of pcpu-vcpu maps for instances and instance spawning
- (optional) Compute resource and capability advertising including free pcpus 
and NUMA topology.
- (optional) Scheduling based on free cpus and NUMA topology.

The implementation is quite simple:

(additional/optional parts)
Nova-computes are advertising free pcpus and NUMA topology in same manner than 
host capabilities. Instances are scheduled based on this information.

(core pinning)
admin can set PCPUs for VCPUs and for emulator process, or select NUMA cell for 
instance vcpus, by adding key:value pairs to flavor's extra specs.

EXAMPLE:
instance has 4 vcpus
:
vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2...
emulator:5 --> emulator pinned to pcpu5
or
numacell:0 --> all vcpus are pinned to pcpus in numa cell 0.

In nova-compute, core pinning information is read from extra specs and added to 
domain xml same way as cpu quota values(cputune).


  
  
  
  
  


What do you think? Implementation alternatives? Is this worth of blueprint? All 
related comments are welcome!

Regards,
Tuomas





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Horizon based UI Updates in Tuskar-UI

2013-11-19 Thread Jiri Tomasek


Hi,

As Horizon is recently undergoing a set of user interface changes, there 
will be needed some effort in

getting Tuskar-UI up to date with new Horizon UI structure and features.

The changes are comming mostly from the update of Twitter Bootstrap 
framework to latest version (3.0) and
Introducing the AngularJS javascript frontend framework. These two 
updates are in review phase and are

counted in for Icehouse.

To adopt these changes I have identified following tasks:

Horizon based UI Updates in Tuskar-UI:

- Get up to date with Bootstrap 3:
  - update templates through the application to match Bootstrap 3 markup
  - get Forms to use django-bootstrap-form
  - update custom less code to match Bootstrap 3 styling
  - update javascript to use Bootstrap 3 features

- Start using AngularJS Directives with our javascript components
  - update graphs js code to use AngularJS Directives
  - update UI code as Horizon transfers to AngularJS if needed, most of 
the AngularJS inovations should not

require code update on Tuskar-UI side

There is also plan to support icon-fonts as a solution for Icons across 
Horizon applications. This might require
adding custom icon-font, if Tuskar-UI will require icons that are not 
present in basic set that Horizon provides.



Related Horizon blueprints:

Update Twitter Bootstrap to version 3
https://blueprints.launchpad.net/horizon/+spec/bootstrap-update

Introduce AngularJS as Frontend Javascript Framework
https://blueprints.launchpad.net/horizon/+spec/angularjs-javascript-frontend-framework

Change current bitmap icons for font icons
https://blueprints.launchpad.net/horizon/+spec/font-icons


Jirka
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Core pinning

On Wed, Nov 13, 2013 at 02:46:06PM +0200, Tuomas Paappanen wrote:
> Hi all,
> 
> I would like to hear your thoughts about core pinning in Openstack.
> Currently nova(with qemu-kvm) supports usage of cpu set of PCPUs
> what can be used by instances. I didn't find blueprint, but I think
> this feature is for isolate cpus used by host from cpus used by
> instances(VCPUs).
> 
> But, from performance point of view it is better to exclusively
> dedicate PCPUs for VCPUs and emulator. In some cases you may want to
> guarantee that only one instance(and its VCPUs) is using certain
> PCPUs.  By using core pinning you can optimize instance performance
> based on e.g. cache sharing, NUMA topology, interrupt handling, pci
> pass through(SR-IOV) in multi socket hosts etc.
> 
> We have already implemented feature like this(PoC with limitations)
> to Nova Grizzly version and would like to hear your opinion about
> it.
> 
> The current implementation consists of three main parts:
> - Definition of pcpu-vcpu maps for instances and instance spawning
> - (optional) Compute resource and capability advertising including
> free pcpus and NUMA topology.
> - (optional) Scheduling based on free cpus and NUMA topology.
> 
> The implementation is quite simple:
> 
> (additional/optional parts)
> Nova-computes are advertising free pcpus and NUMA topology in same
> manner than host capabilities. Instances are scheduled based on this
> information.
> 
> (core pinning)
> admin can set PCPUs for VCPUs and for emulator process, or select
> NUMA cell for instance vcpus, by adding key:value pairs to flavor's
> extra specs.
> 
> EXAMPLE:
> instance has 4 vcpus
> :
> vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2...
> emulator:5 --> emulator pinned to pcpu5
> or
> numacell:0 --> all vcpus are pinned to pcpus in numa cell 0.
> 
> In nova-compute, core pinning information is read from extra specs
> and added to domain xml same way as cpu quota values(cputune).
> 
> 
>   
>   
>   
>   
>   
> 
> 
> What do you think? Implementation alternatives? Is this worth of
> blueprint? All related comments are welcome!

I think there are several use cases mixed up in your descriptions
here which should likely be considered independantly

 - pCPU/vCPU pinning

   I don't really think this is a good idea as a general purpose
   feature in its own right. It tends to lead to fairly inefficient
   use of CPU resources when you consider that a large % of guests
   will be mostly idle most of the time. It has a fairly high
   administrative burden to maintain explicit pinning too. This
   feels like a data center virt use case rather than cloud use
   case really.

 - Dedicated CPU reservation

   The ability of an end user to request that their VM (or their
   group of VMs) gets assigned a dedicated host CPU set to run on.
   This is obviously something that would have to be controlled
   at a flavour level, and in a commercial deployment would carry
   a hefty pricing premium.

   I don't think you want to expose explicit pCPU/vCPU placement
   for this though. Just request the high level concept and allow
   the virt host to decide actual placement

 - Host NUMA placement.

   By not taking NUMA into account currently the libvirt driver
   at least is badly wasting resources. Having too much cross-numa
   node memory access by guests just kills scalability. The virt
   driver should really automaticall figure out cpu & memory pinning
   within the scope of a NUMA node automatically. No admin config
   should be required for this.

 - Guest NUMA topology

   If the flavour memory size / cpu count exceeds the size of a
   single NUMA node, then the flavour should likely have a way to
   express that the guest should see multiple NUMA nodes. The
   virt host would then set guest NUMA topology to match the way
   it places vCPUs & memory on host NUMA nodes. Again you don't
   want explicit pcpu/vcpu mapping done by the admin for this.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Core pinning

On Wed, Nov 13, 2013 at 11:57:22AM -0600, Chris Friesen wrote:
> On 11/13/2013 11:40 AM, Jiang, Yunhong wrote:
> 
> >>But, from performance point of view it is better to exclusively
> >>dedicate PCPUs for VCPUs and emulator. In some cases you may want
> >>to guarantee that only one instance(and its VCPUs) is using certain
> >>PCPUs.  By using core pinning you can optimize instance performance
> >>based on e.g. cache sharing, NUMA topology, interrupt handling, pci
> >>pass through(SR-IOV) in multi socket hosts etc.
> >
> >My 2 cents. When you talking about " performance point of view", are
> >you talking about guest performance, or overall performance? Pin PCPU
> >is sure to benefit guest performance, but possibly not for overall
> >performance, especially if the vCPU is not consume 100% of the CPU
> >resources.
> 
> It can actually be both.  If a guest has several virtual cores that
> both access the same memory, it can be highly beneficial all around
> if all the memory/cpus for that guest come from a single NUMA node
> on the host.  That way you reduce the cross-NUMA-node memory
> traffic, increasing overall efficiency.  Alternately, if a guest has
> several cores that use lots of memory bandwidth but don't access the
> same data, you might want to ensure that the cores are on different
> NUMA nodes to equalize utilization of the different NUMA nodes.
> 
> Similarly, once you start talking about doing SR-IOV networking I/O
> passthrough into a guest (for SDN/NFV stuff) for optimum efficiency
> it is beneficial to be able to steer interrupts on the physical host
> to the specific cpus on which the guest will be running.  This
> implies some form of pinning.

I would say intelligent NUMA placement is something that virt drivers
should address automatically without any need for admin defined pinning.
The latter is just imposing too much admin burden, for something we can
figure out automatically to a good enough extent.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][heat][[keystone] RFC: introducing "request identification"

2013-11-19 Thread Dolph Mathews

Related BP:

  Create a unified request identifier
  https://blueprints.launchpad.net/nova/+spec/cross-service-request-id

On Tue, Nov 19, 2013 at 5:04 AM, haruka tanizawa wrote:

> Hi stackers!!
>
> I'd like to ask for your opinions about my idea of identifying request.
>
> Challenges
> ==
>
> We have no way to know the final result of an API request.
> Indeed we can continuously get the status of allocated resources,
> but this is just resource status, not request status.
>
> It doesn't matter so much for manual operations.
> But it does for automated clients like heat.
> We need request-oriented-status and it should be disclosed to clients.
>
> Literally, we need to address two items for it.
>  1. how to identify request from clients
>  2. how to disclose status of request to clients
>
> Note that this email includes only 1 for initiating the discussion.
> Also, bp:instance-tasks-api[0] should include both two items above.
>
> Proposal: introducing "request identification"
> =
>
> I'd like to introduce "request identification", which is disclosed to
> clients.
> There are two characteristics:
>
>  - "request identification" is unique ID for each request
>so that we can identify tell a specific request from others.
>  - "request identification" is available for clients
>so that we can enable any after-request-operations like check, retry[1]
> or cancel[2].
>(of course we need to add more logic for check/retry/cancel,
> but I'm pretty sure that indentifying request is the starting point
> for these features)
>
> In my understandings, main objections should be 'who should generate and
> maintain such IDs?'.
>
> How to implement "request identification"
> =
>
> There are several options at least. (See also recent discussion at
> openstack-dev[3])
>
> 1. Enable user to provide his/her own "request identification" within API
> request.
>This should be the simplest way. But providing id from outside looks
> hard to be accepted.
>For example, Idempotency for OpenStack API[4].
>Or instance-tasks-api enable to user to put his/her own "request
> identification".
>
> 2. Use correlation_id in oslo as "request identification".
>correlation_id looks similar concept of "request indentification",
>but correlation_id in nova was already rejected[5].
>
> 3. Enable keystone to generate "request identification" (we can call it
> 'request-token', for example).
>Before sending actual API request to nova-api, client sends a request
> to keystone to get 'request-token'.
>

-2


>Then client sends an actual API request with 'request-token'.
>Nova-api will consult it to keystone whether it was really generated.
>Sounds like a auth-token generated by keystone, differences are:
>  [lifecycle] auth-token is used for multiple API requests before it
> expires,
> 'request-token' is used for only single API request.
>  [reusing] if the same 'request-token' was specified twice or more
> times,
> nova-api simply returns 20x (works like client token in AWS[6]).
> Keystone needs to maintain 'request-tokens' until they expire.
>For backward compatibility, actual API request without 'request-token'
> should work as before.
>We can consider several options for uniqueness of 'request-token':
>  uuid, any string with uniqueness per tennant, etc.
>
> IMO, since adding new implementation to Keystone is a little bit hard
> work,
> so implement of 1 is reasonable for me, just idea.
>
> Any comments will be appreciated.
>
> Sincerely, Haruka Tanizawa
>
> [0] https://blueprints.launchpad.net/nova/+spec/instance-tasks-api
> [1] https://wiki.openstack.org/wiki/Support-retry-with-idempotency
> [2] https://blueprints.launchpad.net/nova/+spec/cancel-swap-volume
> [3]
> http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg09023.html
> [4] https://blueprints.launchpad.net/nova/+spec/idempotentcy-client-token
> [5] https://review.openstack.org/#/c/29480/
> [6]
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration

2013-11-19 Thread Bartosz Górski

Hi,

On 11/15/13 9:17 PM, "Georgy Okrokvertskhov"
wrote:

You are right. At the same time heat feature should not enforce some
specific deployment requirements to other openstack components
especially taking into account different security considerations. I am
trying to rise a concern about possible security implications of that
particular approach of using exposed openstack APIs and bring security
requirements to the table for discussion. Probably this will help to
design better solution for heat to heat or DC to DC communication if
it exists. I hope that there is a room for discussion and it is
possible to influence on the final design and implementation. I really
want this feature to be flexible and useful for most of OpenStack
deployments rather then for some specific deployment case.s,

Georgy Okrokvertskhov

Thank you for your security concerns. There is a room for discussion and
possibility to influence the final design. This is the reason why I
started this discussion.

On 11/15/2013 01:41 PM, Zane Bitter wrote:
Good news, everyone! I have created the missing whiteboard diagram
that we all needed at the design summit:

https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram

I've documented 5 possibilities. (1) is the current implementation,
which we agree we want to get away from. I strongly favour (2) for the
reasons listed. I don't think (3) has many friends. (4) seems to be
popular despite the obvious availability problem and doubts that it is
even feasible. Finally, I can save us all some time by simply stating
that I will -2 on sight any attempt to implement (5).

Zane thank you for creating this diagram! I think it was really the
missing piece. Without it people were confused. Right now I think they
have better understanding of the possible solutions.

On 11/15/2013 01:41 PM, Zane Bitter wrote:

On 11/15/13 2:58 PM, "Zane Bitter" wrote:

So, to be clear, this is option (4) from the diagram I put together here:
https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram

It's got a couple of major problems:

* When a whole region goes down, you can lose access to the Heat
instance that was managing still-available resources. This makes it
more or less impossible to use Heat to manage a highly-available
global application.

* Instances have to communicate back to the Heat instance that owns
them (e.g. for WaitConditions), and it's not yet clear that this is
feasible in general.

There are also a number of other things I really don't like about this
solution (listed on the wiki page), though reasonable people may
disagree.

cheers,
Zane.

Yes. You are right. The proof of concept version I implemented is the
option (4). I agree with you about the problems you listed. I am not so
concerned about the first one (region goes down) but the second one will
be a bigger problem in production and I did not notice it before. Also
exposing the API endpoint for all the services maybe not the best
solution from the security perspective. We probably should go with
option (2).

On 11/16/2013 09:20 AM, Zane Bitter wrote:

On 15/11/13 22:17, Keith Bray wrote:
use Heat in that region to do it, you can Adopt the resources into a
Heat

stack in that region. I don't see how 2 is "Robust against failure of
whole region" because if the region on the left part of the picture in 2
goes down, you can't manage your global stack or any of the resources in
the left region that are part of that global stack. All you could
manage
is a subset of resources by manipulating the substack in the right
region,
but you can do this in 4 as well using Adopt. 4 is a simpler
starting use
case and easier (IMO) for a user of the system to digest, and has the
HUGE

advantage of being able to orchestrate deploying resources to multiple
regions without a service operator having to have Heat setup and
installed

in EVERY region. This is particular important for a private cloud/heat
person trying to deploy to public cloud.. If doing so requires the
public
cloud operator have Heat running, then you can't deploy there. If no
Heat
in that region is required, then you can use your own instance of
Heat to
deploy to any available openstack cloud. That is a HUGE benefit of 4.
Good point, I've added that to the Pro/Con in the wiki.

It is the benefit but I think for now we can assume that we have Heat
service deployed in each region.

The implementation differences between option (2) and (4) does not look
like to be big.
May be we can add new option in heat configuration file to choose the
way to create in different region in case when heat service is not
available.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Best,
Bartosz

[openstack-dev] [Neutron][LBaaS] SSL Termination write-up

2013-11-19 Thread Vijay Venkatachalam

Hi Sam, Eugene, & Avishay, etal,

Today I spent some time to create a write-up for SSL 
Termination not exactly design doc. Please share your comments!

https://docs.google.com/document/d/1tFOrIa10lKr0xQyLVGsVfXr29NQBq2nYTvMkMJ_inbo/edit

Would like comments/discussion especially on the following note:

SSL Termination requires certificate management. The ideal way is to handle 
this via an independent IAM service. This would take time to implement so the 
thought was to add the certificate details in VIP resource and send them 
directly to device. Basically don't store the certificate key in the DB there 
by avoiding security concerns of maintaining certificates in controller.

I would expect the certificates to become an independent resource in future 
thereby causing backward compatibility issues.

Any ideas how to achieve this?

My thought was to have independent certificate resource with VIP uuid as one of 
the properties. VIP is already created and will help to identify the 
driver/device. The VIP property can be depreciated in the long term.
Thanks,
Vijay V.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Reg : Security groups implementation using openflows in quantum ovs plugin

2013-11-19 Thread Édouard Thuleau

Hi,

It's an interesting feature.
But just to understand, what do you blame to the actual implementation with
iptables and linux bridge?

The OVS release 1.11.0 implements a new feature calls 'megaflows'
which reduce the number of kernel/usespace crossings.
Actually, OVS neutron agent uses simple default "normal" flow (simple mac
learning switch) which the wildcarding (use by megaflow) will be very good
(just the L2 headers will be matched).
But if we implement security group as OVS flows, the performance will be
reduced. Wildcarding will be worse (L2, L3, and L4headers will be mateched).
Here [1] and [2] a post from the OVS mailing list that explain that.
Perhaps we can create security group OVS flow smarter to improve the
wilcarding.

Another improvement, I see, is the simplification of the interfaces on the
compute node. All interfaces qbr, qvo and qvb will disappear.
But another simple improvement about that could be to use only one Linux
bridge by network instead of one per VNIC and continous to use Linux bridge
and iptables.

Another problem is the OVS flows are not stateful [3] but is it necessary?

[1] http://www.mail-archive.com/discuss@openvswitch.org/msg07715.html
[2] http://www.mail-archive.com/discuss@openvswitch.org/msg07582.html
[3] http://www.mail-archive.com/discuss@openvswitch.org/msg01919.html

Édouard.


On Tue, Nov 19, 2013 at 10:31 AM, Kanthi P wrote:

> Hi All,
>
> Thanks for the response!
> Amir,Mike: Is your implementation being done according to ML2 plugin
>
> Regards,
> Kanthi
>
>
> On Tue, Nov 19, 2013 at 1:43 AM, Mike Wilson  wrote:
>
>> Hi Kanthi,
>>
>> Just to reiterate what Kyle said, we do have an internal implementation
>> using flows that looks very similar to security groups. Jun Park was the
>> guy that wrote this and is looking to get it upstreamed. I think he'll be
>> back in the office late next week. I'll point him to this thread when he's
>> back.
>>
>> -Mike
>>
>>
>> On Mon, Nov 18, 2013 at 3:39 PM, Kyle Mestery (kmestery) <
>> kmest...@cisco.com> wrote:
>>
>>> On Nov 18, 2013, at 4:26 PM, Kanthi P  wrote:
>>> > Hi All,
>>> >
>>> > We are planning to implement quantum security groups using openflows
>>> for ovs plugin instead of iptables which is the case now.
>>> >
>>> > Doing so we can avoid the extra linux bridge which is connected
>>> between the vnet device and the ovs bridge, which is given as a work around
>>> since ovs bridge is not compatible with iptables.
>>> >
>>> > We are planning to create a blueprint and work on it. Could you please
>>> share your views on this
>>> >
>>> Hi Kanthi:
>>>
>>> Overall, this idea is interesting and removing those extra bridges would
>>> certainly be nice. Some people at Bluehost gave a talk at the Summit [1] in
>>> which they explained they have done something similar, you may want to
>>> reach out to them since they have code for this internally already.
>>>
>>> The OVS plugin is in feature freeze during Icehouse, and will be
>>> deprecated in favor of ML2 [2] at the end of Icehouse. I would advise you
>>> to retarget your work at ML2 when running with the OVS agent instead. The
>>> Neutron team will not accept new features into the OVS plugin anymore.
>>>
>>> Thanks,
>>> Kyle
>>>
>>> [1]
>>> http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/towards-truly-open-and-commoditized-software-defined-networks-in-openstack
>>> [2] https://wiki.openstack.org/wiki/Neutron/ML2
>>>
>>> > Thanks,
>>> > Kanthi
>>> > ___
>>> > OpenStack-dev mailing list
>>> > OpenStack-dev@lists.openstack.org
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Unable to access outside world from Openstack instance after assining floating ip

2013-11-19 Thread Priti Sarap

Hello All,

I deployed latest devstack code and tried to access outside world from
openstack
instance. Before assigning floating-ip to an instance I am able to ping
google.com,
but after assigning floating-ip to that instance I am not able to ping
google.com from that instance.

Does anyone has any idea?
Any help would be highly appreciated.


Thanks and Regards,
Priti Sarap
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [savanna] bug day - Nov 19

2013-11-19 Thread Alexander Ignatov

Sergey,

Thank you. I’ve edited information section related to assigning tags in bug 
title. 
Just added more examples.

Regards,
Alexander Ignatov



On 19 Nov 2013, at 15:13, Sergey Lukjanov  wrote:

> Reminder: today is the bug triage day on Savanna project.
> 
> I’ve prepared the wiki page for it - 
> https://wiki.openstack.org/wiki/Savanna/BugTriage (based on 
> https://wiki.openstack.org/wiki/BugTriage).
> 
> See you in #savanna channel.
> 
> Thank you.
> 
> Sincerely yours,
> Sergey Lukjanov
> Savanna Technical Lead
> Mirantis Inc.
> 
> On Nov 15, 2013, at 11:34 AM, Sergey Lukjanov  wrote:
> 
>> Hi team,
>> 
>> we have some unmanaged bugs in LP for Savanna project and, so, we’ll have a 
>> bug day to triage/cleanup them at November, 19 (Tuesday) as we discussed on 
>> the last irc team meeting.
>> 
>> I’ll send a reminder at the start of the day.
>> 
>> Thanks.
>> 
>> Sincerely yours,
>> Sergey Lukjanov
>> Savanna Technical Lead
>> Mirantis Inc.
>> 
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Unable to access outside world from Openstack instance after assining floating ip

2013-11-19 Thread Flavio Percoco


On 19/11/13 19:37 +0530, Priti Sarap wrote:

Hello All,

I deployed latest devstack code and tried to access outside world from
openstack
instance. Before assigning floating-ip to an instance I am able to ping
google.com,
but after assigning floating-ip to that instance I am not able to ping
google.com from that instance.

Does anyone has any idea?
Any help would be highly appreciated.



Hey,

This is a development-only mailing list. For support please post
messages to the 'openstack' m-l[0].

[0] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Chuck Short

Hi

I am excited to see containers getting such traction in the openstack
project.


On Mon, Nov 18, 2013 at 7:30 PM, Russell Bryant  wrote:

> On 11/18/2013 06:30 PM, Dan Smith wrote:
> >> Not having been at the summit (maybe the next one), could somebody
> >> give a really short explanation as to why it needs to be a separate
> >> service? It sounds like it should fit within the Nova area. It is,
> >> after all, just another hypervisor type, or so it seems.
> >
> > But it's not just another hypervisor. If all you want from your
> > containers is lightweight VMs, then nova is a reasonable place to put
> > that (and it's there right now). If, however, you want to expose the
> > complex and flexible attributes of a container, such as being able to
> > overlap filesystems, have fine-grained control over what is shared with
> > the host OS, look at the processes within a container, etc, then nova
> > ends up needing quite a bit of change to support that.
> >
> > I think the overwhelming majority of folks in the room, after discussing
> > it, agreed that Nova is infrastructure and containers is more of a
> > platform thing. Making it a separate service lets us define a mechanism
> > to manage these that makes much more sense than treating them like VMs.
> > Using Nova to deploy VMs that run this service is the right approach,
> > IMHO. Clayton put it very well, I think:
> >
> >   If the thing you want to deploy has a kernel, then you need Nova. If
> >   your thing runs on a kernel, you want $new_service_name.
> >
> > I agree.
> >
> > Note that this is just another service under the compute project (or
> > program, or whatever the correct terminology is this week).
>
> The Compute program is correct.  That is established terminology as
> defined by the TC in the last cycle.
>
> > So while
> > distinct from Nova in terms of code, development should be tightly
> > integrated until (and if at some point) it doesn't make sense.
>
> And it may share a whole bunch of the code.
>
> Another way to put this:  The API requirements people have for
> containers include a number of features considered outside of the
> current scope of Nova (short version: Nova's scope stops before going
> *inside* the servers it creates, except file injection, which we plan to
> remove anyway).  That presents a problem.  A new service is one possible
> solution.
>
> My view of the outcome of the session was not "it *will* be a new
> service".  Instead, it was, "we *think* it should be a new service, but
> let's do some more investigation to decide for sure".
>
> The action item from the session was to go off and come up with a
> proposal for what a new service would look like.  In particular, we
> needed a proposal for what the API would look like.  With that in hand,
> we need to come back and ask the question again of whether a new service
> is the right answer.
>
> I see 3 possible solutions here:
>
> 1) Expand the scope of Nova to include all of the things people want to
> be able to do with containers.
>
> This is my least favorite option.  Nova is already really big.  We've
> worked to split things out (Networking, Block Storage, Images) to keep
> it under control.  I don't think a significant increase in scope is a
> smart move for Nova's future.
>
>
This is my least favorite option. Like a lot of other responses already I
see a lot of code duplication  because Nova and the new nova container's
project. This just doesn't include the scheduler but  things like config
driver, etc.


> 2) Declare containers as explicitly out of scope and start a new project
> with its own API.
>
> That is what is being proposed here.
>
> 3) Some middle ground that is a variation of #2.  Consider Ironic.  The
> idea is that Nova's API will still be used for basic provisioning, which
> Nova will implement by talking to Ironic.  However, there are a lot of
> baremetal management things that don't fit in Nova at all, and those
> only exist in Ironic's API.
>

This is my preferred choice  as well. If we could leverage the existing
nova API and extend it to include containers and features that users who
use containers in their existing production environment wants.

>
> I wanted to mention this option for completeness, but I don't actually
> think it's the right choice here.  With Ironic you have a physical
> resource (managed by Ironic), and then instances of an image running on
> these physical resources (managed by Nova).
>
> With containers, there's a similar line.  You have instances of
> containers (managed either by Nova or the new service) running on
> servers (managed by Nova).  I think there is a good line for separating
> concerns, with a container service on top of Nova.
>
>
> Let's ask ourselves:  How much overlap is there between the current
> compute API and a proposed containers API?  Effectively, what's the
> diff?  How much do we expect this diff to change in the coming years?
>
> The current diff demonstrates a significant clash with the cu

[openstack-dev] [infra] How to determine patch set load for a given project

2013-11-19 Thread Matt Riedemann

We have a team working on getting CI setup for DB2 10.5 in 
sqlalchemy-migrate and they were asking me if there was a way to 
calculate the patch load through that project.


I asked around in the infra IRC channel and Jeremy Stanley pointed out 
that there might be something available in 
http://graphite.openstack.org/ by looking for the project's test stats.


I found that if you expand stats_counts > zuul > job and then search for 
your project (sqlalchemy-migrate in this case), you can find the jobs 
and their graphs for load. In my case I care about stats for 
gate-sqlalchemy-migrate-python27.


I'm having a little trouble interpreting the data though. From looking 
at what's out there for review now, there is one new patch created on 
11/19 and the last new one before that was on 11/15. I see spikes in the 
graph around 11/15, 11/18 and 11/19, but I'm not sure what the 11/18 
spike is from?


--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

2013-11-19 Thread Peter Feiner

A substantive reason for switching from mox to mock is the derelict
state of mox releases. There hasn't been a release of mox in three
years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
in the past 3 years, substantial bugs have been fixed in upstream mox.
For example, with the year-old fix to
https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
in nova would have been caught by an existing test [3].

Alternatively, a copy of the upstream mox code could be added in-tree.

[1] mox releases: https://code.google.com/p/pymox/downloads/list
[2] mox on pypi: https://pypi.python.org/pypi/mox
[3] see comments 5 and 6 in https://bugs.launchpad.net/nova/+bug/1251792

On Wed, Nov 13, 2013 at 2:24 PM, Matt Riedemann
 wrote:
>
>
> On 11/12/2013 5:04 PM, Chuck Short wrote:
>>
>>
>>
>>
>> On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin > > wrote:
>>
>> On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:
>>  >
>>  > Hi
>>  >
>>  >
>>  > On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin
>> mailto:mar...@redhat.com>>
>>
>>  > wrote:
>>  > On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
>>  > > Maybe we should have some 60% rule... that is: If you
>> change
>>  > more than
>>  > > half of a test... you should *probably* rewrite the test
>> in
>>  > Mock.
>>  >
>>  >
>>  > A rule needs a reasoning attached to it :)
>>  >
>>  > Why do we want people to use mock?
>>  >
>>  > Is it really for Python3? If so, I assume that means we've
>>  > ruled out the
>>  > python3 port of mox? (Ok by me, but would be good to hear
>> why)
>>  > And, if
>>  > that's the case, then we should encourage whoever wants to
>>  > port mox
>>  > based tests to mock.
>>  >
>>  >
>>  >
>>  > The upstream maintainer is not going to port mox to python3 so we
>> have
>>  > a fork of mox called mox3. Ideally, we would drop the usage of mox
>> in
>>  > favour of mock so we don't have to carry a forked mox.
>>
>> Isn't that the opposite conclusion you came to here:
>>
>>
>> http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html
>>
>> i.e. using mox3 results in less code churn?
>>
>> Mark.
>>
>>
>>
>> Yes that was my original position but I though we agreed in thread
>> (further on) that we would use mox3 and then migrate to mock further on.
>>
>> Regards
>> chuck
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> So it sounds like we're good with using mox for new tests again? Given Chuck
> got it into global-requirements here:
>
> https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d
>
> We can stave off the need to transition everything from mox to mock?
>
> I can't seem to find the nova blueprint to convert everything from mox to
> mock, maybe it was obsoleted already.
>
> Anyway, if mox(3) is OK and we don't need to use mock, it seems like we
> could add something to the developer guide here because I think this
> question comes up frequently:
>
> http://docs.openstack.org/developer/nova/devref/unit_tests.html
>
> Does anyone disagree?
>
> BTW, I care about this because I've been keeping in mind the mox/mock
> transition when doing code reviews and giving a -1 when new tests are using
> mox (since I thought that was a no-no now).
> --
>
> Thanks,
>
> Matt Riedemann
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] When is a blueprint unnecessary?

Greetings,

One of the bits of feedback that came from the "Nova Project Structure
and Process" session at the design summit was that it would be nice to
skip having blueprints for smaller items.

In an effort to capture this, I updated the blueprint review criteria
[1] with the following:

  Some blueprints are closed as unnecessary. Blueprints are used for
  tracking significant development efforts. In general, small and/or
  straight forward cleanups do not need blueprints. A blueprint should
  be filed if:

   - it impacts the release notes
   - it covers significant internal development or refactoring efforts

So, to pick out some examples (not trying to pick on anything in
particular here):

An example of a feature that should be in release notes:
  https://blueprints.launchpad.net/nova/+spec/xenapi-resize-ephemeral-disks

An example of a major refactoring effort that justifies a blueprint:
  https://blueprints.launchpad.net/nova/+spec/live-migration-to-conductor

An example of something I think is small enough to not need a blueprint:
  https://blueprints.launchpad.net/nova/+spec/no-import-uuid


Determining when a blueprint is and is not necessary is a bit difficult
to define clearly.  I would appreciate input on how we can define it
more cleanly so that it can be enforced by the blueprint reviewers
consistently.

Thanks,

[1] https://wiki.openstack.org/wiki/Blueprints#Blueprint_Review_Criteria

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Search Project - summit follow up

2013-11-19 Thread Jaromir Coufal


Hey OpenStackers,

I hope you all enjoyed Hong Kong Summit and for those who couldn't 
attend, I hope we will see each other next time!


For all of you, I'd like to follow up on conversation from Summit about 
Search project [0], which Dmitri introduced and I believe it has great 
potential for enhancing OpenStack's user experience, especially in 
Horizon, but not just in there.


I'd like to thank to Dmitri and all folks who were working on this 
project for sharing and their will for contributing.


It would be great, if we can follow up on this topic, start discussion 
around the project here on mailing list (since not everybody could 
attend the summit) and gather some feedback. There are lot of questions 
like where to accommodate the code, what are next steps, etc.


I am cc'ing Dmitri, so he can bring more light into this discussion and 
I hope we can move forward.


Thanks
-- Jarda

[0] https://wiki.openstack.org/wiki/SearchProject
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Hyper-V meeting agenda

2013-11-19 Thread Peter Pouliot

Hi All,

The agenda for todays Hyper-V meeting

* Cloudbase-init improvements

* Ceilometer fixes

* Puppet Module Status

* Ci Update
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] When is a blueprint unnecessary?

On Tue, Nov 19, 2013 at 10:56:10AM -0500, Russell Bryant wrote:
> Greetings,
> 
> One of the bits of feedback that came from the "Nova Project Structure
> and Process" session at the design summit was that it would be nice to
> skip having blueprints for smaller items.
> 
> In an effort to capture this, I updated the blueprint review criteria
> [1] with the following:
> 
>   Some blueprints are closed as unnecessary. Blueprints are used for
>   tracking significant development efforts. In general, small and/or
>   straight forward cleanups do not need blueprints. A blueprint should
>   be filed if:
> 
>- it impacts the release notes
>- it covers significant internal development or refactoring efforts

"Impacts release notes" can seem a bit nebulous. It shifts the problem
from
 
   "what is a blueprint required for?"

to

   "what is a release note required for?"

which can be just as ill-defined for many people. So we might want
to be more explicit to enumerate all the things that would typically
trigger release notes, and thus blueprints, explicitly

 - Adds configuration parameters
 - Adds glance/cinder metdata properties
 - Changes public APIs
 - Affects database schema
   ...probably many other things...

> So, to pick out some examples (not trying to pick on anything in
> particular here):
> 
> An example of a feature that should be in release notes:
>   https://blueprints.launchpad.net/nova/+spec/xenapi-resize-ephemeral-disks
> 
> An example of a major refactoring effort that justifies a blueprint:
>   https://blueprints.launchpad.net/nova/+spec/live-migration-to-conductor
> 
> An example of something I think is small enough to not need a blueprint:
>   https://blueprints.launchpad.net/nova/+spec/no-import-uuid
> 
> 
> Determining when a blueprint is and is not necessary is a bit difficult
> to define clearly.  I would appreciate input on how we can define it
> more cleanly so that it can be enforced by the blueprint reviewers
> consistently.

I'd also suggest another criteria for

 - Impacts cross-project interaction/interfaces

 eg calls between Nova & Neutron for VIF setup
   or betweeen Nova & Cinder for volume setup

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [style] () vs \ continuations

2013-11-19 Thread David Stanek

On Mon, Nov 18, 2013 at 8:23 PM, Vishvananda Ishaya
wrote:
>
>
> +1 to sticking something in hacking. FWIW I would probably do the following
> to avoid the debate altogether:
>
> result = self._path_file_exists(ds_browser, folder_path, file_name)
> folder_exists, file_exists, file_size_in_kb, disk_extents = result
>
> Vish
>
>
I'm working on some test code and came up with what I think is a valid use
of a line ending in \.  I'm not escaping the newline from a Python syntax
point of view.  I'm doing in inside of a string literal.  Example:

mismatches_description = textwrap.dedent("""\
expected = http://docs.openstack.org/identity/api/v2.0
">
  


actual = http://docs.openstack.org/identity/api/v2.0";>
  

""")

This can be written in several different ways, but I think that none of
them is as clear as the trailing slash.  This is, I think, the clearest
alternative example:

mismatches_description = textwrap.dedent("""
expected = http://docs.openstack.org/identity/api/v2.0
">
  


actual = http://docs.openstack.org/identity/api/v2.0";>
  

""").lstrip()

Would this use be forbidden too?

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [UX] Automatically post new threads from AskBot to the list

2013-11-19 Thread Julie Pichon

Hi folks,

I've been thinking about the AskBot UX website [0] and its lack of
visibility, particularly for new community members.

I think it would be valuable to have the first post from new
conversation threads be posted to the -dev list with the appropriate
[UX] tag, with a link to AskBot and a request to continue the
conversation over there. This would help newcomers realising there is a
space for UX conversations, and existing community members to pop by
when a topic comes up around an area of interest. The traffic has been
low so far, and I wouldn't expect it to become more overwhelming than
any of the current projects currently communicating via the developers
list.

Would there be any concern or objections?

Julie

[0] ask-openstackux.rhcloud.com/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

2013-11-19 Thread Chuck Short

Hi


On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner  wrote:

> A substantive reason for switching from mox to mock is the derelict
> state of mox releases. There hasn't been a release of mox in three
> years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
> in the past 3 years, substantial bugs have been fixed in upstream mox.
> For example, with the year-old fix to
> https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
> in nova would have been caught by an existing test [3].
>
> Alternatively, a copy of the upstream mox code could be added in-tree.
>
> Please no, I think we are in an agreement with mox3 and mock.



> [1] mox releases: https://code.google.com/p/pymox/downloads/list
> [2] mox on pypi: https://pypi.python.org/pypi/mox
> [3] see comments 5 and 6 in https://bugs.launchpad.net/nova/+bug/1251792
>
> On Wed, Nov 13, 2013 at 2:24 PM, Matt Riedemann
>  wrote:
> >
> >
> > On 11/12/2013 5:04 PM, Chuck Short wrote:
> >>
> >>
> >>
> >>
> >> On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin  >> > wrote:
> >>
> >> On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:
> >>  >
> >>  > Hi
> >>  >
> >>  >
> >>  > On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin
> >> mailto:mar...@redhat.com>>
> >>
> >>  > wrote:
> >>  > On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> >>  > > Maybe we should have some 60% rule... that is: If you
> >> change
> >>  > more than
> >>  > > half of a test... you should *probably* rewrite the
> test
> >> in
> >>  > Mock.
> >>  >
> >>  >
> >>  > A rule needs a reasoning attached to it :)
> >>  >
> >>  > Why do we want people to use mock?
> >>  >
> >>  > Is it really for Python3? If so, I assume that means
> we've
> >>  > ruled out the
> >>  > python3 port of mox? (Ok by me, but would be good to hear
> >> why)
> >>  > And, if
> >>  > that's the case, then we should encourage whoever wants
> to
> >>  > port mox
> >>  > based tests to mock.
> >>  >
> >>  >
> >>  >
> >>  > The upstream maintainer is not going to port mox to python3 so we
> >> have
> >>  > a fork of mox called mox3. Ideally, we would drop the usage of
> mox
> >> in
> >>  > favour of mock so we don't have to carry a forked mox.
> >>
> >> Isn't that the opposite conclusion you came to here:
> >>
> >>
> >>
> http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html
> >>
> >> i.e. using mox3 results in less code churn?
> >>
> >> Mark.
> >>
> >>
> >>
> >> Yes that was my original position but I though we agreed in thread
> >> (further on) that we would use mox3 and then migrate to mock further on.
> >>
> >> Regards
> >> chuck
> >>
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> > So it sounds like we're good with using mox for new tests again? Given
> Chuck
> > got it into global-requirements here:
> >
> >
> https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d
> >
> > We can stave off the need to transition everything from mox to mock?
> >
> > I can't seem to find the nova blueprint to convert everything from mox to
> > mock, maybe it was obsoleted already.
> >
> > Anyway, if mox(3) is OK and we don't need to use mock, it seems like we
> > could add something to the developer guide here because I think this
> > question comes up frequently:
> >
> > http://docs.openstack.org/developer/nova/devref/unit_tests.html
> >
> > Does anyone disagree?
> >
> > BTW, I care about this because I've been keeping in mind the mox/mock
> > transition when doing code reviews and giving a -1 when new tests are
> using
> > mox (since I thought that was a no-no now).
> > --
> >
> > Thanks,
> >
> > Matt Riedemann
> >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Reg : Security groups implementation using openflows in quantum ovs plugin

2013-11-19 Thread Mike Wilson

The current implementation is fairly generic, the plan is to get it into
the ML2 plugin.

-Mike


On Tue, Nov 19, 2013 at 2:31 AM, Kanthi P  wrote:

> Hi All,
>
> Thanks for the response!
> Amir,Mike: Is your implementation being done according to ML2 plugin
>
> Regards,
> Kanthi
>
>
> On Tue, Nov 19, 2013 at 1:43 AM, Mike Wilson  wrote:
>
>> Hi Kanthi,
>>
>> Just to reiterate what Kyle said, we do have an internal implementation
>> using flows that looks very similar to security groups. Jun Park was the
>> guy that wrote this and is looking to get it upstreamed. I think he'll be
>> back in the office late next week. I'll point him to this thread when he's
>> back.
>>
>> -Mike
>>
>>
>> On Mon, Nov 18, 2013 at 3:39 PM, Kyle Mestery (kmestery) <
>> kmest...@cisco.com> wrote:
>>
>>> On Nov 18, 2013, at 4:26 PM, Kanthi P  wrote:
>>> > Hi All,
>>> >
>>> > We are planning to implement quantum security groups using openflows
>>> for ovs plugin instead of iptables which is the case now.
>>> >
>>> > Doing so we can avoid the extra linux bridge which is connected
>>> between the vnet device and the ovs bridge, which is given as a work around
>>> since ovs bridge is not compatible with iptables.
>>> >
>>> > We are planning to create a blueprint and work on it. Could you please
>>> share your views on this
>>> >
>>> Hi Kanthi:
>>>
>>> Overall, this idea is interesting and removing those extra bridges would
>>> certainly be nice. Some people at Bluehost gave a talk at the Summit [1] in
>>> which they explained they have done something similar, you may want to
>>> reach out to them since they have code for this internally already.
>>>
>>> The OVS plugin is in feature freeze during Icehouse, and will be
>>> deprecated in favor of ML2 [2] at the end of Icehouse. I would advise you
>>> to retarget your work at ML2 when running with the OVS agent instead. The
>>> Neutron team will not accept new features into the OVS plugin anymore.
>>>
>>> Thanks,
>>> Kyle
>>>
>>> [1]
>>> http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/towards-truly-open-and-commoditized-software-defined-networks-in-openstack
>>> [2] https://wiki.openstack.org/wiki/Neutron/ML2
>>>
>>> > Thanks,
>>> > Kanthi
>>> > ___
>>> > OpenStack-dev mailing list
>>> > OpenStack-dev@lists.openstack.org
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] When is a blueprint unnecessary?

On 11/19/2013 11:11 AM, Daniel P. Berrange wrote:
> On Tue, Nov 19, 2013 at 10:56:10AM -0500, Russell Bryant wrote:
>> Greetings,
>>
>> One of the bits of feedback that came from the "Nova Project Structure
>> and Process" session at the design summit was that it would be nice to
>> skip having blueprints for smaller items.
>>
>> In an effort to capture this, I updated the blueprint review criteria
>> [1] with the following:
>>
>>   Some blueprints are closed as unnecessary. Blueprints are used for
>>   tracking significant development efforts. In general, small and/or
>>   straight forward cleanups do not need blueprints. A blueprint should
>>   be filed if:
>>
>>- it impacts the release notes
>>- it covers significant internal development or refactoring efforts
> 
> "Impacts release notes" can seem a bit nebulous. It shifts the problem
> from
>  
>"what is a blueprint required for?"
> 
> to
> 
>"what is a release note required for?"
> 
> which can be just as ill-defined for many people. So we might want
> to be more explicit to enumerate all the things that would typically
> trigger release notes, and thus blueprints, explicitly
> 
>  - Adds configuration parameters
>  - Adds glance/cinder metdata properties
>  - Changes public APIs
>  - Affects database schema
>...probably many other things...
> 

Good point.  In general, I think of release notes as

"Does a user care about this?"

They care about

   - features being added, removed, changed
  - includes config, special properties, public APIs
   - changes that affect deployment or upgrades

I'll see if I can get this incorporated into the wiki.

>> So, to pick out some examples (not trying to pick on anything in
>> particular here):
>>
>> An example of a feature that should be in release notes:
>>   https://blueprints.launchpad.net/nova/+spec/xenapi-resize-ephemeral-disks
>>
>> An example of a major refactoring effort that justifies a blueprint:
>>   https://blueprints.launchpad.net/nova/+spec/live-migration-to-conductor
>>
>> An example of something I think is small enough to not need a blueprint:
>>   https://blueprints.launchpad.net/nova/+spec/no-import-uuid
>>
>>
>> Determining when a blueprint is and is not necessary is a bit difficult
>> to define clearly.  I would appreciate input on how we can define it
>> more cleanly so that it can be enforced by the blueprint reviewers
>> consistently.
> 
> I'd also suggest another criteria for
> 
>  - Impacts cross-project interaction/interfaces
> 
>  eg calls between Nova & Neutron for VIF setup
>or betweeen Nova & Cinder for volume setup

ACK, will add.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Neutron Tempest code sprint - 2nd week of January, Montreal, QC, Canada

2013-11-19 Thread Rossella Sblendido

Hi all,

sorry if this is a bit OT now.
I contacted some hotels to see if we could get a special price if we book
many rooms. According to my research the difference in price is not much.
Also, as Anita was saying, booking for everybody is more complicated.
So I decided to booked a room for myself.
I share the name of the hotel, in case you want to stay in the same place
http://www.lenouvelhotel.com/.
It's close to the meeting room and the price is one of the best I have
found.

cheers,

Rossella




On Sat, Nov 16, 2013 at 7:39 PM, Anita Kuno  wrote:

>  On 11/16/2013 01:14 PM, Anita Kuno wrote:
>
> On 11/16/2013 12:37 PM, Sean Dague wrote:
>
> On 11/15/2013 10:36 AM, Russell Bryant wrote:
>
>  On 11/13/2013 11:10 AM, Anita Kuno wrote:
>
>  Neutron Tempest code sprint
>
> In the second week of January in Montreal, Quebec, Canadathere will be a
> Neutron Tempest code sprint to improve the status of Neutron tests in
> Tempest and to add new tests.
>
>  First off, I think anything regarding putting more effort into this is
> great.  However, I *beg* the Neutron team not to wait until this week to
> make significant progress.
>
> To be clear, IMO, this is already painfully late.  It's one of the
> largest items blocking deprecation of nova-network and moving forward
> with Neutron.  This spring is just a couple weeks before icehouse-2.
> Come i2, mid-cycle, we're going to have make a call on whether or not
> nova-network deprecation seems likely.  If not, we really can't wait
> around and I will probably propose un-freezing nova-network.
>
>  100% agreed. Realistically this has to be a capping session, not when
> the real work starts. If the agent rewrite isn't done, and the parallel
> gate basically working before then, it won't work.
>
> Honestly, I think we're going to need to checkpoint before Christmas to
> figure out if the preconditions are met, because if they aren't, then
> it's not clear the 3 days of code sprint are really going to generate
> the results we need.
>
> I'm encouraged by the level of activity in the neutron channel, so I
> think right now planning for success is good. But again, we need some
> aggressive check points, and people need to realize this is not the
> beginning of this work, it is the closing session.
>
>   -Sean
>
>  Great. What kind of check points in a mutual location would we like to
> see?
>
> Etherpad?
>
> Meeting logs?
>
> Other suggestions?
>
> Thanks Sean,
> Anita.
>
> Actually I am going to answer my own question with a proposal.
>
> I would like to see a special sub-committee of the TC convened comprised
> of (5?) TC members and/or appointees for the express purpose of being
> available to help meet this goal, be available in the -neutron irc channel
> for questions and support and also to guide the process so the goal is
> achieved.
>
> Thoughts?
>
>
> Anita.
>
>
>
> ___
> OpenStack-dev mailing 
> listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Swift] Metadata Search API

2013-11-19 Thread Paula Ta-Shma


> Hi Paula,
>
> Where can we see the source code for the prototype and any API specs
> that you may have?
>
> Best,
> -jay

Hi Jay,

Thanks for your interest. Our prototype intercepts Swift REST requests
using proxy server middleware and uses Solr as the indexing/search back
end.
Our search API is similar to the one proposed by HP.  Rather than propose
yet another API (we already have the HP proposal and the SoftLayer API out
there), we would be glad to participate in the discussion on search API,
functionality, and which parts should be open source and which vendor
specific. We haven't put our code out in the open yet but may be able to
contribute to a reference implementation.

In general we would propose that the API be general and allow different
vendors to implement subsets, for example by publishing which 
are implemented as in the HP proposal. So the SoftLayer API would be one
example subset.

Some things to consider as optional additions to the API (could be included
as optional  as in the HP proposal):
- allowing to search for accounts/containers/objects having certain
attribute names, where the attribute names could possibly have wildcards
- allowing user defined types with their own sort orders to be
indexed/searched
- allowing multiple search criteria where each one has a different scope
i.e. account, container etc., for example, search for objects in containers
created in 2013 whose color is red

regards
Paula



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] SSL Termination write-up

Excerpts from Vijay Venkatachalam's message of 2013-11-19 05:48:43 -0800:
> Hi Sam, Eugene, & Avishay, etal,
> 
> Today I spent some time to create a write-up for SSL 
> Termination not exactly design doc. Please share your comments!
> 
> https://docs.google.com/document/d/1tFOrIa10lKr0xQyLVGsVfXr29NQBq2nYTvMkMJ_inbo/edit
> 
> Would like comments/discussion especially on the following note:
> 
> SSL Termination requires certificate management. The ideal way is to handle 
> this via an independent IAM service. This would take time to implement so the 
> thought was to add the certificate details in VIP resource and send them 
> directly to device. Basically don't store the certificate key in the DB there 
> by avoiding security concerns of maintaining certificates in controller.
> 
> I would expect the certificates to become an independent resource in future 
> thereby causing backward compatibility issues.
> 

Perhaps Barbican can be leveraged for this, it seems that it was
specifically designed for the use case. Quoting from their README:

Design Goals

 1. Provide a central secret-store capable of distributing secret / keying 
material to all types of deployments including ephemeral Cloud instances.
 2. Support reasonable compliance regimes through reporting and auditability.
 3. Application adoption costs should be minimal or non-existent.
 4. Build a community and ecosystem by being open-source and extensible.
 5. Improve security through sane defaults and centralized management of 
policies for all secrets.
 6. Out of band communication mechanism to notify and protect sensitive assets.

https://github.com/stackforge/barbican

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] TaskFlow 0.1 integration

2013-11-19 Thread Walter A. Boring IV


Awesome guys,
  Thanks for picking this up.   I'm looking forward to the reviews :)

Walt

On 19.11.2013 10:38, Kekane, Abhishek wrote:

Hi All,

Greetings!!!

Hi there!

And thanks for your interest in cinder and taskflow!


We are in process of implementing the TaskFlow 0.1 in Cinder for "copy
volume to image" and "delete volume".

I have added two blueprints for the same.
1. https://blueprints.launchpad.net/cinder/+spec/copy-volume-to-image-task-flow
2. https://blueprints.launchpad.net/cinder/+spec/delete-volume-task-flow

I would like to know if any other developers/teams are working or
planning to work on any cinder api apart from above two api's.

Your help is appreciated.

Anastasia Karpinska works on updating existing flows to use released
TaskFlow 0.1.1 instead of internal copy:

https://review.openstack.org/53922

It was blocked because taskflow was not in openstack/requirements, but
now we're there, and Anastasia promised to finish the work and submit
updated changeset for review in couple of days.

There are also two changesets that convert cinder APIs to use TaskFlow:
- https://review.openstack.org/53480 for create_backup by Victor
   Rodionov
- https://review.openstack.org/55134 for create_snapshot by Stanislav
   Kudriashev

As far as I know both Stanislav and Victor suspended their work unitil
Anastasia's change lands.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

2013-11-19 Thread Peter Feiner

On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short  wrote:
> Hi
>
>
> On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner  wrote:
>>
>> A substantive reason for switching from mox to mock is the derelict
>> state of mox releases. There hasn't been a release of mox in three
>> years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
>> in the past 3 years, substantial bugs have been fixed in upstream mox.
>> For example, with the year-old fix to
>> https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
>> in nova would have been caught by an existing test [3].
>>
>> Alternatively, a copy of the upstream mox code could be added in-tree.
>>
> Please no, I think we are in an agreement with mox3 and mock.

That's cool. As long as the mox* is phased out, the false-positive
test results will be fixed.

Of course, there's _another_ alternative, which is to retrofit mox3
with the upstream mox fixes (e.g., the bug I cited above exists in
mox3). However, the delta between mox3 and upstream mox is pretty huge
(I just checked), so effort is probably better spent switching to
mock. To that end, I plan on changing the tests I cited above.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] TaskFlow 0.1 integration

Sweet!

Feel free to ask lots of questions and jump on both irc channels 
(#openstack-state-management and #openstack-cinder) if u need any help that can 
be better solved in real time chat.

Thanks for helping getting this ball rolling :-)

Sent from my really tiny device...

> On Nov 19, 2013, at 8:46 AM, "Walter A. Boring IV"  
> wrote:
> 
> Awesome guys,
>  Thanks for picking this up.   I'm looking forward to the reviews :)
> 
> Walt
>>> On 19.11.2013 10:38, Kekane, Abhishek wrote:
>>> Hi All,
>>> 
>>> Greetings!!!
>> Hi there!
>> 
>> And thanks for your interest in cinder and taskflow!
>> 
>>> We are in process of implementing the TaskFlow 0.1 in Cinder for "copy
>>> volume to image" and "delete volume".
>>> 
>>> I have added two blueprints for the same.
>>> 1. 
>>> https://blueprints.launchpad.net/cinder/+spec/copy-volume-to-image-task-flow
>>> 2. https://blueprints.launchpad.net/cinder/+spec/delete-volume-task-flow
>>> 
>>> I would like to know if any other developers/teams are working or
>>> planning to work on any cinder api apart from above two api's.
>>> 
>>> Your help is appreciated.
>> Anastasia Karpinska works on updating existing flows to use released
>> TaskFlow 0.1.1 instead of internal copy:
>> 
>> https://review.openstack.org/53922
>> 
>> It was blocked because taskflow was not in openstack/requirements, but
>> now we're there, and Anastasia promised to finish the work and submit
>> updated changeset for review in couple of days.
>> 
>> There are also two changesets that convert cinder APIs to use TaskFlow:
>> - https://review.openstack.org/53480 for create_backup by Victor
>>   Rodionov
>> - https://review.openstack.org/55134 for create_snapshot by Stanislav
>>   Kudriashev
>> 
>> As far as I know both Stanislav and Victor suspended their work unitil
>> Anastasia's change lands.
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Search Project - summit follow up

2013-11-19 Thread Dmitri Zimin(e) | StackStorm

Thanks Jarda for the intro;

Hi Stackers,

The project Search is a service providing fast full-text search for
resources across OpenStack services.

The idea was introduced and discussed at the Summit. We confirmed the
need for Search service: 1) from core Horizon team and UX 2) from
integrators building custom consoles 3) some suggested CLI use-cases.
We established it’s likely be a new service. I spoke with Julien @
Ceilometer about leveraging some overlapping functionality and he
suggested an integration point to address it.

Now let’s discuss it with the broader community and have you guys
guide it through the proper next steps.

DETAILS:
* Wiki - basic project description, and 2 min video showing Search in action.
  https://wiki.openstack.org/wiki/SearchProject
* Etherpad - captures Summit discussions:
https://etherpad.openstack.org/p/search
* Github - the rough prototype:
   * Search backend: https://github.com/StackStorm/search
   * Horizon plugin: https://github.com/StackStorm/search-horizon

QUESTIONS:

1) Is it relevant to OpenStack vision and directions?
--> yes, based on Summit discussions

2) What is a good home for it? A a new standalone program or a part of
existing program?
--> So far: Standalone program (input from David Lyle, Matthias Runge,
Doug Helmann)

3) Who else wants to be involved? Pigs? Chickens?

4) What do we do next?
--> Create an Launchpad?
--> Form the team?
--> Give it a name?
--> Begin flashing out the design?

Thanks to David Lyle, Mattihas Runge, Nicolas Simonds, Julien Danjou,
Noy Itzikowitz, Ryan Lane and Doug Helmann for your help and active
input.

Cheers, Dmitri.

On Tue, Nov 19, 2013 at 7:56 AM, Jaromir Coufal  wrote:
> Hey OpenStackers,
>
> I hope you all enjoyed Hong Kong Summit and for those who couldn't attend, I
> hope we will see each other next time!
>
> For all of you, I'd like to follow up on conversation from Summit about
> Search project [0], which Dmitri introduced and I believe it has great
> potential for enhancing OpenStack's user experience, especially in Horizon,
> but not just in there.
>
> I'd like to thank to Dmitri and all folks who were working on this project
> for sharing and their will for contributing.
>
> It would be great, if we can follow up on this topic, start discussion
> around the project here on mailing list (since not everybody could attend
> the summit) and gather some feedback. There are lot of questions like where
> to accommodate the code, what are next steps, etc.
>
> I am cc'ing Dmitri, so he can bring more light into this discussion and I
> hope we can move forward.
>
> Thanks
> -- Jarda
>
> [0] https://wiki.openstack.org/wiki/SearchProject

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Edgar Magana

Isaku,

Do you have in mind any implementation, any BP?
We could actually work on this together, all plugins will get the benefits
of a better implementation.

Thanks,

Edgar

On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:

>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
>Robert Kukura  wrote:
>
>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
>> > Developers,
>> > 
>> > This topic has been discussed before but I do not remember if we have
>>a
>> > good solution or not.
>> 
>> The ML2 plugin addresses this by calling each MechanismDriver twice. The
>> create_network_precommit() method is called as part of the DB
>> transaction, and the create_network_postcommit() method is called after
>> the transaction has been committed. Interactions with devices or
>> controllers are done in the postcommit methods. If the postcommit method
>> raises an exception, the plugin deletes that partially-created resource
>> and returns the exception to the client. You might consider a similar
>> approach in your plugin.
>
>Splitting works into two phase, pre/post, is good approach.
>But there still remains race window.
>Once the transaction is committed, the result is visible to outside.
>So the concurrent request to same resource will be racy.
>There is a window after pre_xxx_yyy before post_xxx_yyy() where
>other requests can be handled.
>
>The state machine needs to be enhanced, I think. (plugins need
>modification)
>For example, adding more states like pending_{create, delete, update}.
>Also we would like to consider serializing between operation of ports
>and subnets. or between operation of subnets and network depending on
>performance requirement.
>(Or carefully audit complex status change. i.e.
>changing port during subnet/network update/deletion.)
>
>I think it would be useful to establish reference locking policy
>for ML2 plugin for SDN controllers.
>Thoughts or comments? If this is considered useful and acceptable,
>I'm willing to help.
>
>thanks,
>Isaku Yamahata
>
>> -Bob
>> 
>> > Basically, if concurrent API calls are sent to Neutron, all of them
>>are
>> > sent to the plug-in level where two actions have to be made:
>> > 
>> > 1. DB transaction ? No just for data persistence but also to collect
>>the
>> > information needed for the next action
>> > 2. Plug-in back-end implementation ? In our case is a call to the
>>python
>> > library than consequentially calls PLUMgrid REST GW (soon SAL)
>> > 
>> > For instance:
>> > 
>> > def create_port(self, context, port):
>> > with context.session.begin(subtransactions=True):
>> > # Plugin DB - Port Create and Return port
>> > port_db = super(NeutronPluginPLUMgridV2,
>> > self).create_port(context,
>> >   
>> port)
>> > device_id = port_db["device_id"]
>> > if port_db["device_owner"] == "network:router_gateway":
>> > router_db = self._get_router(context, device_id)
>> > else:
>> > router_db = None
>> > try:
>> > LOG.debug(_("PLUMgrid Library: create_port() called"))
>> > # Back-end implementation
>> > self._plumlib.create_port(port_db, router_db)
>> > except Exception:
>> > 
>> > 
>> > The way we have implemented at the plugin-level in Havana (even in
>> > Grizzly) is that both action are wrapped in the same "transaction"
>>which
>> > automatically rolls back any operation done to its original state
>> > protecting mostly the DB of having any inconsistency state or left
>>over
>> > data if the back-end part fails.=.
>> > The problem that we are experiencing is when concurrent calls to the
>> > same API are sent, the number of operation at the plug-in back-end are
>> > long enough to make the next concurrent API call to get stuck at the
>>DB
>> > transaction level, which creates a hung state for the Neutron Server
>>to
>> > the point that all concurrent API calls will fail.
>> > 
>> > This can be fixed if we include some "locking" system such as calling:
>> > 
>> > from neutron.common import utile
>> > 
>> > 
>> > @utils.synchronized('any-name', external=True)
>> > def create_port(self, context, port):
>> > 
>> > 
>> > Obviously, this will create a serialization of all concurrent calls
>> > which will ends up in having a really bad performance. Does anyone
>>has a
>> > better solution?
>> > 
>> > Thanks,
>> > 
>> > Edgar
>> > 
>> > 
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > 
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>-- 
>Isaku Yamahata 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listin

Re: [openstack-dev] [nova][heat][[keystone] RFC: introducing "request identification"

2013-11-19 Thread Adam Young


On 11/19/2013 08:21 AM, Dolph Mathews wrote:

Related BP:

  Create a unified request identifier
https://blueprints.launchpad.net/nova/+spec/cross-service-request-id


And we have discussed  workplans as well, which would be data to be 
carried along with the request id


https://blueprints.launchpad.net/keystone/+spec/delegation-workplans
https://etherpad.openstack.org/keystone_workplans



On Tue, Nov 19, 2013 at 5:04 AM, haruka tanizawa > wrote:


Hi stackers!!

I'd like to ask for your opinions about my idea of identifying
request.

Challenges
==

We have no way to know the final result of an API request.
Indeed we can continuously get the status of allocated resources,
but this is just resource status, not request status.

It doesn't matter so much for manual operations.
But it does for automated clients like heat.
We need request-oriented-status and it should be disclosed to clients.

Literally, we need to address two items for it.
 1. how to identify request from clients
 2. how to disclose status of request to clients

Note that this email includes only 1 for initiating the discussion.
Also, bp:instance-tasks-api[0] should include both two items above.

Proposal: introducing "request identification"
=

I'd like to introduce "request identification", which is disclosed
to clients.
There are two characteristics:

 - "request identification" is unique ID for each request
   so that we can identify tell a specific request from others.
 - "request identification" is available for clients
   so that we can enable any after-request-operations like check,
retry[1] or cancel[2].
   (of course we need to add more logic for check/retry/cancel,
but I'm pretty sure that indentifying request is the starting
point for these features)

In my understandings, main objections should be 'who should
generate and maintain such IDs?'.

How to implement "request identification"
=

There are several options at least. (See also recent discussion at
openstack-dev[3])

1. Enable user to provide his/her own "request identification"
within API request.
   This should be the simplest way. But providing id from outside
looks hard to be accepted.
   For example, Idempotency for OpenStack API[4].
   Or instance-tasks-api enable to user to put his/her own
"request identification".

2. Use correlation_id in oslo as "request identification".
   correlation_id looks similar concept of "request indentification",
   but correlation_id in nova was already rejected[5].

3. Enable keystone to generate "request identification" (we can
call it 'request-token', for example).
   Before sending actual API request to nova-api, client sends a
request to keystone to get 'request-token'.


-2

   Then client sends an actual API request with 'request-token'.
   Nova-api will consult it to keystone whether it was really
generated.
   Sounds like a auth-token generated by keystone, differences are:
 [lifecycle] auth-token is used for multiple API requests
before it expires,
'request-token' is used for only single API request.
 [reusing] if the same 'request-token' was specified twice or
more times,
nova-api simply returns 20x (works like client token in
AWS[6]).
Keystone needs to maintain 'request-tokens' until they expire.
   For backward compatibility, actual API request without
'request-token' should work as before.
   We can consider several options for uniqueness of 'request-token':
 uuid, any string with uniqueness per tennant, etc.

IMO, since adding new implementation to Keystone is a little bit
hard work,
so implement of 1 is reasonable for me, just idea.

Any comments will be appreciated.

Sincerely, Haruka Tanizawa

[0] https://blueprints.launchpad.net/nova/+spec/instance-tasks-api
[1] https://wiki.openstack.org/wiki/Support-retry-with-idempotency
[2] https://blueprints.launchpad.net/nova/+spec/cancel-swap-volume
[3]
http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg09023.html
[4]
https://blueprints.launchpad.net/nova/+spec/idempotentcy-client-token
[5] https://review.openstack.org/#/c/29480/
[6]

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--

-Dolph


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openst

Re: [openstack-dev] [Ironic][Ceilometer] get IPMI data for ceilometer

2013-11-19 Thread Doug Hellmann

On Mon, Nov 18, 2013 at 8:58 PM, Lu, Lianhao  wrote:

>
> Doug Hellmann wrote on 2013-11-19:
> >
> > On Mon, Nov 18, 2013 at 12:25 PM, Devananda van der Veen <
> devananda@gmail.com> wrote:
> >
> >
> >   Hi Lianhao Lu,
> >
> >   I briefly summarized my recollection of that session in this
> blueprint:
> >
> >   https://blueprints.launchpad.net/ironic/+spec/add-ceilometer-agent
> >
> >
> >   I've responded to your questions inline as well.
> >
> >
> >   On Sun, Nov 17, 2013 at 10:24 PM, Lu, Lianhao <
> lianhao...@intel.com> wrote:
> >
> >
> >   Hi stackers,
> >
> >   During the summit session Expose hardware sensor (IPMI)
> data
> >
> https://etherpad.openstack.org/p/icehouse-summit-ceilometer-hardware-sensors,
> it was proposed to deploy a ceilometer agent next to
> > the ironic conductor to the get the ipmi data. Here I'd like to ask some
> questions to figure out what's the current missing pieces in ironic
> > and ceilometer for that proposal.
> >
> >   1. Just double check, ironic won't provide API to get IPMI
> data, right?
> >
> >
> >
> >   Correct. This was generally felt to be unnecessary.
> >
> >
> >   2. If deploying a ceilometer agent next to the ironic
> conductor, how does the agent talk to the conductor? Through rpc?
> >
> >
> >
> >   My understanding is that ironic-conductor will emit messages to
> the ceilimeter agent, and the communication is one-way. These could
> > be triggered by a periodic task, or by some other event within Ironic,
> such as a change in the power state of a node.
> >
> >
> > Cool, so this eliminates the need for a separate agent. The ceilometer
> work can be done in the collector, to receive the new messages.
> >
> Does this means we lose the ability to specify the different polling
> interval for different monitoring data, like we have in ceilometer pipeline?
>

It means that would need to be implemented in the ironic code that is doing
the polling. 

Doug


>
> >
> >   3. Does the current ironic conductor have rpc_method to
> support getting generic ipmi data, i.e. let the rpc_method caller
> > specifying arbitrary netfn/command to get any type of ipmi data?
> >
> >
> >
> >   No, and as far as I understand, it doesn't need one.
> >
> >
> > It would only need that if we were going to poll for the data, but if
> ironic is emitting notifications then we don't have to do that.
> >
> >   4. I believe the ironic conductor uses some kind of
> node_id to associate the bmc with its credentials, right? If so, how can the
> > ceilometer agent get those node_ids to ask the ironic conductor to poll
> the ipmi data? And how can the ceilometer agent extract
> > meaningful information from that node_id to set those fields in the
> ceilometer Sample(e.g. recource_id, project_id, user_id, etc.) to identify
> > which physical node the ipmi data is coming from?
> >
> >   This question perhaps requires a longer answer.
> >
> >   Ironic references physical machines (nodes) internally with an
> integer node_id and externally with a standard uuid. When a Nova
> > instance is created, it will be associated to a node, that node will
> have a reference to the nova instance_uuid which is exposed in our API,
> > and can be passed to Ceilometer's agent. I believe that nova
> instance_uuid will enable ceilometer to detect the project, user, etc.
> >
> >
> > If ironic has those values (at least the project), and can include them
> in the notification payload, that will make processing the incoming
> > notifications significantly less expensive.
> >
> >
> >   Should Ironic emit messages regarding nodes which are not
> provisioned? Physical machines that don't have a tenant instance on them
> > are not associated to any project, user, tenant, quota, etc, so I
> suspect that we shouldn't notify about them. It would be like tracking the
> > unused disks in a SAN.
> >
> >
> > I don't think it would be useful, but if someone else does then it seems
> OK to include them.
>
> I think it'd better for Ironic to emit those data in case some users want
> to collect them, at least Ironic should have a configuration setting to
> emit those kind of data.
>
> -Lianhao
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-security] Neutron security groups for L2 networks in Havana

2013-11-19 Thread Aaron Rosen

Hi Kanthi,

The issue is that the nova-api says that by default every instance needs to
be in a default security group that blocks all ingress traffic from outside
and allows all ingress from instances that belong to the same security
group. If an instance does not have an ip address we are unable to enforce
the instance->instance communication of members that are part of the same
security group (or at least the iptables implementation in place doesn't).

There is an extension in neutron that implements port_security that allows
one to disable this so that one can do L2 networking as you want to. Though
unfortunately, this does not work correctly at this time with nova as
currently it's calling into neutron to create the security group and attach
it to the instance anways. See: https://bugs.launchpad.net/nova/+bug/1175464 .
I'm hoping to resolve this issue in the next few weeks.

Best,

Aaron


On Fri, Nov 8, 2013 at 1:45 AM, Kanthi P  wrote:

> Hi Aaron,
>
> Thanks for the reply !
> Yes security groups are mapped to L3/L4 headers, these security rules are
> being converted to iptables.
>
> If we remove the subnet check, we will be able to launch instances without
> ipaddress, they just have the mac address.
> Users can configure their own ipaddress after they are booted.
>
> If we use neutron security groups, it provides a default group (on port
> basis) which blocks all ingress and allows all egress traffic.
>
> Later users can configure security groups based on the ip address what
> they provided to the vnics.
>
> I mean to say, ports will have subnet but just that this subnet is not
> known to openstack during the instance boot time.
>
>
>
>
> On Fri, Nov 8, 2013 at 9:42 AM, Aaron Rosen  wrote:
>
>>
>>
>>
>> On Thu, Nov 7, 2013 at 12:23 PM, Kanthi P wrote:
>>
>>> Hi,
>>>
>>> I am trying to boot a VM which has a network without subnet in Havana,
>>> but it throws an exception saying that subnet is mandatory if quantum
>>> security groups are enabled.
>>>
>>> Here are the commands I used.
>>>
>>> neutron net-create testNet
>>> neutron net-show testNet
>>> +---+--+
>>> | Field | Value|
>>> +---+--+
>>> | admin_state_up| True |
>>> | id| 47208beb-2801-4729-bc7b-6532717232fc |
>>> | name  | testNet  |
>>> | provider:network_type | local|
>>> | provider:physical_network |  |
>>> | provider:segmentation_id  |  |
>>> | router:external   | False|
>>> | shared| False|
>>> | status| ACTIVE   |
>>> | subnets   |  |
>>> | tenant_id | b5b591dcda2645cd9d15a4fe3eb1b50d |
>>> +---+--+
>>>
>>> nova boot --flavor 2 --image 30c1984c-8a4f-4e3f-8c9a-7de0b321caa2 --nic
>>> net-id=47208beb-2801-4729-bc7b-6532717232fc testServer1
>>>
>>> Here is the piece of code which generated this exception
>>>
>>> nova/network/neutronv2/api.py
>>>
>>> if (security_groups and not (
>>> network['subnets']
>>> and network.get('port_security_enabled', True))):
>>>
>>> raise exception.SecurityGroupCannotBeApplied()
>>>
>>>
>>> Can someone please explain why do we need this check?
>>>
>>
>> Hi Kanthi,
>>
>> We need this check because because in order to enforce security groups
>> the port needs to have an ip_address (i.e: network needs a subnet) since
>> Security groups only map to L3/4 headers. Today, nova automatically adds a
>> default security group to all instances when booted. Hopefully we can punt
>> this task off to neutron in this release by moving the port-creation up on
>> the stack to nova-api instead of nova-compute though this isn't the case
>> right now.
>>
>> Aaron
>>
>>>
>>> To my understanding subnet is used for two purposes in terms of security
>>> groups
>>> 1. To allow dhcp traffic if dhcp is enabled on the subnet, example given
>>> below
>>> -A quantum-openvswi-i7bf776d2-b -s 192.168.1.3/32 -p udp -m udp
>>> --sport 67 --dport 68 -j RETURN
>>> 2. To avoid ip spoof
>>> -A quantum-openvswi-o7bf776d2-b ! -s 192.168.1.2/32 -j DROP
>>>
>>> Can we remove this so that we can have guests which has nic with just
>>> MAC address, guest can configure its own ipaddress. Later if needed they
>>> can enable their own security rules via quantum api.
>>>
>>> ___
>>> Openstack-security mailing list
>>> openstack-secur...@lists.openstack.org
>>> http://lists.op

Re: [openstack-dev] [Swift] Metadata Search API

2013-11-19 Thread Jay Pipes


On 11/19/2013 11:33 AM, Paula Ta-Shma wrote:



Hi Paula,

Where can we see the source code for the prototype and any API specs
that you may have?

Best,
-jay


Hi Jay,

Thanks for your interest. Our prototype intercepts Swift REST requests
using proxy server middleware and uses Solr as the indexing/search back
end.


OK, sounds interesting.


Our search API is similar to the one proposed by HP.


My apologies, I'm apparently coming into this quite late :) Would you 
mind sharing a link to the HP proposal? I wasn't at the summit 
unfortunately and am playing a bit of catch up.


> Rather than propose

yet another API (we already have the HP proposal and the SoftLayer API out
there), we would be glad to participate in the discussion on search API,
functionality, and which parts should be open source and which vendor
specific. We haven't put our code out in the open yet but may be able to
contribute to a reference implementation.


The FLOSSian in me encourages you and your team to share early and 
release often. Open code breeds innovation! :)



In general we would propose that the API be general and allow different
vendors to implement subsets, for example by publishing which 
are implemented as in the HP proposal. So the SoftLayer API would be one
example subset.


K, sounds good. If there's already a good API proposal, I totally agree 
with you on not creating another one and instead working to improve 
existing ones if necessary.



Some things to consider as optional additions to the API (could be included
as optional  as in the HP proposal):
- allowing to search for accounts/containers/objects having certain
attribute names, where the attribute names could possibly have wildcards
- allowing user defined types with their own sort orders to be
indexed/searched
- allowing multiple search criteria where each one has a different scope
i.e. account, container etc., for example, search for objects in containers
created in 2013 whose color is red


++

Best,
-jay


regards
Paula



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Core pinning

2013-11-19 Thread yunhong jiang

On Tue, 2013-11-19 at 12:52 +, Daniel P. Berrange wrote:
> On Wed, Nov 13, 2013 at 02:46:06PM +0200, Tuomas Paappanen wrote:
> > Hi all,
> > 
> > I would like to hear your thoughts about core pinning in Openstack.
> > Currently nova(with qemu-kvm) supports usage of cpu set of PCPUs
> > what can be used by instances. I didn't find blueprint, but I think
> > this feature is for isolate cpus used by host from cpus used by
> > instances(VCPUs).
> > 
> > But, from performance point of view it is better to exclusively
> > dedicate PCPUs for VCPUs and emulator. In some cases you may want to
> > guarantee that only one instance(and its VCPUs) is using certain
> > PCPUs.  By using core pinning you can optimize instance performance
> > based on e.g. cache sharing, NUMA topology, interrupt handling, pci
> > pass through(SR-IOV) in multi socket hosts etc.
> > 
> > We have already implemented feature like this(PoC with limitations)
> > to Nova Grizzly version and would like to hear your opinion about
> > it.
> > 
> > The current implementation consists of three main parts:
> > - Definition of pcpu-vcpu maps for instances and instance spawning
> > - (optional) Compute resource and capability advertising including
> > free pcpus and NUMA topology.
> > - (optional) Scheduling based on free cpus and NUMA topology.
> > 
> > The implementation is quite simple:
> > 
> > (additional/optional parts)
> > Nova-computes are advertising free pcpus and NUMA topology in same
> > manner than host capabilities. Instances are scheduled based on this
> > information.
> > 
> > (core pinning)
> > admin can set PCPUs for VCPUs and for emulator process, or select
> > NUMA cell for instance vcpus, by adding key:value pairs to flavor's
> > extra specs.
> > 
> > EXAMPLE:
> > instance has 4 vcpus
> > :
> > vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2...
> > emulator:5 --> emulator pinned to pcpu5
> > or
> > numacell:0 --> all vcpus are pinned to pcpus in numa cell 0.
> > 
> > In nova-compute, core pinning information is read from extra specs
> > and added to domain xml same way as cpu quota values(cputune).
> > 
> > 
> >   
> >   
> >   
> >   
> >   
> > 
> > 
> > What do you think? Implementation alternatives? Is this worth of
> > blueprint? All related comments are welcome!
> 
> I think there are several use cases mixed up in your descriptions
> here which should likely be considered independantly
> 
>  - pCPU/vCPU pinning
> 
>I don't really think this is a good idea as a general purpose
>feature in its own right. It tends to lead to fairly inefficient
>use of CPU resources when you consider that a large % of guests
>will be mostly idle most of the time. It has a fairly high
>administrative burden to maintain explicit pinning too. This
>feels like a data center virt use case rather than cloud use
>case really.
> 
>  - Dedicated CPU reservation
> 
>The ability of an end user to request that their VM (or their
>group of VMs) gets assigned a dedicated host CPU set to run on.
>This is obviously something that would have to be controlled
>at a flavour level, and in a commercial deployment would carry
>a hefty pricing premium.
> 
>I don't think you want to expose explicit pCPU/vCPU placement
>for this though. Just request the high level concept and allow
>the virt host to decide actual placement
> 
>  - Host NUMA placement.
> 
>By not taking NUMA into account currently the libvirt driver
>at least is badly wasting resources. Having too much cross-numa
>node memory access by guests just kills scalability. The virt
>driver should really automaticall figure out cpu & memory pinning
>within the scope of a NUMA node automatically. No admin config
>should be required for this.
> 
>  - Guest NUMA topology
> 
>If the flavour memory size / cpu count exceeds the size of a
>single NUMA node, then the flavour should likely have a way to
>express that the guest should see multiple NUMA nodes. The
>virt host would then set guest NUMA topology to match the way
>it places vCPUs & memory on host NUMA nodes. Again you don't
>want explicit pcpu/vcpu mapping done by the admin for this.
> 
> 
> 
> Regards,
> Daniel

Quite clear splitting and +1 for P/V pin option.

--jyh



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?


On 11/18/2013 06:47 PM, Joshua Harlow wrote:

An idea related to this, what would need to be done to make the DB have
the exact state that a compute node is going through (and therefore the
scheduler would not make unreliable/racey decisions, even when there are
multiple schedulers). It's not like we are dealing with a system which
can not know the exact state (as long as the compute nodes are connected
to the network, and a network partition does not occur).


How would you synchronize the various schedulers with each other? 
Suppose you have multiple scheduler nodes all trying to boot multiple 
instances each.


Even if each at the start of the process each scheduler has a perfect 
view of the system, each scheduler would need to have a view of what 
every other scheduler is doing in order to not make racy decisions.


I see a few options:

1) Push scheduling down into the database itself.  Implement scheduler 
filters as SQL queries or stored procedures.


2) Get rid of the DB for scheduling.  It looks like people are working 
on this: https://blueprints.launchpad.net/nova/+spec/no-db-scheduler


3) Do multi-stage scheduling.  Do a "tentative" schedule, then try and 
update the DB to reserve all the necessary resources.  If that fails, 
someone got there ahead of you so try again with the new data.


Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Reg : Security groups implementation using openflows in quantum ovs plugin

2013-11-19 Thread Amir Sadoughi

Yes, my work has been on ML2 with neutron-openvswitch-agent.  I’m interested to 
see what Jun Park has. I might have something ready before he is available 
again, but would like to collaborate regardless.

Amir

On Nov 19, 2013, at 3:31 AM, Kanthi P 
mailto:pavuluri.kan...@gmail.com>> wrote:

Hi All,

Thanks for the response!
Amir,Mike: Is your implementation being done according to ML2 plugin

Regards,
Kanthi

On Tue, Nov 19, 2013 at 1:43 AM, Mike Wilson 
mailto:geekinu...@gmail.com>> wrote:
Hi Kanthi,

Just to reiterate what Kyle said, we do have an internal implementation using 
flows that looks very similar to security groups. Jun Park was the guy that 
wrote this and is looking to get it upstreamed. I think he'll be back in the 
office late next week. I'll point him to this thread when he's back.

-Mike

On Mon, Nov 18, 2013 at 3:39 PM, Kyle Mestery (kmestery) 
mailto:kmest...@cisco.com>> wrote:
On Nov 18, 2013, at 4:26 PM, Kanthi P 
mailto:pavuluri.kan...@gmail.com>> wrote:
> Hi All,
>
> We are planning to implement quantum security groups using openflows for ovs 
> plugin instead of iptables which is the case now.
>
> Doing so we can avoid the extra linux bridge which is connected between the 
> vnet device and the ovs bridge, which is given as a work around since ovs 
> bridge is not compatible with iptables.
>
> We are planning to create a blueprint and work on it. Could you please share 
> your views on this
>
Hi Kanthi:

Overall, this idea is interesting and removing those extra bridges would 
certainly be nice. Some people at Bluehost gave a talk at the Summit [1] in 
which they explained they have done something similar, you may want to reach 
out to them since they have code for this internally already.

The OVS plugin is in feature freeze during Icehouse, and will be deprecated in 
favor of ML2 [2] at the end of Icehouse. I would advise you to retarget your 
work at ML2 when running with the OVS agent instead. The Neutron team will not 
accept new features into the OVS plugin anymore.

Thanks,
Kyle

[1] 
http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/towards-truly-open-and-commoditized-software-defined-networks-in-openstack
[2] https://wiki.openstack.org/wiki/Neutron/ML2

> Thanks,
> Kanthi
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

2013-11-19 Thread Caitlin Bestler


On 11/18/2013 11:35 AM, Mike Spreitzer wrote:

There were some concerns expressed at the summit about scheduler
scalability in Nova, and a little recollection of Boris' proposal to
keep the needed state in memory.  I also heard one guy say that he
thinks Nova does not really need a general SQL database, that a NOSQL
database with a bit of denormalization and/or client-maintained
secondary indices could suffice.  Has that sort of thing been considered
before?  What is the community's level of interest in exploring that?

Thanks,
Mike



How the data is stored is not the central question. The real issue is 
how is the data normalized and distributed.


Data that is designed to be distributed deals with temporary 
inconsistencies and only worries about eventual consistency.

Once you have that you can store the data in Objects, or in
a distributed database.

If you define your data so that you need global synchronization then
you will always be fighting scaling issues.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] - Vendor specific erros

2013-11-19 Thread Salvatore Orlando

Thanks Avishay.

I think the status description error was introduced with this aim.
Whether vendor-specific error descriptions can make sense to a tenant,
that's a good question.
Personally, I feel like as a tenant that information would not be a lot
useful to me, as I would not be able to do any debug or maintenance on the
appliance where the error was generated; on the other hand, as a deployer I
might find that message very useful, but probably I would look for it in
the logs rather than in API responses; furthermore, as a deployer I might
find more convenient to not provide tenants any detail about the peculiar
driver being used.

On this note however, this is just my personal opinion. I'm sure there are
plenty of valid use cases for giving tenants vendor-specific error messages.

Salvatore


On 19 November 2013 13:00, Avishay Balderman  wrote:

>  Hi Salvatore
>
> I think you are mixing between the state machine (ACTIVE,PENDEING_XYZ,etc)
>  and the status description
>
> All I want to do is to write a vendor specific error message when the
> state is ERROR.
>
> I DO NOT want to touch the state machine.
>
>
>
> See: https://bugs.launchpad.net/neutron/+bug/1248423
>
>
>
> Thanks
>
>
>
> Avishay
>
>
>
> *From:* Salvatore Orlando [mailto:sorla...@nicira.com]
> *Sent:* Thursday, November 14, 2013 1:15 PM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [Neutron] - Vendor specific erros
>
>
>
> In general, an error state make sense.
>
> I think you might want to send more details about how this state plugs
> into the load balancer state machine, but I reckon it is a generally
> non-recoverable state which could be reached by any other state; in that
> case it would be a generic enough case which might be supported by all
> drivers.
>
>
>
> It is good to point out that driver-specific state transitions however, in
> my opinion, are to avoid; application using the Neutron API will become
> non-portable, or at least users of the Neutron API would need to be aware
> that an entity might have a different state machine from driver to driver,
> which I reckon would be bad enough for a developer to decide to switch over
> to Cloudstack or AWS APIs!
>
>
>
> Salvatore
>
>
>
> PS: On the last point I am obviously joking, but not so much.
>
>
>
>
>
> On 12 November 2013 08:00, Avishay Balderman  wrote:
>
>
>
> Hi
>
> Some of the DB entities in the LBaaS domain inherit from
> HasStatusDescription
>
> With this we can set the entity status (ACTIVE, PENDING_CREATE,etc) and a
> description for the status.
>
> There are flows in the Radware LBaaS driver that the  driver needs to set
> the entity status to ERROR and it is able to set the description of the
> error –  the description is Radware specific.
>
> My question is:  Does it make sense to do that?
>
> After all the tenant is aware to the fact that he works against Radware
> load balancer -  the tenant selected Radware as the lbaas provider in the
> UI.
>
> Any reason not to do that?
>
>
>
> This is a generic issue/question and does not relate  to a specific plugin
> or driver.
>
>
>
> Thanks
>
>
>
> Avishay
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread James Bottomley

On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
> Hey all
> 
> Not having been at the summit (maybe the next one), could somebody
> give a really short explanation as to why it needs to be a separate
> service?
> It sounds like it should fit within the Nova area. It is, after all,
> just another hypervisor type, or so it seems.

I can take a stab at this:  Firstly, a container is *not* a hypervisor.
Hypervisor based virtualisation is done at the hardware level (so with
hypervisors you boot a second kernel on top of the virtual hardware),
container based virtualisation is done at the OS (kernel) level (so all
containers share the same kernel ... and sometimes even huge chunks of
the OS). With recent advances in the Linux Kernel, we can make a
container behave like a hypervisor (full OS/IaaS virtualisation), but
quite a bit of the utility of containers is that they can do much more
than hypervisors, so they shouldn't be constrained by hypervisor APIs
(which are effectively virtual hardware APIs).

It is possible to extend the Nova APIs to control containers more fully,
but there was resistance do doing this on the grounds that it's
expanding the scope of Nova, hence the new project.

James

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration

Excerpts from Zane Bitter's message of 2013-11-15 12:41:53 -0800:
> Good news, everyone! I have created the missing whiteboard diagram that 
> we all needed at the design summit:
> 
> https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram
> 
> I've documented 5 possibilities. (1) is the current implementation, 
> which we agree we want to get away from. I strongly favour (2) for the 
> reasons listed. I don't think (3) has many friends. (4) seems to be 
> popular despite the obvious availability problem and doubts that it is 
> even feasible. Finally, I can save us all some time by simply stating 
> that I will -2 on sight any attempt to implement (5).
> 
> When we're discussing this, please mention explicitly the number of the 
> model you are talking about at any given time.
> 
> If you have a suggestion for a different model, make your own diagram! 
> jk, you can sketch it or something for me and I'll see if I can add it.

Thanks for putting this together Zane. I just now got around to looking
closely.

Option 2 is good. I'd love for option 1 to be made automatic by making
the client smarter, but parsing templates in the client will require
some deep thought before we decide it is a good idea.

I'd like to consider a 2a, which just has the same Heat engines the user
is talking to being used to do the orchestration in whatever region
they are in. I think that is actually the intention of the diagram,
but it looks like there is a "special" one that talks to the engines
that actually do the work.

2 may morph into 3 actually, if users don't like the nested stack
requirement for 2, we can do the work to basically make the engine create
a nested stack per region. So that makes 2 a stronger choice for first
implementation.

4 has an unstated pro, which is that attack surface is reduced. This
makes more sense when you consider the TripleO case where you may want
the undercloud (hardware cloud) to orchestrate things existing in the
overcloud (vm cloud) but you don't want the overcloud administrators to
be able to control your entire stack.

Given CAP theorem, option 5, the global orchestrator, would be doable
with not much change as long as partition tolerance were the bit we gave
up. We would just have to have a cross-region RPC bus and database. Of
course, since regions are most likely to be partitioned, that is not
really a good choice. Trading partition tolerance for consistency lands
us in the complexity black hole. Trading out availability makes it no
better than option 4.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Swift] Metadata Search API

2013-11-19 Thread Paula Ta-Shma


> My apologies, I'm apparently coming into this quite late :) Would you
> mind sharing a link to the HP proposal? I wasn't at the summit
> unfortunately and am playing a bit of catch up.

Hi Jay,

Take a look at https://wiki.openstack.org/wiki/MetadataSearch
There are links there to the REST API proposed by HP and the slides
presented at the summit.

regards
Paula



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] How to determine patch set load for a given project

2013-11-19 Thread Ilya Shakhat

Matt,

As an option you may estimate the load using Stackalytics data on number of
commits -
http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=sqlalchemy-migrateNumber
of commits is certainly less than number of patches, but for project
sqlalchemy-migrate the multiplier 2 will give a good estimation.

Thanks,
Ilya


2013/11/19 Matt Riedemann 

> We have a team working on getting CI setup for DB2 10.5 in
> sqlalchemy-migrate and they were asking me if there was a way to calculate
> the patch load through that project.
>
> I asked around in the infra IRC channel and Jeremy Stanley pointed out
> that there might be something available in http://graphite.openstack.org/by 
> looking for the project's test stats.
>
> I found that if you expand stats_counts > zuul > job and then search for
> your project (sqlalchemy-migrate in this case), you can find the jobs and
> their graphs for load. In my case I care about stats for
> gate-sqlalchemy-migrate-python27.
>
> I'm having a little trouble interpreting the data though. From looking at
> what's out there for review now, there is one new patch created on 11/19
> and the last new one before that was on 11/15. I see spikes in the graph
> around 11/15, 11/18 and 11/19, but I'm not sure what the 11/18 spike is
> from?
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

2013-11-19 Thread Christopher Armstrong

On Mon, Nov 18, 2013 at 5:57 AM, Zane Bitter  wrote:

> On 16/11/13 11:15, Angus Salkeld wrote:
>
>> On 15/11/13 08:46 -0600, Christopher Armstrong wrote:
>>
>>> On Fri, Nov 15, 2013 at 3:57 AM, Zane Bitter  wrote:
>>>
>>>  On 15/11/13 02:48, Christopher Armstrong wrote:

  On Thu, Nov 14, 2013 at 5:40 PM, Angus Salkeld  > wrote:
>
> On 14/11/13 10:19 -0600, Christopher Armstrong wrote:
>
> http://docs.heatautoscale.__apiary.io/
>
> 
>
> I've thrown together a rough sketch of the proposed API for
> autoscaling.
> It's written in API-Blueprint format (which is a simple subset
> of Markdown)
> and provides schemas for inputs and outputs using JSON-Schema.
> The source
> document is currently at
> https://github.com/radix/heat/__raw/as-api-spike/
> autoscaling.__apibp
>
>
>  >
>
>
> Things we still need to figure out:
>
> - how to scope projects/domains. put them in the URL? get them
> from the
> token?
> - how webhooks are done (though this shouldn't affect the API
> too much;
> they're basically just opaque)
>
> Please read and comment :)
>
>
> Hi Chistopher
>
> In the group create object you have 'resources'.
> Can you explain what you expect in there? I thought we talked at
> summit about have a unit of scaling as a nested stack.
>
> The thinking here was:
> - this makes the new config stuff easier to scale (config get
> applied
> Â  per scaling stack)
>
> - you can potentially place notification resources in the scaling
> Â  stack (think marconi message resource - on-create it sends a
> Â  message)
>
> - no need for a launchconfig
> - you can place a LoadbalancerMember resource in the scaling stack
> Â  that triggers the loadbalancer to add/remove it from the lb.
>
>
> I guess what I am saying is I'd expect an api to a nested stack.
>
>
> Well, what I'm thinking now is that instead of "resources" (a
> mapping of
> resources), just have "resource", which can be the template definition
> for a single resource. This would then allow the user to specify a
> Stack
> resource if they want to provide multiple resources. How does that
> sound?
>
>
 My thought was this (digging into the implementation here a bit):

 - Basically, the autoscaling code works as it does now: creates a
 template
 containing OS::Nova::Server resources (changed from AWS::EC2::Instance),
 with the properties obtained from the LaunchConfig, and creates a
 stack in
 Heat.
 - LaunchConfig can now contain any properties you like (I'm not 100%
 sure
 about this one*).
 - The user optionally supplies a template. If the template is
 supplied, it
 is passed to Heat and set in the environment as the provider for the
 OS::Nova::Server resource.


  I don't like the idea of binding to OS::Nova::Server specifically for
>>> autoscaling. I'd rather have the ability to scale *any* resource,
>>> including
>>> nested stacks or custom resources. It seems like jumping through hoops to
>>>
>>
>> big +1 here, autoscaling should not even know what it is scaling, just
>> some resource. solum might want to scale all sorts of non-server
>> resources (and other users).
>>
>
> I'm surprised by the negative reaction to what I suggested, which is a
> completely standard use of provider templates. Allowing a user-defined
> stack of resources to stand in for an unrelated resource type is the entire
> point of providers. Everyone says that it's a great feature, but if you try
> to use it for something they call it a "hack". Strange.
>

To clarify this position (which I already did in IRC), replacing one
concrete resource with another that means something in a completely
different domain is a hack -- say, replacing "server" with "group of
related resources". However, replacing OS::Nova::Server with something
which still does something very much like creating a server is reasonable
-- e.g., using a different API like one for creating containers or using a
different cloud provider's API.


>
> So, allow me to make a slight modification to my proposal:
>
> - The autoscaling service manages a template containing
> OS::Heat::ScaledResource resources. This is an imaginary resource type that
> is not backed by a plugin in Heat.
> - If no template is supplied by the user, the environment declares another
> resource plugin as the provider for OS::Heat::ScaledResource (by default it
> would be OS::Nova::Serv

Re: [openstack-dev] [Swift] Metadata Search API

2013-11-19 Thread Jay Pipes


On 11/19/2013 01:08 PM, Paula Ta-Shma wrote:



My apologies, I'm apparently coming into this quite late :) Would you
mind sharing a link to the HP proposal? I wasn't at the summit
unfortunately and am playing a bit of catch up.


Hi Jay,

Take a look at https://wiki.openstack.org/wiki/MetadataSearch
There are links there to the REST API proposed by HP and the slides
presented at the summit.


Thank you, Paula!

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Personally I would prefer #3 from the below. #2 I think will still have to
deal with consistency issues, just switching away from a DB doesn't make
magical ponies and unicorns appear (in-fact it can potentially make the
problem worse if its done incorrectly - and its pretty easy to get it
wrong IMHO). #1 could also work, but then u hit a vertical scaling limit
(works if u paid oracle for there DB or IBM for DB2 I suppose). I prefer
#2 since I think it is honestly needed under all solutions.

On 11/19/13 9:29 AM, "Chris Friesen"  wrote:

>On 11/18/2013 06:47 PM, Joshua Harlow wrote:
>> An idea related to this, what would need to be done to make the DB have
>> the exact state that a compute node is going through (and therefore the
>> scheduler would not make unreliable/racey decisions, even when there are
>> multiple schedulers). It's not like we are dealing with a system which
>> can not know the exact state (as long as the compute nodes are connected
>> to the network, and a network partition does not occur).
>
>How would you synchronize the various schedulers with each other?
>Suppose you have multiple scheduler nodes all trying to boot multiple
>instances each.
>
>Even if each at the start of the process each scheduler has a perfect
>view of the system, each scheduler would need to have a view of what
>every other scheduler is doing in order to not make racy decisions.
>
>I see a few options:
>
>1) Push scheduling down into the database itself.  Implement scheduler
>filters as SQL queries or stored procedures.
>
>2) Get rid of the DB for scheduling.  It looks like people are working
>on this: https://blueprints.launchpad.net/nova/+spec/no-db-scheduler
>
>3) Do multi-stage scheduling.  Do a "tentative" schedule, then try and
>update the DB to reserve all the necessary resources.  If that fails,
>someone got there ahead of you so try again with the new data.
>
>Chris
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Sam Alba

On Tue, Nov 19, 2013 at 6:45 AM, Chuck Short  wrote:
> Hi
>
> I am excited to see containers getting such traction in the openstack
> project.
>
>
> On Mon, Nov 18, 2013 at 7:30 PM, Russell Bryant  wrote:
>>
>> On 11/18/2013 06:30 PM, Dan Smith wrote:
>> >> Not having been at the summit (maybe the next one), could somebody
>> >> give a really short explanation as to why it needs to be a separate
>> >> service? It sounds like it should fit within the Nova area. It is,
>> >> after all, just another hypervisor type, or so it seems.
>> >
>> > But it's not just another hypervisor. If all you want from your
>> > containers is lightweight VMs, then nova is a reasonable place to put
>> > that (and it's there right now). If, however, you want to expose the
>> > complex and flexible attributes of a container, such as being able to
>> > overlap filesystems, have fine-grained control over what is shared with
>> > the host OS, look at the processes within a container, etc, then nova
>> > ends up needing quite a bit of change to support that.
>> >
>> > I think the overwhelming majority of folks in the room, after discussing
>> > it, agreed that Nova is infrastructure and containers is more of a
>> > platform thing. Making it a separate service lets us define a mechanism
>> > to manage these that makes much more sense than treating them like VMs.
>> > Using Nova to deploy VMs that run this service is the right approach,
>> > IMHO. Clayton put it very well, I think:
>> >
>> >   If the thing you want to deploy has a kernel, then you need Nova. If
>> >   your thing runs on a kernel, you want $new_service_name.
>> >
>> > I agree.
>> >
>> > Note that this is just another service under the compute project (or
>> > program, or whatever the correct terminology is this week).
>>
>> The Compute program is correct.  That is established terminology as
>> defined by the TC in the last cycle.
>>
>> > So while
>> > distinct from Nova in terms of code, development should be tightly
>> > integrated until (and if at some point) it doesn't make sense.
>>
>> And it may share a whole bunch of the code.
>>
>> Another way to put this:  The API requirements people have for
>> containers include a number of features considered outside of the
>> current scope of Nova (short version: Nova's scope stops before going
>> *inside* the servers it creates, except file injection, which we plan to
>> remove anyway).  That presents a problem.  A new service is one possible
>> solution.
>>
>> My view of the outcome of the session was not "it *will* be a new
>> service".  Instead, it was, "we *think* it should be a new service, but
>> let's do some more investigation to decide for sure".
>>
>> The action item from the session was to go off and come up with a
>> proposal for what a new service would look like.  In particular, we
>> needed a proposal for what the API would look like.  With that in hand,
>> we need to come back and ask the question again of whether a new service
>> is the right answer.
>>
>> I see 3 possible solutions here:
>>
>> 1) Expand the scope of Nova to include all of the things people want to
>> be able to do with containers.
>>
>> This is my least favorite option.  Nova is already really big.  We've
>> worked to split things out (Networking, Block Storage, Images) to keep
>> it under control.  I don't think a significant increase in scope is a
>> smart move for Nova's future.
>>
>
> This is my least favorite option. Like a lot of other responses already I
> see a lot of code duplication  because Nova and the new nova container's
> project. This just doesn't include the scheduler but  things like config
> driver, etc.

Can we dig into this option? Honestly, I'd be glad to find a way to
avoid reimplementing everything again (a new compute service with
Keystone, Glance, Horizon integration, etc...). But I do understand
the limitation of changing Nova to improve containers support.

Can someone bring more details (maybe in the spec etherpad, in a new
section) about this 3rd option?

Since the API (in the front) and the virt API (in the back) have to be
different, I barely see how we can reuse most of Nova's code.

>>
>> 2) Declare containers as explicitly out of scope and start a new project
>> with its own API.
>>
>> That is what is being proposed here.
>>
>> 3) Some middle ground that is a variation of #2.  Consider Ironic.  The
>> idea is that Nova's API will still be used for basic provisioning, which
>> Nova will implement by talking to Ironic.  However, there are a lot of
>> baremetal management things that don't fit in Nova at all, and those
>> only exist in Ironic's API.
>
>
> This is my preferred choice  as well. If we could leverage the existing nova
> API and extend it to include containers and features that users who use
> containers in their existing production environment wants.
>>
>>
>> I wanted to mention this option for completeness, but I don't actually
>> think it's the right choice here.  With Ironic you have a physical

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

If you start adding these states you might really want to consider the
following work that is going on in other projects.

It surely appears that everyone is starting to hit the same problem (and
joining efforts would produce a more beneficial result).

Relevant icehouse etherpads:
- https://etherpad.openstack.org/p/CinderTaskFlowFSM
- https://etherpad.openstack.org/p/icehouse-oslo-service-synchronization

And of course my obvious plug for taskflow (which is designed to be a
useful library to help in all these usages).

- https://wiki.openstack.org/wiki/TaskFlow

The states u just mentioned start to line-up with
https://wiki.openstack.org/wiki/TaskFlow/States_of_Task_and_Flow

If this sounds like a useful way to go (joining efforts) then lets see how
we can make it possible.

IRC: #openstack-state-management is where I am usually at.

On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:

>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
>Robert Kukura  wrote:
>
>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
>> > Developers,
>> > 
>> > This topic has been discussed before but I do not remember if we have
>>a
>> > good solution or not.
>> 
>> The ML2 plugin addresses this by calling each MechanismDriver twice. The
>> create_network_precommit() method is called as part of the DB
>> transaction, and the create_network_postcommit() method is called after
>> the transaction has been committed. Interactions with devices or
>> controllers are done in the postcommit methods. If the postcommit method
>> raises an exception, the plugin deletes that partially-created resource
>> and returns the exception to the client. You might consider a similar
>> approach in your plugin.
>
>Splitting works into two phase, pre/post, is good approach.
>But there still remains race window.
>Once the transaction is committed, the result is visible to outside.
>So the concurrent request to same resource will be racy.
>There is a window after pre_xxx_yyy before post_xxx_yyy() where
>other requests can be handled.
>
>The state machine needs to be enhanced, I think. (plugins need
>modification)
>For example, adding more states like pending_{create, delete, update}.
>Also we would like to consider serializing between operation of ports
>and subnets. or between operation of subnets and network depending on
>performance requirement.
>(Or carefully audit complex status change. i.e.
>changing port during subnet/network update/deletion.)
>
>I think it would be useful to establish reference locking policy
>for ML2 plugin for SDN controllers.
>Thoughts or comments? If this is considered useful and acceptable,
>I'm willing to help.
>
>thanks,
>Isaku Yamahata
>
>> -Bob
>> 
>> > Basically, if concurrent API calls are sent to Neutron, all of them
>>are
>> > sent to the plug-in level where two actions have to be made:
>> > 
>> > 1. DB transaction ? No just for data persistence but also to collect
>>the
>> > information needed for the next action
>> > 2. Plug-in back-end implementation ? In our case is a call to the
>>python
>> > library than consequentially calls PLUMgrid REST GW (soon SAL)
>> > 
>> > For instance:
>> > 
>> > def create_port(self, context, port):
>> > with context.session.begin(subtransactions=True):
>> > # Plugin DB - Port Create and Return port
>> > port_db = super(NeutronPluginPLUMgridV2,
>> > self).create_port(context,
>> >   
>> port)
>> > device_id = port_db["device_id"]
>> > if port_db["device_owner"] == "network:router_gateway":
>> > router_db = self._get_router(context, device_id)
>> > else:
>> > router_db = None
>> > try:
>> > LOG.debug(_("PLUMgrid Library: create_port() called"))
>> > # Back-end implementation
>> > self._plumlib.create_port(port_db, router_db)
>> > except Exception:
>> > Š
>> > 
>> > The way we have implemented at the plugin-level in Havana (even in
>> > Grizzly) is that both action are wrapped in the same "transaction"
>>which
>> > automatically rolls back any operation done to its original state
>> > protecting mostly the DB of having any inconsistency state or left
>>over
>> > data if the back-end part fails.=.
>> > The problem that we are experiencing is when concurrent calls to the
>> > same API are sent, the number of operation at the plug-in back-end are
>> > long enough to make the next concurrent API call to get stuck at the
>>DB
>> > transaction level, which creates a hung state for the Neutron Server
>>to
>> > the point that all concurrent API calls will fail.
>> > 
>> > This can be fixed if we include some "locking" system such as calling:
>> > 
>> > from neutron.common import utile
>> > Š
>> > 
>> > @utils.synchronized('any-name', external=True)
>> > def create_port(self, context, port):
>> > Š
>> > 
>> > Obviously, this will create a serialization of all concurrent calls
>> > which will ends up in having a really bad performance.

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Excerpts from Chris Friesen's message of 2013-11-19 09:29:00 -0800:
> On 11/18/2013 06:47 PM, Joshua Harlow wrote:
> > An idea related to this, what would need to be done to make the DB have
> > the exact state that a compute node is going through (and therefore the
> > scheduler would not make unreliable/racey decisions, even when there are
> > multiple schedulers). It's not like we are dealing with a system which
> > can not know the exact state (as long as the compute nodes are connected
> > to the network, and a network partition does not occur).
> 
> How would you synchronize the various schedulers with each other? 
> Suppose you have multiple scheduler nodes all trying to boot multiple 
> instances each.
> 
> Even if each at the start of the process each scheduler has a perfect 
> view of the system, each scheduler would need to have a view of what 
> every other scheduler is doing in order to not make racy decisions.
> 

Your question assumes they need to be "in sync" at a granular level.

Each scheduler process can own a different set of resources. If they
each grab instance requests in a round-robin fashion, then they will
fill their resources up in a relatively well balanced way until one
scheduler's resources are exhausted. At that time it should bow out of
taking new instances. If it can't fit a request in, it should kick the
request out for retry on another scheduler.

In this way, they only need to be in sync in that they need a way to
agree on who owns which resources. A distributed hash table that gets
refreshed whenever schedulers come and go would be fine for that.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

And also of course, nearly forgot a similar situation/review in heat.

https://review.openstack.org/#/c/49440/

Except theres was/is dealing with stack locking (a heat concept).

On 11/19/13 10:33 AM, "Joshua Harlow"  wrote:

>If you start adding these states you might really want to consider the
>following work that is going on in other projects.
>
>It surely appears that everyone is starting to hit the same problem (and
>joining efforts would produce a more beneficial result).
>
>Relevant icehouse etherpads:
>- https://etherpad.openstack.org/p/CinderTaskFlowFSM
>- https://etherpad.openstack.org/p/icehouse-oslo-service-synchronization
>
>And of course my obvious plug for taskflow (which is designed to be a
>useful library to help in all these usages).
>
>- https://wiki.openstack.org/wiki/TaskFlow
>
>The states u just mentioned start to line-up with
>https://wiki.openstack.org/wiki/TaskFlow/States_of_Task_and_Flow
>
>If this sounds like a useful way to go (joining efforts) then lets see how
>we can make it possible.
>
>IRC: #openstack-state-management is where I am usually at.
>
>On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:
>
>>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
>>Robert Kukura  wrote:
>>
>>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
>>> > Developers,
>>> > 
>>> > This topic has been discussed before but I do not remember if we have
>>>a
>>> > good solution or not.
>>> 
>>> The ML2 plugin addresses this by calling each MechanismDriver twice.
>>>The
>>> create_network_precommit() method is called as part of the DB
>>> transaction, and the create_network_postcommit() method is called after
>>> the transaction has been committed. Interactions with devices or
>>> controllers are done in the postcommit methods. If the postcommit
>>>method
>>> raises an exception, the plugin deletes that partially-created resource
>>> and returns the exception to the client. You might consider a similar
>>> approach in your plugin.
>>
>>Splitting works into two phase, pre/post, is good approach.
>>But there still remains race window.
>>Once the transaction is committed, the result is visible to outside.
>>So the concurrent request to same resource will be racy.
>>There is a window after pre_xxx_yyy before post_xxx_yyy() where
>>other requests can be handled.
>>
>>The state machine needs to be enhanced, I think. (plugins need
>>modification)
>>For example, adding more states like pending_{create, delete, update}.
>>Also we would like to consider serializing between operation of ports
>>and subnets. or between operation of subnets and network depending on
>>performance requirement.
>>(Or carefully audit complex status change. i.e.
>>changing port during subnet/network update/deletion.)
>>
>>I think it would be useful to establish reference locking policy
>>for ML2 plugin for SDN controllers.
>>Thoughts or comments? If this is considered useful and acceptable,
>>I'm willing to help.
>>
>>thanks,
>>Isaku Yamahata
>>
>>> -Bob
>>> 
>>> > Basically, if concurrent API calls are sent to Neutron, all of them
>>>are
>>> > sent to the plug-in level where two actions have to be made:
>>> > 
>>> > 1. DB transaction ? No just for data persistence but also to collect
>>>the
>>> > information needed for the next action
>>> > 2. Plug-in back-end implementation ? In our case is a call to the
>>>python
>>> > library than consequentially calls PLUMgrid REST GW (soon SAL)
>>> > 
>>> > For instance:
>>> > 
>>> > def create_port(self, context, port):
>>> > with context.session.begin(subtransactions=True):
>>> > # Plugin DB - Port Create and Return port
>>> > port_db = super(NeutronPluginPLUMgridV2,
>>> > self).create_port(context,
>>> >  
>>> port)
>>> > device_id = port_db["device_id"]
>>> > if port_db["device_owner"] == "network:router_gateway":
>>> > router_db = self._get_router(context, device_id)
>>> > else:
>>> > router_db = None
>>> > try:
>>> > LOG.debug(_("PLUMgrid Library: create_port()
>>>called"))
>>> > # Back-end implementation
>>> > self._plumlib.create_port(port_db, router_db)
>>> > except Exception:
>>> > Š
>>> > 
>>> > The way we have implemented at the plugin-level in Havana (even in
>>> > Grizzly) is that both action are wrapped in the same "transaction"
>>>which
>>> > automatically rolls back any operation done to its original state
>>> > protecting mostly the DB of having any inconsistency state or left
>>>over
>>> > data if the back-end part fails.=.
>>> > The problem that we are experiencing is when concurrent calls to the
>>> > same API are sent, the number of operation at the plug-in back-end
>>>are
>>> > long enough to make the next concurrent API call to get stuck at the
>>>DB
>>> > transaction level, which creates a hung state for the Neutron Server
>>>to
>>> > the point that all concurrent API calls will fail.
>>> > 
>>> > This can be fi

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Eric Windisch

On Tue, Nov 19, 2013 at 1:02 PM, James Bottomley
 wrote:
> On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
>> Hey all
>>
>> Not having been at the summit (maybe the next one), could somebody
>> give a really short explanation as to why it needs to be a separate
>> service?
>> It sounds like it should fit within the Nova area. It is, after all,
>> just another hypervisor type, or so it seems.
>
> I can take a stab at this:  Firstly, a container is *not* a hypervisor.
> Hypervisor based virtualisation is done at the hardware level (so with
> hypervisors you boot a second kernel on top of the virtual hardware),
> container based virtualisation is done at the OS (kernel) level (so all
> containers share the same kernel ... and sometimes even huge chunks of
> the OS). With recent advances in the Linux Kernel, we can make a
> container behave like a hypervisor (full OS/IaaS virtualisation), but
> quite a bit of the utility of containers is that they can do much more
> than hypervisors, so they shouldn't be constrained by hypervisor APIs
> (which are effectively virtual hardware APIs).
>
> It is possible to extend the Nova APIs to control containers more fully,
> but there was resistance do doing this on the grounds that it's
> expanding the scope of Nova, hence the new project.

It might be worth noting that it was also brought up that
hypervisor-based virtualization can offer a number of features that
bridge some of these gaps, but are not supported in, nor may ever be
supported in Nova.

For example, Daniel brings up an interesting point with the
libvirt-sandbox feature. This is one of those features that bridges
some of the gaps. There are also solutions, however brittle, for
introspection that work on hypervisor-driven VMs. It is not clear what
the scope or desire for these features might be, how they might be
sufficiently abstracted between hypervisors and guest OSes, nor how
these would fit into any of the existing or planned compute API
buckets.

Having a separate service for managing containers draws a thick line
in the sand that will somewhat stiffen innovation around
hypervisor-based virtualization. That isn't necessarily a bad thing,
it will help maintain stability in the project. However, the choice
and the implications shouldn't be ignored.

-- 
Regards,
Eric Windisch

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Sorry that was I prefer #3 (not #2) at the end there. Keyboard failure ;)

On 11/19/13 10:27 AM, "Joshua Harlow"  wrote:

>Personally I would prefer #3 from the below. #2 I think will still have to
>deal with consistency issues, just switching away from a DB doesn't make
>magical ponies and unicorns appear (in-fact it can potentially make the
>problem worse if its done incorrectly - and its pretty easy to get it
>wrong IMHO). #1 could also work, but then u hit a vertical scaling limit
>(works if u paid oracle for there DB or IBM for DB2 I suppose). I prefer
>#2 since I think it is honestly needed under all solutions.
>
>On 11/19/13 9:29 AM, "Chris Friesen"  wrote:
>
>>On 11/18/2013 06:47 PM, Joshua Harlow wrote:
>>> An idea related to this, what would need to be done to make the DB have
>>> the exact state that a compute node is going through (and therefore the
>>> scheduler would not make unreliable/racey decisions, even when there
>>>are
>>> multiple schedulers). It's not like we are dealing with a system which
>>> can not know the exact state (as long as the compute nodes are
>>>connected
>>> to the network, and a network partition does not occur).
>>
>>How would you synchronize the various schedulers with each other?
>>Suppose you have multiple scheduler nodes all trying to boot multiple
>>instances each.
>>
>>Even if each at the start of the process each scheduler has a perfect
>>view of the system, each scheduler would need to have a view of what
>>every other scheduler is doing in order to not make racy decisions.
>>
>>I see a few options:
>>
>>1) Push scheduling down into the database itself.  Implement scheduler
>>filters as SQL queries or stored procedures.
>>
>>2) Get rid of the DB for scheduling.  It looks like people are working
>>on this: https://blueprints.launchpad.net/nova/+spec/no-db-scheduler
>>
>>3) Do multi-stage scheduling.  Do a "tentative" schedule, then try and
>>update the DB to reserve all the necessary resources.  If that fails,
>>someone got there ahead of you so try again with the new data.
>>
>>Chris
>>
>>___
>>OpenStack-dev mailing list
>>OpenStack-dev@lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] [Ironic] [TripleO] scheduling flow with Ironic?

2013-11-19 Thread Devananda van der Veen

On Wed, Nov 13, 2013 at 10:11 PM, Alex Glikson  wrote:

> Thanks, I understand the Nova scheduler part. One of the gaps there is
> related to the blueprint we have are working on [1]. I was wondering
> regarding the role of Ironic, and the exact interaction between the user,
> Nova and Ironic.
>

The interaction from the point of "nova boot" onwards will be the same --
nova maintains a list of available (host, node) resources, the scheduler
picks one according to the request, dispatches the work to n-cond / n-cpu,
which in turn calls down to various methods in the nova/virt/driver API.
The implementation of the ironic driver is a wrapper around
python-ironicclient library, which will make calls out to the ironic API
service, which in turn performs the necessary work.

Where the interaction is different is around the management of physical
machines; eg, enrolling them with Ironic, temporarily marking a machine as
unavailable while doing maintenance on it, and other sorts of things we
haven't actually written the code for yet.

> In particular, initially I thought that Ironic is going to have its own
> scheduler, resolving some of the issues and complexity within Nova (which
> could focus on VM management, maybe even getting rid of hosts versus nodes,
> etc).

I'm not sure how putting a scheduler in Ironic would solve this problem at
all.

Conversely, I don't think there's any need for the whole (host, node)
thing. Chris Behrens and I talked at the last summit about a possible
rewrite to nova-conductor that would remove the need for this distinction
entirely. I would love to see Nova just track node, and I think this can
work for typical hypervisors (kvm, xen, ...) as well.

> But it seems that Ironic aims to stay at the level of virt driver API.. It
> is a bit unclear to me what is the desired architecture going forward -
> e.g., if the idea is to standardize virt driver APIs but keep the
> scheduling centralized,

AFAIK, the nova.virt.driver API is the standard that all the virt drivers
are written to. Unless you're referring to libvirt's API, in which case, I
don't understand the question.

> maybe we should take the rest of virt drivers into separate projects as
> well, and extend Nova to schedule beyond just compute (if it is already
> doing so for virt + bare-metal).

Why would Nova need to schedule anything besides compute resources? In this
context, Ironic is merely providing a different type of compute resource,
and Nova is still scheduling compute workloads. That this hypervisor driver
has different scheduling characteristics (eg, flavor == node resource;
extra_specs:cpu_arch == node arch; and so on) than other hypervisor drivers
doesn't mean it's not still a compute resource.

> Alternatively, each of them could have its own scheduler (like the
> approach we took when splitting out cinder, for example) - and then someone
> on top (e.g., Heat) would need to do the cross-project logic. Taking
> different architectural approaches in different cases confuses me a bit.
>

Yes, well, Cinder is a different type of resource (block storage).

HTH,
-Deva
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] RFC: Potential to increase min required libvirt version to 0.9.8 ?

Currently the Nova libvirt driver is declaring that it wants a minimum
of libvirt 0.9.6.

For cases where we use features newer than this, we have to do conditional
logic to ensure we operate correctly on old libvirt. We don't want to keep
adding conditionals forever since they complicate the code and thus impose
an ongoing dev burden.

I think it is wise to re-evaluate the min required libvirt version at the
start of each dev cycle. To do this we need to understand what versions
are available in the OS distributions we care about supporting for the
Icehouse release.

To this end I've created a wiki page covering Debian, Fedora, OpenSUSE,
RHEL and Ubuntu

  https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix

Considering those distros it looks like we can potentially increase the
min required libvirt to 0.9.8 for Icehouse.

If there are other distros I've missed which expect to support deployment
of Icehouse please add them to this list. Hopefully there won't be any
with libvirt software older than Ubuntu 12.04 LTS


The reason I'm asking this now, is that we're working to make the libvirt
python module a separate tar.gz that can build with multiple libvirt
versions, and I need to decide how ancient a libvirt we should support
for it.

Regards.
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread James Bottomley

On Tue, 2013-11-19 at 13:46 -0500, Eric Windisch wrote:
> On Tue, Nov 19, 2013 at 1:02 PM, James Bottomley
>  wrote:
> > On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
> >> Hey all
> >>
> >> Not having been at the summit (maybe the next one), could somebody
> >> give a really short explanation as to why it needs to be a separate
> >> service?
> >> It sounds like it should fit within the Nova area. It is, after all,
> >> just another hypervisor type, or so it seems.
> >
> > I can take a stab at this:  Firstly, a container is *not* a hypervisor.
> > Hypervisor based virtualisation is done at the hardware level (so with
> > hypervisors you boot a second kernel on top of the virtual hardware),
> > container based virtualisation is done at the OS (kernel) level (so all
> > containers share the same kernel ... and sometimes even huge chunks of
> > the OS). With recent advances in the Linux Kernel, we can make a
> > container behave like a hypervisor (full OS/IaaS virtualisation), but
> > quite a bit of the utility of containers is that they can do much more
> > than hypervisors, so they shouldn't be constrained by hypervisor APIs
> > (which are effectively virtual hardware APIs).
> >
> > It is possible to extend the Nova APIs to control containers more fully,
> > but there was resistance do doing this on the grounds that it's
> > expanding the scope of Nova, hence the new project.
> 
> It might be worth noting that it was also brought up that
> hypervisor-based virtualization can offer a number of features that
> bridge some of these gaps, but are not supported in, nor may ever be
> supported in Nova.
> 
> For example, Daniel brings up an interesting point with the
> libvirt-sandbox feature. This is one of those features that bridges
> some of the gaps. There are also solutions, however brittle, for
> introspection that work on hypervisor-driven VMs. It is not clear what
> the scope or desire for these features might be, how they might be
> sufficiently abstracted between hypervisors and guest OSes, nor how
> these would fit into any of the existing or planned compute API
> buckets.

It's certainly possible, but some of them are possible in the same way
as it's possible to get a square peg into a round hole by beating the
corners flat with a sledge hammer ... it works, but it's much less
hassle just to use a round peg because it actually fits the job.

> Having a separate service for managing containers draws a thick line
> in the sand that will somewhat stiffen innovation around
> hypervisor-based virtualization. That isn't necessarily a bad thing,
> it will help maintain stability in the project. However, the choice
> and the implications shouldn't be ignored.

How about this: we get the container API agreed and we use a driver
model like Nova (we have to anyway since there about four different
container technologies interested in this), then we see if someone wants
to do a hypervisor driver emulating the features.

James



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

On Tue, Nov 19, 2013 at 10:02:45AM -0800, James Bottomley wrote:
> On Mon, 2013-11-18 at 14:28 -0800, Stuart Fox wrote:
> > Hey all
> > 
> > Not having been at the summit (maybe the next one), could somebody
> > give a really short explanation as to why it needs to be a separate
> > service?
> > It sounds like it should fit within the Nova area. It is, after all,
> > just another hypervisor type, or so it seems.
> 
> I can take a stab at this:  Firstly, a container is *not* a hypervisor.
> Hypervisor based virtualisation is done at the hardware level (so with
> hypervisors you boot a second kernel on top of the virtual hardware),
> container based virtualisation is done at the OS (kernel) level (so all
> containers share the same kernel ... and sometimes even huge chunks of
> the OS). With recent advances in the Linux Kernel, we can make a
> container behave like a hypervisor (full OS/IaaS virtualisation), but
> quite a bit of the utility of containers is that they can do much more
> than hypervisors, so they shouldn't be constrained by hypervisor APIs
> (which are effectively virtual hardware APIs).
> 
> It is possible to extend the Nova APIs to control containers more fully,
> but there was resistance do doing this on the grounds that it's
> expanding the scope of Nova, hence the new project.

You're focusing on the low level technical differences between containers
and hypervisor here which IMHO is not the right comparison to be making.
Once you look at the high level use cases many of the referenced container
virt apps are trying to address things don't look anywhere near as clear
cut. As mentioned elsewhere libvirt-sandbox which provides a application
sandboxing toolkit is able to leverage either LXC or KVM virt to achieve
its high level goals. Based on my understanding of Docker, I believe it
would actually be possible to run Docker images under KVM without much
difficultly. There will certainly be some setups that aren't possible
todo with hypervisors, but I don't think those will be in the majority,
nor require starting again from scratch, throwing out Nova.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

2013-11-19 Thread Steven Dake



On 11/17/2013 01:57 PM, Steve Baker wrote:

On 11/15/2013 05:19 AM, Christopher Armstrong wrote:

http://docs.heatautoscale.apiary.io/

I've thrown together a rough sketch of the proposed API for
autoscaling. It's written in API-Blueprint format (which is a simple
subset of Markdown) and provides schemas for inputs and outputs using
JSON-Schema. The source document is currently
at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp


Apologies if I'm about to re-litigate an old argument, but...

At summit we discussed creating a new endpoint (and new pythonclient)
for autoscaling. Instead I think the autoscaling API could just be added
to the existing heat-api endpoint.

Arguments for just making auto scaling part of heat api include:
* Significantly less development, packaging and deployment configuration
of not creating a heat-autoscaling-api and python-autoscalingclient
* Autoscaling is orchestration (for some definition of orchestration) so
belongs in the orchestration service endpoint
* The autoscaling API includes heat template snippets, so a heat service
is a required dependency for deployers anyway
* End-users are still free to use the autoscaling portion of the heat
API without necessarily being aware of (or directly using) heat
templates and stacks
* It seems acceptable for single endpoints to manage many resources (eg,
the increasingly disparate list of resources available via the neutron API)

Arguments for making a new auto scaling api include:
* Autoscaling is not orchestration (for some narrower definition of
orchestration)
* Autoscaling implementation will be handled by something other than
heat engine (I have assumed the opposite)
(no doubt this list will be added to in this thread)
A separate process can be autoscaled independently of heat-api which is 
a big plus architecturally.


They really do different things, and separating their concerns at the 
process level is a good goal.


I prefer a separate process for these reasons.

Regards
-steve


What do you think?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?


On 11/19/2013 12:35 PM, Clint Byrum wrote:


Each scheduler process can own a different set of resources. If they
each grab instance requests in a round-robin fashion, then they will
fill their resources up in a relatively well balanced way until one
scheduler's resources are exhausted. At that time it should bow out of
taking new instances. If it can't fit a request in, it should kick the
request out for retry on another scheduler.

In this way, they only need to be in sync in that they need a way to
agree on who owns which resources. A distributed hash table that gets
refreshed whenever schedulers come and go would be fine for that.


That has some potential, but at high occupancy you could end up refusing 
to schedule something because no one scheduler has sufficient resources 
even if the cluster as a whole does.


This gets worse once you start factoring in things like heat and 
instance groups that will want to schedule whole sets of resources 
(instances, IP addresses, network links, cinder volumes, etc.) at once 
with constraints on where they can be placed relative to each other.


Chris


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Version Negotiation Middleware Accept Header issue

2013-11-19 Thread Fuente, Pablo A

Hi,
I noticed that the Accept HTTP Header checked in the Version
Negotiation Middleware by Heat is the same MIME type used by Glance
("application/vnd.openstack.images-")
Is this OK, or should it be something like
"application/vnd.openstack.orchestration-"?
If this is the case I would proceed to file a bug.

Thanks.
Pablo.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?


On 11/19/2013 12:27 PM, Joshua Harlow wrote:

Personally I would prefer #3 from the below. #2 I think will still have to
deal with consistency issues, just switching away from a DB doesn't make
magical ponies and unicorns appear (in-fact it can potentially make the
problem worse if its done incorrectly - and its pretty easy to get it
wrong IMHO). #1 could also work, but then u hit a vertical scaling limit
(works if u paid oracle for there DB or IBM for DB2 I suppose). I prefer
#3 since I think it is honestly needed under all solutions.


Personally I think we need a combination of #3 (resource reservation) 
with something else to speed up scheduling.


We have multiple filters that currently loop over all the compute nodes, 
gathering a bunch of data from the DB and then ignoring most of that 
data while doing some simple logic in python.


There is really no need for the bulk of the resource information to be 
stored in the DB.  The compute nodes could broadcast their current state 
to all scheduler nodes, and the scheduler nodes could reserve resources 
directly from the compute nodes (triggering an update of all the other 
scheduler nodes).


Failing that, it should be possible to push at least some of the 
filtering down into the DB itself. Stuff like ramfilter or cpufilter 
would be trival (and fast) as an SQL query.


Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:
> On 11/19/2013 12:35 PM, Clint Byrum wrote:
> 
> > Each scheduler process can own a different set of resources. If they
> > each grab instance requests in a round-robin fashion, then they will
> > fill their resources up in a relatively well balanced way until one
> > scheduler's resources are exhausted. At that time it should bow out of
> > taking new instances. If it can't fit a request in, it should kick the
> > request out for retry on another scheduler.
> >
> > In this way, they only need to be in sync in that they need a way to
> > agree on who owns which resources. A distributed hash table that gets
> > refreshed whenever schedulers come and go would be fine for that.
> 
> That has some potential, but at high occupancy you could end up refusing 
> to schedule something because no one scheduler has sufficient resources 
> even if the cluster as a whole does.
> 

I'm not sure what you mean here. What resource spans multiple compute
hosts?

> This gets worse once you start factoring in things like heat and 
> instance groups that will want to schedule whole sets of resources 
> (instances, IP addresses, network links, cinder volumes, etc.) at once 
> with constraints on where they can be placed relative to each other.
> 

Actually that is rather simple. Such requests have to be serialized
into a work-flow. So if you say "give me 2 instances in 2 different
locations" then you allocate 1 instance, and then another one with
'not_in_location(1)' as a condition.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

2013-11-19 Thread Rick Jones


On 11/19/2013 10:02 AM, James Bottomley wrote:

It is possible to extend the Nova APIs to control containers more fully,
but there was resistance do doing this on the grounds that it's
expanding the scope of Nova, hence the new project.


How well received would another CLI/API to learn be among the end-users?

rick jones

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Does Nova really need an SQL database?

On 11/19/2013 01:51 PM, Clint Byrum wrote:

Excerpts from Chris Friesen's message of 2013-11-19 11:37:02 -0800:

On 11/19/2013 12:35 PM, Clint Byrum wrote:

Each scheduler process can own a different set of resources. If they
each grab instance requests in a round-robin fashion, then they will
fill their resources up in a relatively well balanced way until one
scheduler's resources are exhausted. At that time it should bow out of
taking new instances. If it can't fit a request in, it should kick the
request out for retry on another scheduler.

In this way, they only need to be in sync in that they need a way to
agree on who owns which resources. A distributed hash table that gets
refreshed whenever schedulers come and go would be fine for that.

That has some potential, but at high occupancy you could end up refusing
to schedule something because no one scheduler has sufficient resources
even if the cluster as a whole does.

I'm not sure what you mean here. What resource spans multiple compute
hosts?

Imagine the cluster is running close to full occupancy, each scheduler
has room for 40 more instances. Now I come along and issue a single
request to boot 50 instances. The cluster has room for that, but none
of the schedulers do.

This gets worse once you start factoring in things like heat and
instance groups that will want to schedule whole sets of resources
(instances, IP addresses, network links, cinder volumes, etc.) at once
with constraints on where they can be placed relative to each other.

Actually that is rather simple. Such requests have to be serialized
into a work-flow. So if you say "give me 2 instances in 2 different
locations" then you allocate 1 instance, and then another one with
'not_in_location(1)' as a condition.

Actually, you don't want to serialize it, you want to hand the whole set
of resource requests and constraints to the scheduler all at once.

If you do them one at a time, then early decisions made with
less-than-complete knowledge can result in later scheduling requests
failing due to being unable to meet constraints, even if there are
actually sufficient resources in the cluster.

The "VM ensembles" document at
https://docs.google.com/document/d/1bAMtkaIFn4ZSMqqsXjs_riXofuRvApa--qo4UTwsmhw/edit?pli=1
has a good example of how one-at-a-time scheduling can cause spurious
failures.

And if you're handing the whole set of requests to a scheduler all at
once, then you want the scheduler to have access to as many resources as
possible so that it has the highest likelihood of being able to satisfy
the request given the constraints.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] SSL Termination write-up

2013-11-19 Thread Eugene Nikanorov

Hi Vijay,

Thanks for working on this. As was discussed at the summit, immediate
solution seems to be passing certificates via transient fields in Vip
object, which will avoid the need for certificate management (incl. storing
them).
If certificate management is concerned then I agree that it needs to be a
separate specialized service.

> My thought was to have independent certificate resource with VIP uuid as
one of the properties. VIP is already created and
> will help to identify the driver/device. The VIP property can be
depreciated in the long term.
I think it's a little bit forward looking at this moment, also I think that
certificates are not specific for load balancing.
Transient fields could help here too: client could pass certificate Id and
driver of corresponding device will communicate with external service
fetching corresponding certificate.

Thanks,
Eugene.


On Tue, Nov 19, 2013 at 5:48 PM, Vijay Venkatachalam <
vijay.venkatacha...@citrix.com> wrote:

>  Hi Sam, Eugene, & Avishay, etal,
>
>
>
> Today I spent some time to create a write-up for SSL
> Termination not exactly design doc. Please share your comments!
>
>
>
>
> https://docs.google.com/document/d/1tFOrIa10lKr0xQyLVGsVfXr29NQBq2nYTvMkMJ_inbo/edit
>
>
>
> Would like comments/discussion especially on the following note:
>
>
>
> SSL Termination requires certificate management. The ideal way is to
> handle this via an independent IAM service. This would take time to
> implement so the thought was to add the certificate details in VIP resource
> and send them directly to device. Basically don’t store the certificate key
> in the DB there by avoiding security concerns of maintaining certificates
> in controller.
>
>
>
> I would expect the certificates to become an independent resource in
> future thereby causing backward compatibility issues.
>
>
>
> Any ideas how to achieve this?
>
>
>
> My thought was to have independent certificate resource with VIP uuid as
> one of the properties. VIP is already created and will help to identify the
> driver/device. The VIP property can be depreciated in the long term.
>
>  Thanks,
>
> Vijay V.
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

On 11/19/2013 03:08 PM, Rick Jones wrote:
> On 11/19/2013 10:02 AM, James Bottomley wrote:
>> It is possible to extend the Nova APIs to control containers more fully,
>> but there was resistance do doing this on the grounds that it's
>> expanding the scope of Nova, hence the new project.
> 
> How well received would another CLI/API to learn be among the end-users?

It depends.  It is it mostly a duplication of the compute API, but just
slightly different enough to be annoying?  Or is it something
significantly different enough that much better suits their use cases?

That's the important question to answer here.  How different is it, and
does the difference justify a split?

The opinions so far are quite vast.  Those interested in pushing this
are going to go off and work on a more detailed API proposal.  I think
we should revisit this discussion when they have something for us to
evaluate against.

Even if this ends up staying in Nova, the API work is still useful.  It
would help us toward defining a container specific extension to the
compute API.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

2013-11-19 Thread Steve Baker

On 11/19/2013 08:37 PM, Thomas Spatzier wrote:
> Steve Baker  wrote on 18.11.2013 21:52:04:
>> From: Steve Baker 
>> To: openstack-dev@lists.openstack.org,
>> Date: 18.11.2013 21:54
>> Subject: Re: [openstack-dev] [Heat] HOT software configuration
>> refined after design summit discussions
>>
>> On 11/19/2013 02:22 AM, Thomas Spatzier wrote:
>>> Hi all,
>>>
>>> I have reworked the wiki page [1] I created last week to reflect
>>> discussions we had on the mail list and in IRC. From ML discussions
> last
>>> week it looked like we were all basically on the same page (with some
>>> details to be worked out), and I hope the new draft eliminates some
>>> confusion that the original draft had.
>>>
>>> [1]
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
>> Thanks Thomas, this looks really good. I've actually started on a POC
>> which maps to this model.
> Good to hear that, Steve :-)
> Now that we are converging, should we consolidate the various wiki pages
> and just have one? E.g. copy the complete contents of
> hot-software-config-WIP to your original hot-software-config, or deprecate
> all others and make hot-software-config-WIP the master?
Lets just bless hot-software-config-WIP and add to it as we flesh out
the implementation.

>> I've used different semantics which you may actually prefer some of,
>> please comment below.
>>
>> Resource types:
>> SoftwareConfig -> SoftwareConfig (yay!)
>> SoftwareDeployment -> SoftwareApplier - less typing, less mouth-fatigue
> I'm ok with SoftwareApplier. If we don't hear objections, I can change it
> in the wiki.
>
>> SoftwareConfig properties:
>> parameters -> inputs - just because parameters is overloaded already.
> Makes sense.
>
>> Although if the CM tool has their own semantics for inputs then that
>> should be used in that SoftwareConfig resource implementation instead.
>> outputs -> outputs
>>
>> SoftwareApplier properties:
>> software_config -> apply_config - because there will sometimes be a
>> corresponding remove_config
> Makes sense, and the remove_config thought is a very good point!
>
>> server -> server
>> parameters -> input_values - to match the 'inputs' schema property in
>> SoftwareConfig
> Agree on input_values.
>
>> Other comments on hot-software-config-WIP:
>>
>> Regarding apply_config/remove_config, if a SoftwareApplier resource is
>> deleted it should trigger any remove_config and wait for the server to
>> acknowledge when that is complete. This allows for any
>> evacuation/deregistering workloads to be executed.
>>
>> I'm unclear yet what the SoftwareConfig 'role' is for, unless the role
>> specifies the contract for a given inputs and outputs schema? How would
>> this be documented or enforced? I'm inclined to leave it out for now.
> So about 'role', as I stated in the wiki, my thinking was that there will
> be different SoftwareConfig and SoftwareApplier implementations per CM tool
> (more on that below), since all CM tools will probably have their specific
> metadata and runtime implementation. So in my example I was using Chef, and
> 'role' is just a Chef concept, i.e. you take a cookbook and configure a
> specific Chef role on a server.
OK, its Chef specific; I'm fine with that.
>> It should be possible to write a SoftwareConfig type for a new CM tool
>> as a provider template. This has some nice implications for deployers
>> and users.
> I think provider templates are a good thing to have clean componentization
> for re-use. However, I think it still would be good to allow users to
> define their SoftwareConfigs inline in a template for simple use cases. I
> heard that requirement in several posts on the ML last week.
> The question is whether we can live with a single implementation of
> SoftwareConfig and SoftwareApplier then (see also below).
Yes, a provider template would encapsulate some base SoftwareConfig
resource type, but users would be free to use this type inline in their
template too.
>> My hope is that there will not need to be a different SoftwareApplier
>> type written for each CM tool. But maybe there will be one for each
>> delivery mechanism. The first implementation will use metadata polling
>> and signals, another might use Marconi. Bootstrapping an image to
>> consume a given CM tool and applied configuration data is something that
>> we need to do, but we can make it beyond the scope of this particular
>> proposal.
> I was thinking about a single implementation, too. However, I cannot really
> imagine how one single implementation could handle both the different
> metadata of different CM tools, and different runtime implementation. I
> think we would want to support at least a handful of the most favorite
> tools, but cannot see at the moment how to cover them all in one
> implementation. My thought was that there could be a super-class for common
> behavior, and then plugins with specific behavior for each tool.
>
> Anyway, all of that needs to be verified, so working on PoC patches is
> def

Re: [openstack-dev] [Ironic][Ceilometer] get IPMI data for ceilometer

2013-11-19 Thread Devananda van der Veen

On Mon, Nov 18, 2013 at 10:35 AM, Ladislav Smola  wrote:

>  Hello. I have a couple of additional questions.
>
> 1. What about IPMI data that we want to get by polling. E.g. temperatures,
> etc. Will the Ironic be polling these kind of
> data and send them directly to collector(or agent)? Not sure if this
> belongs to Ironic. It would have to support some
> pluggable architecture for vendor specific pollsters like Ceilometer.
>
>
If there is a fixed set of information (eg, temp, fan speed, etc) that
ceilometer will want, let's make a list of that and add a driver interface
within Ironic to abstract the collection of that information from physical
nodes. Then, each driver will be able to implement it as necessary for that
vendor. Eg., an iLO driver may poll its nodes differently than a generic
IPMI driver, but the resulting data exported to Ceilometer should have the
same structure.

I don't think we should, at least right now, support pluggable pollsters on
the Ceilometer->Ironic side. Let's start with a small set of data that
Ironic exposes, make it pluggable internally for different types of
hardware, and iterate if necessary.


> 2. I've seen in the etherpad that the SNMP agent(pollster) will be also
> part of the Ironic(running next to conductor). Is it true?
> Or that will be placed in Ceilometer central agent?
>

An SNMP agent doesn't fit within the scope of Ironic, as far as I see, so
this would need to be implemented by Ceilometer.

As far as where the SNMP agent would need to run, it should be on the same
host(s) as ironic-conductor so that it has access to the management network
(the physically-separate network for hardware management, IPMI, etc). We
should keep the number of applications with direct access to that network
to a minimum, however, so a thin agent that collects and forwards the SNMP
data to the central agent would be preferable, in my opinion.


Regards,
Devananda



>
>
> Thanks for response.
> Ladislav
>
>
>
> On 11/18/2013 06:25 PM, Devananda van der Veen wrote:
>
> Hi Lianhao Lu,
>
>  I briefly summarized my recollection of that session in this blueprint:
>
>  https://blueprints.launchpad.net/ironic/+spec/add-ceilometer-agent
>
>  I've responded to your questions inline as well.
>
>
> On Sun, Nov 17, 2013 at 10:24 PM, Lu, Lianhao wrote:
>
>> Hi stackers,
>>
>> During the summit session Expose hardware sensor (IPMI) data
>> https://etherpad.openstack.org/p/icehouse-summit-ceilometer-hardware-sensors,
>> it was proposed to deploy a ceilometer agent next to the ironic conductor
>> to the get the ipmi data. Here I'd like to ask some questions to figure out
>> what's the current missing pieces in ironic and ceilometer for that
>> proposal.
>>
>> 1. Just double check, ironic won't provide API to get IPMI data, right?
>>
>
>  Correct. This was generally felt to be unnecessary.
>
>>
>> 2. If deploying a ceilometer agent next to the ironic conductor, how does
>> the agent talk to the conductor? Through rpc?
>>
>
>  My understanding is that ironic-conductor will emit messages to the
> ceilimeter agent, and the communication is one-way. These could be
> triggered by a periodic task, or by some other event within Ironic, such as
> a change in the power state of a node.
>
>
>>
>> 3. Does the current ironic conductor have rpc_method to support getting
>> generic ipmi data, i.e. let the rpc_method caller specifying arbitrary
>> netfn/command to get any type of ipmi data?
>>
>
>  No, and as far as I understand, it doesn't need one.
>
>
>>
>> 4. I believe the ironic conductor uses some kind of node_id to associate
>> the bmc with its credentials, right? If so, how can the ceilometer agent
>> get those node_ids to ask the ironic conductor to poll the ipmi data? And
>> how can the ceilometer agent extract meaningful information from that
>> node_id to set those fields in the ceilometer Sample(e.g. recource_id,
>> project_id, user_id, etc.) to identify which physical node the ipmi data is
>> coming from?
>>
>
>  This question perhaps requires a longer answer.
>
>  Ironic references physical machines (nodes) internally with an integer
> node_id and externally with a standard uuid. When a Nova instance is
> created, it will be associated to a node, that node will have a reference
> to the nova instance_uuid which is exposed in our API, and can be passed to
> Ceilometer's agent. I believe that nova instance_uuid will enable
> ceilometer to detect the project, user, etc.
>
>  Should Ironic emit messages regarding nodes which are not provisioned?
> Physical machines that don't have a tenant instance on them are not
> associated to any project, user, tenant, quota, etc, so I suspect that we
> shouldn't notify about them. It would be like tracking the unused disks in
> a SAN.
>
>  Regards,
> Devananda
>
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> 
> Regarding apply_config/remove_config, if a SoftwareApplier resource is
> deleted it should trigger any remove_config and wait for the server to
> acknowledge when that is complete. This allows for any
> evacuation/deregistering workloads to be executed.
> 

I'm a little worried about the road that leads us down. Most configuration
software defines forward progress only. Meaning, if you want something
not there, you don't remove it from your assertions, you assert that it
is not there.

The reason this is different than the way we operate with resources is
that resources are all under Heat's direct control via well defined
APIs. In-instance things, however, will be indirectly controlled. So I
feel like focusing on a "diff" mechanism for user-deployed tools may be
unnecessary and might confuse. I'd much rather have a "converge"
mechanism for the users to focus on.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

2013-11-19 Thread Steve Baker

On 11/20/2013 09:50 AM, Clint Byrum wrote:
> Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
>> Regarding apply_config/remove_config, if a SoftwareApplier resource is
>> deleted it should trigger any remove_config and wait for the server to
>> acknowledge when that is complete. This allows for any
>> evacuation/deregistering workloads to be executed.
>>
> I'm a little worried about the road that leads us down. Most configuration
> software defines forward progress only. Meaning, if you want something
> not there, you don't remove it from your assertions, you assert that it
> is not there.
>
> The reason this is different than the way we operate with resources is
> that resources are all under Heat's direct control via well defined
> APIs. In-instance things, however, will be indirectly controlled. So I
> feel like focusing on a "diff" mechanism for user-deployed tools may be
> unnecessary and might confuse. I'd much rather have a "converge"
> mechanism for the users to focus on.
>
>
A specific use-case I'm trying to address here is tripleo doing an
update-replace on a nova compute node. The remove_config contains the
workload to evacuate VMs and signal heat when the node is ready to be
shut down. This is more involved than just "uninstall the things".

Could you outline in some more detail how you think this could be done?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

Excerpts from Steve Baker's message of 2013-11-19 13:06:21 -0800:
> On 11/20/2013 09:50 AM, Clint Byrum wrote:
> > Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> >> Regarding apply_config/remove_config, if a SoftwareApplier resource is
> >> deleted it should trigger any remove_config and wait for the server to
> >> acknowledge when that is complete. This allows for any
> >> evacuation/deregistering workloads to be executed.
> >>
> > I'm a little worried about the road that leads us down. Most configuration
> > software defines forward progress only. Meaning, if you want something
> > not there, you don't remove it from your assertions, you assert that it
> > is not there.
> >
> > The reason this is different than the way we operate with resources is
> > that resources are all under Heat's direct control via well defined
> > APIs. In-instance things, however, will be indirectly controlled. So I
> > feel like focusing on a "diff" mechanism for user-deployed tools may be
> > unnecessary and might confuse. I'd much rather have a "converge"
> > mechanism for the users to focus on.
> >
> >
> A specific use-case I'm trying to address here is tripleo doing an
> update-replace on a nova compute node. The remove_config contains the
> workload to evacuate VMs and signal heat when the node is ready to be
> shut down. This is more involved than just "uninstall the things".
> 
> Could you outline in some more detail how you think this could be done?
> 

So for that we would not remove the software configuration for the
nova-compute, we would assert that the machine needs vms evacuated.
We want evacuation to be something we explicitly do, not a side effect
of deleting things. Perhaps having delete hooks for starting delete
work-flows is right, but it set off a red flag for me so I want to make
sure we think it through.

Also IIRC, evacuation is not necessarily an in-instance thing. It looks
more like the weird thing we've been talking about lately which is
"how do we orchestrate tenant API's":

https://etherpad.openstack.org/p/orchestrate-tenant-apis

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration

2013-11-19 Thread Zane Bitter

On 19/11/13 19:03, Clint Byrum wrote:

Excerpts from Zane Bitter's message of 2013-11-15 12:41:53 -0800:

Good news, everyone! I have created the missing whiteboard diagram that
we all needed at the design summit:

https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram

When we're discussing this, please mention explicitly the number of the
model you are talking about at any given time.

If you have a suggestion for a different model, make your own diagram!
jk, you can sketch it or something for me and I'll see if I can add it.

Thanks for putting this together Zane. I just now got around to looking
closely.

Option 2 is good. I'd love for option 1 to be made automatic by making
the client smarter, but parsing templates in the client will require
some deep thought before we decide it is a good idea.

I'd like to consider a 2a, which just has the same Heat engines the user
is talking to being used to do the orchestration in whatever region
they are in. I think that is actually the intention of the diagram,
but it looks like there is a "special" one that talks to the engines
that actually do the work.

Yes, I think you're saying the same thing as the diagram was intended to
convey. (Each orange box is meant to be a Heat engine.)

So the user talks to the Heat endpoint in a region of their choice
(doesn't matter which, they're all the same). When a OS::Heat::Stack
resource has Region and/or Endpoint specified, it will use
python-heatclient to connect to the appropriate engine for that nested
stack.

2 may morph into 3 actually, if users don't like the nested stack
requirement for 2, we can do the work to basically make the engine create
a nested stack per region. So that makes 2 a stronger choice for first
implementation.

Yeah, if you run your own standalone Heat engine, you've effectively
created 3 with no code changes from 2 afaict. I wouldn't recommend that
operators deploy this way though.

4 has an unstated pro, which is that attack surface is reduced. This
makes more sense when you consider the TripleO case where you may want
the undercloud (hardware cloud) to orchestrate things existing in the
overcloud (vm cloud) but you don't want the overcloud administrators to
be able to control your entire stack.

Given CAP theorem, option 5, the global orchestrator, would be doable
with not much change as long as partition tolerance were the bit we gave

Well, in the real world partitions _do_ happen, so the choice is to give
up consistency, availability or both.

up. We would just have to have a cross-region RPC bus and database. Of
course, since regions are most likely to be partitioned, that is not
really a good choice. Trading partition tolerance for consistency lands
us in the complexity black hole. Trading out availability makes it no
better than option 4.

Yep, exactly.

cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Easier way of trying TripleO

2013-11-19 Thread James Slagle

I'd like to propose an idea around a simplified and complimentary version of
devtest that makes it easier for someone to get started and try TripleO.  

The goal being to get people using TripleO as a way to experience the
deployment of OpenStack, and not necessarily a way to get an experience of a
useable OpenStack cloud itself.

To that end, we could:

1) Provide an undercloud vm image so that you could effectively skip the entire
   seed setup.
2) Provide pre-built downloadable images for the overcloud and deployment
   kernel and ramdisk.
3) Instructions on how to use these images to deploy a running
   overcloud.

Images could be provided for Ubuntu and Fedora, since both those work fairly
well today.

The instructions would look something like:

1) Download all the images.
2) Perform initial host setup.  This would be much smaller than what is
   required for devtest and off the top of my head would mostly be:
   - openvswitch bridge setup
   - libvirt configuration
   - ssh configuration (for the baremetal virtual power driver)
3) Start the undercloud vm.  It would need to be bootstrapped with an initial
   static json file for the heat metadata, same as the seed works today.
4) Any last mile manual configuration, such as nova.conf edits for the virtual
   power driver user.
6) Use tuskar+horizon (running on the undercloud)  to deploy the overcloud.
7) Overcloud configuration (don't see this being much different than what is
   there today).

All the openstack clients, heat templates, etc., are on the undercloud vm, and
that's where they're used from, as opposed to from the host (results in less 
stuff
to install/configure on the host).

We could also provide instructions on how to configure the undercloud vm to
provision baremetal.  I assume this would be possible, given the correct
bridged networking setup.

It could make sense to use an all in one overcloud for this as well, given it's
going for simplification.

Obviously, this approach implies some image management on the community's part,
and I think we'd document and use all the existing tools (dib, elements) to
build images, etc.

Thoughts on this approach?  

--
-- James Slagle
--

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Icehouse roadmap status

Greetings,

We've made a lot of progress on reviewing blueprints over the last week
and have the icehouse-1 [1] list in reasonable shape.  Note that
icehouse-1 development must be completed two weeks from today, and that
time frame includes a major holiday in the US.  Please adjust plans and
expectations accordingly.

Here are the goals we need to be aiming for over this next week:

1) File blueprints!  We talked about a *lot* of stuff in the design
summit that is still not reflected in blueprints.  Please get those
blueprints filed and targerted to the appropriate icehouse milestone ASAP!

2) nova-core reviewers should consider committing to reviewing
blueprints they care most about.  That will give us a better picture of
which blueprints are most likely to get reviewed in time to be merged.

The process here is to just add a note to the blueprint whiteboard
indicating yourself as a review sponsor.  For example:

"nova-core sponsors: russellb, alaski"

Once a blueprint has at least two sponsors, its priority can be raised
above Low.

3) nova-drivers: We need to keep up the pace on blueprint reviews.
There's still a lot to review in the overall icehouse list [2].  It's a
lot of work right now, early in the cycle while there's a lot of
planning going on.  It will start to taper off in coming weeks.


Please let me know if you have any questions.

Thanks everyone!


[1] https://launchpad.net/nova/+milestone/icehouse-1
[2] https://blueprints.launchpad.net/nova/icehouse

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] PTL election

2013-11-19 Thread Thierry Carrez

Thierry Carrez wrote:
> Thierry Carrez wrote:
>> The two candidates who nominated themselves in time for this election are:
>>
>> * David Lyle
>> * Matthias Runge
>>
>> The election will be set up tomorrow, and will stay open for voting for
>> a week.

The poll is now closed, and the winner is David Lyle !

You can see results at:

http://www.cs.cornell.edu/w8/~andru/cgi-perl/civs/results.pl?id=E_cd16dd051e519ef2

Congrats, David!

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] PTL election

2013-11-19 Thread Matthias Runge

On 11/19/2013 11:02 PM, Thierry Carrez wrote:

> 
> The poll is now closed, and the winner is David Lyle !
> 
David, well done and honestly deserved!

Matthias


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation

2013-11-19 Thread Salvatore Orlando

For what is worth we have considered this aspect from the perspective of
the Neutron plugin my team maintains (NVP) during the past release cycle.

The synchronous model that most plugins with a controller on the backend
currently implement is simple and convenient, but has some flaws:

- reliability: the current approach where the plugin orchestrates the
backend is not really optimal when it comes to ensuring your running
configuration (backend/control plane) is in sync with your desired
configuration (neutron/mgmt plane); moreover in some case, due to neutron
internals, API calls to the backend are wrapped in a transaction too,
leading to very long SQL transactions, which are quite dangerous indeed. It
is not easy to recover from a failure due to an eventlet thread deadlocking
with a mysql transaction, where by 'recover' I mean ensuring neutron and
backend state are in sync.

- maintainability: since handling rollback in case of failures on the
backend and/or the db is cumbersome, this often leads to spaghetti code
which is very hard to maintain regardless of the effort (ok, I agree here
that this also depends on how good the devs are - most of the guys in my
team are very good, but unfortunately they have me too...).

- performance & scalability:
-  roundtrips to the backend take a non-negligible toll on the duration
of an API call, whereas most Neutron API calls should probably just
terminate at the DB just like a nova boot call does not wait for the VM to
be ACTIVE to return.
- we need to keep some operation serialized in order to avoid the
mentioned race issues

For this reason we're progressively moving toward a change in the NVP
plugin with a series of patches under this umbrella-blueprint [1].

For answering the issues mentioned by Isaku, we've been looking at a task
management library with an efficient and reliable set of abstractions for
ensuring operations are properly ordered thus avoiding those races (I agree
on the observation on the pre/post commit solution).
We are currently looking at using celery [2] rather than taskflow; mostly
because we've already have expertise on how to use it into our
applications, and has very easy abstractions for workflow design, as well
as for handling task failures.
Said that, I think we're still open to switch to taskflow should we become
aware of some very good reason for using it.

Regards,
Salvatore

[1]
https://blueprints.launchpad.net/neutron/+spec/nvp-async-backend-communication
[2] http://docs.celeryproject.org/en/master/index.html



On 19 November 2013 19:42, Joshua Harlow  wrote:

> And also of course, nearly forgot a similar situation/review in heat.
>
> https://review.openstack.org/#/c/49440/
>
> Except theres was/is dealing with stack locking (a heat concept).
>
> On 11/19/13 10:33 AM, "Joshua Harlow"  wrote:
>
> >If you start adding these states you might really want to consider the
> >following work that is going on in other projects.
> >
> >It surely appears that everyone is starting to hit the same problem (and
> >joining efforts would produce a more beneficial result).
> >
> >Relevant icehouse etherpads:
> >- https://etherpad.openstack.org/p/CinderTaskFlowFSM
> >- https://etherpad.openstack.org/p/icehouse-oslo-service-synchronization
> >
> >And of course my obvious plug for taskflow (which is designed to be a
> >useful library to help in all these usages).
> >
> >- https://wiki.openstack.org/wiki/TaskFlow
> >
> >The states u just mentioned start to line-up with
> >https://wiki.openstack.org/wiki/TaskFlow/States_of_Task_and_Flow
> >
> >If this sounds like a useful way to go (joining efforts) then lets see how
> >we can make it possible.
> >
> >IRC: #openstack-state-management is where I am usually at.
> >
> >On 11/19/13 3:57 AM, "Isaku Yamahata"  wrote:
> >
> >>On Mon, Nov 18, 2013 at 03:55:49PM -0500,
> >>Robert Kukura  wrote:
> >>
> >>> On 11/18/2013 03:25 PM, Edgar Magana wrote:
> >>> > Developers,
> >>> >
> >>> > This topic has been discussed before but I do not remember if we have
> >>>a
> >>> > good solution or not.
> >>>
> >>> The ML2 plugin addresses this by calling each MechanismDriver twice.
> >>>The
> >>> create_network_precommit() method is called as part of the DB
> >>> transaction, and the create_network_postcommit() method is called after
> >>> the transaction has been committed. Interactions with devices or
> >>> controllers are done in the postcommit methods. If the postcommit
> >>>method
> >>> raises an exception, the plugin deletes that partially-created resource
> >>> and returns the exception to the client. You might consider a similar
> >>> approach in your plugin.
> >>
> >>Splitting works into two phase, pre/post, is good approach.
> >>But there still remains race window.
> >>Once the transaction is committed, the result is visible to outside.
> >>So the concurrent request to same resource will be racy.
> >>There is a window after pre_xxx_yyy before post_xxx_yyy() where
> >>other requests can be handled.
> >>
>

Re: [openstack-dev] [Heat] Version Negotiation Middleware Accept Header issue

2013-11-19 Thread Angus Salkeld


On 19/11/13 19:40 +, Fuente, Pablo A wrote:

Hi,
I noticed that the Accept HTTP Header checked in the Version
Negotiation Middleware by Heat is the same MIME type used by Glance
("application/vnd.openstack.images-")
Is this OK, or should it be something like
"application/vnd.openstack.orchestration-"?
If this is the case I would proceed to file a bug.


Yeah, that  looks wrong, I posted a fix here:
https://review.openstack.org/57338

-Angus



Thanks.
Pablo.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

2013-11-19 Thread Zane Bitter

On 19/11/13 19:14, Christopher Armstrong wrote:

On Mon, Nov 18, 2013 at 5:57 AM, Zane Bitter mailto:zbit...@redhat.com>> wrote:

On 16/11/13 11:15, Angus Salkeld wrote:

On 15/11/13 08:46 -0600, Christopher Armstrong wrote:

On Fri, Nov 15, 2013 at 3:57 AM, Zane Bitter
mailto:zbit...@redhat.com>> wrote:

On 15/11/13 02:48, Christopher Armstrong wrote:

On Thu, Nov 14, 2013 at 5:40 PM, Angus Salkeld
mailto:asalk...@redhat.com>
>> wrote:

 On 14/11/13 10:19 -0600, Christopher Armstrong
wrote:

http://docs.heatautoscale.__ap__iary.io/

 >

 I've thrown together a rough sketch of the
proposed API for
 autoscaling.
 It's written in API-Blueprint format (which
is a simple subset
 of Markdown)
 and provides schemas for inputs and outputs
using JSON-Schema.
 The source
 document is currently at
https://github.com/radix/heat/raw/as-api-spike/

autoscaling.__apibp

 >

 Things we still need to figure out:

 - how to scope projects/domains. put them
in the URL? get them
 from the
 token?
 - how webhooks are done (though this
shouldn't affect the API
 too much;
 they're basically just opaque)

 Please read and comment :)

 Hi Chistopher

 In the group create object you have 'resources'.
 Can you explain what you expect in there? I
thought we talked at
 summit about have a unit of scaling as a nested
stack.

 The thinking here was:
 - this makes the new config stuff easier to
scale (config get
applied
 Â  per scaling stack)

 - you can potentially place notification
resources in the scaling
 Â  stack (think marconi message resource -
on-create it sends a
 Â  message)

 - no need for a launchconfig
 - you can place a LoadbalancerMember resource
in the scaling stack
 Â  that triggers the loadbalancer to add/remove
it from the lb.

 I guess what I am saying is I'd expect an api
to a nested stack.

Well, what I'm thinking now is that instead of
"resources" (a
mapping of
resources), just have "resource", which can be the
template definition
for a single resource. This would then allow the
user to specify a
Stack
resource if they want to provide multiple resources.
How does that
sound?

My thought was this (digging into the implementation
here a bit):

- Basically, the autoscaling code works as it does now:
creates a
template
containing OS::Nova::Server resources (changed from
AWS::EC2::Instance),
with the properties obtained from the LaunchConfig, and
creates a
stack in
Heat.
- LaunchConfig can now contain any properties you like
(I'm not 100%
sure
about this one*).
- The user optionally supplies a template. If the
template is
supplied, it
is passed to Heat and set in the environment as the
provider for the
OS::Nova::Server resource.

Re: [openstack-dev] [Neutron] Race condition between DB layer and plugin back-end implementation