Re: [openstack-dev] [cinder] Berlin Forum Proposals

[Gorka Eguileor]

On 19/09, Jay S Bryant wrote:
> Gorka,
>
> Oh man!  Sorry for the duplication.  I will update the link on the Forum
> page if you are able to move your content over.  Think it will confused
> people less if we use the page I most recently sent out.  Does that make
> sense?
>
Hi Jay,

Yup, it makes sense.

I moved the contents and updated the wiki to point to your etherpad.

> Thanks for catching this mistake!
>

It was my mistake for not mentioning the existing etherpad during the
PTG... XD

Cheers,
Gorka.


> Jay
>
>
> On 9/19/2018 4:42 AM, Gorka Eguileor wrote:
> > On 18/09, Jay S Bryant wrote:
> > > Team,
> > >
> > > I have created an etherpad for our Forum Topic Planning:
> > > https://etherpad.openstack.org/p/cinder-berlin-forum-proposals
> > >
> > > Please add your ideas to the etherpad.  Thank you!
> > >
> > > Jay
> > >
> > Hi Jay,
> >
> > After our last IRC meeting, a couple of weeks ago, I created an etherpad
> > [1] and added it to the Forum wiki [2] (though I failed to mention it).
> >
> > I had added a possible topic to this etherpad [1], but I can move it to
> > yours and update the wiki if you like.
> >
> > Cheers,
> > Gorka.
> >
> >
> > [1]: https://etherpad.openstack.org/p/cinder-forum-stein
> > [2]: https://wiki.openstack.org/wiki/Forum/Berlin2018
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg][cinder] Stein PTG Summary Page Ready ...

[Gorka Eguileor]

On 18/09, Jay S Bryant wrote:
> Team,
>
> I have put together the following page with a summary of all our discussions
> at the PTG: https://wiki.openstack.org/wiki/CinderSteinPTGSummary
>
> Please review the contents and let me know if anything needs to be changed.
>
> Jay
>
>

Hi Jay,

Thank you for the great summary, it looks great.

After reading it, I can't think of anything that's missing.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Berlin Forum Proposals

[Gorka Eguileor]

On 18/09, Jay S Bryant wrote:
> Team,
>
> I have created an etherpad for our Forum Topic Planning:
> https://etherpad.openstack.org/p/cinder-berlin-forum-proposals
>
> Please add your ideas to the etherpad.  Thank you!
>
> Jay
>

Hi Jay,

After our last IRC meeting, a couple of weeks ago, I created an etherpad
[1] and added it to the Forum wiki [2] (though I failed to mention it).

I had added a possible topic to this etherpad [1], but I can move it to
yours and update the wiki if you like.

Cheers,
Gorka.


[1]: https://etherpad.openstack.org/p/cinder-forum-stein
[2]: https://wiki.openstack.org/wiki/Forum/Berlin2018

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg][cinder][placement] etherpad for this afternoon's meeting

[Gorka Eguileor]

On 11/09, Jay Pipes wrote:
> Hi Jay, where is this discussion taking place?
>

Hi,

It was on another email:

  Big Thompson Room on Tuesday from 15:15 to 17:00

Cheers,
Gorka.

> On Tue, Sep 11, 2018, 11:10 AM Jay S Bryant  wrote:
>
> > All,
> >
> > I have created an etherpad to take notes during our meeting this
> > afternoon:
> > https://etherpad.openstack.org/p/cinder-placement-denver-ptg-2018
> >
> > If you have information you want to get in there before the meeting I
> > would appreciate you pre-populating the pad.
> >
> > Jay
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][ptg] Topics scheduled for next week ...

[Gorka Eguileor]

On 07/09, Jay S Bryant wrote:
> Team,
>
> I have created an etherpad for each of the days of the PTG and split out the
> proposed topics from the planning etherpad into the individual days for
> discussion: [1] [2] [3]
>
> If you want to add an additional topic please add it to Friday or find some
> time on one of the other days.
>
> I look forward to discussing all these topics with you all next week.
>
> Thanks!
>
> Jay

Thanks Jay.

I have added to the Cinder general etherpad the shared_target discussion
topic, as I believe we should be discussing it in the Cinder room first
before Thursday's meeting with Nova.

I saw that on Wednesday the 2:30 to 3:00 privsep topic is a duplicate of
the 12:00 to 12:30 slot, so I have taken the liberty of replacing it
with the shared_targets one.  I hope that's alright.

Cheers,
Gorka.

>
> [1] https://etherpad.openstack.org/p/cinder-ptg-stein-wednesday
>
> [2] https://etherpad.openstack.org/p/cinder-ptg-stein-thursday
>
> [3] https://etherpad.openstack.org/p/cinder-ptg-stein-friday
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

[Gorka Eguileor]

On 29/08, Matt Riedemann wrote:
> On 8/29/2018 3:21 PM, Tim Bell wrote:
> > Sounds like a good topic for PTG/Forum?
>
> Yeah it's already on the PTG agenda [1][2]. I started the thread because I
> wanted to get the ball rolling as early as possible, and with people that
> won't attend the PTG and/or the Forum, to weigh in on not only the known
> issues with cross-cell migration but also the things I'm not thinking about.
>
> [1] https://etherpad.openstack.org/p/nova-ptg-stein
> [2] https://etherpad.openstack.org/p/nova-ptg-stein-cells
>
> --
>
> Thanks,
>
> Matt
>

Should we also add the topic to the Thursday Cinder-Nova slot in case
there are some questions where the Cinder team can assist?

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder][neutron] Cross-cell cold migration

[Gorka Eguileor]

On 24/08, Jay S Bryant wrote:
>
>
> On 8/23/2018 12:07 PM, Gorka Eguileor wrote:
> > On 23/08, Dan Smith wrote:
> > > > I think Nova should never have to rely on Cinder's hosts/backends
> > > > information to do migrations or any other operation.
> > > >
> > > > In this case even if Nova had that info, it wouldn't be the solution.
> > > > Cinder would reject migrations if there's an incompatibility on the
> > > > Volume Type (AZ, Referenced backend, capabilities...)
> > > I think I'm missing a bunch of cinder knowledge required to fully grok
> > > this situation and probably need to do some reading. Is there some
> > > reason that a volume type can't exist in multiple backends or something?
> > > I guess I think of volume type as flavor, and the same definition in two
> > > places would be interchangeable -- is that not the case?
> > >
> > Hi,
> >
> > I just know the basics of flavors, and they are kind of similar, though
> > I'm sure there are quite a few differences.
> >
> > Sure, multiple storage arrays can meet the requirements of a Volume
> > Type, but then when you create the volume you don't know where it's
> > going to land. If your volume type is too generic you volume could land
> > somewhere your cell cannot reach.
> >
> >
> > > > I don't know anything about Nova cells, so I don't know the specifics of
> > > > how we could do the mapping between them and Cinder backends, but
> > > > considering the limited range of possibilities in Cinder I would say we
> > > > only have Volume Types and AZs to work a solution.
> > > I think the only mapping we need is affinity or distance. The point of
> > > needing to migrate the volume would purely be because moving cells
> > > likely means you moved physically farther away from where you were,
> > > potentially with different storage connections and networking. It
> > > doesn't *have* to mean that, but I think in reality it would. So the
> > > question I think Matt is looking to answer here is "how do we move an
> > > instance from a DC in building A to building C and make sure the
> > > volume gets moved to some storage local in the new building so we're
> > > not just transiting back to the original home for no reason?"
> > >
> > > Does that explanation help or are you saying that's fundamentally hard
> > > to do/orchestrate?
> > >
> > > Fundamentally, the cells thing doesn't even need to be part of the
> > > discussion, as the same rules would apply if we're just doing a normal
> > > migration but need to make sure that storage remains affined to compute.
> > >
> > We could probably work something out using the affinity filter, but
> > right now we don't have a way of doing what you need.
> >
> > We could probably rework the migration to accept scheduler hints to be
> > used with the affinity filter and to accept calls with the host or the
> > hints, that way it could migrate a volume without knowing the
> > destination host and decide it based on affinity.
> >
> > We may have to do more modifications, but it could be a way to do it.
> >
> >
> >
> > > > I don't know how the Nova Placement works, but it could hold an
> > > > equivalency mapping of volume types to cells as in:
> > > >
> > > >   Cell#1Cell#2
> > > >
> > > > VolTypeA <--> VolTypeD
> > > > VolTypeB <--> VolTypeE
> > > > VolTypeC <--> VolTypeF
> > > >
> > > > Then it could do volume retypes (allowing migration) and that would
> > > > properly move the volumes from one backend to another.
> > > The only way I can think that we could do this in placement would be if
> > > volume types were resource providers and we assigned them traits that
> > > had special meaning to nova indicating equivalence. Several of the words
> > > in that sentence are likely to freak out placement people, myself
> > > included :)
> > >
> > > So is the concern just that we need to know what volume types in one
> > > backend map to those in another so that when we do the migration we know
> > > what to ask for? Is "they are the same name" not enough? Going back to
> > > the flavor analogy, you could kinda compare two flavor definitions and
> > > have a good idea if they're equivalent or not...
> > >
> > > --Dan
> > In Cinder you don't get that from Volu

Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)

[Gorka Eguileor]

On 24/08, Matt Riedemann wrote:
> On 8/22/2018 4:46 AM, Gorka Eguileor wrote:
> > The solution is conceptually simple.  We add a new API microversion in
> > Cinder that adds and optional parameter called "generic_keep_source"
> > (defaults to False) to both migrate and retype operations.
>
> But if the problem is that users are not using the retype API and instead
> are hitting the compute swap volume API instead, they won't use this new
> parameter anyway. Again, retype is admin-or-owner but volume migration (in
> cinder) and swap volume (in nova) are both admin-only, so are admins calling
> swap volume directly or are people easing up the policy restrictions so
> non-admins can use these migration APIs?
>
> --
>
> Thanks,
>
> Matt

Hi,

These are two different topics, and I thought we had already closed that
part of the discussion.  Nova needs to fix that issue, a good option
could be not allowing non Cinder callers to do that operation.

As mbooth mentioned, these issues come from real user needs, and this is
a different topic, and the topic I was talking about with the proposed
solution.  I agree with mbooth that we should find a way to address
these customer needs, even if it's in a limited way.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)

[Gorka Eguileor]

On 25/08, Sean McGinnis wrote:
> On Fri, Aug 24, 2018 at 04:20:21PM -0500, Matt Riedemann wrote:
> > On 8/20/2018 10:29 AM, Matthew Booth wrote:
> > > Secondly, is there any reason why we shouldn't just document then you
> > > have to delete snapshots before doing a volume migration? Hopefully
> > > some cinder folks or operators can chime in to let me know how to back
> > > them up or somehow make them independent before doing this, at which
> > > point the volume itself should be migratable?
> >
> > Coincidentally the volume migration API never had API reference
> > documentation. I have that here now [1]. It clearly states the preconditions
> > to migrate a volume based on code in the volume API. However, volume
> > migration is admin-only by default and retype (essentially like resize) is
> > admin-or-owner so non-admins can do it and specify to migrate. In general I
> > think it's best to have preconditions for *any* API documented, so anything
> > needed to perform a retype should be documented in the API, like that the
> > volume can't have snapshots.
>
> That's where things get tricky though. There aren't really reconditions we can
> have as a blanket statement with the retype API.
>
> A retype can do a lot of different things, all dependent on what type you are
> coming from and trying to go to. There are some retypes where all it does is
> enable vendor flag ``foo`` on the volume with no change in any other state.
> Then there are other retypes (using --migrate-policy on-demand) that 
> completely
> move the volume from one backend to another one, copying every block along the
> way from the original to the new volume. It really depends on what types you
> are trying to retype to.
>

We can say that retypes that require migration between different vendor
backends cannot be performed with snapshots, and between arrays from the
same vendor will depend on the driver (though I don't know if any driver
can actually pull this off).

Cheers,
Gorka.


> >
> > [1] https://review.openstack.org/#/c/595379/
> >
> > --
> >
> > Thanks,
> >
> > Matt
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder][neutron] Cross-cell cold migration

[Gorka Eguileor]

On 23/08, Dan Smith wrote:
> > I think Nova should never have to rely on Cinder's hosts/backends
> > information to do migrations or any other operation.
> >
> > In this case even if Nova had that info, it wouldn't be the solution.
> > Cinder would reject migrations if there's an incompatibility on the
> > Volume Type (AZ, Referenced backend, capabilities...)
>
> I think I'm missing a bunch of cinder knowledge required to fully grok
> this situation and probably need to do some reading. Is there some
> reason that a volume type can't exist in multiple backends or something?
> I guess I think of volume type as flavor, and the same definition in two
> places would be interchangeable -- is that not the case?
>

Hi,

I just know the basics of flavors, and they are kind of similar, though
I'm sure there are quite a few differences.

Sure, multiple storage arrays can meet the requirements of a Volume
Type, but then when you create the volume you don't know where it's
going to land. If your volume type is too generic you volume could land
somewhere your cell cannot reach.


> > I don't know anything about Nova cells, so I don't know the specifics of
> > how we could do the mapping between them and Cinder backends, but
> > considering the limited range of possibilities in Cinder I would say we
> > only have Volume Types and AZs to work a solution.
>
> I think the only mapping we need is affinity or distance. The point of
> needing to migrate the volume would purely be because moving cells
> likely means you moved physically farther away from where you were,
> potentially with different storage connections and networking. It
> doesn't *have* to mean that, but I think in reality it would. So the
> question I think Matt is looking to answer here is "how do we move an
> instance from a DC in building A to building C and make sure the
> volume gets moved to some storage local in the new building so we're
> not just transiting back to the original home for no reason?"
>
> Does that explanation help or are you saying that's fundamentally hard
> to do/orchestrate?
>
> Fundamentally, the cells thing doesn't even need to be part of the
> discussion, as the same rules would apply if we're just doing a normal
> migration but need to make sure that storage remains affined to compute.
>

We could probably work something out using the affinity filter, but
right now we don't have a way of doing what you need.

We could probably rework the migration to accept scheduler hints to be
used with the affinity filter and to accept calls with the host or the
hints, that way it could migrate a volume without knowing the
destination host and decide it based on affinity.

We may have to do more modifications, but it could be a way to do it.



> > I don't know how the Nova Placement works, but it could hold an
> > equivalency mapping of volume types to cells as in:
> >
> >  Cell#1Cell#2
> >
> > VolTypeA <--> VolTypeD
> > VolTypeB <--> VolTypeE
> > VolTypeC <--> VolTypeF
> >
> > Then it could do volume retypes (allowing migration) and that would
> > properly move the volumes from one backend to another.
>
> The only way I can think that we could do this in placement would be if
> volume types were resource providers and we assigned them traits that
> had special meaning to nova indicating equivalence. Several of the words
> in that sentence are likely to freak out placement people, myself
> included :)
>
> So is the concern just that we need to know what volume types in one
> backend map to those in another so that when we do the migration we know
> what to ask for? Is "they are the same name" not enough? Going back to
> the flavor analogy, you could kinda compare two flavor definitions and
> have a good idea if they're equivalent or not...
>
> --Dan

In Cinder you don't get that from Volume Types, unless all your backends
have the same hardware and are configured exactly the same.

There can be some storage specific information there, which doesn't
correlate to anything on other hardware.  Volume types may refer to a
specific pool that has been configured in the array to use specific type
of disks.  But even the info on the type of disks is unknown to the
volume type.

I haven't checked the PTG agenda yet, but is there a meeting on this?
Because we may want to have one to try to understand the requirements
and figure out if there's a way to do it with current Cinder
functionality of if we'd need something new.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder][neutron] Cross-cell cold migration

[Gorka Eguileor]

On 22/08, Matt Riedemann wrote:
> Hi everyone,
>
> I have started an etherpad for cells topics at the Stein PTG [1]. The main
> issue in there right now is dealing with cross-cell cold migration in nova.
>
> At a high level, I am going off these requirements:
>
> * Cells can shard across flavors (and hardware type) so operators would like
> to move users off the old flavors/hardware (old cell) to new flavors in a
> new cell.
>
> * There is network isolation between compute hosts in different cells, so no
> ssh'ing the disk around like we do today. But the image service is global to
> all cells.
>
> Based on this, for the initial support for cross-cell cold migration, I am
> proposing that we leverage something like shelve offload/unshelve
> masquerading as resize. We shelve offload from the source cell and unshelve
> in the target cell. This should work for both volume-backed and
> non-volume-backed servers (we use snapshots for shelved offloaded
> non-volume-backed servers).
>
> There are, of course, some complications. The main ones that I need help
> with right now are what happens with volumes and ports attached to the
> server. Today we detach from the source and attach at the target, but that's
> assuming the storage backend and network are available to both hosts
> involved in the move of the server. Will that be the case across cells? I am
> assuming that depends on the network topology (are routed networks being
> used?) and storage backend (routed storage?). If the network and/or storage
> backend are not available across cells, how do we migrate volumes and ports?
> Cinder has a volume migrate API for admins but I do not know how nova would
> know the proper affinity per-cell to migrate the volume to the proper host
> (cinder does not have a routed storage concept like routed provider networks
> in neutron, correct?). And as far as I know, there is no such thing as port
> migration in Neutron.

Hi Matt,

I think Nova should never have to rely on Cinder's hosts/backends
information to do migrations or any other operation.

In this case even if Nova had that info, it wouldn't be the solution.
Cinder would reject migrations if there's an incompatibility on the
Volume Type (AZ, Referenced backend, capabilities...)

I don't know anything about Nova cells, so I don't know the specifics of
how we could do the mapping between them and Cinder backends, but
considering the limited range of possibilities in Cinder I would say we
only have Volume Types and AZs to work a solution.

>
> Could Placement help with the volume/port migration stuff? Neutron routed
> provider networks rely on placement aggregates to schedule the VM to a
> compute host in the same network segment as the port used to create the VM,
> however, if that segment does not span cells we are kind of stuck, correct?
>

I don't know how the Nova Placement works, but it could hold an
equivalency mapping of volume types to cells as in:

 Cell#1Cell#2

VolTypeA <--> VolTypeD
VolTypeB <--> VolTypeE
VolTypeC <--> VolTypeF

Then it could do volume retypes (allowing migration) and that would
properly move the volumes from one backend to another.

Cheers,
Gorka.


> To summarize the issues as I see them (today):
>
> * How to deal with the targeted cell during scheduling? This is so we can
> even get out of the source cell in nova.
>
> * How does the API deal with the same instance being in two DBs at the same
> time during the move?
>
> * How to handle revert resize?
>
> * How are volumes and ports handled?
>
> I can get feedback from my company's operators based on what their
> deployment will look like for this, but that does not mean it will work for
> others, so I need as much feedback from operators, especially those running
> with multiple cells today, as possible. Thanks in advance.
>
> [1] https://etherpad.openstack.org/p/nova-ptg-stein-cells
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)

[Gorka Eguileor]

On 22/08, Matthew Booth wrote:
> On Wed, 22 Aug 2018 at 10:47, Gorka Eguileor  wrote:
> >
> > On 20/08, Matthew Booth wrote:
> > > For those who aren't familiar with it, nova's volume-update (also
> > > called swap volume by nova devs) is the nova part of the
> > > implementation of cinder's live migration (also called retype).
> > > Volume-update is essentially an internal cinder<->nova api, but as
> > > that's not a thing it's also unfortunately exposed to users. Some
> > > users have found it and are using it, but because it's essentially an
> > > internal cinder<->nova api it breaks pretty easily if you don't treat
> > > it like a special snowflake. It looks like we've finally found a way
> > > it's broken for non-cinder callers that we can't fix, even with a
> > > dirty hack.
> > >
> > > volume-updateessentially does a live copy of the
> > > data on  volume to  volume, then seamlessly swaps the
> > > attachment to  from  to . The guest OS on 
> > > will not notice anything at all as the hypervisor swaps the storage
> > > backing an attached volume underneath it.
> > >
> > > When called by cinder, as intended, cinder does some post-operation
> > > cleanup such that  is deleted and  inherits the same
> > > volume_id; that is  effectively becomes . When called any
> > > other way, however, this cleanup doesn't happen, which breaks a bunch
> > > of assumptions. One of these is that a disk's serial number is the
> > > same as the attached volume_id. Disk serial number, in KVM at least,
> > > is immutable, so can't be updated during volume-update. This is fine
> > > if we were called via cinder, because the cinder cleanup means the
> > > volume_id stays the same. If called any other way, however, they no
> > > longer match, at least until a hard reboot when it will be reset to
> > > the new volume_id. It turns out this breaks live migration, but
> > > probably other things too. We can't think of a workaround.
> > >
> > > I wondered why users would want to do this anyway. It turns out that
> > > sometimes cinder won't let you migrate a volume, but nova
> > > volume-update doesn't do those checks (as they're specific to cinder
> > > internals, none of nova's business, and duplicating them would be
> > > fragile, so we're not adding them!). Specifically we know that cinder
> > > won't let you migrate a volume with snapshots. There may be other
> > > reasons. If cinder won't let you migrate your volume, you can still
> > > move your data by using nova's volume-update, even though you'll end
> > > up with a new volume on the destination, and a slightly broken
> > > instance. Apparently the former is a trade-off worth making, but the
> > > latter has been reported as a bug.
> > >
> >
> > Hi Matt,
> >
> > As you know, I'm in favor of making this REST API call only authorized
> > for Cinder to avoid messing the cloud.
> >
> > I know you wanted Cinder to have a solution to do live migrations of
> > volumes with snapshots, and while this is not possible to do in a
> > reasonable fashion, I kept thinking about it given your strong feelings
> > to provide a solution for users that really need this, and I think we
> > may have a "reasonable" compromise.
> >
> > The solution is conceptually simple.  We add a new API microversion in
> > Cinder that adds and optional parameter called "generic_keep_source"
> > (defaults to False) to both migrate and retype operations.
> >
> > This means that if the driver optimized migration cannot do the
> > migration and the generic migration code is the one doing the migration,
> > then, instead of our final step being to swap the volume id's and
> > deleting the source volume, what we would do is to swap the volume id's
> > and move all the snapshots to reference the new volume.  Then we would
> > create a user message with the new ID of the volume.
> >
> > This way we can preserve the old volume with all its snapshots and do
> > the live migration.
> >
> > The implementation is a little bit tricky, as we'll have to add anew
> > "update_migrated_volume" mechanism to support the renaming of both
> > volumes, since the old one wouldn't work with this among other things,
> > but it's doable.
> >
> > Unfortunately I don't have the time right now to work on this...
>
> Sounds promising, and honestly more than I'd have hoped for.
>
> Matt
>

Hi Matt,

Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)

[Gorka Eguileor]

On 22/08, Sean McGinnis wrote:
> >
> > The solution is conceptually simple.  We add a new API microversion in
> > Cinder that adds and optional parameter called "generic_keep_source"
> > (defaults to False) to both migrate and retype operations.
> >
> > This means that if the driver optimized migration cannot do the
> > migration and the generic migration code is the one doing the migration,
> > then, instead of our final step being to swap the volume id's and
> > deleting the source volume, what we would do is to swap the volume id's
> > and move all the snapshots to reference the new volume.  Then we would
> > create a user message with the new ID of the volume.
> >
>
> How would you propose to "move all the snapshots to reference the new volume"?
> Most storage does not allow a snapshot to be moved from one volume to another.
> really the only way a migration of a snapshot can work across all storage 
> types
> would be to incrementally copy the data from a source to a destination up to
> the point of the oldest snapshot, create a new snapshot on the new volume, 
> then
> proceed through until all snapshots have been rebuilt on the new volume.
>

Hi Sean,

Sorry, I phrased that wrong. When I say move the snapshots to the new
volume I mean to the "New Volume DB entry", which is now pointing to the
old volume.

So we wouldn't really be moving the snapshots, we would just be leaving
the old volume with its snapshots under a new UUID, and the old UUID
that the user had attached to Nova will be referencing the new volume.

Again, sorry for the confusion.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)

[Gorka Eguileor]

On 20/08, Matthew Booth wrote:
> For those who aren't familiar with it, nova's volume-update (also
> called swap volume by nova devs) is the nova part of the
> implementation of cinder's live migration (also called retype).
> Volume-update is essentially an internal cinder<->nova api, but as
> that's not a thing it's also unfortunately exposed to users. Some
> users have found it and are using it, but because it's essentially an
> internal cinder<->nova api it breaks pretty easily if you don't treat
> it like a special snowflake. It looks like we've finally found a way
> it's broken for non-cinder callers that we can't fix, even with a
> dirty hack.
>
> volume-updateessentially does a live copy of the
> data on  volume to  volume, then seamlessly swaps the
> attachment to  from  to . The guest OS on 
> will not notice anything at all as the hypervisor swaps the storage
> backing an attached volume underneath it.
>
> When called by cinder, as intended, cinder does some post-operation
> cleanup such that  is deleted and  inherits the same
> volume_id; that is  effectively becomes . When called any
> other way, however, this cleanup doesn't happen, which breaks a bunch
> of assumptions. One of these is that a disk's serial number is the
> same as the attached volume_id. Disk serial number, in KVM at least,
> is immutable, so can't be updated during volume-update. This is fine
> if we were called via cinder, because the cinder cleanup means the
> volume_id stays the same. If called any other way, however, they no
> longer match, at least until a hard reboot when it will be reset to
> the new volume_id. It turns out this breaks live migration, but
> probably other things too. We can't think of a workaround.
>
> I wondered why users would want to do this anyway. It turns out that
> sometimes cinder won't let you migrate a volume, but nova
> volume-update doesn't do those checks (as they're specific to cinder
> internals, none of nova's business, and duplicating them would be
> fragile, so we're not adding them!). Specifically we know that cinder
> won't let you migrate a volume with snapshots. There may be other
> reasons. If cinder won't let you migrate your volume, you can still
> move your data by using nova's volume-update, even though you'll end
> up with a new volume on the destination, and a slightly broken
> instance. Apparently the former is a trade-off worth making, but the
> latter has been reported as a bug.
>

Hi Matt,

As you know, I'm in favor of making this REST API call only authorized
for Cinder to avoid messing the cloud.

I know you wanted Cinder to have a solution to do live migrations of
volumes with snapshots, and while this is not possible to do in a
reasonable fashion, I kept thinking about it given your strong feelings
to provide a solution for users that really need this, and I think we
may have a "reasonable" compromise.

The solution is conceptually simple.  We add a new API microversion in
Cinder that adds and optional parameter called "generic_keep_source"
(defaults to False) to both migrate and retype operations.

This means that if the driver optimized migration cannot do the
migration and the generic migration code is the one doing the migration,
then, instead of our final step being to swap the volume id's and
deleting the source volume, what we would do is to swap the volume id's
and move all the snapshots to reference the new volume.  Then we would
create a user message with the new ID of the volume.

This way we can preserve the old volume with all its snapshots and do
the live migration.

The implementation is a little bit tricky, as we'll have to add anew
"update_migrated_volume" mechanism to support the renaming of both
volumes, since the old one wouldn't work with this among other things,
but it's doable.

Unfortunately I don't have the time right now to work on this...

Cheers,
Gorka.


> I'd like to make it very clear that nova's volume-update, isn't
> expected to work correctly except when called by cinder. Specifically
> there was a proposal that we disable volume-update from non-cinder
> callers in some way, possibly by asserting volume state that can only
> be set by cinder. However, I'm also very aware that users are calling
> volume-update because it fills a need, and we don't want to trap data
> that wasn't previously trapped.
>
> Firstly, is anybody aware of any other reasons to use nova's
> volume-update directly?
>
> Secondly, is there any reason why we shouldn't just document then you
> have to delete snapshots before doing a volume migration? Hopefully
> some cinder folks or operators can chime in to let me know how to back
> them up or somehow make them independent before doing this, at which
> point the volume itself should be migratable?
>
> If we can establish that there's an acceptable alternative to calling
> volume-update directly for all use-cases we're aware of, I'm going to
> propose heading off this class of bug by disabling it for 

Re: [openstack-dev] [cinder][nova] Proper behavior for os-force_detach

[Gorka Eguileor]

On 17/07, Sean McGinnis wrote:
> On Tue, Jul 17, 2018 at 04:06:29PM -0300, Erlon Cruz wrote:
> > Hi Cinder and Nova folks,
> >
> > Working on some tests for our drivers, I stumbled upon this tempest test
> > 'force_detach_volume'
> > that is calling Cinder API passing a 'None' connector. At the time this was
> > added several CIs
> > went down, and people started discussing whether this (accepting/sending a
> > None connector)
> > would be the proper behavior for what is expected to a driver to do[1]. So,
> > some of CIs started
> > just skipping that test[2][3][4] and others implemented fixes that made the
> > driver to disconnected
> > the volume from all hosts if a None connector was received[5][6][7].
>
> Right, it was determined the correct behavior for this was to disconnect the
> volume from all hosts. The CIs that are skipping this test should stop doing 
> so
> (once their drivers are fixed of course).
>
> >
> > While implementing this fix seems to be straightforward, I feel that just
> > removing the volume
> > from all hosts is not the correct thing to do mainly considering that we
> > can have multi-attach.
> >
>
> I don't think multiattach makes a difference here. Someone is forcibly
> detaching the volume and not specifying an individual connection. So based on
> that, Cinder should be removing any connections, whether that is to one or
> several hosts.
>

Hi,

I agree with Sean, drivers should remove all connections for the volume.

Even without multiattach there are cases where you'll have multiple
connections for the same volume, like in a Live Migration.

It's also very useful when Nova and Cinder get out of sync and your
volume has leftover connections. In this case if you try to delete the
volume you get a "volume in use" error from some drivers.

Cheers,
Gorka.


> > So, my questions are: What is the best way to fix this problem? Should
> > Cinder API continue to
> > accept detachments with None connectors? If, so, what would be the effects
> > on other Nova
> > attachments for the same volume? Is there any side effect if the volume is
> > not multi-attached?
> >
> > Additionally to this thread here, I should bring this topic to tomorrow's
> > Cinder's meeting,
> > so please join if you have something to share.
> >
>
> +1 - good plan.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] about block device driver

[Gorka Eguileor]

On 16/07, Rambo wrote:
> Well,in my opinion,the BlockDeviceDriver is more suitable than any other 
> solution for data processing scenarios.Does the community will agree to merge 
> the BlockDeviceDriver to the Cinder repository again if our company hold the 
> maintainer and CI?
>

Hi,

I'm sure the community will be happy to merge the driver back into the
repository.

Still, I would recommend you looking at the "How To Contribute a driver to
Cinder" guide [1] and the "Third Party CI Requirement Policy"
documentation [2], and then adding this topic to Wednesday's meeting [3]
and go to the meeting to ensure that everybody is on board with it.

Best regards,
Gorka.


[1]: https://wiki.openstack.org/wiki/Cinder/how-to-contribute-a-driver
[2]: https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers
[3]: https://etherpad.openstack.org/p/cinder-rocky-meeting-agendas

>
> ------ Original --
> From: "Gorka Eguileor";
> Date: 2018年7月16日(星期一) 下午5:20
> To: "OpenStack Developmen";
> Subject: Re: [openstack-dev] [cinder] about block device driver
>
>
> On 16/07, Rambo wrote:
> > Hi,all
> >
> >
> >  In the Cinder repository, I noticed that the BlockDeviceDriver driver 
> > is being deprecated, and was eventually be removed with the Queens release.
> >
> >
> > https://github.com/openstack/cinder/blob/stable/ocata/cinder/volume/drivers/block_device.py
> >
> >
> > In my use case, the instances using Cinder perform intense I/O, thus iSCSI 
> > or LVM is not a viable option - benchmarked them several times, since Juno, 
> > unsatisfactory results.For data processing scenarios is always better to 
> > use local storage than any SAN/NAS solution.
> >
> >
> > So I felt a great need to know why we deprecated it.If there has any better 
> > one to replace it? What do you suggest to use once BlockDeviceDriver is 
> > removed?Can you tell me about this?Thank you very much!
> >
> > Best Regards
> > Rambo
>
> Hi,
>
> If I remember correctly the driver was deprecated because it had no
> maintainer or CI.  In Cinder we require our drivers to have both,
> otherwise we can't guarantee that they actually work or that anyone will
> fix it if it gets broken.
>
> Cheers,
> Gorka.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-community] Give cinder-backup more CPU resources

[Gorka Eguileor]

On 06/07, Amy Marrich wrote:
> Hey,
>
> Forwarding to the Dev list as you may get a better response from there.
>
> Thanks,
>
> Amy (spotz)
>

Hi,

I included a good number of improvements regarding bottlenecks on the
backup service in recent releases, some of them are automatic, and
others need configuration tweaks.  But even in older releases there are
a couple of things that can be done to mitigate/fix this situation,
depending on the release you are using.

In general I recommend:

- Using more than one process:

  Use the "backup_processes" option, if available, to run multiple
  subprocesses (like Alan suggested).

  If this option is not available run multiple backup services on the
  same machine (like Duncan suggests), but use a configuration overlay
  to change the host configuration option. That way you won't have
  problems with one service messing up the others' resources at start
  time.

- Increase the native thread pool size: This is even more critical if
  you are using RBD backend:

  Using the "backend_native_threads_pool_size" configuration option [1],
  if available, we can increase the default size.

  If your deployment doesn't have this option, you can still use
  environmental variable "EVENTLET_THREADPOOL_SIZE" to set it in any
  release.

Cheers,
Gorka.

[1]: 
https://github.com/openstack/cinder/commit/e570436d1cca5cfa89388aec8b2daa63d01d0250

> On Thu, Jul 5, 2018 at 11:30 PM, Keynes Lee/WHQ/Wistron <
> keynes_...@wistron.com> wrote:
>
> > Hi
> >
> >
> >
> > When making “cinder backup-create”
> >
> > We found the process “cinder-backup” use 100% util of 1 CPU core on an
> > OpenStack Controller node.
> >
> > It not just causes a bad backup performance, also make the
> > openstack-cinder-backup unstable.
> >
> > Especially when we make several backup at the same time.
> >
> >
> >
> > The Controller Node has 40 CPU cores.
> >
> > Can we assign more CPU resources to cinder-backup ?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > [image: cid:image007.jpg@01D1747D.DB260110]
> >
> > *Keynes  Lee**李* *俊* *賢*
> >
> > Direct:
> >
> > +886-2-6612-1025
> >
> > Mobile:
> >
> > +886-9-1882-3787
> >
> > Fax:
> >
> > +886-2-6612-1991
> >
> >
> >
> > E-Mail:
> >
> > keynes_...@wistron.com
> >
> >
> >
> >
> >
> >
> > *---*
> >
> > *This email contains confidential or legally privileged information and is
> > for the sole use of its intended recipient. *
> >
> > *Any unauthorized review, use, copying or distribution of this email or
> > the content of this email is strictly prohibited.*
> >
> > *If you are not the intended recipient, you may reply to the sender and
> > should delete this e-mail immediately.*
> >
> >
> > *---*
> >
> > ___
> > Community mailing list
> > commun...@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/community
> >
> >



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] about block device driver

[Gorka Eguileor]

On 16/07, Rambo wrote:
> Hi,all
>
>
>  In the Cinder repository, I noticed that the BlockDeviceDriver driver is 
> being deprecated, and was eventually be removed with the Queens release.
>
>
> https://github.com/openstack/cinder/blob/stable/ocata/cinder/volume/drivers/block_device.py
>
>
> In my use case, the instances using Cinder perform intense I/O, thus iSCSI or 
> LVM is not a viable option - benchmarked them several times, since Juno, 
> unsatisfactory results.For data processing scenarios is always better to use 
> local storage than any SAN/NAS solution.
>
>
> So I felt a great need to know why we deprecated it.If there has any better 
> one to replace it? What do you suggest to use once BlockDeviceDriver is 
> removed?Can you tell me about this?Thank you very much!
>
> Best Regards
> Rambo

Hi,

If I remember correctly the driver was deprecated because it had no
maintainer or CI.  In Cinder we require our drivers to have both,
otherwise we can't guarantee that they actually work or that anyone will
fix it if it gets broken.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Cinder] Reusing Cinder drivers' code directly without running Cinder: In our applications, from Ansible, and for Containers

[Gorka Eguileor]

Hi,

During the last OpenStack PTG, I announced in the Cinder room the
development of cinderlib, and explained how this library allowed any
Python application to use Cinder storage drivers (there are over 80)
without running any services.

This takes the standalone effort one step further. Now you don't need to
run any Cinder services (API, Scheduler, Volume), RabbitMQ, or even a
DB, to manage and attach volumes and snapshots.

Even though we don't need a DB we still need to persist the metadata,
but the library supports JSON serialization so we can save it wherever
we want.  I'm also finishing a metadata persistence plugin mechanism to
allow external plugins for different storage solutions (DB, K8s CRDs,
Key-Value systems...).

This library opens a broad range of possibilities for the Cinder
drivers, and I have explored a couple of them: Using it from Ansible and
in containers with CSI driver that includes the latest features
including snapshots that were introduced last week.

The projects' documentation is lacking, but I've written a couple of
blog posts with a brief introduction to these POCs for anybody that is
interested:

- Cinderlib: https://gorka.eguileor.com/cinderlib
- Ansible storage role: https://gorka.eguileor.com/ansible-role-storage
- Cinderlib-CSI: https://gorka.eguileor.com/cinderlib-csi

And the repositories can be found in GitHub:

- Cinderlib: https://github.com/akrog/cinderlib
- Ansible storage role: https://github.com/akrog/ansible-role-storage
- Cinderlib-CSI:https://github.com/akrog/cinderlib-csi

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?

[Gorka Eguileor]

On 09/04, Michael Bayer wrote:
> On Mon, Apr 9, 2018 at 5:53 AM, Gorka Eguileor <gegui...@redhat.com> wrote:
> > On 06/04, Michael Bayer wrote:
> >> On Wed, Apr 4, 2018 at 5:00 AM, Gorka Eguileor <gegui...@redhat.com> wrote:
> >> > On 03/04, Jay Pipes wrote:
> >> >> On 04/03/2018 11:07 AM, Michael Bayer wrote:
> >> >> > The MySQL / MariaDB variants we use nowadays default to
> >> >> > innodb_file_per_table=ON and we also set this flag to ON in installer
> >> >> > tools like TripleO. The reason we like file per table is so that
> >> >> > we don't grow an enormous ibdata file that can't be shrunk without
> >> >> > rebuilding the database.  Instead, we have lots of little .ibd
> >> >> > datafiles for each table throughout each openstack database.
> >> >> >
> >> >> > But now we have the issue that these files also can benefit from
> >> >> > periodic optimization which can shrink them and also have a beneficial
> >> >> > effect on performance.   The OPTIMIZE TABLE statement achieves this,
> >> >> > but as would be expected it itself can lock tables for potentially a
> >> >> > long time.   Googling around reveals a lot of controversy, as various
> >> >> > users and publications suggest that OPTIMIZE is never needed and would
> >> >> > have only a negligible effect on performance.   However here we seek
> >> >> > to use OPTIMIZE so that we can reclaim disk space on tables that have
> >> >> > lots of DELETE activity, such as keystone "token" and ceilometer
> >> >> > "sample".
> >> >> >
> >> >> > Questions for the group:
> >> >> >
> >> >> > 1. is OPTIMIZE table worthwhile to be run for tables where the
> >> >> > datafile has grown much larger than the number of rows we have in the
> >> >> > table?
> >> >>
> >> >> Possibly, though it's questionable to use MySQL/InnoDB for storing 
> >> >> transient
> >> >> data that is deleted often like ceilometer samples and keystone tokens. 
> >> >> A
> >> >> much better solution is to use RDBMS partitioning so you can simply 
> >> >> ALTER
> >> >> TABLE .. DROP PARTITION those partitions that are no longer relevant 
> >> >> (and
> >> >> don't even bother DELETEing individual rows) or, in the case of 
> >> >> Ceilometer
> >> >> samples, don't use a traditional RDBMS for timeseries data at all...
> >> >>
> >> >> But since that is unfortunately already the case, yes it is probably a 
> >> >> good
> >> >> idea to OPTIMIZE TABLE on those tables.
> >> >>
> >> >> > 2. from people's production experience how safe is it to run OPTIMIZE,
> >> >> > e.g. how long is it locking tables, etc.
> >> >>
> >> >> Is it safe? Yes.
> >> >>
> >> >> Does it lock the entire table for the duration of the operation? No. It 
> >> >> uses
> >> >> online DDL operations:
> >> >>
> >> >> https://dev.mysql.com/doc/refman/5.7/en/innodb-file-defragmenting.html
> >> >>
> >> >> Note that OPTIMIZE TABLE is mapped to ALTER TABLE tbl_name FORCE for 
> >> >> InnoDB
> >> >> tables.
> >> >>
> >> >> > 3. is there a heuristic we can use to measure when we might run this
> >> >> > -.e.g my plan is we measure the size in bytes of each row in a table
> >> >> > and then compare that in some ratio to the size of the corresponding
> >> >> > .ibd file, if the .ibd file is N times larger than the logical data
> >> >> > size we run OPTIMIZE ?
> >> >>
> >> >> I don't believe so, no. Most things I see recommended is to simply run
> >> >> OPTIMIZE TABLE in a cron job on each table periodically.
> >> >>
> >> >> > 4. I'd like to propose this job of scanning table datafile sizes in
> >> >> > ratio to logical data sizes, then running OPTIMIZE, be a utility
> >> >> > script that is delivered via oslo.db, and would run for all innodb
> >> >> > tables within a target MySQL/ MariaDB server generically.  That is, I
> >> >> > really *dont*

Re: [openstack-dev] [cinder][nova] about re-image the volume

[Gorka Eguileor]

On 09/04, Sean McGinnis wrote:
> On Mon, Apr 09, 2018 at 07:00:56PM +0100, Duncan Thomas wrote:
> > Hopefully this flow means we can do rebuild root filesystem from
> > snapshot/backup too? It seems rather artificially limiting to only do
> > restore-from-image. I'd expect restore-from-snap to be a more common
> > use case, personally.
> >
>
> That could get tricky. We only support reverting to the last snapshot if we
> reuse the same volume. Otherwise, we can create volume from snapshot, but I
> don't think it's often that the first thing a user does is create a snapshot 
> on
> initial creation of a boot image. If it was created from image cache, and the
> backend creates those cached volume by using a snapshot, then that might be an
> option.
>
> But these are a lot of ifs, so that seems like it would make the logic for 
> this
> much more complicated.
>
> Maybe a phase II optimization we can look into?
>

From the Cinder side of things I think these two would be easier than
the re-image, because we would have even fewer steps, and the
functionality to do the copying is exactly what we have now, as it will
copy the data to the same volume, so we wouldn't need to fiddle with the
UUID fields etc.

Moreover I know customers who have asked about this functionality in the
past, mostly interested in restoring the root volume of an existing VM
from a backup to preserve the system ID and not break licenses.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?

[Gorka Eguileor]

On 06/04, Michael Bayer wrote:
> On Wed, Apr 4, 2018 at 5:00 AM, Gorka Eguileor <gegui...@redhat.com> wrote:
> > On 03/04, Jay Pipes wrote:
> >> On 04/03/2018 11:07 AM, Michael Bayer wrote:
> >> > The MySQL / MariaDB variants we use nowadays default to
> >> > innodb_file_per_table=ON and we also set this flag to ON in installer
> >> > tools like TripleO. The reason we like file per table is so that
> >> > we don't grow an enormous ibdata file that can't be shrunk without
> >> > rebuilding the database.  Instead, we have lots of little .ibd
> >> > datafiles for each table throughout each openstack database.
> >> >
> >> > But now we have the issue that these files also can benefit from
> >> > periodic optimization which can shrink them and also have a beneficial
> >> > effect on performance.   The OPTIMIZE TABLE statement achieves this,
> >> > but as would be expected it itself can lock tables for potentially a
> >> > long time.   Googling around reveals a lot of controversy, as various
> >> > users and publications suggest that OPTIMIZE is never needed and would
> >> > have only a negligible effect on performance.   However here we seek
> >> > to use OPTIMIZE so that we can reclaim disk space on tables that have
> >> > lots of DELETE activity, such as keystone "token" and ceilometer
> >> > "sample".
> >> >
> >> > Questions for the group:
> >> >
> >> > 1. is OPTIMIZE table worthwhile to be run for tables where the
> >> > datafile has grown much larger than the number of rows we have in the
> >> > table?
> >>
> >> Possibly, though it's questionable to use MySQL/InnoDB for storing 
> >> transient
> >> data that is deleted often like ceilometer samples and keystone tokens. A
> >> much better solution is to use RDBMS partitioning so you can simply ALTER
> >> TABLE .. DROP PARTITION those partitions that are no longer relevant (and
> >> don't even bother DELETEing individual rows) or, in the case of Ceilometer
> >> samples, don't use a traditional RDBMS for timeseries data at all...
> >>
> >> But since that is unfortunately already the case, yes it is probably a good
> >> idea to OPTIMIZE TABLE on those tables.
> >>
> >> > 2. from people's production experience how safe is it to run OPTIMIZE,
> >> > e.g. how long is it locking tables, etc.
> >>
> >> Is it safe? Yes.
> >>
> >> Does it lock the entire table for the duration of the operation? No. It 
> >> uses
> >> online DDL operations:
> >>
> >> https://dev.mysql.com/doc/refman/5.7/en/innodb-file-defragmenting.html
> >>
> >> Note that OPTIMIZE TABLE is mapped to ALTER TABLE tbl_name FORCE for InnoDB
> >> tables.
> >>
> >> > 3. is there a heuristic we can use to measure when we might run this
> >> > -.e.g my plan is we measure the size in bytes of each row in a table
> >> > and then compare that in some ratio to the size of the corresponding
> >> > .ibd file, if the .ibd file is N times larger than the logical data
> >> > size we run OPTIMIZE ?
> >>
> >> I don't believe so, no. Most things I see recommended is to simply run
> >> OPTIMIZE TABLE in a cron job on each table periodically.
> >>
> >> > 4. I'd like to propose this job of scanning table datafile sizes in
> >> > ratio to logical data sizes, then running OPTIMIZE, be a utility
> >> > script that is delivered via oslo.db, and would run for all innodb
> >> > tables within a target MySQL/ MariaDB server generically.  That is, I
> >> > really *dont* want this to be a script that Keystone, Nova, Ceilometer
> >> > etc. are all maintaining delivering themselves.   this should be done
> >> > as a generic pass on a whole database (noting, again, we are only
> >> > running it for very specific InnoDB tables that we observe have a poor
> >> > logical/physical size ratio).
> >>
> >> I don't believe this should be in oslo.db. This is strictly the purview of
> >> deployment tools and should stay there, IMHO.
> >>
> >
> > Hi,
> >
> > As far as I know most projects do "soft deletes" where we just flag the
> > rows as deleted and don't remove them from the DB, so it's only when we
> > use a management tool and run the "purge" command that we actually
> > remove these 

Re: [openstack-dev] [cinder][nova] about re-image the volume

[Gorka Eguileor]

On 06/04, Matt Riedemann wrote:
> On 4/6/2018 5:09 AM, Matthew Booth wrote:
> > I think you're talking at cross purposes here: this won't require a
> > swap volume. Apart from anything else, swap volume only works on an
> > attached volume, and as previously discussed Nova will detach and
> > re-attach.
> >
> > Gorka, the Nova api Matt is referring to is called volume update
> > externally. It's the operation required for live migrating an attached
> > volume between backends. It's called swap volume internally in Nova.
>
> Yeah I was hoping we were just having a misunderstanding of what 'swap
> volume' in nova is, which is the blockRebase for an already attached volume
> to the guest, called from cinder during a volume retype or migration.
>
> As for the re-image thing, nova would be detaching the volume from the guest
> prior to calling the new cinder re-image API, and then re-attach to the
> guest afterward - similar to how shelve and unshelve work, and for that
> matter how rebuild works today with non-root volumes.
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi,

Thanks for the clarification.  When I was talking about "swapping" I was
referring to the fact that Nova will have to not only detach the volume
locally using OS-Brick, but it will also need to use new connection
information to do the attach after the volume has been re-imaged.

As I see it, the process would look something like this:

- Nova detaches volume using OS-Brick
- Nova calls Cinder re-image passing the node's info (like we do when
  attaching a new volume)
- Cinder would:
  - Ensure only that node is connected to the volume
  - Terminate connection to the original volume
  - If we can do optimized volume creation:
- If encrypted volume we create a copy of the encryption key on
  Barbican or copy the ID field from the DB and ensure we don't
  delete the Barbican key on the delete.
- Create new volume from image
- Swap DB fields to preserve the UUID
- Delete original volume
  - If it cannot do optimized volume creation:
- Initialize+Attach volume to Cinder node
- DD the new image into the volume
- Detach+Terminate volume
  - Initialize connection for the new volume to the Nova node
  - Return connection information to the volume
- Nova attaches volume with OS-Brick using returned connection
  information.

So I agree, it's not a blockRebase operation, just a change in the
volume that is used.

Regards,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova] about re-image the volume

[Gorka Eguileor]

On 05/04, Matt Riedemann wrote:
> On 4/5/2018 3:15 AM, Gorka Eguileor wrote:
> > But just to be clear, Nova will have to initialize the connection with
> > the re-imagined volume and attach it again to the node, as in all cases
> > (except when defaulting to downloading the image and dd-ing it to the
> > volume) the result will be a new volume in the backend.
>
> Yeah I think I pointed this out earlier in this thread on what I thought the
> steps would be on the nova side with respect to creating a new empty
> attachment to keep the volume 'reserved' while we delete the old attachment,
> re-image the volume, and then update the volume attachment for the new
> connection. I think that would be similar to how shelve and unshelve works
> in nova.
>
> Would this really require a swap volume call from Cinder? I'd hope not since
> swap volume in itself is a pretty gross operation on the nova side.
>
> --
>
> Thanks,
>
> Matt
>

Hi Matt,

Yes, it will require a volume swap, with the worst case scenario
exception where we dd the image into the volume.

In the same way that anyone would expect a re-imaging preserving the
volume id, one would also expect it to behave like creating a new volume
from the same image: be as fast and take up as much space on the
backend.

And to do so we have to use existing optimized mechanisms that will only
work when creating a new volume.

The alternative would be to have the worst case scenario as the default
(attach and dd the image) and make *ALL* Cinder drivers implement the
optimized mechanism where they can efficiently re-imagine a volume.  I
can't talk for the Cinder team, but I for one would oppose this
alternative.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova] about re-image the volume

[Gorka Eguileor]

On 04/04, Matt Riedemann wrote:
> On 4/2/2018 6:59 AM, Gorka Eguileor wrote:
> > I can only see one benefit from implementing this feature in Cinder
> > versus doing it in Nova, and that is that we can preserve the volume's
> > UUID, but I don't think this is even relevant for this use case, so why
> > is it better to implement this in Cinder than in Nova?
>
> With a new image, the volume_image_metadata in the volume would also be
> wrong, and I don't think nova should (or even can) update that information.
> So nova re-imaging the volume doesn't seem like a good fit to me given
> Cinder "owns" the volume along with any metadata about it.
>
> If Cinder isn't agreeable to this new re-image API, then I think we're stuck

Hi Matt,

I didn't mean to imply that the Cinder team is against this proposal, I
just want to make sure that Cinder is the right place to do it and that
we will actually get some benefits from doing it in Cinder, because
right now I don't see that many...



> with the original proposal of creating a new volume and swapping out the
> root disk, along with all of the problems that can arise from that (original
> volume type is gone, tenant goes over-quota, what do we do with the original
> volume (delete it?), etc).
>
> --
>
> Thanks,
>
> Matt
>

This is what I thought the Nova alternative was, so that's why I didn't
understand the image metadata issue.

For clarification, the original volume type cannot be gone, as the type
delete operation prevents used volume types to be deleted, and if for
some reason it were gone (though I don't see how) Cinder would find
itself with the exact same problem, so there's no difference here.

The flow you are describing is basically what the generic implementation
for that functionality would do in Cinder:

- Create a new volume from image using the same volume type
- Swap the volume information like we do in the live migration case
- Delete the original volume
- Nova will have to swap the root volume (request new connection
  information for that volume and attach it to the node).

Because the alternative is for Cinder to download the image and dd it
into the original volume, which breaks all the optimizations that Cinder
has for speed and storage saving in the backend (there would be no
cloning).

So reading your response I expand the benefits to 2 if done by Cinder:

- Preserve volume UUID
- Remove unlikely race condition of someone deleting the volume type
  between Nova deleting the original volume and creating the new one (in
  this order to avoid the quota issue) when there is no other volume
  using that volume type.

I guess the user facing volume UUID preservation is good enough reason
to have this API in Cinder, as one would assume re-imaging a volume
would never result in having a new volume ID.

But just to be clear, Nova will have to initialize the connection with
the re-imagined volume and attach it again to the node, as in all cases
(except when defaulting to downloading the image and dd-ing it to the
volume) the result will be a new volume in the backend.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?

[Gorka Eguileor]

On 03/04, Jay Pipes wrote:
> On 04/03/2018 11:07 AM, Michael Bayer wrote:
> > The MySQL / MariaDB variants we use nowadays default to
> > innodb_file_per_table=ON and we also set this flag to ON in installer
> > tools like TripleO. The reason we like file per table is so that
> > we don't grow an enormous ibdata file that can't be shrunk without
> > rebuilding the database.  Instead, we have lots of little .ibd
> > datafiles for each table throughout each openstack database.
> >
> > But now we have the issue that these files also can benefit from
> > periodic optimization which can shrink them and also have a beneficial
> > effect on performance.   The OPTIMIZE TABLE statement achieves this,
> > but as would be expected it itself can lock tables for potentially a
> > long time.   Googling around reveals a lot of controversy, as various
> > users and publications suggest that OPTIMIZE is never needed and would
> > have only a negligible effect on performance.   However here we seek
> > to use OPTIMIZE so that we can reclaim disk space on tables that have
> > lots of DELETE activity, such as keystone "token" and ceilometer
> > "sample".
> >
> > Questions for the group:
> >
> > 1. is OPTIMIZE table worthwhile to be run for tables where the
> > datafile has grown much larger than the number of rows we have in the
> > table?
>
> Possibly, though it's questionable to use MySQL/InnoDB for storing transient
> data that is deleted often like ceilometer samples and keystone tokens. A
> much better solution is to use RDBMS partitioning so you can simply ALTER
> TABLE .. DROP PARTITION those partitions that are no longer relevant (and
> don't even bother DELETEing individual rows) or, in the case of Ceilometer
> samples, don't use a traditional RDBMS for timeseries data at all...
>
> But since that is unfortunately already the case, yes it is probably a good
> idea to OPTIMIZE TABLE on those tables.
>
> > 2. from people's production experience how safe is it to run OPTIMIZE,
> > e.g. how long is it locking tables, etc.
>
> Is it safe? Yes.
>
> Does it lock the entire table for the duration of the operation? No. It uses
> online DDL operations:
>
> https://dev.mysql.com/doc/refman/5.7/en/innodb-file-defragmenting.html
>
> Note that OPTIMIZE TABLE is mapped to ALTER TABLE tbl_name FORCE for InnoDB
> tables.
>
> > 3. is there a heuristic we can use to measure when we might run this
> > -.e.g my plan is we measure the size in bytes of each row in a table
> > and then compare that in some ratio to the size of the corresponding
> > .ibd file, if the .ibd file is N times larger than the logical data
> > size we run OPTIMIZE ?
>
> I don't believe so, no. Most things I see recommended is to simply run
> OPTIMIZE TABLE in a cron job on each table periodically.
>
> > 4. I'd like to propose this job of scanning table datafile sizes in
> > ratio to logical data sizes, then running OPTIMIZE, be a utility
> > script that is delivered via oslo.db, and would run for all innodb
> > tables within a target MySQL/ MariaDB server generically.  That is, I
> > really *dont* want this to be a script that Keystone, Nova, Ceilometer
> > etc. are all maintaining delivering themselves.   this should be done
> > as a generic pass on a whole database (noting, again, we are only
> > running it for very specific InnoDB tables that we observe have a poor
> > logical/physical size ratio).
>
> I don't believe this should be in oslo.db. This is strictly the purview of
> deployment tools and should stay there, IMHO.
>

Hi,

As far as I know most projects do "soft deletes" where we just flag the
rows as deleted and don't remove them from the DB, so it's only when we
use a management tool and run the "purge" command that we actually
remove these rows.

Since running the optimize without purging would be meaningless, I'm
wondering if we should trigger the OPTIMIZE also within the purging
code.  This way we could avoid innefective runs of the optimize command
when no purge has happened and even when we do the optimization we could
skip the ratio calculation altogether for tables where no rows have been
deleted (the ratio hasn't changed).

Ideally the ratio calculation and optimization code would be provided by
oslo.db to reduce code duplication between projects.

Cheers,
Gorka.


> > 5. for Galera this gets more tricky, as we might want to run OPTIMIZE
> > on individual nodes directly.  The script at [1] illustrates how to
> > run this on individual nodes one at a time.
> >
> > More succinctly, the Q is:
> >
> > a. OPTIMIZE, yes or no?
>
> Yes.
>
> > b. oslo.db script to run generically, yes or no?
>
> No. Just have Triple-O install galera_innoptimizer and run it in a cron job.
>
> Best,
> -jay
>
> > thanks for your thoughts!
> >
> >
> >
> > [1] https://github.com/deimosfr/galera_innoptimizer
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: 

Re: [openstack-dev] [cinder][nova] about re-image the volume

[Gorka Eguileor]

On 29/03, Sean McGinnis wrote:
> >   This is the spec [0] about rebuild the volumed backed server.
> > The question raised in the spec is about how to bandle the root volume.
> > Finally,in Nova team,we think that the cleanest / best solution to this is 
> > to
> > add a volume action API to cinder for re-imaging the volume.Once that is
> > available in a new cinder v3 microversion, nova can use it. The reason I
> > ...
> >   So Nova team want Cinder to achieve the re-image api.But, I see a spec
> > about volume revert by snapshot[1].It is so good for rebuild operation.In
> > short,I have two ideas,one is change the volume revert by snapshot spec to
> > re-image spec,not only it can let the volume revert by snapshot,but also can
> > re-image the volume which the image's size is greater than 0;another idea is
> > add a only re-image spec,it only can re-image the volume which the image's
> > size is greater than 0.
> >
>
> I do not think changing the revert to snapshot implementation is appropriate
> here. There may be some cases where this can get the desired result, but there
> is no guarantee that there is a snapshot on the volume's base image state to
> revert to. It also would not make sense to overload this functionality to
> "revert to snapshot if you can, otherwise do all this other stuff instead."
>
> This would need to be a new API (microversioned) to add a reimage call. I
> wouldn't expect implementation to be too difficult as we already have that
> functionality for new volumes. We would just need to figure out the most
> appropriate way to take an already in-use volume, detach it, rewrite the 
> image,
> then reattach it.
>

Hi,

The implementation may be more complex that we think, as we have 4
create volume from image mechanisms we have to consider:

- When Glance is using Cinder as backend
- When using Glance image location to do cloning
- When using Cinder cache and we do cloning
- Basic case where we download the image, attach the volume, and copy
  the data.

The only simple, yet efficient, solution I can see is calling the
driver's delete volume method (without soft-deleting it from the DB),
clear the volume DB information of the image metadata, and then run the
create volume from image flow with the same volume information but the
new image metadata.

I can only see one benefit from implementing this feature in Cinder
versus doing it in Nova, and that is that we can preserve the volume's
UUID, but I don't think this is even relevant for this use case, so why
is it better to implement this in Cinder than in Nova?

Cheers,
Gorka.


> Ideally, from my perspective, Nova would take care of the detach/attach 
> portion
> and Cinder would only need to take care of imaging the volume.
>
> Sean
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Support share backup to different projects?

[Gorka Eguileor]

On 21/03, TommyLike Hu wrote:
> Hey Gorka,
> Thanks for your input:) I think I need to clarify that our idea is to
> share the backup resource to another tenants, that is different with
> transfer as the original tenant still can fully control the backup
> resource, and the tenants that have been shared to only have the ability to
> see and read the content of that backup.
>

Hi TommyLike,

Thanks for correcting my misconception of the feature.

Sean brought this up in yesterday's meeting and, after explaining to me
that we were talking about sharing and not transferring, we agreed that
this is not an acceptable feature.

I am OK with transferring backups, but certainly not sharing them.

For sharing data you would use Glance images, and maybe you can even use
the "image_upload_use_cinder_backend" feature to get what you want,
though I have never used it myself.

Or if the client wants to use the volumes to boot from them they could
attach the "golden volume" to a nova instance, create a "nova instance
snapshot" which will result in a Glance image placeholder for a Cinder
snapshot.

But as you can see all the solutions I can think of go through Glance
images, which is the right way to share volume data between users.

I think our efforts would be better spent removing the bottlenecks we
currently have on all create volume from source operations if what the
customer is trying to do is workaround it via the backups.

Cheers,
Gorka.

> Gorka Eguileor <gegui...@redhat.com>于2018年3月21日周三 下午7:15写道：
>
> > On 20/03, Jay S Bryant wrote:
> > >
> > >
> > > On 3/19/2018 10:55 PM, TommyLike Hu wrote:
> > > > Now Cinder can transfer volume (with or without snapshots) to different
> > > > projects,  and this make it possbile to transfer data across tenant via
> > > > volume or image. Recently we had a conversation with our customer from
> > > > Germany, they mentioned they are more pleased if we can support
> > transfer
> > > > data accross tenant via backup not image or volume, and these below are
> > > > some of their concerns:
> > > >
> > > > 1. There is a use case that they would like to deploy their
> > > > develop/test/product systems in the same region but within different
> > > > tenants, so they have the requirment to share/transfer data across
> > > > tenants.
> > > >
> > > > 2. Users are more willing to use backups to secure/store their volume
> > > > data since backup feature is more advanced in product openstack version
> > > > (incremental backups/periodic backups/etc.).
> > > >
> > > > 3. Volume transfer is not a valid option as it's in AZ and it's a
> > > > complicated process if we would like to share the data to multiple
> > > > projects (keep copy in all the tenants).
> > > >
> > > > 4. Most of the users would like to use image for bootable volume only
> > > > and share volume data via image means the users have to maintain lots
> > of
> > > > image copies when volume backup changed as well as the whole system
> > > > needs to differentiate bootable images and none bootable images, most
> > > > important, we can not restore volume data via image now.
> > > >
> > > > 5. The easiest way for this seems to support sharing backup to
> > different
> > > > projects, the owner project have the full authority while shared
> > > > projects only can view/read the backups.
> > > >
> > > > 6. AWS has the similar concept, share snapshot. We can share it by
> > > > modify the snapshot's create volume permissions [1].
> > > >
> > > > Looking forward to any like or dislike or suggestion on this idea
> > > > accroding to my feature proposal experience:)
> > > >
> > > >
> > > > Thanks
> > > > TommyLike
> > > >
> > > >
> > > > [1]:
> > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modifying-snapshot-permissions.html
> > > >
> > > >
> > > >
> > __
> > > > OpenStack Development Mailing List (not for usage questions)
> > > > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > Tommy,
> > >
> > > As discussed at the PTG, this still sounds like improper usage of
> > Backup.
> > > Happy to hear input f

Re: [openstack-dev] [cinder] Support share backup to different projects?

[Gorka Eguileor]

On 20/03, Jay S Bryant wrote:
>
>
> On 3/19/2018 10:55 PM, TommyLike Hu wrote:
> > Now Cinder can transfer volume (with or without snapshots) to different
> > projects,  and this make it possbile to transfer data across tenant via
> > volume or image. Recently we had a conversation with our customer from
> > Germany, they mentioned they are more pleased if we can support transfer
> > data accross tenant via backup not image or volume, and these below are
> > some of their concerns:
> >
> > 1. There is a use case that they would like to deploy their
> > develop/test/product systems in the same region but within different
> > tenants, so they have the requirment to share/transfer data across
> > tenants.
> >
> > 2. Users are more willing to use backups to secure/store their volume
> > data since backup feature is more advanced in product openstack version
> > (incremental backups/periodic backups/etc.).
> >
> > 3. Volume transfer is not a valid option as it's in AZ and it's a
> > complicated process if we would like to share the data to multiple
> > projects (keep copy in all the tenants).
> >
> > 4. Most of the users would like to use image for bootable volume only
> > and share volume data via image means the users have to maintain lots of
> > image copies when volume backup changed as well as the whole system
> > needs to differentiate bootable images and none bootable images, most
> > important, we can not restore volume data via image now.
> >
> > 5. The easiest way for this seems to support sharing backup to different
> > projects, the owner project have the full authority while shared
> > projects only can view/read the backups.
> >
> > 6. AWS has the similar concept, share snapshot. We can share it by
> > modify the snapshot's create volume permissions [1].
> >
> > Looking forward to any like or dislike or suggestion on this idea
> > accroding to my feature proposal experience:)
> >
> >
> > Thanks
> > TommyLike
> >
> >
> > [1]: 
> > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modifying-snapshot-permissions.html
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Tommy,
>
> As discussed at the PTG, this still sounds like improper usage of Backup. 
> Happy to hear input from others but I am having trouble getting my head
> around it.
>
> The idea of sharing a snapshot, as you mention AWS supports sounds like it
> could be a more sensible approach.  Why are you not proposing that?
>
> Jay
>

Hi,

I agree with Jay that this sounds like an improper use of Backups, and I
believe that this feature, just like trying to transfer snapshots, would
incur in a lot of code changes as well as an ever greater number of
bugs, because the ownership structure in Cinder is hierarchical and well
defined.

So if you transferred a snapshot then you would lose that snapshot
information on the source volume, which means that we could not prevent
a volume deletion with a snapshot, or we could prevent it but would
either have to prevent the deletion from happening (creating a terrible
user experience since the user can't delete the volume now because
somebody else still has one of its snapshots) or we have to implement
some kind of "trash" mechanism to postpone cleanup until all the
snapshots have been deleted, which would make our quota code more
complex as well as make our stats reporting and scheduling diverge from
what the user think has actually happened (they deleted a bunch of
volumes but the data has not been freed from the backend).

As for backups, you have an even worse situation because of our
incremental backups, since transferring ownership of an incremental
backup will create similar deletion issues as the snapshots but we also
require access to all all incremental snapshots to restore a volume.  So
the only alternative would be to only allow transferring a full Backup
and this would carry all the incremental backups with it.

All in all I think this would be an abuse of the Backups, and as stated
by TommyLike we already have mechanisms to do this via images and volume
transfers.

Although I have to admit that after giving this some thought there is a
very good case where it wouldn't be an abuse and where we should allow
transferring full backups together with all their incremental backups,
and that is when you transfer a volume.  If we transfer a volume with
all its snapshots, it makes sense that we should also allow transferring
its backups, after all the original source of the backups no longer
belongs to the owner of the backups.

To summarize, if we are talking about transferring only full backups
with all their dependent incremental backup then I probably won't oppose
the change.

Cheers,
Gorka.

Re: [openstack-dev] [cinder] [manila] Performance concern on new quota system

[Gorka Eguileor]

On 09/03, Sean McGinnis wrote:
>
> > On Mar 9, 2018, at 07:37, TommyLike Hu  wrote:
> >
> > Thanks Gorka,
> > To be clear, I started this discussion not because I reject this 
> > feature, instead I like it as it's much more clean and simple, compared 
> > with performance impact it solves several other issues which we hate badly. 
> > I wrote this is to point out we may have this issue, and to see whether we 
> > could improve it before it's actually landed. Better is better:)
> >
> >
> > - Using a single query to retrieve both counts and sums instead of 2
> >  queries.
> >
> > For this advice, I think I already combined count and sum into single query.
> >

Yes, but we would be doing 2 count and sum queries, one for the volumes
and another one for the per volume types, the idea I was proposing is
doing just 1 query for both calculations, that way even if you increase
the payload of the response from the DB you are getting rid of a round
trip to the DB as well as a pass through  all the volumes for the volume
type.



> >  - DB triggers to do the actual counting.
> >
>
> Please, no DB triggers. :)
>
> > This seems a good idea, but not sure whether it could cover all of the 
> > cases we have in our quota system and whether can be easily integrated into 
> > cinder, can you share more detail on this?
> >
> > Thanks
> > TommyLike
> >
>


As a general rule I agree with Sean that it's better to not have
triggers, but I wanted to mention them as an alternative in case we
really, really, really have problems with the other alternatives.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] [manila] Performance concern on new quota system

[Gorka Eguileor]

On 09/03, TommyLike Hu wrote:
> Hey all,
> During the Cinder and Manila's cross project discussion on quota system
> last week, I raise our concern of performance impact on the count resource
> feature, and Gorka pointed out we might miss some of the indexes when
> testing. So I reshaped the environment and share some test results here as
> below:
>
> *Test patch:*
>  Basically this [1] is borrowed from Nova for Cinder, and only the
> create volume process has been updated to use the new system.
>
> *Test Environment on AWS:*
> 1. 3 EC2 t2.large(2 vCPU, 8GiB，80G SSD) each one deployed cinder API
> service with 10 api workers (coordinated by haproxy)
> 2. 1 RDS db.m4.xlarge(4 vCPU, 16GiB，200G SSD) MySQL 5.7.19
>
> *Database upgrade:*
> Composite index has been added to volume table (volumes__composite_index):
> +-++--+--+-+---+-+--++--++-+---+
> | Table   | Non_unique | Key_name  | Seq_in_index |
> Column_name | Collation | Cardinality | Sub_part | Packed | Null |
> Index_type | Comment | Index_comment |
> +-++--+--+-+---+-+--++--++-+---+
> |   .other index... |
> | volumes | 1 | volumes__composite_index | 1 |
> project_id  | A |   2 | NULL | NULL   | YES  |
> BTREE  | |   |
> | volumes | 1 | volumes__compoite_index |   2 |
> deleted | A |   2 | NULL | NULL   | YES  |
> BTREE  | |   |
> | volumes | 1 | volumes__composite_index | 3 |
> volume_type_id  | A |   2 | NULL | NULL   | YES  |
> BTREE  | |   |
> +-++--+--+-+---+-+--++--++-+---+
>  Explain result for one of the sql statements for new quota system (*explain
> select count(*), sum(size) from volumes where project_id={project_id} and
> volume_type_id={type_id} and deleted=false*):
>
>  
> ++-+-++--+--+--+-+---+--+--+---+
> | id | select_type | table   | partitions | type | possible_keys
>  | key  | key_len | ref   | rows | filtered
> | Extra |
>
>  
> ++-+-++--+--+--+-+---+--+--+---+
> |  1 | SIMPLE  | volumes | NULL   | ref  | volumes__composite_index
> | volumes__composite_index | 881 | const,const,const |1 |   100.00
> | NULL  |
>
>  
> ++-+-++--+--+--+-+---+--+--+---+
>
> *Time comparsion between two system (in Seconds, * *Conc = Concurrency**):*
>
> *NOTE**:*
>
>   1. *QUOTAS.check_deltas*  stands for the total time consumed
> including two sql statements as below when creating single volume:
>
>   1. SELECT count(id), sum(size) from volumes where
> project_id=$(project_id) and deleted=False
>
>   2. SELECT count(id), sum(size) from volumes where
> project_id=$(project_id) and deleted=False and volume_type=$(volume_type)
>

Hi,

As I see it there are 3 questions here:

1. Do we need to change the quota system?

   I believe we all agree that the Quota system in Cinder is bad, so bad
   that we can divide Cinder deployments in 2 clear categories, those
   that don't use quotas and those that have out of sync quotas.

2. Will the DB implementation used by Nova solve our current problems?

   As the Nova team kindly explained us, the new quotas may not be
   perfect for limiting (we may allow going slightly above allowed quota
   or just get short), but it will at least solve our problems of having
   out of sync quotas that require manually changing the DB to fix it.

3. Will the new solution introduce new problems?

   To me introducing a small performance impact on resource creation is
   an acceptable trade-off compared with the alternative, moreover
   considering that most resource creation procedures are somewhat slow
   operations.

   And let's not forget that we can always look for more efficient ways
   of doing the counting:

   - Using a single query to retrieve both counts and sums instead of 2
 queries.

   - DB triggers to do the actual counting.

To me comparing the performance of something that doesn't work with
something that does doesn't seem fair.

Cheers,

Re: [openstack-dev] [cinder] Cinder volume revert to snapshot with Ceph

[Gorka Eguileor]

On 06/03, 李杰 wrote:
> Hi,all
>
>
>   This is the patch [0] about volume revert to snapshot with Ceph.Is 
> anyone working on this patchset or maybe new patchset was proposed to 
> implement RBD specific functionality?Can you tell me more about this ?Thank 
> you very much.
>   The link is here.
>   Re:https://review.openstack.org/#/c/481566/
>
>
> Best Regards
> Lijie

Hi,

As far as I know nobody else has been working on that feature, and if
you are looking to work on it please keep in mind the comments I wrote
on that patch during the review.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][cinder]Question about cinder as glance store

[Gorka Eguileor]

On 07/02, Erlon Cruz wrote:
> That will depend on the Cinder/OS-brick iscsiadm versions right? Can you
> tell what are the versions from where the problem was fixed?
>
> Erlon
>

Hi,

Like you said it depends on your Cinder and OS-Brick versions, and the
open-iscsi package will be different depending on your distro, for RHOS
7.4 this is iscsi-initiator-utils version 6.2.0.874-2 or later.

Things working fine may also depend on your multipath and Cinder/Nova
configuration as well as LVM filters (if your deployment can have images
that have LVM).

I think that's all that need to be properly setup.

Cheers,
Gorka.

> 2018-02-07 8:27 GMT-02:00 Gorka Eguileor <gegui...@redhat.com>:
>
> > On 07/02, Rikimaru Honjo wrote:
> > > Hello,
> > >
> > > I'm planning to use cinder as glance store.
> > > And, I'll setup cinder to connect storage by iSCSI multipath.
> > >
> > > In this case, can I run glance-api and cinder-volume on the same node?
> > >
> > > In my understanding, glance-api will attach a volume to own node and
> > > write a uploaded image to the volume if glance backend is cinder.
> > > I afraid that the race condition of cinder-volume's iSCSI operations
> > > and glance-api's iSCSI operations.
> > > Is there possibility of occurring it?
> > > --
> > > _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
> > > Rikimaru Honjo
> > > E-mail:honjo.rikim...@po.ntt-tx.co.jp
> > >
> > >
> > >
> > > 
> > __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> > unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > Hi,
> >
> > When properly set with the right configuration and the right system and
> > OpenStack packages, Cinder, OS-Brick, and Nova no longer have race
> > conditions with iSCSI operations anymore (single or multipathed), not
> > even with drivers that do "shared target".
> >
> > So I would assume that Glance won't have any issues either as long as
> > it's properly making the Cinder and OS-Brick calls.
> >
> > Cheers,
> > Gorka.
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][cinder]Question about cinder as glance store

[Gorka Eguileor]

On 07/02, Rikimaru Honjo wrote:
> Hello,
>
> I'm planning to use cinder as glance store.
> And, I'll setup cinder to connect storage by iSCSI multipath.
>
> In this case, can I run glance-api and cinder-volume on the same node?
>
> In my understanding, glance-api will attach a volume to own node and
> write a uploaded image to the volume if glance backend is cinder.
> I afraid that the race condition of cinder-volume's iSCSI operations
> and glance-api's iSCSI operations.
> Is there possibility of occurring it?
> --
> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
> Rikimaru Honjo
> E-mail:honjo.rikim...@po.ntt-tx.co.jp
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi,

When properly set with the right configuration and the right system and
OpenStack packages, Cinder, OS-Brick, and Nova no longer have race
conditions with iSCSI operations anymore (single or multipathed), not
even with drivers that do "shared target".

So I would assume that Glance won't have any issues either as long as
it's properly making the Cinder and OS-Brick calls.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][ceilometer] cinder capacity notifications

[Gorka Eguileor]

On 12/12, gordon chung wrote:
> hi,
>
> i'm attempting to add support for the cinder capacity notifications that
> were added a while back[1].
>
> adding measurement support in ceilometer is pretty simple where in most
> cases you just add an entry to a yaml file[2]. the problem i have right
> now is i have no idea what resource these measurement are measuring so
> i'm unsure how to name the measurement in ceilometer so a user can
> understand what it's measuring.
>
> the following are probably stupid questions but i see it measures two
> different resource types, a 'pool' and a 'backend' and they are
> ultimately tied to a host. are these measuring the amount of disk on
> each host? our pools confined to a host?
>
> anyone with better knowledge is welcomed to take over my patch.
>
> [1] https://review.openstack.org/#/c/206923
> [2] https://review.openstack.org/#/c/526538
>
> cheers,
>
> --
> gord
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi,

Here's a brief explanation of the 3 names you mention:

- host: A node were cinder-volume service is running
- backend: A set of driver configurations used to access a specific
  storage.  A cinder-volume service can have multiple backends.
- pool: A logical concept to describe a set of storage resource that can
  be used to serve core Cinder requests.  A backend can have multiple
  pools, each with different characteristics.

Cinder-volume services are named taking these 3 names and joining them
like: host@backend#pool

As for the data that is being reported, this is mostly explained in the
latest spec [1], except the virtual_free which is basically the
available space taking into account the over provisioning.

More information on the pools can be found in the Juno specs [2].

I hope this helps.

Cheers,
Gorka.

[1] 
https://specs.openstack.org/openstack/cinder-specs/specs/queens/provisioning-improvements.html
[2] 
https://specs.openstack.org/openstack/cinder-specs/specs/juno/pool-aware-cinder-scheduler.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] May I run iscsiadm --op show & update 100 times?

[Gorka Eguileor]

On 03/10, Rikimaru Honjo wrote:
> Hello Gorka,
>
> On 2017/10/02 20:37, Gorka Eguileor wrote:
> > On 02/10, Rikimaru Honjo wrote:
> > > Hello,
> > >
> > > I'd like to discuss about the following bug of os-brick.
> > >
> > > * os-brick's iscsi initiator unexpectedly reverts node.startup from 
> > > "automatic" to "manual".
> > >https://bugs.launchpad.net/os-brick/+bug/1670237
> > >
> > > The important point of this bug is:
> > >
> > > When os-brick initializes iscsi connections:
> > > 1. os-brick will run "iscsiadm -m discovery" command if we use iscsi 
> > > multipath.
> >
> > This only happens with a small number of cinder drivers, since most
> > drivers try to avoid the discovery path due to the number of
> > disadvantages it presents for a reliable deployment.  The most notorious
> > issue is that the path to the discovery portal on the attaching node is
> > down you cannot attach the volume no matter how many of the other paths
> > are up.
> >
> >
> >
> > > 2. os-brick will update node.startup values to "automatic" if we use 
> > > iscsi.
> > > 3. "iscsiadm -m discovery" command will recreate iscsi node 
> > > repositories.[1]
> > > As a result, node.startup values of already attached volumes will be 
> > > revert
> > > to default(=manual).
> > >
> > > Gorka Eguileor and I discussed how do I fix this bug[2].
> > > Our idea is this:
> > >
> > > 1. Confirm node.startup values of all the iscsi targets before running 
> > > discovery.
> > > 2. Re-update node.startup values of all the iscsi targets after running 
> > > discovery.
> > >
> > > But, I afraid that this operation will take a long time.
> > > I ran showing & updating node.startup values 100 times for researching.
> > > As a result, it took about 4 seconds.
> > > When I ran 200 times, it took about 8 seconds.
> > > I think this is a little long.
> > >
> > > If we use multipath and attach 25 volumes, 100 targets will be created.
> > > I think that updating 100 times is a possible use case.
> > >
> > > How do you think about it?
> > > Can I implement the above idea?
> > >
> >
> > The approach I proposed is on the review is valid, the flaw is in the
> > specific implementation, you are doing 100 request where 4 would
> > suffice.
> >
> > You don't need to do a request for each target-portal tuple, you only
> > need to do 1 request per portal, which reduces the number of calls to
> > iscsiadm from 100 to 4 in the case you mention.
> >
> > You can check all targets for an IP with:
> >iscsiadm -m node -p IP
> >
> > This means that the performance hit from having 100 or 200 targets
> > should be negligible.
>
> I have one question.
>
> I can see node.startup values by 1 request per portal as you said.
>
> But, may I update values by 1 request per portal?
> Updating values has been done by 1 request per target until now.
> So I think my patch should update values in same way(=1 request per target).
>

If all the targets on the same portal had the original same value you
should just make a single call after the discovery that changes them
all, at least that would be my preference.



> > Cheers,
> > Gorka.
> >
> >
> >
> > > [1]This is correct behavior of iscsiadm.
> > > 
> > > https://github.com/open-iscsi/open-iscsi/issues/58#issuecomment-325528315
> > > [2]https://bugs.launchpad.net/os-brick/+bug/1670237
> > > --
> > > _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
> > > Rikimaru Honjo
> > > E-mail:honjo.rikim...@po.ntt-tx.co.jp
> > >
> > >
> > >
> > > __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
>
> --
> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
> ★社名とメールアドレスが変わりました。
>
> NTTテクノクロス株式会社
> クラウド&セキュリティ事業部 第二事業ユニット（CS2BU）
> 本上力丸
> TEL.  :045-212-7539
> E-mail:honjo.rikim...@po.ntt-tx.co.jp
> 〒220-0012
>   横浜市西区みなとみらい4丁目4番5号
>   横浜アイマークプレイス 13階
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova]May I run iscsiadm --op show & update 100 times?

[Gorka Eguileor]

On 02/10, Rikimaru Honjo wrote:
> Hello,
>
> I'd like to discuss about the following bug of os-brick.
>
> * os-brick's iscsi initiator unexpectedly reverts node.startup from 
> "automatic" to "manual".
>   https://bugs.launchpad.net/os-brick/+bug/1670237
>
> The important point of this bug is:
>
> When os-brick initializes iscsi connections:
> 1. os-brick will run "iscsiadm -m discovery" command if we use iscsi 
> multipath.

This only happens with a small number of cinder drivers, since most
drivers try to avoid the discovery path due to the number of
disadvantages it presents for a reliable deployment.  The most notorious
issue is that the path to the discovery portal on the attaching node is
down you cannot attach the volume no matter how many of the other paths
are up.



> 2. os-brick will update node.startup values to "automatic" if we use iscsi.
> 3. "iscsiadm -m discovery" command will recreate iscsi node repositories.[1]
>As a result, node.startup values of already attached volumes will be revert
>to default(=manual).
>
> Gorka Eguileor and I discussed how do I fix this bug[2].
> Our idea is this:
>
> 1. Confirm node.startup values of all the iscsi targets before running 
> discovery.
> 2. Re-update node.startup values of all the iscsi targets after running 
> discovery.
>
> But, I afraid that this operation will take a long time.
> I ran showing & updating node.startup values 100 times for researching.
> As a result, it took about 4 seconds.
> When I ran 200 times, it took about 8 seconds.
> I think this is a little long.
>
> If we use multipath and attach 25 volumes, 100 targets will be created.
> I think that updating 100 times is a possible use case.
>
> How do you think about it?
> Can I implement the above idea?
>

The approach I proposed is on the review is valid, the flaw is in the
specific implementation, you are doing 100 request where 4 would
suffice.

You don't need to do a request for each target-portal tuple, you only
need to do 1 request per portal, which reduces the number of calls to
iscsiadm from 100 to 4 in the case you mention.

You can check all targets for an IP with:
  iscsiadm -m node -p IP

This means that the performance hit from having 100 or 200 targets
should be negligible.

Cheers,
Gorka.



> [1]This is correct behavior of iscsiadm.
>https://github.com/open-iscsi/open-iscsi/issues/58#issuecomment-325528315
> [2]https://bugs.launchpad.net/os-brick/+bug/1670237
> --
> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
> Rikimaru Honjo
> E-mail:honjo.rikim...@po.ntt-tx.co.jp
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Identify Openstack commands and manipulate them

[Gorka Eguileor]

On 03/07, david jhon wrote:
> Thanks for the help.
>
> Openstack commands mean some administrative commands as listed here:
>
> https://docs.openstack.org/developer/python-openstackclient/command-objects/server.html
>
> I want to identify incoming openstack server commands on controller machine
> and manipulate their execution.
>

Hi,

Are you trying to track the execution from the client to the server and
then within services?

Usually you would use the `--debug` parameter on the client (this works
not only on the openstack common client but also on individual service
clients), then you'll see what REST API entries are being called on the
server.

From there you can go and check the server logs (better if services are
in debug mode) and see the code path these are following.

If you want to follow a particular tenant/project and user it will
depend on the configured logging format, but they usually include the
project_id or project_name and the user_id or user_name.

Also, on the server side most projects have an "api" directory where all
REST API entries first hit the code (well, sometimes it's not really the
first hit, but close enough), for example in cinder we'll have
cinder/api.

Cheers,
Gorka.

>
> On Mon, Jul 3, 2017 at 1:05 AM, Maciej Kucia  wrote:
>
> > What is "OpenStack command"? You mean Rest API, RPC? What services?
> >
> > Maybe the following will be of help.
> > http://vmartinezdelacruz.com/in-a-nutshell-how-openstack-works/
> >
> > Regards,
> > Maciej
> >
> > 2017-07-02 15:16 GMT+02:00 david jhon :
> >
> >> Hello folks,
> >>
> >> I've been digging into openstack code for quite some time, I'm wondering
> >> how to identify some specific openstack commands possibly coming from
> >> hosted tenants or projects.
> >>
> >> Where should I look into the code if I want to suspend or block their
> >> execution. OR
> >> What is the starting point or any useful resource which can help me
> >> further.
> >>
> >> Any suggestion/link/help might be great.
> >>
> >> Thanks.
> >>
> >> 
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
> >> e
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][Cinder][backup]Get total files/objects disk size of backup

[Gorka Eguileor]

On 30/06, int32bit wrote:
> Currently the size in backup is src volume size rather than backup
> file/object size, it's used to record original volume size that avoid
> damaging target in restore operation if the volume has been resized.
>
> But it may be more useful for end user to tell them backup files/objects
> size(object count really make no sense for user). It's also useful for
> cloud provider to compute their bill.
>
> I think It's not very hard to implement it if we use or extend
> `chunkeddriver`.  Firstly we need add `get_backup_size` new interface in
> driver API and let's update backup size in `_finalize_backup`. The backend
> driver should implement this interface if possible. If the backend driver
> is not possible to implement like `CephBackupDriver`, we raise
> NotImplementedError. And then we need add a new field(named volume_size) in
> backup object as well as db model associated to backup, and store
> files/objects size in original size field. At last, we add this new
> field volume_size to backup ViewBuilder in API.

Hi,

I think this is a good idea, just a couple of comments.

We should not change the meaning of the size field in the backup model,
size should remain the original volume's size, and the new field should
refer to the allocated/consumed size.

Besides adding the size of the hash table to the size of the chunks when
calculating the consumed size of the backups, maybe we should also
include a mechanism for drivers to account for the overhead of having
additional objects.

Last I would like to thank you for being thorough and considering the
Ceph backup driver case as well, although it won't need to be an
exception, as it could also report the real size of the backup with one
of these 2 mechanisms:

- Piping the output from export-diff and processing the metadata in
  there to calculate the size.

- Once the backup has completed using the diff command to get the list
  of byte extents and calculate the real size of the image from the
  parent backup.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova][os-brick] Testing for proposed iSCSI OS-Brick code

[Gorka Eguileor]

On 31/05, Matt Riedemann wrote:
> On 5/31/2017 6:58 AM, Gorka Eguileor wrote:
> > Hi,
> >
> > As some of you may know I've been working on improving iSCSI connections
> > on OpenStack to make them more robust and prevent them from leaving
> > leftovers on attach/detach operations.
> >
> > There are a couple of posts [1][2] going in more detail, but a good
> > summary would be that to fix this issue we require a considerable rework
> > in OS-Brick, changes in Open iSCSI, Cinder, Nova and specific tests.
> >
> > Relevant changes for those projects are:
> >
> > - Open iSCSI: iscsid behavior is not a perfect fit for the OpenStack use
> >case, so a new feature was added to disable automatic scans that added
> >unintended devices to the systems.  Done and merged [3][4], it will be
> >available on RHEL with iscsi-initiator-utils-6.2.0.874-2.el7
> >
> > - OS-Brick: rework iSCSI to make it robust on unreliable networks, to
> >add a `force` detach option that prioritizes leaving a clean system
> >over possible data loss, and to support the new Open iSCSI feature.
> >Done and pending review [5][6][7]
> >
> > - Cinder: Handle some attach/detach errors a little better and add
> >support to the force detach option for some operations where data loss
> >on error is acceptable, ie: create volume from image, restore backup,
> >etc. Done and pending review [8][9]
> >
> > - Nova: I haven't looked into the code here, but I'm sure there will be
> >cases where using the force detach operation will be useful.
> >
> > - Tests: While we do have tempest tests that verify that attach/detach
> >operations work both in Nova and in cinder volume creation operations,
> >they are not meant to test the robustness of the system, so new tests
> >will be required to validate the code.  Done [10]
> >
> > Proposed tests are simplified versions of the ones I used to validate
> > the code; but hey, at least these are somewhat readable ;-)
> > Unfortunately they are not in line with the tempest mission since they
> > are not meant to be run in a production environment due to their
> > disruptive nature while injecting errors.  They need to be run
> > sequentially and without any other operations running on the deployment.
> > They also run sudo commands via local bash or SSH for the verification
> > and error generation bits.
> >
> > We are testing create volume from image and attaching a volume to an
> > instance under the following networking error scenarios:
> >
> >   - No errors
> >   - All paths have 10% incoming packets dropped
> >   - All paths have 20% incoming packets dropped
> >   - All paths have 100% incoming packets dropped
> >   - Half the paths have 20% incoming packets dropped
> >   - The other half of the paths have 20% incoming packets dropped
> >   - Half the paths have 100% incoming packets dropped
> >   - The other half of the paths have 100% incoming packets dropped
> >
> > There are single execution versions as well as 10 consecutive operations
> > variants.
> >
> > Since these are big changes I'm sure we would all feel a lot more
> > confident to merge them if storage vendors would run the new tests to
> > confirm that there are no issues with their backends.
> >
> > Unfortunately to fully test the solution you may need to build the
> > latest Open-iSCSI package and install it in the system, then you can
> > just use an all-in-one DevStack with a couple of changes in the local.conf:
> >
> > enable_service tempest
> >
> > CINDER_REPO=https://review.openstack.org/p/openstack/cinder
> > CINDER_BRANCH=refs/changes/45/469445/1
> >
> > LIBS_FROM_GIT=os-brick
> >
> > OS_BRICK_REPO=https://review.openstack.org/p/openstack/os-brick
> > OS_BRICK_BRANCH=refs/changes/94/455394/11
> >
> > [[post-config|$CINDER_CONF]]
> > [multipath-backend]
> > use_multipath_for_image_xfer=true
> >
> > [[post-config|$NOVA_CONF]]
> > [libvirt]
> > volume_use_multipath = True
> >
> > [[post-config|$KEYSTONE_CONF]]
> > [token]
> > expiration = 14400
> >
> > [[test-config|$TEMPEST_CONFIG]]
> > [volume-feature-enabled]
> > multipath = True
> > [volume]
> > build_interval = 10
> > multipath_type = $MULTIPATH_VOLUME_TYPE
> > backend_protocol_tcp_port = 3260
> > multipath_backend_addresses = $STORAGE_BACKEND_IP1,$STORAGE_BACKEND_IP2
&

[openstack-dev] [cinder][nova][os-brick] Testing for proposed iSCSI OS-Brick code

[Gorka Eguileor]

Hi,

As some of you may know I've been working on improving iSCSI connections
on OpenStack to make them more robust and prevent them from leaving
leftovers on attach/detach operations.

There are a couple of posts [1][2] going in more detail, but a good
summary would be that to fix this issue we require a considerable rework
in OS-Brick, changes in Open iSCSI, Cinder, Nova and specific tests.

Relevant changes for those projects are:

- Open iSCSI: iscsid behavior is not a perfect fit for the OpenStack use
  case, so a new feature was added to disable automatic scans that added
  unintended devices to the systems.  Done and merged [3][4], it will be
  available on RHEL with iscsi-initiator-utils-6.2.0.874-2.el7

- OS-Brick: rework iSCSI to make it robust on unreliable networks, to
  add a `force` detach option that prioritizes leaving a clean system
  over possible data loss, and to support the new Open iSCSI feature.
  Done and pending review [5][6][7]

- Cinder: Handle some attach/detach errors a little better and add
  support to the force detach option for some operations where data loss
  on error is acceptable, ie: create volume from image, restore backup,
  etc. Done and pending review [8][9]

- Nova: I haven't looked into the code here, but I'm sure there will be
  cases where using the force detach operation will be useful.

- Tests: While we do have tempest tests that verify that attach/detach
  operations work both in Nova and in cinder volume creation operations,
  they are not meant to test the robustness of the system, so new tests
  will be required to validate the code.  Done [10]

Proposed tests are simplified versions of the ones I used to validate
the code; but hey, at least these are somewhat readable ;-)
Unfortunately they are not in line with the tempest mission since they
are not meant to be run in a production environment due to their
disruptive nature while injecting errors.  They need to be run
sequentially and without any other operations running on the deployment.
They also run sudo commands via local bash or SSH for the verification
and error generation bits.

We are testing create volume from image and attaching a volume to an
instance under the following networking error scenarios:

 - No errors
 - All paths have 10% incoming packets dropped
 - All paths have 20% incoming packets dropped
 - All paths have 100% incoming packets dropped
 - Half the paths have 20% incoming packets dropped
 - The other half of the paths have 20% incoming packets dropped
 - Half the paths have 100% incoming packets dropped
 - The other half of the paths have 100% incoming packets dropped

There are single execution versions as well as 10 consecutive operations
variants.

Since these are big changes I'm sure we would all feel a lot more
confident to merge them if storage vendors would run the new tests to
confirm that there are no issues with their backends.

Unfortunately to fully test the solution you may need to build the
latest Open-iSCSI package and install it in the system, then you can
just use an all-in-one DevStack with a couple of changes in the local.conf:

   enable_service tempest

   CINDER_REPO=https://review.openstack.org/p/openstack/cinder
   CINDER_BRANCH=refs/changes/45/469445/1

   LIBS_FROM_GIT=os-brick

   OS_BRICK_REPO=https://review.openstack.org/p/openstack/os-brick
   OS_BRICK_BRANCH=refs/changes/94/455394/11

   [[post-config|$CINDER_CONF]]
   [multipath-backend]
   use_multipath_for_image_xfer=true

   [[post-config|$NOVA_CONF]]
   [libvirt]
   volume_use_multipath = True

   [[post-config|$KEYSTONE_CONF]]
   [token]
   expiration = 14400

   [[test-config|$TEMPEST_CONFIG]]
   [volume-feature-enabled]
   multipath = True
   [volume]
   build_interval = 10
   multipath_type = $MULTIPATH_VOLUME_TYPE
   backend_protocol_tcp_port = 3260
   multipath_backend_addresses = $STORAGE_BACKEND_IP1,$STORAGE_BACKEND_IP2

Multinode configurations are also supported using SSH with use/password or
private key to introduce the errors or check that the systems didn't leave any
leftovers, the tests can also run a cleanup command between tests, etc., but
that's beyond the scope of this email.

Then you can run them all from /opt/stack/tempest with:

 $ cd /opt/stack/tempest
 $ OS_TEST_TIMEOUT=7200 ostestr -r 
cinder.tests.tempest.scenario.test_multipath.*

But I would recommend first running the simplest one without errors and
manually checking that the multipath is being created.

 $ ostestr -n 
cinder.tests.tempest.scenario.test_multipath.TestMultipath.test_create_volume_with_errors_1

Then doing the same with one with errors and verify the presence of the
filters in iptables and that the packet drop for those filters is non zero:

 $ ostestr -n 
cinder.tests.tempest.scenario.test_multipath.TestMultipath.test_create_volume_with_errors_2
 $ sudo iptables -nvL INPUT

Then doing the same with a Nova test just to verify that it is correctly
configured to use multipathing:

 $ ostestr -n 

Re: [openstack-dev] [cinder] Is Cinder still maintained?

[Gorka Eguileor]

On 27/04, Julien Danjou wrote:
> Hi,
>
> I've posted a refactoring patch that simplifies tooz (read: remove
> technical debt) usage more than a month ago, and I got 0 review since
> then:
>
>   https://review.openstack.org/#/c/447079
>
> I'm a bit worried to see this zero review on such patches. It seems the
> most recently merged things are all vendor specific. Is the core of
> Cinder still maintained? Is there any other reason for such patches to
> be ignored for so long?
>
> --
> Julien Danjou
> // Free Software hacker
> // https://julien.danjou.info

Hi,

Patches usually get attention, you may have been unfortunate, and it's
faster to ask for eyes on IRC than here anyway.

So, you mention that the feature has been there for a while, but funny
enough documentation has not been updated and it's still reflecting it
should be done how we are doing it in Cinder [1].

As far as I can tell looking at Tooz's code for the heartbeat it doesn't
do the same thing our code does, because it doesn't reconnect on
failure...  Am I missing something?

Cheers,
Gorka.


[1]: https://docs.openstack.org/developer/tooz/tutorial/coordinator.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] About cinder-volume HA A/A

[Gorka Eguileor]

On 21/04, Jae Sang Lee wrote:
> Hello, cinder developers.
>
> What progress about cinder-volume HA A/A? In the release notes[1],
> cinder-volume HA A/A is still progress from Newton to Ocata and there is so
> many patches and progress on the feature page[2], but it doesn't look like
> up-to-date, It's difficult to understand the current situation.
>
> Is there any developer or know people about this feature? How's it going? I
> wait this feature from long ago and I want to help in any way.
>
> Thanks.
>
> [1] https://releases.openstack.org/teams/cinder.html
>
> [2] https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1583009

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi,

The HA A/A work can be thought of as having 2 different parts, one is
the core functionality in Cinder and the other is the part relative to
the drivers.

The BP for the feature has been tracking the core functionality, which
can be considered as done, now we need to start working on the
individual drivers to ensure that they support HA A/A operations.

For this we are going to write a list of tests to help vendors determine
the readiness state of their drivers and a guide explaining the
different kind of locks that are available for them to use and what
mutual exclusion cases are already covered by the cinder core
functionality.

Regards,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][gate] functional job busted

[Gorka Eguileor]

On 14/03, Ihar Hrachyshka wrote:
> Hi all,
>
> the patch that started to produce log index file for logstash [1] and
> the patch that switched metadata proxy to haproxy [2] landed and
> together busted the functional job because the latter produces log
> messages with null-bytes inside, while os-log-merger is not resilient
> against it.
>
> If functional job would be in gate and not just in check queue, that
> would not happen.
>
> Attempt to fix the situation in multiple ways at [3]. (For
> os-log-merger patches, we will need new release and then bump the
> version used in gate, so short term neutron patches seem more viable.)
>
> I will need support from both authors of os-log-merger as well as
> other neutron members to unravel that. I am going offline in a moment,
> and hope someone will take care of patches up for review, and land
> what's due.
>
> [1] https://review.openstack.org/#/c/442804/ [2]
> https://review.openstack.org/#/c/431691/ [3]
> https://review.openstack.org/#/q/topic:fix-os-log-merger-crash
>
> Thanks,
> Ihar

Hi Ihar,

That is an unexpected case that never came up during our tests or usage,
but it is indeed something the script should take into account.

Thanks for the os-log-merger patches, I've reviewed them and they look
good to me, so hopefully they'll land before you come back online.  ;-)

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] BlockDeviceDriver deprecated?

[Gorka Eguileor]

On 03/02, Pieter Baele wrote:
> Red Hat tells me the Cinder BlockDeviceDriver is going to be deprecated
> upstream. It isn't supported in RHOSP in any case.
> Is this true?

Yes, that is correct, it's in the release notes:

 
https://github.com/openstack/cinder/blob/master/releasenotes/notes/deprecate-block-device-driver-d30232547a31fe1e.yaml

>
> Bakground information:
> BDD is one of the things I would have used for important HDFS DataNodes in
> a private cloud.
> Red Hat proposes Ceph, but running a replicated HDFS on top of Ceph RBD
> which also replicated.?
>
> So my only option left is bare-metal Hadoop DataNodes (?)
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


The first thing that would come to mind in this case would be to remove
the replication from one of the 2 filesystems, since both of them allow
this:

 - You can create the ceph pool for the volumes with a replication size
   of 1
 - Set your HDFS file replication to 1

I haven't done tests or analyzed your specific case, so I wouldn't be
able to say which one is the most appropriate for your case.

You could always go with LVM nodes, although I'm not sure that's such a
great idea; and you probably already know that Red Hat doesn't support
it for production environments, as it's only approved for PoC
deployments.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder]Using DEFAULT section to configure drivers is not supported since Ocata

[Gorka Eguileor]

On 02/02, Gyorgy Szombathelyi wrote:
> Hi!
>
> Because of the deprecated driver configuration in the DEFAULT section doesn't 
> work anymore in Ocata, I would like to ask if any migration tool exists for 
> using the previously created volumes?
>
> E.g. if the existing volumes have the attribute os-vol-host-attr:host like 
> hostname#RBD, cinder operations won't work until one doesn't do a database 
> update to the format like hostname#ceph#ceph (if the new backend section name 
> is ceph).
> It is not that hard to do a UPDATE SQL on the database, but I think it is not 
> a good thing to force users to do it. Maybe a migration script, or a fallback 
> code in cinder-volume would be appreciated.
> Thinking about the migration script, maybe it should update the old style 
> host attribute to the first defined backend. Or am I just oversighting 
> something obvious?
>
> Br,
> György
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi György,

There is a manage command that allows you to change the host without
using SQL:

 cinder-manage volume update_host --currenthost CURRENTHOST --newhost NEWHOST


If you are using consistency groups you'll also have to change those:

 cinder-manage cg update_cg_host --currenthost CURRENTHOST --newhost NEWHOST


I hope these help.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] How to retrieve volume_driver

[Gorka Eguileor]

On 07/10, cr...@interia.pl wrote:
> Hi Erlon.
>
> Thank you for the reply. I need to collect this information to generate sort 
> of overview of drivers used in given environment. Obviously, it potentially 
> is a multiple backed one. This information does not need to be pulled from 
> the cinder API - I can assume I work under root on host on which cinder 
> service is running.
>
> Parsing of cinder.conf comes with usual problems related to trying to get 
> runtime information from config file. It is very prone to suddenly stop 
> working due to changes in OpenStack itself. We may also potentially run into 
> a situation in which config file has been modified, but Cinder has not been 
> restarted (or worse, file is currently being edited and we read garbage). 
> Now, if it comes to that I can live with those problems, but I was hoping a 
> cleaner solution exists (as mentioned earlier - volume_backend_name, defined 
> in same config, is very easy to retrieve).
>
> Thanks again and best regards,
> Lukasz
>

Hi Lukasz,

Every approach I can think of has its own disadvantages, so you should
evaluate which disadvantages you can live with:

- Reading from the configuration with a Python script using cinder,
  oslo_config, or manually.  This has the out of sync issues you
  mentioned.

- Reading it from Cinder's log files, this requires that the volume
  service is running in debug log level.

- Patching Cinder to add a new API to get this information.  This means
  you have custom code and your data collecting program doesn't work
  with standard Cinder services.

- Using GDB 7 and later, that supports extending GDB with Python, to get
  this information from the running service.  This has a couple of
  disadvantages, one is you have to be really careful with what you are
  doing since you could "break" the running Cinder service, and the
  other disadvantage is that it will pause the whole service while you
  attach, get the info and detach from the service.  I will be talking
  about how to do this sort of thing in a Barcelona Summit talk [1]
  showcasing a case where the benefits outweigh the risks.

Cheers,
Gorka.

[1]: 
https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/15159/cinder-always-on-reliability-and-scalability-guide


> Użytkownik "Erlon Cruz"  napisał(a):
> > Temat: Re: [openstack-dev] [cinder] How to retrieve volume_driver
> > Data: 2016-10-07 14:19
> > Nadawca: "Erlon Cruz" 
> > Adresat: "OpenStack Development Mailing List (not for usage questions)" 
> > ;
> >
> > Hi Luzasz, 
> > This information (volume_driver) is only used on driver loading so, its not 
> > available outside Cinder context. Can you tell how and why you need that? 
> > And why parsing the conf file is not enough for you?
> > Erlon
> > On Fri, Oct 7, 2016 at 9:05 AM,  cr...@interia.pl wrote:
> > Dear all.
> >
> >
> > I'm a PHD student from Poland and have found out about this list from 
> > Openstack wiki. Could you kindly tell me, if there is a way to retrieve, 
> > from outside of OpenStack code, volume_driver for given backend? Preferably 
> > - without a need to parse the cinder.conf file?
> >
> >
> > It is fairly easy to retrieve the volume_backend_name, which in most 
> > scenarios seems to describe the driver type used. However, in general 
> > scenario, this is just a custom string entered by the admin. I would 
> > greatly appreciate, if someone could point me to a way to acquire the 
> > unique stirng, identifying the driver type used - the ona that is passed as 
> > volume_driver in cinder.conf.
> >
> >
> > Thank you in advance for your time.
> >
> >
> > Best regards,
> >
> > Lukasz
> >
> > __
> >
> > OpenStack Development Mailing List (not for usage questions)
> >
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] FFE request for RBD replication

[Gorka Eguileor]

On 12/09, Sean McGinnis wrote:
> On Mon, Sep 12, 2016 at 10:24:23AM +0200, Michał Dulko wrote:
> > +1, thanks for taking care of that!
> >
> > On 09/12/2016 03:35 AM, Huang Zhiteng wrote:
> > > +1 for this long-waited feature to land in Newton.
> > >
> > > On Sun, Sep 11, 2016 at 1:09 AM, Jay S. Bryant
> > > <jsbry...@electronicjungle.net <mailto:jsbry...@electronicjungle.net>>
> > > wrote:
> > >
> > > +1 from me.  It is making good progress and is low risk.
> > >
> > > -Jay
> > >
> > >
> > >
> > > On 09/09/2016 02:32 PM, Gorka Eguileor wrote:
> > >
> > > Hi,
> > >
> > > As some of you may know, Jon Bernard (jbernard on IRC) has
> > > been working
> > > on the RBD v2.1 replication implementation [1] for a while,
> > > and we would
> > > like to request a Feature Freeze Exception for that work, as
> > > we believe
> > > it is a good candidate being a low risk change for the
> > > integrity of
> > > the existing functionality in the driver:
> > >
> > > - It's non intrusive if it's not enabled (enabled using
> > >replication_device configuration option).
> > > - It doesn't affect existing deployments (disabled by default).
> > > - Changes are localized to the driver itself (rbd.py) and the
> > > driver
> > >unit tests file (test_rbd.py).
>
> This looks fine to me. My only concern is the risk accepting it would
> introduce, but as you point out that risk is low and should be well
> contained. I think it should be fine to still get this in.
>
>
> > >
> > > Jon would have liked to make this request himself, but due to the
> > > untimely arrival of his newborn baby this is not possible.
> > >
> > > For obvious reasons Jon will not be available for a little
> > > while, but
> > > this will not be a problem, as I am well acquainted with the
> > > code -and
> > > I'll be able to reach Jon if necessary- and will be taking
> > > care of the
> > > final steps of the review process of his patch: replying to
> > > comments in
> > > a timely fashion, making changes to the code as required, and
> > > answering
> > > pings on IRC regarding the patch.
> > >
> > > Since some people may be interested in testing this
> > > functionality during
> > > the reviewing process -or just for fun- I'll be publishing a
> > > post with
> > > detailed explanation on how to deploy and test this feature as
> > > well as
> > > an automated way to deploy 2 Ceph clusters -linked to be
> > > mirroring one
> > > another-, and one devstack node with everything ready to test the
> > > functionality (configuration and keys for the Ceph clusters,
> > > cinder
> > > configuration, the latest upstream patch, and a volume type
> > > with the
> > > right configuration).
> > >
> > > Please, do not hesitate to ask if there are any questions to
> > > or concerns
> > > related to this request.
> > >
> > > Thank you for taking the time to evaluate this request.
> > >
> > > Cheers,
> > > Gorka.
> > >
> > > [1]: https://review.openstack.org/333565
> > > <https://review.openstack.org/333565>
> > >
> > > 
> > > __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > 
> > > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > 
> > > <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> > >
> > >
> > >
> > > 
> > > __
> > > OpenStack Development M

Re: [openstack-dev] Spain Visa for Indian contributors

[Gorka Eguileor Gimeno]

Hi,

As a native Spanish speaker I offer my help with the translation if needed.
I assume the OpenStack visa team would be the one required to issue the
translated invitation letter for it to be valid.

Cheers,
Gorka.

On Thu, Sep 15, 2016 at 2:14 PM, M Ranga Swami Reddy 
wrote:

> Page#2:  http://www.vfsglobal.com/spain/india/pdf/Spain-Checklist.pdf
> ==
> For business visas:
> □ Invitation letter from the Spanish company in Spanish language or both
> in English and Spanish.Letters issued only in English will not be
> accepted. This will be an obligatory requirement.
> ===
>
> On Thu, Sep 15, 2016 at 3:35 PM, M Ranga Swami Reddy  > wrote:
>
>> That's good. Please share the appropriate link/url to openstack visa team.
>>
>> On Thu, Sep 15, 2016 at 1:14 PM, Kekane, Abhishek <
>> abhishek.kek...@nttdata.com> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> Yes it’s request from embassy than for Indian citizens invitation letter
>>> should be in Spanish as well English language.
>>>
>>> I have sent mail to openstack visa team.
>>>
>>>
>>>
>>> Thank you,
>>>
>>>
>>>
>>> Abhishek Kekane
>>>
>>>
>>>
>>> *From:* M Ranga Swami Reddy [mailto:swamire...@gmail.com]
>>> *Sent:* Thursday, September 15, 2016 1:06 PM
>>>
>>> *To:* OpenStack Development Mailing List (not for usage questions)
>>> *Subject:* Re: [openstack-dev] Spain Visa for Indian contributors
>>>
>>>
>>>
>>> Hi,
>>>
>>> Who asked the invitation letter in Spanish? Is it embassy, please share
>>> the same info openstack visa team.
>>>
>>>
>>>
>>> JYI - for Tokyo - invitation letter was in Japanese only.
>>>
>>>
>>>
>>> Thanks
>>>
>>> Swami
>>>
>>>
>>>
>>> On Thu, Sep 15, 2016 at 11:58 AM, Kekane, Abhishek <
>>> abhishek.kek...@nttdata.com> wrote:
>>>
>>> Hi Janki,
>>>
>>>
>>>
>>> I will keep you posted about this. Mean time sent mail to ‘
>>> eventv...@openstack.org’ to explain your issue and also give reference
>>> of this ML.
>>>
>>>
>>>
>>> Thank you,
>>>
>>>
>>>
>>> Abhishek Kekane
>>>
>>>
>>>
>>> *From:* Janki Chhatbar [mailto:jankih...@gmail.com]
>>> *Sent:* Thursday, September 15, 2016 11:51 AM
>>>
>>>
>>> *To:* OpenStack Development Mailing List (not for usage questions)
>>> *Subject:* Re: [openstack-dev] Spain Visa for Indian contributors
>>>
>>>
>>>
>>> Hi Abhishek
>>>
>>> I am Janki. I am applying for visa from India. I haven't applied yet but
>>> a friend of mine is facing the same issue of needing an invitation letter
>>> in Spanish.
>>>
>>> May be everyone applying from India can request to have the letter in
>>> Spanish too.
>>>
>>> Let me know how you proceed further. It would help me while applying.
>>>
>>> Thanks
>>>
>>> Janki
>>>
>>>
>>>
>>> On Thu, Sep 15, 2016 at 11:30 AM, Kekane, Abhishek <
>>> abhishek.kek...@nttdata.com> wrote:
>>>
>>> Hi John,
>>>
>>> I have sent mail to 'eventv...@openstack.org', waiting for positive
>>> reply.
>>>
>>> One reply I got from Kendall is,
>>>
>>> We only send the letter in English. No one else applying for a visa has
>>> had a problem getting their visa with just the English letter so the letter
>>> you received should be sufficient.
>>>
>>> I will show this reply mail at VFS center and hopefully they will take
>>> no objection about this.
>>>
>>> Thank you for your time,
>>>
>>> Abhishek Kekane
>>>
>>> -Original Message-
>>> From: John Villalovos [mailto:openstack@sodarock.com]
>>>
>>> Sent: Thursday, September 15, 2016 11:17 AM
>>> To: OpenStack Development Mailing List (not for usage questions)
>>> Subject: Re: [openstack-dev] Spain Visa for Indian contributors
>>>
>>> On that page they seem to list a contact email:
>>>
>>> eventv...@openstack.org
>>>
>>> Hopefully they can help with the issue.
>>>
>>> John
>>>
>>> On Wed, Sep 14, 2016 at 9:24 PM, Kekane, Abhishek <
>>> abhishek.kek...@nttdata.com> wrote:
>>> > Hi John,
>>> >
>>> > I have read the information given at
>>> > https://www.openstack.org/summit/barcelona-2016/travel/#visa
>>> > and got the invitation letter but it's in English language. Problem is
>>> that Visa centers in India are demanding this invitation letter in English
>>> as well as Spanish language.
>>> >
>>> > Thank you,
>>> >
>>> > Abhishek Kekane
>>> >
>>> > -Original Message-
>>> > From: John Villalovos [mailto:openstack@sodarock.com]
>>> > Sent: Thursday, September 15, 2016 9:45 AM
>>> > To: OpenStack Development Mailing List (not for usage questions)
>>> > Subject: Re: [openstack-dev] Spain Visa for Indian contributors
>>> >
>>> > There is information on the Visa process at:
>>> > https://www.openstack.org/summit/barcelona-2016/travel/#visa
>>> >
>>> > Not sure if you have already read that information.
>>> >
>>> > They talk about the steps needed to get an invitation letter.
>>> >
>>> > Good luck!
>>> >
>>> >
>>> > On Wed, Sep 14, 2016 at 8:51 PM, Kekane, Abhishek <
>>> abhishek.kek...@nttdata.com> wrote:
>>> >> Hi Devs, Stackers,
>>> >>
>>> >>
>>> >>
>>> >> While applying visa from Pune (India), I came to know that 

Re: [openstack-dev] [Cinder] FFE request for RBD replication

[Gorka Eguileor Gimeno]

Thanks everyone for the quick and positive responses.

I'm on PTO this week with limited internet access, so my replies may have
some delays. Next week it will go back to normal.  :-)

Cheers,
Gorka.

On Mon, Sep 12, 2016 at 3:04 PM, Sean McGinnis <sean.mcgin...@gmx.com>
wrote:

> On Mon, Sep 12, 2016 at 10:24:23AM +0200, Michał Dulko wrote:
> > +1, thanks for taking care of that!
> >
> > On 09/12/2016 03:35 AM, Huang Zhiteng wrote:
> > > +1 for this long-waited feature to land in Newton.
> > >
> > > On Sun, Sep 11, 2016 at 1:09 AM, Jay S. Bryant
> > > <jsbry...@electronicjungle.net <mailto:jsbry...@electronicjungle.net>>
> > > wrote:
> > >
> > > +1 from me.  It is making good progress and is low risk.
> > >
> > > -Jay
> > >
> > >
> > >
> > > On 09/09/2016 02:32 PM, Gorka Eguileor wrote:
> > >
> > > Hi,
> > >
> > > As some of you may know, Jon Bernard (jbernard on IRC) has
> > > been working
> > > on the RBD v2.1 replication implementation [1] for a while,
> > > and we would
> > > like to request a Feature Freeze Exception for that work, as
> > > we believe
> > > it is a good candidate being a low risk change for the
> > > integrity of
> > > the existing functionality in the driver:
> > >
> > > - It's non intrusive if it's not enabled (enabled using
> > >replication_device configuration option).
> > > - It doesn't affect existing deployments (disabled by default).
> > > - Changes are localized to the driver itself (rbd.py) and the
> > > driver
> > >unit tests file (test_rbd.py).
>
> This looks fine to me. My only concern is the risk accepting it would
> introduce, but as you point out that risk is low and should be well
> contained. I think it should be fine to still get this in.
>
>
> > >
> > > Jon would have liked to make this request himself, but due to
> the
> > > untimely arrival of his newborn baby this is not possible.
> > >
> > > For obvious reasons Jon will not be available for a little
> > > while, but
> > > this will not be a problem, as I am well acquainted with the
> > > code -and
> > > I'll be able to reach Jon if necessary- and will be taking
> > > care of the
> > > final steps of the review process of his patch: replying to
> > > comments in
> > > a timely fashion, making changes to the code as required, and
> > > answering
> > > pings on IRC regarding the patch.
> > >
> > > Since some people may be interested in testing this
> > > functionality during
> > > the reviewing process -or just for fun- I'll be publishing a
> > > post with
> > > detailed explanation on how to deploy and test this feature as
> > > well as
> > > an automated way to deploy 2 Ceph clusters -linked to be
> > > mirroring one
> > > another-, and one devstack node with everything ready to test
> the
> > > functionality (configuration and keys for the Ceph clusters,
> > > cinder
> > > configuration, the latest upstream patch, and a volume type
> > > with the
> > > right configuration).
> > >
> > > Please, do not hesitate to ask if there are any questions to
> > > or concerns
> > > related to this request.
> > >
> > > Thank you for taking the time to evaluate this request.
> > >
> > > Cheers,
> > > Gorka.
> > >
> > > [1]: https://review.openstack.org/333565
> > > <https://review.openstack.org/333565>
> > >
> > > 
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > <http://openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe>
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack-dev
> > > <http://lists.openstack.or

[openstack-dev] [Cinder] FFE request for RBD replication

[Gorka Eguileor]

Hi,

As some of you may know, Jon Bernard (jbernard on IRC) has been working
on the RBD v2.1 replication implementation [1] for a while, and we would
like to request a Feature Freeze Exception for that work, as we believe
it is a good candidate being a low risk change for the integrity of
the existing functionality in the driver:

- It's non intrusive if it's not enabled (enabled using
  replication_device configuration option).
- It doesn't affect existing deployments (disabled by default).
- Changes are localized to the driver itself (rbd.py) and the driver
  unit tests file (test_rbd.py).

Jon would have liked to make this request himself, but due to the
untimely arrival of his newborn baby this is not possible.

For obvious reasons Jon will not be available for a little while, but
this will not be a problem, as I am well acquainted with the code -and
I'll be able to reach Jon if necessary- and will be taking care of the
final steps of the review process of his patch: replying to comments in
a timely fashion, making changes to the code as required, and answering
pings on IRC regarding the patch.

Since some people may be interested in testing this functionality during
the reviewing process -or just for fun- I'll be publishing a post with
detailed explanation on how to deploy and test this feature as well as
an automated way to deploy 2 Ceph clusters -linked to be mirroring one
another-, and one devstack node with everything ready to test the
functionality (configuration and keys for the Ceph clusters, cinder
configuration, the latest upstream patch, and a volume type with the
right configuration).

Please, do not hesitate to ask if there are any questions to or concerns
related to this request.

Thank you for taking the time to evaluate this request.

Cheers,
Gorka.

[1]: https://review.openstack.org/333565

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] Nominating Scott D'Angelo to Cinder core

[Gorka Eguileor]

+2

Congrats Scott!!


On 27/06, Sean McGinnis wrote:
> I would like to nominate Scott D'Angelo to core. Scott has been very
> involved in the project for a long time now and is always ready to help
> folks out on IRC. His contributions [1] have been very valuable and he
> is a thorough reviewer [2].
>
> Please let me know if there are any objects to this within the next
> week. If there are none I will switch Scott over by next week, unless
> all cores approve prior to then.
>
> Thanks!
>
> Sean McGinnis (smcginnis)
>
> [1] 
> https://review.openstack.org/#/q/owner:%22Scott+DAngelo+%253Cscott.dangelo%2540hpe.com%253E%22+status:merged
> [2] http://cinderstats-dellstorage.rhcloud.com/cinder-reviewers-90.txt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-operators][cinder] max_concurrent_builds in Cinder

[Gorka Eguileor]

On 23/05, Ivan Kolodyazhny wrote:
> Hi developers and operators,
> I would like to get any feedback from you about my idea before I'll start
> work on spec.
> 
> In Nova, we've got max_concurrent_builds option [1] to set 'Maximum number
> of instance builds to run concurrently' per each compute. There is no
> equivalent Cinder.

Hi,

First I want to say that I think this is a good idea because I know this
message will get diluted once I start with my mumbling.  ;-)

The first thing we should allow to control is the number of workers per
service, since we currently only allow setting it for the API nodes and
all other nodes will use a default of 1000.  I posted a patch [1] to
allow this and it's been sitting there for the last 3 months.  :'-(

As I see it not all mentioned problems are equal, and the main
distinction is caused by Cinder being not only in the control path but
also in the data path. Resulting in some of the issues being backend
specific limitations, that I believe should be address differently in
the specs.

For operations where Cinder is in the control path we should be
limiting/queuing operations in the cinder core code (for example the
manager) whereas when the limitation only applies to some drivers this
should be addressed by the drivers themselves.  Although the spec should
provide a clear mechanism/pattern to solve it in the drivers as well so
all drivers can use a similar pattern which will provide consistency,
making it easier to review and maintain.

The queuing should preserve the order of arrival of operations, which
file locks from Oslo concurrency and Tooz don't do.

> 
> Why do we need it for Cinder? IMO, it could help us to address following
> issues:
> 
>- Creation of N volumes at the same time increases a lot of resource
>usage by cinder-volume service. Image caching feature [2] could help us a
>bit in case when we create volume form image. But we still have to upload N
>images to the volumes backend at the same time.

This is an example where we are in the data path.

>- Deletion on N volumes at parallel. Usually, it's not very hard task
>for Cinder, but if you have to delete 100+ volumes at once, you can fit
>different issues with DB connections, CPU and memory usages. In case of
>LVM, it also could use 'dd' command to cleanup volumes.

This is a case where it is a backend limitation and should be handled by
the drivers.

I know some people say that deletion and attaching have problems when a
lot of them are requested to the c-vol nodes and that cinder cannot
handle the workload properly, but in my experience these cases are
always due to suboptimal cinder configuration, like a low number of DB
connections configured in cinder that make operations fight for a DB
connection creating big delays to complete operations.

>- It will be some kind of load balancing in HA mode: if cinder-volume
>process is busy with current operations, it will not catch message from
>RabbitMQ and other cinder-volume service will do it.

I don't understand what you mean with this.  Do you mean that Cinder
service will stop listening to the message queue when it reaches a
certain workload on the "heavy" operations?  Then wouldn't it also stop
processing "light" operations?

>- From users perspective, it seems that better way is to create/delete N
>volumes a bit slower than fail after X volumes were created/deleted.

I agree, it's better not to fail.  :-)

Cheers,
Gorka.

> 
> 
> [1]
> https://github.com/openstack/nova/blob/283da2bbb74d0a131a66703516d16698a05817c7/nova/conf/compute.py#L161-L163
> [2]
> https://specs.openstack.org/openstack/cinder-specs/specs/liberty/image-volume-cache.html
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] Nominating Michał Dulko to Cinder Core

[Gorka Eguileor]

On 03/05, Sean McGinnis wrote:
> Hey everyone,
> 
> I would like to nominate Michał Dulko to the Cinder core team. Michał's
> contributions with both code reviews [0] and code contributions [1] have
> been significant for some time now.
> 
> His persistence with versioned objects has been instrumental in getting
> support in the Mitaka release for rolling upgrades.
> 
> If there are no objections from current cores by next week, I will add
> Michał to the core group.
> 
> [0] http://cinderstats-dellstorage.rhcloud.com/cinder-reviewers-90.txt
> [1]
> https://review.openstack.org/#/q/owner:%22Michal+Dulko+%253Cmichal.dulko%2540intel.com%253E%22++status:merged
> 
> Thanks!
> 
> Sean McGinnis (smcginnis)
> 

+1  Definitively a great addition!

Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Let's do presentations/sessions on Mitaka's new complex features in Design Summit

[Gorka Eguileor]

On 16/03, D'Angelo, Scott wrote:
> I can do a presentation on microversions.
> 

"I love it when a plan comes together".
This one had your name written all over it.  ;-)

> 
> 
> Scott D'Angelo (scottda)
> 
> -- Original Message --
> From : Gorka Eguileor
> Subject : [openstack-dev] [cinder] Let's do presentations/sessions on 
> Mitaka's new complex features in Design Summit
> 
> 
> Hi,
> 
> As you all probably know, during this cycle we have introduced quite a
> big number of changes in cinder that will have a great impact in the
> development of the new functionality as well as changes to existing ones
> moving forward from an implementation perspective.
> 
> These changes to the cinder code include, but are not limited to,
> microversions, rolling upgrades, and conditional DB update functionality
> to remove API races, and while the latter has a good number of examples
> already merged and more patches under review, the other 2 have just been
> introduced and there are no patches in cinder that can serve as easy
> reference on how to use them.
> 
> As cinder developers we will all have to take these changes into account
> in our new patches, but it is hard to do so when one doesn't have an
> in-depth knowledge of them, and while we all probably know quite a bit
> about them, it will take some time to get familiar enough to be aware of
> *all* the implications of the changes made by newer patches.
> 
> And it's for this reason that I would like to suggest that during this
> summit's cinder design sessions we take the time to go through the
> changes giving not only an example of how they should be used in a
> patch, but also the do's, dont's and gotchas.
> 
> A possible format for these explanations could be a presentation -around
> 30 minutes- by the people that were involved in the development,
> followed by Q
> 
> I would have expected to see some of these in the "Upstream Dev" track,
> but unfortunately I don't (maybe I'm just missing them with all the cool
> title names).  And maybe these talks are not relevant for that track,
> being so specific and only relevant to cinder developers and all.
> 
> I believe these presentations would help the cinder team increase the
> adoption speed of these features while reducing the learning curve and
> the number of bugs introduced in the code caused by gaps in our
> knowledge and misinterpretations of the new functionality.
> 
> I would take lead on the conditional DB updates functionality, and I
> would have no problem doing the Rolling upgrades presentation as well.
> But I believe there are people more qualified and more deserving of
> doing that one; though I offer my help if they want it.
> 
> I have added those 3 topics to the Etherpad with Newton Cinder Design
> Summit Ideas [1] so people can volunteer and express their ideas in
> there.
> 
> Cheers,
> Gorka.
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Let's do presentations/sessions on Mitaka's new complex features in Design Summit

[Gorka Eguileor]

On 15/03, Michał Dulko wrote:
> On 03/14/2016 09:52 PM, Gorka Eguileor wrote:
> > Hi,
> >
> > As you all probably know, during this cycle we have introduced quite a
> > big number of changes in cinder that will have a great impact in the
> > development of the new functionality as well as changes to existing ones
> > moving forward from an implementation perspective.
> >
> > These changes to the cinder code include, but are not limited to,
> > microversions, rolling upgrades, and conditional DB update functionality
> > to remove API races, and while the latter has a good number of examples
> > already merged and more patches under review, the other 2 have just been
> > introduced and there are no patches in cinder that can serve as easy
> > reference on how to use them.
> >
> > As cinder developers we will all have to take these changes into account
> > in our new patches, but it is hard to do so when one doesn't have an
> > in-depth knowledge of them, and while we all probably know quite a bit
> > about them, it will take some time to get familiar enough to be aware of
> > *all* the implications of the changes made by newer patches.
> >
> > And it's for this reason that I would like to suggest that during this
> > summit's cinder design sessions we take the time to go through the
> > changes giving not only an example of how they should be used in a
> > patch, but also the do's, dont's and gotchas.
> >
> > A possible format for these explanations could be a presentation -around
> > 30 minutes- by the people that were involved in the development,
> > followed by Q
> >
> > I would have expected to see some of these in the "Upstream Dev" track,
> > but unfortunately I don't (maybe I'm just missing them with all the cool
> > title names).  And maybe these talks are not relevant for that track,
> > being so specific and only relevant to cinder developers and all.
> >
> > I believe these presentations would help the cinder team increase the
> > adoption speed of these features while reducing the learning curve and
> > the number of bugs introduced in the code caused by gaps in our
> > knowledge and misinterpretations of the new functionality.
> >
> > I would take lead on the conditional DB updates functionality, and I
> > would have no problem doing the Rolling upgrades presentation as well.
> > But I believe there are people more qualified and more deserving of
> > doing that one; though I offer my help if they want it.
> >
> > I have added those 3 topics to the Etherpad with Newton Cinder Design
> > Summit Ideas [1] so people can volunteer and express their ideas in
> > there.
> >
> > Cheers,
> > Gorka.
>
> I can certainly do one on rolling upgrades from developer's perspective.
> I think I've got this knowledge summed up in patch to enhance the devref
> [1], but of course presentation and Q would be beneficial.

Jeje, I was actually thinking of you for that one, but didn't want to be
pushy about it.  ;-)

>
> And by the way - I think that for all the stuff that's worthy of such
> presentation, we should have a detailed devref page.
>
> [1] https://review.openstack.org/#/c/279186/

You are totally right!

I'll review that one and work on adding a new one for the conditional
update stuff.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Maintaining httplib2 python library

[Gorka Eguileor]

On 14/03, Sean McGinnis wrote:
> On Mon, Mar 14, 2016 at 10:37:52AM -0400, Sean Dague wrote:
> > On 03/14/2016 10:24 AM, Ian Cordasco wrote:
> > >  
> > > 
> > > -Original Message-
> > > From: Davanum Srinivas 
> > > Reply: OpenStack Development Mailing List (not for usage questions) 
> > > 
> > > Date: March 14, 2016 at 09:18:50
> > > To: OpenStack Development Mailing List (not for usage questions) 
> > > 
> > > Subject:  [openstack-dev] [all] Maintaining httplib2 python library
> > > 
> > >> Team,
> > >>  
> > >> fyi, http://bitworking.org/news/2016/03/an_update_on_httplib2
> > >>  
> > >> We have httplib2 in our global requirements and lots of projects are
> > >> using it[1]. Is there anyone willing to step up?
> > > 
> > > Is it really worth our time to dedicate extra resources to that? Glance 
> > > has been discussing (but it's been a low priority) to switing all our 
> > > dependence on httplib2 to requests (and maybe urllib3 directly) as 
> > > necessary.
> > > 
> > > We have other tools and libraries we can use without taking over 
> > > maintenance of yet another library.
> > > 
> > > I think the better question than "Can people please maintain this for the 
> > > community?" is "What benefits does httplib2 have over something that is 
> > > actively maintained (and has been actively maintaiend) like urllib3, 
> > > requests, etc.?"
> > > 
> > > And then we can (and should) also ask "Why have we been using this? How 
> > > much work do cores think it would be to remove this from our global 
> > > requirements?"
> 
> Cinder only has it in a new backup driver for Google Cloud Storage. The
> googleapiclient docs actually say to use httplib2 for one of the calls
> "or something that acts like it."
> 
> I will see if we can switch this over to an appropriate duck type. I
> would much rather get rid of usage than unnecessarily keep a project on
> life support.

Looking at Google's api python client I see it has a dependency on
httplib2 [1] but it looks like Sean's duck typing suggestion would work
in our case since we are not using batch http requests and Google's
library would mostly use httplib2 for constants.  Although we would
still have an indirect dependency on httplib2.

If we really want to remove cinder's dependency from httplib2, indirect
or otherwise, we could switch to another Google cloud library, like
(shameless plug here) gcs-client [2] that uses requests instead of
httplib2 and which would not only be easy to replace it with, but has
also already been tested as a Google Cloud Backup driver in cinder [3].

Cheers,
Gorka.


[1]: https://github.com/google/google-api-python-client/blob/master/setup.py#L66
[2]: https://github.com/Akrog/gcs-client
[3]: 
https://github.com/Akrog/cinder/blob/akrog/gcs_backup/cinder/backup/drivers/gcs.py


> 
> > 
> > +1.
> > 
> > Here is the non comprehensive list of usages based on what trees I
> > happen to have checked out (which is quite a few, but not all of
> > OpenStack for sure).
> > 
> > I think before deciding to take over ownership of an upstream lib (which
> > is a large commitment over space and time), we should figure out the
> > migration cost. All the uses in Tempest come from usage in Glance IIRC
> > (and dealing with chunked encoding).
> > 
> > Neutron seems to use it for a couple of proxies, but that seems like
> > requests/urllib3 might be sufficient.
> > 
> > In Horizon it's only used for a couple of tests.
> > 
> > EC2 uses it as a proxy client to the Nova metadata service. Again, I
> > can't imagine that requests wouldn't be sufficient.
> > 
> > Trove doesn't seem to actually use it (though it's listed), though maybe
> > wsgi_intercept uses it directly?
> > 
> > run_tests.py:from wsgi_intercept.httplib2_intercept import install as
> > wsgi_install
> > 
> > python-muranoclient lists it as a requirement, there is no reference in
> > the source tree for it.
> > 
> > 
> > I suspect Glance is really the lynchpin here (as it actually does some
> > low level stuff with it). If there can be a Glance plan to get off of
> > it, the rest can follow pretty easily.
> > 
> > -Sean
> > 
> > -- 
> > Sean Dague
> > http://dague.net
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

[openstack-dev] [cinder] Let's do presentations/sessions on Mitaka's new complex features in Design Summit

[Gorka Eguileor]

Hi,

As you all probably know, during this cycle we have introduced quite a
big number of changes in cinder that will have a great impact in the
development of the new functionality as well as changes to existing ones
moving forward from an implementation perspective.

These changes to the cinder code include, but are not limited to,
microversions, rolling upgrades, and conditional DB update functionality
to remove API races, and while the latter has a good number of examples
already merged and more patches under review, the other 2 have just been
introduced and there are no patches in cinder that can serve as easy
reference on how to use them.

As cinder developers we will all have to take these changes into account
in our new patches, but it is hard to do so when one doesn't have an
in-depth knowledge of them, and while we all probably know quite a bit
about them, it will take some time to get familiar enough to be aware of
*all* the implications of the changes made by newer patches.

And it's for this reason that I would like to suggest that during this
summit's cinder design sessions we take the time to go through the
changes giving not only an example of how they should be used in a
patch, but also the do's, dont's and gotchas.

A possible format for these explanations could be a presentation -around
30 minutes- by the people that were involved in the development,
followed by Q

I would have expected to see some of these in the "Upstream Dev" track,
but unfortunately I don't (maybe I'm just missing them with all the cool
title names).  And maybe these talks are not relevant for that track,
being so specific and only relevant to cinder developers and all.

I believe these presentations would help the cinder team increase the
adoption speed of these features while reducing the learning curve and
the number of bugs introduced in the code caused by gaps in our
knowledge and misinterpretations of the new functionality.

I would take lead on the conditional DB updates functionality, and I
would have no problem doing the Rolling upgrades presentation as well.
But I believe there are people more qualified and more deserving of
doing that one; though I offer my help if they want it.

I have added those 3 topics to the Etherpad with Newton Cinder Design
Summit Ideas [1] so people can volunteer and express their ideas in
there.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] PTL Candidacy

[Gorka Eguileor]

Thanks for your leadership Sean, I think you've done a terrific job.

Gorka.

On 11/03, Sean McGinnis wrote:
> Hey everyone,
> 
> Wow, how six months flies! I'd like to announce my candidacy to continue on as
> Cinder PTL for the Newton release cycle.
> 
> A lot has been accomplished in the Mitaka cycle. After a lot of work by many
> folks, over a couple development cycles, we now have what we consider a "tech
> preview" of rolling upgrades. It just hasn't had enough runtime and testing 
> for
> us to say it's "official". We will likely need to fix a few minor things in 
> the
> Newton timeframe before it's fully baked and reliable. But it has come a long
> way and I'm really happy with the progress that has been made.
> 
> Another priority we had identified for Mitaka was active/active high
> availability of the c-vol service. We were not able to complete that work, but
> many pieces have been put in place to support that in Newton. We fixed several
> API races and added the ability to use something like tooz for locking. These
> are foundation pieces for us to be able to start breaking out things and
> running in a reliable active/active configuration.
> 
> Microversion support has been added and there is now a new v3 API endpoint.
> This was a bit of a controversy as we really had just started to get folks to
> move off of v1 to v2. To be safe though I decided it would protect end users
> better to have a clearly separate new API endpoint for the microversion
> compatibility. And now hopefully it is our last.
> 
> Replication was another slightly controversial feature implemented. Late in
> Liberty we finally agreed on a spec for a v2 version of replication. The v2
> spec was approved so late that no one actually had time to implement it for
> that release. As we started to implement it for Mitaka we found that a lot of
> compromises had crept in during the spec review that it had the risk of being
> too complex and having some of the issues we were trying to get rid of by
> moving away from replication v1. At our midcycle we had a lot of discussion on
> replication and finally decided to change course before it was too late.
> Whether that ends up being the best choice when we look back a year from now 
> or
> not, I'm proud of the team that we were willing to put on the brakes and make
> changes - even though it was more work for us - before we released something
> out to end users that would have caused problems or a poor experience.
> 
> Other than that, there's mostly been a lot of good bug fixes. Eight new 
> drivers
> have been added from (I think) five different vendors. The os-brick library is
> now 1.0 (actually 1.1.0) and is in use by both Cinder and Nova for common
> storage management operations so there is not a duplication and disconnect of
> code between the two projects. We were also able to add a Brick cinder client
> extension to be able to perform storage management on nodes without Nova (bare
> metal, etc.).
> 
> None of this goodness was from me.
> 
> We have a bunch of smart and active members of the Cinder community. They are
> the ones that are making a difference, working across the community, and
> making sure Cinder is a solid component in an OpenStack cloud.
> 
> Being part of the Cinder community has been one of the best and most engaging
> parts of my career. I am lucky enough to have support from my company to be
> able to devote time to being a part of this. I would love the opportunity to
> continue as PTL to not just contribute where I can, but to make sure the folks
> doing the heavy lifting have the support and project organization they need to
> avoid distractions and be able to focus on getting the important stuff done.
> 
> I think in Newton we need to continue the momentum and get Active/Active 
> Cinder
> volume service support implemented. We need to continue to work closely with
> the Nova team to make sure our interaction is correct and solid. But also work
> to make Cinder a useful storage management interface in environments without
> Nova. I will continue to encourage developer involvement and vendor support.
> We need to improve the user experience with better error reporting when things
> go wrong. And last, but definitely not least, we need to continue to expand 
> our
> testing - unit, functional, and tempest - to make sure we can avoid those
> errors and deliver a high quality and solid solution.
> 
> I really feel I'm just getting into the swing of things. I would love the
> opportunity to serve as PTL for the Newton release.
> 
> Thank you for your consideration.
> 
> Sean McGinnis (smcginnis)
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Nested quota -1 limit race condition

[Gorka Eguileor]

On 03/03, Ryan McNair wrote:
> Hey all,
> 
> Nested quotas has officially started fighting back - while writing Tempest
> tests for the nested quota support [1], I hit a race-condition related to
> the nested quota -1 support that has me stumped. I opened a bug for it [2]
> with more details, but basically the issue occurs when if you change a
> limit to or from -1 for a project while at the same time creating volumes
> on the project (or on it's subtree if child is using -1 also).
> 
> Trying to figure out how to best handle this. Whether suggestions for how
> to fix this, thoughts on how critical this situation is, whether we should
> disable -1 support for now, etc. - all input is welcome.
> 
> Thanks,
> Ryan McNair (mc_nair)
> 
> [1] https://review.openstack.org/#/c/285640/2
> [2] https://bugs.launchpad.net/cinder/+bug/1552944

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi,

I haven't looked in depth into the nested quotas, but from reading the
BZ and what I can remember this is not a trivial feature and it seems
like the DB implementation is not properly thought of for the
concurrency issues that usually arise in systems like OpenStack, though
I could be totally wrong here.

My best suggestion from my ignorance on this specific topic is that you
try looking into the conditional updates functionality that was added to
Cinder to remove the API races and if that's not enough you should look
into DB transaction isolation for those operations.

If the information provided in the conditional updates docstrings and
the race removing code that's been merged into the API is not enough and
you need a hand to understand how it works, feel free to ping me on IRC
(geguileo).

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Gorka Eguileor]

On 02/03, Ivan Kolodyazhny wrote:
> I'll try to implement such scenario and step-by-step guideline soon.
> 

That would be fantastic!! Thank you very much

Looking forward to it.  :-)

Cheers,
Gorka.

> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> On Wed, Mar 2, 2016 at 5:16 PM, Eric Harney  wrote:
> 
> > On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote:
> > > Eric,
> > >
> > > For now, we test Cinder API with some concurrency only with Rally, so,
> > IMO,
> > > it's reasonable get more scenarios for API races fixes.
> > >
> > > It's not a hard task to implement new scenarios, they are pretty simple:
> > > [11] and [12]
> > >
> >
> > Sure, these are simple, but I think it's nowhere near that simple to
> > write a scenario which will prove that "remove API races" works correctly.
> >
> > > [11]
> > >
> > https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535
> > > [12]
> > >
> > https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516
> > >
> > > Regards,
> > > Ivan Kolodyazhny,
> > > http://blog.e0ne.info/
> > >
> > > On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney  wrote:
> > >
> > >> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
> > >>> Eric,
> > >>>
> > >>> There are Gorka's patches [10] to remove API Races
> > >>>
> > >>>
> > >>> [10]
> > >>>
> > >>
> > https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> > >>>
> > >>> Regards,
> > >>> Ivan Kolodyazhny,
> > >>> http://blog.e0ne.info/
> > >>>
> > >>
> > >> So the second part of my question is, is writing a Rally job to prove
> > >> out that code a reasonable task?
> > >>
> > >> How hard is that to do and what does it look like?
> > >>
> > >>> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney 
> > wrote:
> > >>>
> >  On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> > > Hi Team,
> > >
> > > Here are my thoughts and proposals how to make Cinder testing process
> > > better. I won't cover "3rd party CI's" topic here. I will share my
> >  opinion
> > > about current and feature jobs.
> > >
> > >
> > > Unit-tests
> > >
> > >- Long-running tests. I hope, everybody will agree that unit-tests
> >  must
> > >be quite simple and very fast. Unit tests which takes more than
> > 3-5
> >  seconds
> > >should be refactored and/or moved to 'integration' tests.
> > >Thanks to Tom Barron for several fixes like [1]. IMO, we it would
> > be
> > >good to have some hacking checks to prevent such issues in a
> > future.
> > >
> > >- Tests coverage. We don't check it in an automatic way on gates.
> > >Usually, we require to add some unit-tests during code review
> >  process. Why
> > >can't we add coverage job to our CI and do not merge new patches,
> > >> with
> > >will decrease tests coverage rate? Maybe, such job could be voting
> > >> in
> >  a
> > >future to not ignore it. For now, there is not simple way to check
> >  coverage
> > >because 'tox -e cover' output is not useful [2].
> > >
> > >
> > > Functional tests for Cinder
> > >
> > > We introduced some functional tests last month [3]. Here is a patch
> > to
> > > infra to add new job [4]. Because these tests were moved from
> >  unit-tests, I
> > > think we're OK to make this job voting. Such tests should not be a
> > > replacement for Tempest. They even could tests Cinder with Fake
> > Driver
> > >> to
> > > make it faster and not related on storage backends issues.
> > >
> > >
> > > Tempest in-tree tests
> > >
> > > Sean started work on it [5] and I think it's a good idea to get them
> > in
> > > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
> > >> real
> > > backend.
> > >
> > >
> > > Functional tests for python-brick-cinderclient-ext
> > >
> > > There are patches that introduces functional tests [6] and new job
> > [7].
> > >
> > >
> > > Functional tests for python-cinderclient
> > >
> > > We've got a very limited set of such tests and non-voting job. IMO,
> > we
> >  can
> > > run them even with Cinder Fake Driver to make them not depended on a
> > > storage backend and make it faster. I believe, we can make this job
> >  voting
> > > soon. Also, we need more contributors to this kind of tests.
> > >
> > >
> > > Integrated tests for python-cinderclient
> > >
> > > We need such tests to make sure that we won't break Nova, Heat or
> > other
> > > python-cinderclient consumers with a next merged patch. There is a
> > >> thread
> > > in openstack-dev ML about such tests [8] and proposal [9] to
> > introduce
> >  them
> > > to python-cinderclient.
> > >
> > >
> > > Rally tests
> > >
> 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Gorka Eguileor]

On 02/03, Ivan Kolodyazhny wrote:
> Eric,
> 
> There are Gorka's patches [10] to remove API Races
> 
> 
> [10]
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> 

I looked at Rally a long time ago so apologies if I'm totally off base
here, but it looked like it was a performance evaluation tool, which
means that it probably won't help to check for API Races (at least I
didn't see how when I looked).

Many of the API races only happen if you simultaneously try the same
operation multiple times against the same resource or if there are
different operations that are trying to operate on the same resource.

On the first case if Rally allowed it we could test it because we know
only 1 of the operations should succeed, but on the second case when we
are talking about preventing races from different operations there is no
way to know what the result should be, since the order in which those
operations are executed on each test run will determine which one will
fail and which one will succeed.

I'm not trying to go against the general idea of adding rally tests, I
just think that they won't help in the case of the API races.

> > > [ ... ]
> > >
> > >- Tests coverage. We don't check it in an automatic way on gates.
> > >Usually, we require to add some unit-tests during code review
> > process. Why
> > >can't we add coverage job to our CI and do not merge new patches, with
> > >will decrease tests coverage rate? Maybe, such job could be voting in
> > a
> > >future to not ignore it. For now, there is not simple way to check
> > coverage
> > >because 'tox -e cover' output is not useful [2].

In my experience preventing patches that reduce test coverage can be
sometimes problematic because it's a percentage, and many times code
refactoring gives false positives.  :-(


Cheers,
Gorka.


> > >
> > >
> > > Functional tests for Cinder
> > >
> > > We introduced some functional tests last month [3]. Here is a patch to
> > > infra to add new job [4]. Because these tests were moved from
> > unit-tests, I
> > > think we're OK to make this job voting. Such tests should not be a
> > > replacement for Tempest. They even could tests Cinder with Fake Driver to
> > > make it faster and not related on storage backends issues.
> > >
> > >
> > > Tempest in-tree tests
> > >
> > > Sean started work on it [5] and I think it's a good idea to get them in
> > > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> > > backend.
> > >
> > >
> > > Functional tests for python-brick-cinderclient-ext
> > >
> > > There are patches that introduces functional tests [6] and new job [7].
> > >
> > >
> > > Functional tests for python-cinderclient
> > >
> > > We've got a very limited set of such tests and non-voting job. IMO, we
> > can
> > > run them even with Cinder Fake Driver to make them not depended on a
> > > storage backend and make it faster. I believe, we can make this job
> > voting
> > > soon. Also, we need more contributors to this kind of tests.
> > >
> > >
> > > Integrated tests for python-cinderclient
> > >
> > > We need such tests to make sure that we won't break Nova, Heat or other
> > > python-cinderclient consumers with a next merged patch. There is a thread
> > > in openstack-dev ML about such tests [8] and proposal [9] to introduce
> > them
> > > to python-cinderclient.
> > >
> > >
> > > Rally tests
> > >
> > > IMO, it would be good to have new Rally scenarios for every patches like
> > > 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> > > Cinder community don't have enough time to implement them, we have to ask
> > > for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> > > needed.
> > >
> >
> > Are there any recent examples of a fix like this recently where it would
> > seem like a reasonable task to write a Rally scenario along with the patch?
> >
> > Not being very familiar with Rally (as I think most of us aren't), I'm
> > having a hard time picturing this.
> >
> > >
> > > [1] https://review.openstack.org/#/c/282861/
> > > [2] http://paste.openstack.org/show/488925/
> > > [3] https://review.openstack.org/#/c/267801/
> > > [4] https://review.openstack.org/#/c/287115/
> > > [5] https://review.openstack.org/#/c/274471/
> > > [6] https://review.openstack.org/#/c/265811/
> > > [7] https://review.openstack.org/#/c/265925/
> > > [8]
> > >
> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> > > [9] https://review.openstack.org/#/c/279432/
> > >
> > >
> > > Regards,
> > > Ivan Kolodyazhny,
> > > http://blog.e0ne.info/
> > >
> > >
> > >
> > >
> > __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> >
> > 

Re: [openstack-dev] [Cinder][DRBD] questions about pep8/flake8 etc.

[Gorka Eguileor]

On 21/12, Walter A. Boring IV wrote:
> On 12/21/2015 06:40 AM, Philipp Marek wrote:
> >Hi everybody,
> >
> >in the current patch https://review.openstack.org/#/c/259973/1 the test
> >script needs to use a lot of the constant definitions of the backend driver
> >it's using (DRBDmanage).
> >
> >As the DRBDmanage libraries need not be installed on the CI nodes, I'm
> >providing a minimum of upstream files, accumulated in a separate directory
> >- they get imported and "fixed" to the expected location, so that the
> >driver that should be tested runs as if DRBDmanage is installed.
> >
> >
> >My problem is now that the upstream project doesn't accept all the pep8
> >conventions like Openstack does; so the CI run
> > 
> > http://logs.openstack.org/73/259973/1/check/gate-cinder-pep8/5032b16/console.html
> >gives a lot of messages like "E221 multiple spaces before operator" and
> >similar. (It even crashes during AST parsing ;)
> >
> >
> >So, I can see these options now:
> >
> >   * Make pep8 ignore these files - they're only used by one test script,
> > and are never used in production anyway.
> > + Simple
> > + New upstream files can simply be dropped in as needed
> > - bad example?
> >   * Reformat the files to conform to pep8
> > - some work for every new version that needs to be incorporated
> > - can't be compared for equality with upstream any more
> > - might result in mismatches later on, ie. production code uses
> >   different values from test code
> I would suggest this option.  We don't want to allow code in cinder that
> bypasses the pep8 checks.  Since you are trying to get drbd support into
> Cinder, it then falls upon you to make sure the code you are submitting
> follows the same standards as the rest of the project.
> 
> Walt
> 

I would normally suggest option #4, but considering listed downsides
this looks like the best alternative, as Walter suggests.

To reduce the work for every release you could probably automate it
using autopep8.

Cheers,
Gorka.

> >
> >   * Throw upstream files away, and do "manual" fakes
> > - A lot of work
> > - Work needed for every new needed constant
> > - lots of duplicated code
> > - might result in mismatches later on, ie. production code uses
> >   different values from test code
> > + whole checkout still "clean" for pep8
> >
> >   * Require DRBDmanage to be installed
> > + uses same values as upstream and production
> > - Need to get it upstream into PyPi
> > - Meaning delay
> > - delay for every new release of DRBDmanage
> > - Might not even be compatible with every used distribution/CI
> >   out there
> >
> >
> >I would prefer the first option - make pep8 ignore these files.
> >But I'm only a small player here, what's the opinion of the Cinder cores?
> >Would that be acceptable?
> >
> >
> >Regards,
> >
> >Phil
> >
> >__
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >.
> >
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Custom fields for versioned objects

[Gorka Eguileor]

On 15/12, Ryan Rossiter wrote:
> Thanks for the review Michal! As for the bp/bug report, there’s four options:
> 
> 1. Tack the work on as part of bp cinder-objects
> 2. Make a new blueprint (bp cinder—object-fields)

I think #2 would be the best option, as this is not part of the effort
to move to VOs but to improve them and the code in general.

Thanks for working on this, as I've been meaning to tackle the issue of
having literal strings all over the code for the status for quite a
while.

> 3. Open a bug to handle all changes for enums/fields
> 4. Open a bug for each changed enum/field
> 
> Personally, I’m partial to #1, but #2 is better if you want to track this 
> work separately from the other objects work. I don’t think we should go with 
> bug reports because #3 will be a lot of Partial-Bug and #4 will be kinda 
> spammy. I don’t know what the spec process is in Cinder compared to Nova, but 
> this is nowhere near enough work to be spec-worthy.
> 
> If this is something you or others think should be discussed in a meeting, I 
> can tack it on to the agenda for tomorrow.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] Is there anyone truly working on this issue https://bugs.launchpad.net/cinder/+bug/1520102?

[Gorka Eguileor]

On 11/12, mtanino wrote:
> Hi Thang, Vincent,
> 
> I guess the root cause is that finish_volume_migration() still
> handles a volume as a dictionary instead of volume object and
> the method returns dict volume.
> 
> And then, 'rpcapi.delete_volume()' in migrate_volume_completion()
> tries to delete dict volume but it fails due to the following error.
> 

I believe that is not entirely correct, the issue is that
'finish_volume_migration' returns an ORM volume that then is passed by
'rpcapi.delete_volume' in the place of a Versioned Object Volume (this
is the recently added optional argument), so this is serialized and
deserialized as a normal dictionary (instead of as a VO dictionary), and
when the manager at the other end sees that it has received something in
the place of the VO Volume argument it tries to access the 'id'
attribute.

But since the ORM volume was not a VO it was passed as a normal
dictionary and therefore has no 'id' attribute.

For reference, Vincent has proposed a patch [1].

Cheers,
Gorka.

[1]: https://review.openstack.org/250216/

> >As far as you know, is there someone working on this issue? If not, I am 
> >gonna fix it.
> 
> Not yet. You can go ahead.
> 
> - Result of 'cinder migrate --force-host-copy True '
> 
> 2015-12-11 20:36:33.395 ERROR oslo_messaging.rpc.dispatcher 
> [req-2c271a5e-7e6a-4b38-97d1-22ef245c7892 f95ea885e1a34a81975c50be63444a0b 
> 56d8eb5cc90242178cf05aedab3c1612] Exception during message handling: 'dict' 
> object has no attribute 'id'
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher Traceback (most 
> recent call last):
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 
> 142, in _dispatch_and_reply
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher 
> executor_callback))
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 
> 186, in _dispatch
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher 
> executor_callback)
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 
> 129, in _do_dispatch
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher result = 
> func(ctxt, **new_args)
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in 
> wrapper
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher return 
> f(*args, **kwargs)
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/opt/stack/cinder/cinder/volume/manager.py", line 152, in lvo_inner1
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher return 
> lvo_inner2(inst, context, volume_id, **kwargs)
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, 
> in inner
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher return 
> f(*args, **kwargs)
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/opt/stack/cinder/cinder/volume/manager.py", line 151, in lvo_inner2
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher return 
> f(*_args, **_kwargs)
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher   File 
> "/opt/stack/cinder/cinder/volume/manager.py", line 603, in delete_volume
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher volume_id = 
> volume.id
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher AttributeError: 
> 'dict' object has no attribute 'id'
> 2015-12-11 20:36:33.395 TRACE oslo_messaging.rpc.dispatcher
> 
> Thanks,
> Mitsuhiro Tanino
> 
> On 12/10/2015 11:24 PM, Thang Pham wrote:
> >I have to try it again myself.  What errors are you seeing?  Is it the same? 
> > Feel free to post a patch if you already have one that would solve it.
> >
> >Regards,
> >Thang
> >
> >On Thu, Dec 10, 2015 at 10:51 PM, Sheng Bo Hou  >> wrote:
> >
> >Hi Mitsuhiro, Thang
> >
> >The patch https://review.openstack.org/#/c/228916is merged, but sadly it 
> > does not cover the issue https://bugs.launchpad.net/cinder/+bug/1520102. 
> > This bug is still valid.
> >As far as you know, is there someone working on this issue? If not, I am 
> > gonna fix it.
> >
> >Best wishes,
> >Vincent Hou (侯胜博)
> >
> >Staff Software Engineer, Open Standards and Open Source Team, Emerging 
> > Technology Institute, IBM China Software Development Lab
> >
> >Tel: 86-10-82450778 Fax: 86-10-82453660
> >Notes ID: Sheng Bo Hou/China/IBM@IBMCNE-mail: sb...@cn.ibm.com 
> > 
> >Address:3F Ring, Building 28 Zhongguancun Software Park, 8 Dongbeiwang 
> > West Road, Haidian District, Beijing, 

Re: [openstack-dev] Is there any way I can completely erase all the data when deleting a cinder volume

[Gorka Eguileor]

On 18/11, Young Yang wrote:
> There are some sensitive data in my volume.
> I hope openstack can completely erase all the data (e.g. overwrite the
> whole volume will 0 bits) when deleting a cinder volume.
> 
> I plan to write some code to make Openstack to mount that volume and
> rewrite the whole volume with 0 bits.
> 
> But I'm wondering if there is any better way to accomplish that.
> 
> Thanks in advance! :)

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Hi,

Cinder already does that by default.

Clearing of deleted volumes is controlled by "volume_clear"
configuration option which has a default of "zero".

Available values are "none", "zero" and "shred".

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] Specifications to support High Availability Active-Active configurations in Cinder

[Gorka Eguileor]

Hi,

We finally have ready for review all specifications required to support
High Availability Active/Active configurations in Cinder's Volume nodes.

There is a Blueprint to track this effort [1] and the specs are as follow:

- General description of the issues and solutions [2]
- Removal of Races on API nodes [3]
- Job distribution to clusters [4]
- Cleanup process of crashed nodes [5]
- Data corruption prevention [6]
- Removing local file locks from the manager [7]
- Removing local file locks from drivers [8]

The only effort that is in progress is the removal of code races on the
API nodes that has a series of patches but still needs reviewing and
more races need to be removed from the code.

I think it would be very beneficial for everyone if these patches were
reviewed before the Summit Sessions related to this topic, as it is a
complicated matter with many moving pieces and a good knowledge of the
topic will be essential to have productive sessions in Tokyo.

While I believe this is a viable path to implement HA A/A in Cinder,
these specifications can also be useful to anyone looking for other
solutions as it gives a detailed description of the problem as well as a
good number of cases that any proper solution needs to be able to
handle.

Alternatives to any of the issues are more than welcome, preferably
simpler ones, but they need to be well thought and must:

a) Take into account all cases of the issue that are trying to fix.

b) Play well with the other individual solutions as to give a reliable
global solution.

Cheers,
Gorka.


[1]: 
https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support
[2]: https://review.openstack.org/232599
[3]: https://review.openstack.org/207101
[4]: https://review.openstack.org/232595
[5]: https://review.openstack.org/236977
[6]: https://review.openstack.org/237076
[7]: https://review.openstack.org/237602
[8]: https://review.openstack.org/237604

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [cinder] [all] The future of Cinder API v1

[Gorka Eguileor]

On 28/09, John Griffith wrote:
> On Mon, Sep 28, 2015 at 6:19 PM, Mark Voelker  wrote:
> 
> > FWIW, the most popular client libraries in the last user survey[1] other
> > than OpenStack’s own clients were: libcloud (48 respondents), jClouds (36
> > respondents), Fog (34 respondents), php-opencloud (21 respondents),
> > DeltaCloud (which has been retired by Apache and hasn’t seen a commit in
> > two years, but 17 respondents are still using it), pkgcloud (15
> > respondents), and OpenStack.NET (14 respondents).  Of those:
> >
> > * libcloud appears to support the nova-volume API but not the cinder API:
> > https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/openstack.py#L251
> >
> > * jClouds appears to support only the v1 API:
> > https://github.com/jclouds/jclouds/tree/jclouds-1.9.1/apis/openstack-cinder/src/main/java/org/jclouds
> >
> > * Fog also appears to only support the v1 API:
> > https://github.com/fog/fog/blob/master/lib/fog/openstack/volume.rb#L99
> >
> > * php-opencloud appears to only support the v1 API:
> > https://php-opencloud.readthedocs.org/en/latest/services/volume/index.html
> >
> > * DeltaCloud I honestly haven’t looked at since it’s thoroughly dead, but
> > I can’t imagine it supports v2.
> >
> > * pkgcloud has beta-level support for Cinder but I think it’s v1 (may be
> > mistaken): https://github.com/pkgcloud/pkgcloud/#block-storagebeta
> > and
> > https://github.com/pkgcloud/pkgcloud/tree/master/lib/pkgcloud/openstack/blockstorage
> >
> > * OpenStack.NET does appear to support v2:
> > http://www.openstacknetsdk.org/docs/html/T_net_openstack_Core_Providers_IBlockStorageProvider.htm
> >
> > Now, it’s anyone’s guess as to whether or not users of those client
> > libraries actually try to use them for volume operations or not
> > (anecdotally I know a few clouds I help support are using client libraries
> > that only support v1), and some users might well be using more than one
> > library or mixing in code they wrote themselves.  But most of the above
> > that support cinder do seem to rely on v1.  Some management tools also
> > appear to still rely on the v1 API (such as RightScale:
> > http://docs.rightscale.com/clouds/openstack/openstack_config_prereqs.html
> > ).  From that perspective it might be useful to keep it around a while
> > longer and disable it by default.  Personally I’d probably lean that way,
> > especially given that folks here on the ops list are still reporting
> > problems too.
> >
> > That said, v1 has been deprecated since Juno, and the Juno release notes
> > said it was going to be removed [2], so there’s a case to be made that
> > there’s been plenty of fair warning too I suppose.
> >
> > [1]
> > http://superuser.openstack.org/articles/openstack-application-developers-share-insights
> > [2] https://wiki.openstack.org/wiki/ReleaseNotes/Juno#Upgrade_Notes_7
> >
> > At Your Service,
> >
> > Mark T. Voelker
> >
> >
> >
> > > On Sep 28, 2015, at 7:17 PM, Sam Morrison  wrote:
> > >
> > > Yeah we’re still using v1 as the clients that are packaged with most
> > distros don’t support v2 easily.
> > >
> > > Eg. with Ubuntu Trusty they have version 1.1.1, I just updated our
> > “volume” endpoint to point to v2 (we have a volumev2 endpoint too) and the
> > client breaks.
> > >
> > > $ cinder list
> > > ERROR: OpenStack Block Storage API version is set to 1 but you are
> > accessing a 2 endpoint. Change its value through --os-volume-api-version or
> > env[OS_VOLUME_API_VERSION].
> > >
> > > Sam
> > >
> > >
> > >> On 29 Sep 2015, at 8:34 am, Matt Fischer  wrote:
> > >>
> > >> Yes, people are probably still using it. Last time I tried to use V2 it
> > didn't work because the clients were broken, and then it went back on the
> > bottom of my to do list. Is this mess fixed?
> > >>
> > >>
> > http://lists.openstack.org/pipermail/openstack-operators/2015-February/006366.html
> > >>
> > >> On Mon, Sep 28, 2015 at 4:25 PM, Ivan Kolodyazhny 
> > wrote:
> > >> Hi all,
> > >>
> > >> As you may know, we've got 2 APIs in Cinder: v1 and v2. Cinder v2 API
> > was introduced in Grizzly and v1 API is deprecated since Juno.
> > >>
> > >> After [1] is merged, Cinder API v1 is disabled in gates by default.
> > We've got a filed bug [2] to remove Cinder v1 API at all.
> > >>
> > >>
> > >> According to Deprecation Policy [3] looks like we are OK to remote it.
> > But I would like to ask Cinder API users if any still use API v1.
> > >> Should we remove it at all Mitaka release or just disable by default in
> > the cinder.conf?
> > >>
> > >> AFAIR, only Rally doesn't support API v2 now and I'm going to implement
> > it asap.
> > >>
> > >> [1] https://review.openstack.org/194726
> > >> [2] https://bugs.launchpad.net/cinder/+bug/1467589
> > >> [3]
> > http://lists.openstack.org/pipermail/openstack-dev/2015-September/073576.html
> > >>
> > >> Regards,
> > >> Ivan Kolodyazhny
> > >>
> > >> 

Re: [openstack-dev] [all] -1 due to line length violation in commit messages

[Gorka Eguileor]

On 28/09, Clint Byrum wrote:
> Excerpts from Kevin Benton's message of 2015-09-28 14:29:14 -0700:
> > I think a blanket statement about what people's motivations are is not
> > fair. We've seen in this thread that some people want to enforce the limit
> > of 72 chars and it's not about padding their stats.
> > 
> > The issue here is that we have a guideline with a very specific number. If
> > we don't care to enforce it, why do we even bother? "Please do this, unless
> > you don't feel like it", is going to be hard for many people to review in a
> > way that pleases everyone.
> > 
> 
> Please do read said guidelines. "Must" would be used if it were to be
> "enforced". It "should" be formatted that way.

Since we are not all native speakers expecting everyone to realize that
difference - which is completely right - may be a little optimistic,
moreover considering that parts of those guidelines may even be written
by non natives.

Let's say I interpret all "should" instances in that guideline as rules
that don't need to be strictly enforced, I see that the Change-Id
"should not be changed when rebasing" - this one would certainly be fun
to watch if we didn't follow it - the blueprint "should give the name of
a Launchpad blueprint" - I don't know any core that would not -1 a patch
if he notices the BP reference missing - and machine targeted metadata
"should all be grouped together at the end of the commit message" - this
one everyone follows instinctively, so no problem.

And if we look at the i18n guidelines, almost everything is using
should, but on reviews these are treated as strict *must* because of the
implications.

Anyway, it's a matter of opinion and afaik in Cinder we don't even have
a real problem with downvoting for the commit message length, I don't
see more than 1 every couple of months or so.

Cheers,
Gorka.

> 
> > On Mon, Sep 28, 2015 at 11:00 PM, Assaf Muller <amul...@redhat.com> wrote:
> > 
> > >
> > >
> > > On Mon, Sep 28, 2015 at 12:40 PM, Zane Bitter <zbit...@redhat.com> wrote:
> > >
> > >> On 28/09/15 05:47, Gorka Eguileor wrote:
> > >>
> > >>> On 26/09, Morgan Fainberg wrote:
> > >>>
> > >>>> As a core (and former PTL) I just ignored commit message -1s unless
> > >>>> there is something majorly wrong (no bug id where one is needed, etc).
> > >>>>
> > >>>> I appreciate well formatted commits, but can we let this one go? This
> > >>>> discussion is so far into the meta-bike-shedding (bike shedding about 
> > >>>> bike
> > >>>> shedding commit messages) ... If a commit message is *that* bad a -1 
> > >>>> (or
> > >>>> just fixing it?) Might be worth it. However, if a commit isn't missing 
> > >>>> key
> > >>>> info (bug id? Bp? Etc) and isn't one long incredibly unbroken sentence
> > >>>> moving from topic to topic, there isn't a good reason to block the 
> > >>>> review.
> > >>>>
> > >>>
> > >> +1
> > >>
> > >> It is not worth having a bot -1 bad commits or even having gerrit muck
> > >>>> with them. Let's do the job of the reviewer and actually review code
> > >>>> instead of going crazy with commit messages.
> > >>>>
> > >>>
> > >> +1
> > >>
> > >> Sent via mobile
> > >>>>
> > >>>>
> > >>> I have to disagree, as reviewers we have to make sure that guidelines
> > >>> are followed, if we have an explicit guideline that states that
> > >>> the limit length is 72 chars, I will -1 any patch that doesn't follow
> > >>> the guideline, just as I would do with i18n guideline violations.
> > >>>
> > >>
> > >> Apparently you're unaware of the definition of the word 'guideline'. It's
> > >> a guide. If it were a hard-and-fast rule then we would have a bot 
> > >> enforcing
> > >> it already.
> > >>
> > >> Is there anything quite so frightening as a large group of people blindly
> > >> enforcing rules with total indifference to any sense of overarching 
> > >> purpose?
> > >>
> > >> A reminder that the reason for this guideline is to ensure that none of
> > >> the broad variety of tools that are available in the Git ecosystem
> > >> effectively become unusable with the 

Re: [openstack-dev] [Cinder] [Manila] Will NFS stay with Cinder as a reference implementation?

[Gorka Eguileor]

On 29/09, Duncan Thomas wrote:
> Cinder provides a block storage abstraction to a vm. Manila provides a
> filesystem abstraction. The two are very different, and complementary. I
> see no reason why the nfs related cinder drivers should be removed based on
> the existence or maturity of manila - manila is not going to suddenly start
> providing block storage to a vm.
> On 29 Sep 2015 06:56, "Sheng Bo Hou"  wrote:
> 
> > Hi folks,
> >
> > I have a question about the file services in OpenStack.
> >
> > As you know there is a generic NFS driver in Cinder and other file system
> > drivers inherit it, while the project Manila is determined to provide the
> > file system service.
> >
> > Will NFS stay with Cinder as the reference implementation for the coming
> > release or releases? Are all the file system drivers going to move to
> > Manila?
> > What is relation between Manila as FSaaS and NFS in Cinder?
> > Any ideas?
> >
> > Thank you.
> >
> > Best wishes,
> > Vincent Hou (侯胜博)
> >
> > Staff Software Engineer, Open Standards and Open Source Team, Emerging
> > Technology Institute, IBM China Software Development Lab
> >
> > Tel: 86-10-82450778 Fax: 86-10-82453660
> > Notes ID: Sheng Bo Hou/China/IBM@IBMCNE-mail: sb...@cn.ibm.com
> > Address:3F Ring, Building 28 Zhongguancun Software Park, 8 Dongbeiwang
> > West Road, Haidian District, Beijing, P.R.C.100193
> > 地址:北京市海淀区东北旺西路8号中关村软件园28号楼环宇大厦3层 邮编：100193
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >

I agree with Duncan,

There is a clear distinction between the projects' objectives:
- Cinder provides the canonical storage provisioning control plane in
  OpenStack for block storage as well as delivering a persistence model
  for instance storage.
- Manila is a File Share Service, in a similar manner, provides
  coordinated access to shared or distributed file systems.

So I wouldn't move out our NFS drivers.

As the relation between those is that while they both use the same
storage type they expose it completely different.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] -1 due to line length violation in commit messages

[Gorka Eguileor]

On 28/09, Andreas Jaeger wrote:
> On 2015-09-28 11:47, Gorka Eguileor wrote:
> >On 26/09, Morgan Fainberg wrote:
> >>As a core (and former PTL) I just ignored commit message -1s unless there 
> >>is something majorly wrong (no bug id where one is needed, etc).
> >>
> >>I appreciate well formatted commits, but can we let this one go? This 
> >>discussion is so far into the meta-bike-shedding (bike shedding about bike 
> >>shedding commit messages) ... If a commit message is *that* bad a -1 (or 
> >>just fixing it?) Might be worth it. However, if a commit isn't missing key 
> >>info (bug id? Bp? Etc) and isn't one long incredibly unbroken sentence 
> >>moving from topic to topic, there isn't a good reason to block the review.
> >>
> >>It is not worth having a bot -1 bad commits or even having gerrit muck with 
> >>them. Let's do the job of the reviewer and actually review code instead of 
> >>going crazy with commit messages.
> >>
> >>Sent via mobile
> >>
> >
> >I have to disagree, as reviewers we have to make sure that guidelines
> >are followed, if we have an explicit guideline that states that
> >the limit length is 72 chars, I will -1 any patch that doesn't follow
> >the guideline, just as I would do with i18n guideline violations.
> > [...]
> 
> You could also tell the committer about the length so that s/he learns for
> the next time. Giving a -1 just for a few lines that are 80 chars long is
> over the top IMHO,
> 
> Andreas
> -- 

I tell the committer of this guideline, just like it was told to me on
my first commits; and I agree that it sucks to give or receive a -1 for
this, but let me put it this way, how many times will you be
getting/giving a -1 to the same person for this?

If it's a first time committer you'll probably say it once, they'll
learn it, fix it and them we have all our commits conforming to our
guidelines, not such a big deal (although I agree with Miguel Angel that
this should be automated) and if it's not a first time committer he
should have known better and he deserves the -1 for not paying attention
and/or not having his dev env properly setup.


>  Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
>   SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>HRB 21284 (AG Nürnberg)
> GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] How to make a mock effactive for all method of a testclass

[Gorka Eguileor]

On 23/09, Eric Harney wrote:
> On 09/23/2015 04:06 AM, liuxinguo wrote:
> > Hi,
> > 
> > In a.py we have a function:
> > def _change_file_mode(filepath):
> > utils.execute('chmod', '600', filepath, run_as_root=True)
> > 
> > In test_xxx.py, there is a testclass:
> > class DriverTestCase(test.TestCase):
> > def test_a(self)
> > ...
> > Call a. _change_file_mode
> > ...
> > 
> > def test_b(self)
> > ...
> > Call a. _change_file_mode
> > ...
> > 
> > I have tried to mock like mock out function _change_file_mode like this:
> > @mock.patch.object(a, '_change_file_mode', return_value=None)
> > class DriverTestCase(test.TestCase):
> > def test_a(self)
> > ...
> > Call a. _change_file_mode
> > ...
> > 
> > def test_b(self)
> > ...
> > Call a. _change_file_mode
> > ...
> > 
> > But the mock takes no effort, the real function _change_file_mode is still 
> > executed.
> > So how to make a mock effactive for all method of a testclass?
> > Thanks for any input!
> > 
> > Wilson Liu
> 
> The simplest way I found to do this was to use mock.patch in the test
> class's setUp() method, and tear it down again in tearDown().
> 
> There may be cleaner ways to do this with tools in oslotest etc. (I'm
> not sure), but this is fairly straightforward.
> 
> See here -- self._clear_patch stores the mock:
> http://git.openstack.org/cgit/openstack/cinder/tree/cinder/tests/unit/test_volume.py?id=8de60a8b#n257
> 

When doing the mock in the setUp it is recommended to add the stop to
the cleanup instead of doing it in the tearDown, in that code it would
be: self.addCleanUp(self._clear_patch.stop)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] How to make a mock effactive for all method of a testclass

[Gorka Eguileor]

On 23/09, liuxinguo wrote:
> Hi,
> 
> In a.py we have a function:
> def _change_file_mode(filepath):
> utils.execute('chmod', '600', filepath, run_as_root=True)
> 
> In test_xxx.py, there is a testclass:
> class DriverTestCase(test.TestCase):
> def test_a(self)
> ...
> Call a. _change_file_mode
> ...
> 
> def test_b(self)
> ...
> Call a. _change_file_mode
> ...
> 
> I have tried to mock like mock out function _change_file_mode like this:
> @mock.patch.object(a, '_change_file_mode', return_value=None)
> class DriverTestCase(test.TestCase):
> def test_a(self)
> ...
> Call a. _change_file_mode
> ...
> 
> def test_b(self)
> ...
> Call a. _change_file_mode
> ...
> 
> But the mock takes no effort, the real function _change_file_mode is still 
> executed.
> So how to make a mock effactive for all method of a testclass?
> Thanks for any input!
> 
> Wilson Liu

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hi,

There is something else going on with your code, as the code you are
showing us here should not even execute at all.  You should be getting a
couple of errors:

 TypeError: test_a() takes exactly 1 argument (2 given)
 TypeError: test_b() takes exactly 1 argument (2 given)

Because mocking the whole class will still pass the mock object as an
argument to the methods.

Anyway, if you accept the mocked object as an argument in your test
methods it would work.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] L3 low pri review queue starvation

[Gorka Eguileor]

On Tue, Sep 01, 2015 at 09:30:26AM -0600, John Griffith wrote:
> On Tue, Sep 1, 2015 at 5:57 AM, Tom Barron  wrote:
> 
> > [Yesterday while discussing the following issue on IRC, jgriffith
> > suggested that I post to the dev list in preparation for a discussion in
> > Wednesday's cinder meeting.]
> >
> > Please take a look at the 10 "Low" priority reviews in the cinder
> > Liberty 3 etherpad that were punted to Mitaka yesterday. [1]
> >
> > Six of these *never* [2] received a vote from a core reviewer. With the
> > exception of the first in the list, which has 35 patch sets, none of the
> > others received a vote before Friday, August 28.  Of these, none had
> > more than -1s on minor issues, and these have been remedied.
> >
> > Review https://review.openstack.org/#/c/213855 "Implement
> > manage/unmanage snapshot in Pure drivers" is a great example:
> >
> >* approved blueprint for a valuable feature
> >* pristine code
> >* passes CI and Jenkins (and by the deadline)
> >* never reviewed
> >
> > We have 11 core reviewers, all of whom were very busy doing reviews
> > during L3, but evidently this set of reviews didn't really have much
> > chance of making it.  This looks like a classic case where the
> > individually rational priority decisions of each core reviewer
> > collectively resulted in starving the Low Priority review queue.
> >

I can't speak for other cores, but in my case reviewing was mostly not
based on my own priorities, I reviewed patches based on the already set
priority of each patch as well as patches that I was already
reviewing.

Some of those medium priority patches took me a lot of time to review,
since they were not trivial (some needed some serious rework).  As for
patches I was already reviewing, as you can imagine it wouldn't be fair
to just ignore a patch that I've been reviewing for some time just when
it's almost ready and the deadline is closing in.

Having said that I have to agree that those patches didn't have much
chances, and I apologize for my part on that.  While it is no excuse I
have to agree with jgriffith when he says that those patches should have
pushed cores for reviews (even if this is clearly not the "right" way to
manage it).

> > One way to remedy would be for the 11 core reviewers to devote a day or
> > two to cleaning up this backlog of 10 outstanding reviews rather than
> > punting all of them out to Mitaka.
> >
> > Thanks for your time and consideration.
> >
> > Respectfully,
> >
> > -- Tom Barron
> >
> > [1] https://etherpad.openstack.org/p/cinder-liberty-3-reviews
> > [2] At the risk of stating the obvious, in this count I ignore purely
> > procedural votes such as the final -2.
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> ​Thanks Tom, this is sadly an ongoing problem every release.  I think we
> have a number of things we can talk about at the summit to try and
> make some of this better.  I honestly think that if people were to
> actually "use" launchpad instead of creating tracking etherpads
> everywhere it would help.  What I mean is that there is a ranked
> targeting of items in Launchpad and we should use it, core team
> members should know that as the source of truth and things that must
> get reviewed.
> 

I agree, we should use Launchpad's functionality to track BPs and Bugs
targeted for each milestone, and maybe we can discuss on a workflow that
helps us reduce starvation at the same time that helps us keep track of
code reviewers responsible for each item.

Just spitballing here, but we could add to BP's work items and Bug's
comments what core members will be responsible for reviewing related
patches.  Although this means cores will have to check this on every
review they do that has a BP or Bug number, so if there are already 2
cores responsible for that feature they should preferably just move on
to other patches and if there are not 2 core reviewers they should add
themselves to LP.  This way patch owners know who they should bug for
reviews on their patches and if there are no core reviewers for them
they should look for some (or wait for them to "claim" that
bug/feature).

> As far as Liberty and your patches; Yesterday was the freeze point, the
> entire Cinder team agreed on that (yourself included both at the mid-cycle
> meet up and at the team meeting two weeks ago when Thingee reiterated the
> deadlines).  If you noticed last week that your patches weren't going
> anywhere YOU should've wrangled up some reviews.
> 
> Furthermore, I've explained every release for the last 3 or 4 years that
> there's no silver bullet, no magic process when it come to review
> throughput.  ESPECIALLY when it comes to the 3'rd milestone.  You can try
> landing 

Re: [openstack-dev] [cinder] Midcycle Sprint Summary

[Gorka Eguileor]

On Mon, Aug 17, 2015 at 08:53:29AM -0700, Mike Perez wrote:
 A *summary* of the Cinder midcycle sprint, in attempt to keep your attention.
 Full meeting notes available [1].
 
 [ ... ]
 
 -- Mike Perez

Thank you very much for the summary.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Tue, Aug 04, 2015 at 08:30:17AM -0700, Joshua Harlow wrote:
 Duncan Thomas wrote:
 On 3 August 2015 at 20:53, Clint Byrum cl...@fewbar.com
 mailto:cl...@fewbar.com wrote:
 
 Excerpts from Devananda van der Veen's message of 2015-08-03
 08:53:21 -0700:
 Also on a side note, I think Cinder's need for this is really subtle,
 and one could just accept that sometimes it's going to break when it
 does
 two things to one resource from two hosts. The error rate there might
 even be lower than the false-error rate that would be caused by a
 twitchy
 DLM with timeouts a little low. So there's a core cinder discussion that
 keeps losing to the shiny DLM discussion, and I'd like to see it played
 out fully: Could Cinder just not do anything, and let the few drivers
 that react _really_ badly, implement their own concurrency controls?
 
 
 
 So the problem here is data corruption. Lots of our races can cause data
 corruption. Not 'my instance didn't come up', not 'my network is screwed
 and I need to tear everything down and do it again', but 'My 1tb of
 customer database is now missing the second half'. This means that we
 *really* need some confidence and understanding in whatever we do. The
 idea of locks timing out and being stolen without fencing is frankly
 scary and begging for data corruption unless we're very careful. I'd
 rather use a persistent lock (e.g. a db record change) and manual
 recovery than a lock timeout that might cause corruption.
 
 So perhaps start off using persistent locks, gain confidence that we have
 all the right fixes in to prevent that data corruption, and then slowly
 remove persistent locks as needed. Sounds like an iterative solution to me,
 and one that will build confidence (hopefully that confidence building can
 be automated via a chaos-monkey like test-suite) as we go :)
 

That was my suggestion as well, it is not that we cannot do without
locks, it's that we have confidence in them and the current code that
uses them, so we can start with an initial solution with distributed
locks, confirm that the rest of the code is running properly (as
distributed locks are not the only change needed) and then, on a second
iteration, proceed to remove locks in the Volume Manager and lastly on
the next iteration remove them in the drivers wherever it is possible,
and for those places where it isn't possible maybe look for alternative
solutions.

This way we can get a solution faster and avoid potential delays that
may raise if we try to do everything at once.

But I can see the point of those who say that why put the ops through
the DLM configuration process if we are probably going to remove the DLM
in a couple of releases. But since we don't know how difficult it will
get to remove all other locks, I think that a bird in the hand is worth
two in the bush and we should still go with the distributed locks and at
least make sure we have a solution.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Tue, Aug 04, 2015 at 08:40:13AM -0700, Joshua Harlow wrote:
 Clint Byrum wrote:
 Excerpts from Devananda van der Veen's message of 2015-08-03 08:53:21 -0700:
 On Mon, Aug 3, 2015 at 8:41 AM Joshua Harlowharlo...@outlook.com  wrote:
 
 Clint Byrum wrote:
 Excerpts from Gorka Eguileor's message of 2015-08-02 15:49:46 -0700:
 On Fri, Jul 31, 2015 at 01:47:22AM -0700, Mike Perez wrote:
 On Mon, Jul 27, 2015 at 12:35 PM, Gorka Eguileorgegui...@redhat.com
 wrote:
 I know we've all been looking at the HA Active-Active problem in
 Cinder
 and trying our best to figure out possible solutions to the different
 issues, and since current plan is going to take a while (because it
 requires that we finish first fixing Cinder-Nova interactions), I've
 been
 looking at alternatives that allow Active-Active configurations
 without
 needing to wait for those changes to take effect.
 
 And I think I have found a possible solution, but since the HA A-A
 problem has a lot of moving parts I ended up upgrading my initial
 Etherpad notes to a post [1].
 
 Even if we decide that this is not the way to go, which we'll probably
 do, I still think that the post brings a little clarity on all the
 moving parts of the problem, even some that are not reflected on our
 Etherpad [2], and it can help us not miss anything when deciding on a
 different solution.
 Based on IRC conversations in the Cinder room and hearing people's
 opinions in the spec reviews, I'm not convinced the complexity that a
 distributed lock manager adds to Cinder for both developers and the
 operators who ultimately are going to have to learn to maintain things
 like Zoo Keeper as a result is worth it.
 
 **Key point**: We're not scaling Cinder itself, it's about scaling to
 avoid build up of operations from the storage backend solutions
 themselves.
 
 Whatever people think ZooKeeper scaling level is going to accomplish
 is not even a question. We don't need it, because Cinder isn't as
 complex as people are making it.
 
 I'd like to think the Cinder team is a great in recognizing potential
 cross project initiatives. Look at what Thang Pham has done with
 Nova's version object solution. He made a generic solution into an
 Oslo solution for all, and Cinder is using it. That was awesome, and
 people really appreciated that there was a focus for other projects to
 get better, not just Cinder.
 
 Have people consider Ironic's hash ring solution? The project Akanda
 is now adopting it [1], and I think it might have potential. I'd
 appreciate it if interested parties could have this evaluated before
 the Cinder midcycle sprint next week, to be ready for discussion.
 
 [1] - https://review.openstack.org/#/c/195366/
 
 -- Mike Perez
 Hi all,
 
 Since my original proposal was more complex that it needed be I have a
 new proposal of a simpler solution, and I describe how we can do it with
 or without a DLM since we don't seem to reach an agreement on that.
 
 The solution description was more rushed than previous one so I may have
 missed some things.
 
 http://gorka.eguileor.com/simpler-road-to-cinder-active-active/
 
 I like the idea of keeping it simpler Gorka. :)
 
 Note that this is punting back to use the database for coordination,
 which is what most projects have done thus far, and has a number of
 advantages and disadvantages.
 
 Note that the stale-lock problem was solved in an interesting way in
 Heat:
 each engine process gets an instance-of-engine uuid that adds to the
 topic queues it listens to. If it holds a lock, it records this UUID in
 the owner field. When somebody wants to steal the lock (due to timeout)
 they send to this queue, and if there's no response, the lock is stolen.
 
 Anyway, I think what might make more sense than copying that directly,
 is implementing Use the database and oslo.messaging to build a DLM
 as a tooz backend. This way as the negative aspects of that approach
 impact an operator, they can pick a tooz driver that satisfies their
 needs, or even write one to their specific backend needs.
 Oh jeez, using 'the database and oslo.messaging to build a DLM' scares
 me :-/
 
 There are already N + 1 DLM like-systems out there (and more every day
 if u consider the list at
 https://raftconsensus.github.io/#implementations) so I'd really rather
 use one that is proven to work by academia vs make a frankenstein one.
 
 
 Joshua,
 
 As has been said on this thread, some projects (eg, Ironic) are already
 using a consistent hash ring backed by a database to meet the requirements
 they have. Could those requirements also be met with some other tech? Yes.
 Would that provide additional functionality or some other benefits? Maybe.
 But that's not what this thread was about.
 
 Distributed hash rings are a well understood technique, as are databases.
 There's no need to be insulting by calling
 not-your-favorite-technology-of-the-day a frankenstein one.
 
 The topic here, which I've been eagerly following, is whether or not Cinder
 needs to 

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Tue, Aug 04, 2015 at 10:32:40AM +0300, Duncan Thomas wrote:
 On 3 August 2015 at 20:53, Clint Byrum cl...@fewbar.com wrote:
 
  Excerpts from Devananda van der Veen's message of 2015-08-03 08:53:21
  -0700:
  Also on a side note, I think Cinder's need for this is really subtle,
  and one could just accept that sometimes it's going to break when it does
  two things to one resource from two hosts. The error rate there might
  even be lower than the false-error rate that would be caused by a twitchy
  DLM with timeouts a little low. So there's a core cinder discussion that
  keeps losing to the shiny DLM discussion, and I'd like to see it played
  out fully: Could Cinder just not do anything, and let the few drivers
  that react _really_ badly, implement their own concurrency controls?
 
 
 
 So the problem here is data corruption. Lots of our races can cause data
 corruption. Not 'my instance didn't come up', not 'my network is screwed
 and I need to tear everything down and do it again', but 'My 1tb of
 customer database is now missing the second half'. This means that we
 *really* need some confidence and understanding in whatever we do. The idea
 of locks timing out and being stolen without fencing is frankly scary and
 begging for data corruption unless we're very careful. I'd rather use a
 persistent lock (e.g. a db record change) and manual recovery than a lock
 timeout that might cause corruption.
 
 
 -- 
 Duncan Thomas


If we end up using a DLM then we have to detect when the connection to
the DLM is lost on a node and stop all ongoing operations to prevent
data corruption.

It may not be trivial to do, but we will have to do it in any solution
we use, even on my last proposal that only uses the DB in Volume Manager
we would still need to stop all operations if we lose connection to the
DB.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Tue, Aug 04, 2015 at 05:47:44AM +1000, Morgan Fainberg wrote:
 
 
  On Aug 4, 2015, at 01:42, Fox, Kevin M kevin@pnnl.gov wrote:
  
  I'm usually for abstraction layers, but they don't always pay off very well 
  due to catering to the lowest common denominator.
  
  Lets clearly define the problem space first. IFF the problem space can be 
  fully implemented using Tooz, then lets do that. Then the operator can 
  choose. If Tooz cant and wont handle the problem space, then we're trying 
  to fit a square peg in a round hole.
  
 
 +1 and specifically around tooz, it is narrow in comparison to the feature 
 sets of some the DLMs (since it has to mostly-implement to the lowest common 
 denominator, as abstraction layers do). Defining the space we are trying to 
 target will let us make the informed decision on what we use. 

Again with this?

We already what we want to get out of Tooz, where we want it and for how
long we'll be using it in each of those places.

To answer those questions all that's needed is to read this thread and
the links referred on some conversations.

Gorka.

 
  Thanks,
  Kevin
  
  From: Gorka Eguileor [gegui...@redhat.com]
  Sent: Monday, August 03, 2015 1:43 AM
  To: OpenStack Development Mailing List (not for usage questions)
  Subject: Re: [openstack-dev] [Cinder] A possible solution for HA
  Active-Active
  
  On Mon, Aug 03, 2015 at 10:22:42AM +0200, Thierry Carrez wrote:
  Flavio Percoco wrote:
  [...]
  So, to summarize, I love the effort behind this. But, as others have
  mentioned, I'd like us to take a step back, run this accross teams and
  come up with an opinonated solution that would work for everyone.
  
  Starting this discussion now would allow us to prepare enough material
  to reach an agreement in Tokyo and work on a single solution for
  Mikata. This sounds like a good topic for a cross-project session.
  
  +1
  
  The last thing we want is to rush a solution that would only solve a
  particular project use case. Personally I'd like us to pick the simplest
  solution that can solve most of the use cases. Each of the solutions
  bring something to the table -- Zookeeper is mature, Consul is
  featureful, etcd is lean and simple... Let's not dive into the best
  solution but clearly define the problem space first.
  
  --
  Thierry Carrez (ttx)
  
  I don't see those as different solutions from the point of view of
  Cinder, they are different implementations to the same solution case,
  using a DLM to lock resources.
  
  We keep circling back to the fancy names like moths to a flame, when we
  are still discussing whether we need or want a DLM for the solution.  I
  think we should stop doing that, we need to decide on the solution from
  an abstract point of view (like you say, define the problem space) and
  not get caught up on discussions of which one of those is best.  If we
  end up deciding to use a DLM, which is unlikely, then we can look into
  available drivers in Tooz and if we are not convinced with the ones we
  have (Redis, ZooKeeper, etc.) then we discuss which one we should be
  using instead and just add it to Tooz.
  
  Cheers,
  Gorka.
  
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Tue, Aug 04, 2015 at 09:28:58AM +, Fox, Kevin M wrote:
 Its been explained for Cinder, but not for other OpenStack projects that also 
 have needs in this area.
 

For that, Flavio started a new thread Does OpenStack need a common
solution for DLM?

We are discussing Cinder specifics on this thread.

Cheers,
Gorka.

 Thanks,
 Kevin
 
 
 From: Gorka Eguileor
 Sent: Tuesday, August 04, 2015 1:39:07 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active
 
 On Mon, Aug 03, 2015 at 06:12:23PM +, Fox, Kevin M wrote:
  For example, to parallel the conversation with databases:
 
  We want a database. Well, that means mongodb, postgres, mysql, 
  berkeleydb, etc
 
  Oh, well, I need it to be a relational db, Well, that means postgresq, 
  mysql, etc
 
  Oh, and I need recursive queries... that excludes even more.
 
  We are pretty sure We want a distributed lock manager. What problems are 
  we trying to solve using it, and what features do they require in the 
  DLM/DLM Abstraction of choice? That will exclude some of them. It also may 
  exclude abstraction layers that don't expose the features needed. 
  (Recursive queries for example)
 
  Thanks,
  Kevin
  
 
 Kevin all that has already been explained:
 
 http://gorka.eguileor.com/a-cinder-road-to-activeactive-ha/
 http://gorka.eguileor.com/simpler-road-to-cinder-active-active/
 
 As well as on IRC and this thread, I don't see the point of repeating it
 over and over again, at some point people need to start doing their
 homework and read what's already been said to get up to speed on the
 topic so we can move on.
 
 Gorka.
 
  From: Gorka Eguileor [gegui...@redhat.com]
  Sent: Monday, August 03, 2015 10:48 AM
  To: OpenStack Development Mailing List (not for usage questions)
  Subject: Re: [openstack-dev] [Cinder] A possible solution for   HA  
  Active-Active
 
  On Mon, Aug 03, 2015 at 03:42:48PM +, Fox, Kevin M wrote:
   I'm usually for abstraction layers, but they don't always pay off very 
   well due to catering to the lowest common denominator.
  
   Lets clearly define the problem space first. IFF the problem space can be 
   fully implemented using Tooz, then lets do that. Then the operator can 
   choose. If Tooz cant and wont handle the problem space, then we're trying 
   to fit a square peg in a round hole.
 
  What do you mean with clearly define the problem space?  We know what we
  want, we just need to agree on the compromises we are willing to make,
  use a DLM and make admins' life a little harder (only for those that
  deploy A-A) but have an A-A solution earlier, or postpone A-A
  functionality but make their life easier.
 
  And we already know that Tooz is not the Holy Grail and will not perform
  the miracle of giving Cinder HA A-A.  It is only a piece of the problem,
  so there's nothing to discuss there, and it's not a square peg on a
  round hole, because it fits perfectly for what it is intended. But once
  you have filled that square hole you need another peg, the round one for
  the round hole.
 
  If people are expecting to find one thing that fixes everything and
  gives us HA A-A on its own, then I believe they are a little bit lost.
 
  Gorka.
 
  
   Thanks,
   Kevin
   
   From: Gorka Eguileor [gegui...@redhat.com]
   Sent: Monday, August 03, 2015 1:43 AM
   To: OpenStack Development Mailing List (not for usage questions)
   Subject: Re: [openstack-dev] [Cinder] A possible solution for HA
   Active-Active
  
   On Mon, Aug 03, 2015 at 10:22:42AM +0200, Thierry Carrez wrote:
Flavio Percoco wrote:
 [...]
 So, to summarize, I love the effort behind this. But, as others have
 mentioned, I'd like us to take a step back, run this accross teams and
 come up with an opinonated solution that would work for everyone.

 Starting this discussion now would allow us to prepare enough material
 to reach an agreement in Tokyo and work on a single solution for
 Mikata. This sounds like a good topic for a cross-project session.
   
+1
   
The last thing we want is to rush a solution that would only solve a
particular project use case. Personally I'd like us to pick the simplest
solution that can solve most of the use cases. Each of the solutions
bring something to the table -- Zookeeper is mature, Consul is
featureful, etcd is lean and simple... Let's not dive into the best
solution but clearly define the problem space first.
   
--
Thierry Carrez (ttx)
   
  
   I don't see those as different solutions from the point of view of
   Cinder, they are different implementations to the same solution case,
   using a DLM to lock resources.
  
   We keep circling back to the fancy names like moths to a flame, when we
   are still discussing

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Mon, Aug 03, 2015 at 06:12:23PM +, Fox, Kevin M wrote:
 For example, to parallel the conversation with databases:
 
 We want a database. Well, that means mongodb, postgres, mysql, berkeleydb, 
 etc
 
 Oh, well, I need it to be a relational db, Well, that means postgresq, 
 mysql, etc
 
 Oh, and I need recursive queries... that excludes even more.
 
 We are pretty sure We want a distributed lock manager. What problems are we 
 trying to solve using it, and what features do they require in the DLM/DLM 
 Abstraction of choice? That will exclude some of them. It also may exclude 
 abstraction layers that don't expose the features needed. (Recursive queries 
 for example)
 
 Thanks,
 Kevin
 

Kevin all that has already been explained:

http://gorka.eguileor.com/a-cinder-road-to-activeactive-ha/
http://gorka.eguileor.com/simpler-road-to-cinder-active-active/

As well as on IRC and this thread, I don't see the point of repeating it
over and over again, at some point people need to start doing their
homework and read what's already been said to get up to speed on the
topic so we can move on.

Gorka.

 From: Gorka Eguileor [gegui...@redhat.com]
 Sent: Monday, August 03, 2015 10:48 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Cinder] A possible solution for   HA  
 Active-Active
 
 On Mon, Aug 03, 2015 at 03:42:48PM +, Fox, Kevin M wrote:
  I'm usually for abstraction layers, but they don't always pay off very well 
  due to catering to the lowest common denominator.
 
  Lets clearly define the problem space first. IFF the problem space can be 
  fully implemented using Tooz, then lets do that. Then the operator can 
  choose. If Tooz cant and wont handle the problem space, then we're trying 
  to fit a square peg in a round hole.
 
 What do you mean with clearly define the problem space?  We know what we
 want, we just need to agree on the compromises we are willing to make,
 use a DLM and make admins' life a little harder (only for those that
 deploy A-A) but have an A-A solution earlier, or postpone A-A
 functionality but make their life easier.
 
 And we already know that Tooz is not the Holy Grail and will not perform
 the miracle of giving Cinder HA A-A.  It is only a piece of the problem,
 so there's nothing to discuss there, and it's not a square peg on a
 round hole, because it fits perfectly for what it is intended. But once
 you have filled that square hole you need another peg, the round one for
 the round hole.
 
 If people are expecting to find one thing that fixes everything and
 gives us HA A-A on its own, then I believe they are a little bit lost.
 
 Gorka.
 
 
  Thanks,
  Kevin
  
  From: Gorka Eguileor [gegui...@redhat.com]
  Sent: Monday, August 03, 2015 1:43 AM
  To: OpenStack Development Mailing List (not for usage questions)
  Subject: Re: [openstack-dev] [Cinder] A possible solution for HA
  Active-Active
 
  On Mon, Aug 03, 2015 at 10:22:42AM +0200, Thierry Carrez wrote:
   Flavio Percoco wrote:
[...]
So, to summarize, I love the effort behind this. But, as others have
mentioned, I'd like us to take a step back, run this accross teams and
come up with an opinonated solution that would work for everyone.
   
Starting this discussion now would allow us to prepare enough material
to reach an agreement in Tokyo and work on a single solution for
Mikata. This sounds like a good topic for a cross-project session.
  
   +1
  
   The last thing we want is to rush a solution that would only solve a
   particular project use case. Personally I'd like us to pick the simplest
   solution that can solve most of the use cases. Each of the solutions
   bring something to the table -- Zookeeper is mature, Consul is
   featureful, etcd is lean and simple... Let's not dive into the best
   solution but clearly define the problem space first.
  
   --
   Thierry Carrez (ttx)
  
 
  I don't see those as different solutions from the point of view of
  Cinder, they are different implementations to the same solution case,
  using a DLM to lock resources.
 
  We keep circling back to the fancy names like moths to a flame, when we
  are still discussing whether we need or want a DLM for the solution.  I
  think we should stop doing that, we need to decide on the solution from
  an abstract point of view (like you say, define the problem space) and
  not get caught up on discussions of which one of those is best.  If we
  end up deciding to use a DLM, which is unlikely, then we can look into
  available drivers in Tooz and if we are not convinced with the ones we
  have (Redis, ZooKeeper, etc.) then we discuss which one we should be
  using instead and just add it to Tooz.
 
  Cheers,
  Gorka.
 
  __
  OpenStack Development Mailing List (not for usage

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Mon, Aug 03, 2015 at 12:28:27AM -0700, Clint Byrum wrote:
 Excerpts from Gorka Eguileor's message of 2015-08-02 15:49:46 -0700:
  On Fri, Jul 31, 2015 at 01:47:22AM -0700, Mike Perez wrote:
   On Mon, Jul 27, 2015 at 12:35 PM, Gorka Eguileor gegui...@redhat.com 
   wrote:
I know we've all been looking at the HA Active-Active problem in Cinder
and trying our best to figure out possible solutions to the different
issues, and since current plan is going to take a while (because it
requires that we finish first fixing Cinder-Nova interactions), I've 
been
looking at alternatives that allow Active-Active configurations without
needing to wait for those changes to take effect.
   
And I think I have found a possible solution, but since the HA A-A
problem has a lot of moving parts I ended up upgrading my initial
Etherpad notes to a post [1].
   
Even if we decide that this is not the way to go, which we'll probably
do, I still think that the post brings a little clarity on all the
moving parts of the problem, even some that are not reflected on our
Etherpad [2], and it can help us not miss anything when deciding on a
different solution.
   
   Based on IRC conversations in the Cinder room and hearing people's
   opinions in the spec reviews, I'm not convinced the complexity that a
   distributed lock manager adds to Cinder for both developers and the
   operators who ultimately are going to have to learn to maintain things
   like Zoo Keeper as a result is worth it.
   
   **Key point**: We're not scaling Cinder itself, it's about scaling to
   avoid build up of operations from the storage backend solutions
   themselves.
   
   Whatever people think ZooKeeper scaling level is going to accomplish
   is not even a question. We don't need it, because Cinder isn't as
   complex as people are making it.
   
   I'd like to think the Cinder team is a great in recognizing potential
   cross project initiatives. Look at what Thang Pham has done with
   Nova's version object solution. He made a generic solution into an
   Oslo solution for all, and Cinder is using it. That was awesome, and
   people really appreciated that there was a focus for other projects to
   get better, not just Cinder.
   
   Have people consider Ironic's hash ring solution? The project Akanda
   is now adopting it [1], and I think it might have potential. I'd
   appreciate it if interested parties could have this evaluated before
   the Cinder midcycle sprint next week, to be ready for discussion.
   
   [1] - https://review.openstack.org/#/c/195366/
   
   -- Mike Perez
  
  Hi all,
  
  Since my original proposal was more complex that it needed be I have a
  new proposal of a simpler solution, and I describe how we can do it with
  or without a DLM since we don't seem to reach an agreement on that.
  
  The solution description was more rushed than previous one so I may have
  missed some things.
  
  http://gorka.eguileor.com/simpler-road-to-cinder-active-active/
  
 
 I like the idea of keeping it simpler Gorka. :)
 
 Note that this is punting back to use the database for coordination,
 which is what most projects have done thus far, and has a number of
 advantages and disadvantages.
 
 Note that the stale-lock problem was solved in an interesting way in Heat:
 each engine process gets an instance-of-engine uuid that adds to the
 topic queues it listens to. If it holds a lock, it records this UUID in
 the owner field. When somebody wants to steal the lock (due to timeout)
 they send to this queue, and if there's no response, the lock is stolen.

I don't think that's a good idea for Cinder, if the node that is holding
the lock is doing a long running CPU bound operation (like backups) it
may not be fast enough to reply to that message, and that would end up
with multiple nodes accessing the same data.

Using the service heartbeat and the startup of the volume nodes we can
do automatic cleanups on failed nodes.  And lets be realistic, failed
nodes will not be the norm, so we should prioritize normal operations
over failure cleanup.

And having inter-node operations like that will not only increase our
message broker workload but it will also set more strict constraint in
our Volume Node responsiveness.  Which could put us in a pinch in some
operations and would require a careful and thorough empirical study to
confirm that we don't have false positives on lock steals.

 
 Anyway, I think what might make more sense than copying that directly,
 is implementing Use the database and oslo.messaging to build a DLM
 as a tooz backend. This way as the negative aspects of that approach
 impact an operator, they can pick a tooz driver that satisfies their
 needs, or even write one to their specific backend needs.
 

I have no problem implementing a locking variant in Tooz using the DB
(not DB locks).  As far as I've seen Tooz community moves really fast
with reviews and we could probably

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Fri, Jul 31, 2015 at 03:18:39PM -0700, Clint Byrum wrote:
 Excerpts from Mike Perez's message of 2015-07-31 10:40:04 -0700:
  On Fri, Jul 31, 2015 at 8:56 AM, Joshua Harlow harlo...@outlook.com wrote:
   ...random thought here, skip as needed... in all honesty orchestration
   solutions like mesos
   (http://mesos.apache.org/assets/img/documentation/architecture3.jpg),
   map-reduce solutions like hadoop, stream processing systems like apache
   storm (...), are already using zookeeper and I'm not saying we should just
   use it cause they are, but the likelihood that they just picked it for no
   reason are imho slim.
  
  I'd really like to see focus cross project. I don't want Ceilometer to
  depend on Zoo Keeper, Cinder to depend on etcd, etc. This is not ideal
  for an operator to have to deploy, learn and maintain each of these
  solutions.

If we finally implement the DLM with Tooz then we wouldn't really need
to use different DLM solution in each project, they could all be using
the same. It's just a matter of choice for the admin. If admin wants to
use the same tool he will set the same configuration in Ceilometer and
Cinder, and he wants more work and use different tools then he will set
different configurations.

The discussion is not about choosing ZooKeeper over Redis or anything
like that, the discussion is more in the lines of we need/don't need a
DLM for HA A-A.


  
  I think this is difficult when you consider everyone wants options of
  their preferred DLM. If we went this route, we should pick one.
  
  Regardless, I want to know if we really need a DLM. Does Ceilometer
  really need a DLM? Does Cinder really need a DLM? Can we just use a
  hash ring solution where operators don't even have to know or care
  about deploying a DLM and running multiple instances of Cinder manager
  just works?
  
 
 So in the Ironic case, if two conductors decide they both own one IPMI
 controller, _chaos_ can ensue. They may, at different times, read that
 the power is up, or down, and issue power control commands that may take
 many seconds, and thus on the next status run of the other command may
 cause the conductor to react by reversing, and they'll just fight over
 the node in a tug-o-war fashion.
 
 Oh wait, except, thats not true. Instead, they use the database as a
 locking mechanism, and AFAIK, no nodes have been torn limb from limb by
 two conductors thus far.

One thing we must not forget is that we are not talking about using
locks just for mutual exclusion or for the sake of it, we are doing it
to maintain current Cinder behavior intact. Something people keep
forgetting.

Right now if you are using volume A for reading, lets say for cloning
into a new volume, and a request that changes the resource status comes
in for that same volume A, like backup or delete, the API will accept
that request and when the message arrives at the Volume Node it will be
queued on the lock, and once that lock is released operation will be
performed.

When locking only using the DB, which I believe is mostly possible with
possible exception of some driver sections of software based storage
systems, you will be probably changing this behavior and no longer
allowing this.  Depending on who you ask this is a big deal (breaking an
implicit contract based on current behavior) or not.

Keeping things backward compatible is a pain.  And I think that's one of
the differences between Ironic and Cinder, at this point we cannot just
decide to change Cinder however we like, we have to take into account
Nova interactions as well as external client interactions.

 
 But, a DLM would be more efficient, and actually simplify failure
 recovery for Ironic's operators. The database locks suffer from being a
 little too conservative, and sometimes you just have to go into the DB
 and delete a lock after something explodes (this was true 6 months ago,
 it may have better automation sometimes now, I don't know).
 
 Anyway, I'm all for the simplest possible solution. But, don't make it
 _too_ simple.
 

I agree, like Einstein said, everything should be made as simple as
possible, but no simpler.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder][Ironic] A possible solution for HA Active-Active

[Gorka Eguileor]

On Fri, Jul 31, 2015 at 12:47:34PM -0700, Joshua Harlow wrote:
 Joshua Harlow wrote:
 Mike Perez wrote:
 On Fri, Jul 31, 2015 at 8:56 AM, Joshua Harlowharlo...@outlook.com
 wrote:
 ...random thought here, skip as needed... in all honesty orchestration
 solutions like mesos
 (http://mesos.apache.org/assets/img/documentation/architecture3.jpg),
 map-reduce solutions like hadoop, stream processing systems like apache
 storm (...), are already using zookeeper and I'm not saying we should
 just
 use it cause they are, but the likelihood that they just picked it
 for no
 reason are imho slim.
 
 I'd really like to see focus cross project. I don't want Ceilometer to
 depend on Zoo Keeper, Cinder to depend on etcd, etc. This is not ideal
 for an operator to have to deploy, learn and maintain each of these
 solutions.
 
 I think this is difficult when you consider everyone wants options of
 their preferred DLM. If we went this route, we should pick one.
 
 +1
 
 
 Regardless, I want to know if we really need a DLM. Does Ceilometer
 really need a DLM? Does Cinder really need a DLM? Can we just use a
 hash ring solution where operators don't even have to know or care
 about deploying a DLM and running multiple instances of Cinder manager
 just works?
 
 All very good questions, although IMHO a hash-ring is just a piece of
 the puzzle, and is more equivalent to sharding resources, which yes is
 one way to scale as long as each shard never touches anything from the
 other shards. If those shards ever start to need to touch anything
 shared then u get back into this same situation again for a DLM (and at
 that point u really do need the 'distributed' part of DLM, because each
 shard is distributed).
 
 And an few (maybe obvious) questions:
 
 - How would re-sharding work?
 - If sharding (the hash-ring partitioning) is based on entities
 (conductors/other) owning a 'bucket' of resources (ie entity 1 manages
 resources A-F, entity 2 manages resources G-M...), what happens if a
 entity dies, does some other entity take over that bucket, what happens
 if that entity really hasn't 'died' but is just disconnected from the
 network (partition tolerance...)? (If the answer is there is a lock on
 the resource/s being used by each entity, then u get back into the LM
 question).
 
 I'm unsure about how ironic handles these problems (although I believe
 they have a hash-ring and still have a locking scheme as well, so maybe
 thats there answer for the dual-entities manipulating the same bucket
 problem).
 
 Code for some of this, maybe ironic folks can chime-in:
 
 https://github.com/openstack/ironic/blob/2015.1.1/ironic/conductor/task_manager.py#L18
 (using DB as DLM)
 
 Afaik, since ironic built-in a hash-ring and the above task manager since
 the start (or from a very earlier commit) they have better been able to
 accomplish the HA goal, retrofitting stuff on-top of nova,cinder,others...
 is not going to as easy...

If you really look at what that code is actually doing you'll see it's
basically implementing a DLM on top of a database.  We can do that as
well, but it will still be a DLM.

So people, please don't get caught up on the implementation details when
making suggestions, if you think that we don't need a DLM but then you
suggest implementing a DLM on top of a DB, then you are thinking too
close to the code.

First you decide if you want/need a DLM, and then you choose whether you
want it implemented with a standard solution like Redis or Zookeper, or
if you want to implement it manually on top of your DB.

Cheers,
Gorka.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Mon, Aug 03, 2015 at 10:22:42AM +0200, Thierry Carrez wrote:
 Flavio Percoco wrote:
  [...]
  So, to summarize, I love the effort behind this. But, as others have
  mentioned, I'd like us to take a step back, run this accross teams and
  come up with an opinonated solution that would work for everyone.
  
  Starting this discussion now would allow us to prepare enough material
  to reach an agreement in Tokyo and work on a single solution for
  Mikata. This sounds like a good topic for a cross-project session.
 
 +1
 
 The last thing we want is to rush a solution that would only solve a
 particular project use case. Personally I'd like us to pick the simplest
 solution that can solve most of the use cases. Each of the solutions
 bring something to the table -- Zookeeper is mature, Consul is
 featureful, etcd is lean and simple... Let's not dive into the best
 solution but clearly define the problem space first.
 
 -- 
 Thierry Carrez (ttx)
 

I don't see those as different solutions from the point of view of
Cinder, they are different implementations to the same solution case,
using a DLM to lock resources.

We keep circling back to the fancy names like moths to a flame, when we
are still discussing whether we need or want a DLM for the solution.  I
think we should stop doing that, we need to decide on the solution from
an abstract point of view (like you say, define the problem space) and
not get caught up on discussions of which one of those is best.  If we
end up deciding to use a DLM, which is unlikely, then we can look into
available drivers in Tooz and if we are not convinced with the ones we
have (Redis, ZooKeeper, etc.) then we discuss which one we should be
using instead and just add it to Tooz.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Mon, Aug 03, 2015 at 07:01:45PM +1000, Morgan Fainberg wrote:
 Lets step back away from tooz. Tooz for the sake of this conversation is as 
 much the same as saying zookeeper or consul or etcd, etc. We should be 
 focused (as both Flavio and Thierry said) on if we need DLM and what it will 
 solve.

What do you mean we should be focused on if we need a DLM and what it
will solve?

I don't know what you mean, as those answers are quite clear:

- The DLM replaces our current local file locks and extends them among
  nodes, it does not provide any additional functionality.

- Do we need a DLM?  Need is a strong word, if you are asking if we can
  do it without a DLM, then the answer is yes, we can do it without it.
  And if you ask if it will take more time than using a DLM and has the
  potential to introduce more bugs, then the answer is yes as well.

- Will we keep using a DLM forever?  No, we will change the DLM locks
  with DB mutual exclusion at the API nodes later.

Gorka.

 
 Once we have all of that defined, the use of an abstraction such as tooz (or 
 just the direct bindings for some specific choice) can be made. 
 
 I want to voice that we should be very picky about the solution (if we decide 
 on a DLM) so that we are implementing to the strengths of the solution rather 
 than try and make everything work seamlessly.  
 
 --Morgan
 
 Sent via mobile
 
  On Aug 3, 2015, at 18:49, Julien Danjou jul...@danjou.info wrote:
  
  On Mon, Aug 03 2015, Thierry Carrez wrote:
  
  The last thing we want is to rush a solution that would only solve a
  particular project use case. Personally I'd like us to pick the simplest
  solution that can solve most of the use cases. Each of the solutions
  bring something to the table -- Zookeeper is mature, Consul is
  featureful, etcd is lean and simple... Let's not dive into the best
  solution but clearly define the problem space first.
  
  Or just start using Tooz – like some of OpenStack are already doing for
  months – and let the operators pick the backend that they are the most
  comfortable with? :)
  
  -- 
  Julien Danjou
  -- Free Software hacker
  -- http://julien.danjou.info
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Mon, Aug 03, 2015 at 03:42:48PM +, Fox, Kevin M wrote:
 I'm usually for abstraction layers, but they don't always pay off very well 
 due to catering to the lowest common denominator.
 
 Lets clearly define the problem space first. IFF the problem space can be 
 fully implemented using Tooz, then lets do that. Then the operator can 
 choose. If Tooz cant and wont handle the problem space, then we're trying to 
 fit a square peg in a round hole.

What do you mean with clearly define the problem space?  We know what we
want, we just need to agree on the compromises we are willing to make,
use a DLM and make admins' life a little harder (only for those that
deploy A-A) but have an A-A solution earlier, or postpone A-A
functionality but make their life easier.

And we already know that Tooz is not the Holy Grail and will not perform
the miracle of giving Cinder HA A-A.  It is only a piece of the problem,
so there's nothing to discuss there, and it's not a square peg on a
round hole, because it fits perfectly for what it is intended. But once
you have filled that square hole you need another peg, the round one for
the round hole.

If people are expecting to find one thing that fixes everything and
gives us HA A-A on its own, then I believe they are a little bit lost.

Gorka.

 
 Thanks,
 Kevin
 
 From: Gorka Eguileor [gegui...@redhat.com]
 Sent: Monday, August 03, 2015 1:43 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Cinder] A possible solution for HA
 Active-Active
 
 On Mon, Aug 03, 2015 at 10:22:42AM +0200, Thierry Carrez wrote:
  Flavio Percoco wrote:
   [...]
   So, to summarize, I love the effort behind this. But, as others have
   mentioned, I'd like us to take a step back, run this accross teams and
   come up with an opinonated solution that would work for everyone.
  
   Starting this discussion now would allow us to prepare enough material
   to reach an agreement in Tokyo and work on a single solution for
   Mikata. This sounds like a good topic for a cross-project session.
 
  +1
 
  The last thing we want is to rush a solution that would only solve a
  particular project use case. Personally I'd like us to pick the simplest
  solution that can solve most of the use cases. Each of the solutions
  bring something to the table -- Zookeeper is mature, Consul is
  featureful, etcd is lean and simple... Let's not dive into the best
  solution but clearly define the problem space first.
 
  --
  Thierry Carrez (ttx)
 
 
 I don't see those as different solutions from the point of view of
 Cinder, they are different implementations to the same solution case,
 using a DLM to lock resources.
 
 We keep circling back to the fancy names like moths to a flame, when we
 are still discussing whether we need or want a DLM for the solution.  I
 think we should stop doing that, we need to decide on the solution from
 an abstract point of view (like you say, define the problem space) and
 not get caught up on discussions of which one of those is best.  If we
 end up deciding to use a DLM, which is unlikely, then we can look into
 available drivers in Tooz and if we are not convinced with the ones we
 have (Redis, ZooKeeper, etc.) then we discuss which one we should be
 using instead and just add it to Tooz.
 
 Cheers,
 Gorka.
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Fri, Jul 31, 2015 at 01:47:22AM -0700, Mike Perez wrote:
 On Mon, Jul 27, 2015 at 12:35 PM, Gorka Eguileor gegui...@redhat.com wrote:
  I know we've all been looking at the HA Active-Active problem in Cinder
  and trying our best to figure out possible solutions to the different
  issues, and since current plan is going to take a while (because it
  requires that we finish first fixing Cinder-Nova interactions), I've been
  looking at alternatives that allow Active-Active configurations without
  needing to wait for those changes to take effect.
 
  And I think I have found a possible solution, but since the HA A-A
  problem has a lot of moving parts I ended up upgrading my initial
  Etherpad notes to a post [1].
 
  Even if we decide that this is not the way to go, which we'll probably
  do, I still think that the post brings a little clarity on all the
  moving parts of the problem, even some that are not reflected on our
  Etherpad [2], and it can help us not miss anything when deciding on a
  different solution.
 
 Based on IRC conversations in the Cinder room and hearing people's
 opinions in the spec reviews, I'm not convinced the complexity that a
 distributed lock manager adds to Cinder for both developers and the
 operators who ultimately are going to have to learn to maintain things
 like Zoo Keeper as a result is worth it.
 
 **Key point**: We're not scaling Cinder itself, it's about scaling to
 avoid build up of operations from the storage backend solutions
 themselves.
 
 Whatever people think ZooKeeper scaling level is going to accomplish
 is not even a question. We don't need it, because Cinder isn't as
 complex as people are making it.
 
 I'd like to think the Cinder team is a great in recognizing potential
 cross project initiatives. Look at what Thang Pham has done with
 Nova's version object solution. He made a generic solution into an
 Oslo solution for all, and Cinder is using it. That was awesome, and
 people really appreciated that there was a focus for other projects to
 get better, not just Cinder.
 
 Have people consider Ironic's hash ring solution? The project Akanda
 is now adopting it [1], and I think it might have potential. I'd
 appreciate it if interested parties could have this evaluated before
 the Cinder midcycle sprint next week, to be ready for discussion.
 
 [1] - https://review.openstack.org/#/c/195366/
 
 -- Mike Perez

Hi all,

Since my original proposal was more complex that it needed be I have a
new proposal of a simpler solution, and I describe how we can do it with
or without a DLM since we don't seem to reach an agreement on that.

The solution description was more rushed than previous one so I may have
missed some things.

http://gorka.eguileor.com/simpler-road-to-cinder-active-active/

Cheers,
Gorka.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

On Fri, Jul 31, 2015 at 01:47:22AM -0700, Mike Perez wrote:
 On Mon, Jul 27, 2015 at 12:35 PM, Gorka Eguileor gegui...@redhat.com wrote:
  I know we've all been looking at the HA Active-Active problem in Cinder
  and trying our best to figure out possible solutions to the different
  issues, and since current plan is going to take a while (because it
  requires that we finish first fixing Cinder-Nova interactions), I've been
  looking at alternatives that allow Active-Active configurations without
  needing to wait for those changes to take effect.
 
  And I think I have found a possible solution, but since the HA A-A
  problem has a lot of moving parts I ended up upgrading my initial
  Etherpad notes to a post [1].
 
  Even if we decide that this is not the way to go, which we'll probably
  do, I still think that the post brings a little clarity on all the
  moving parts of the problem, even some that are not reflected on our
  Etherpad [2], and it can help us not miss anything when deciding on a
  different solution.
 
 Based on IRC conversations in the Cinder room and hearing people's
 opinions in the spec reviews, I'm not convinced the complexity that a
 distributed lock manager adds to Cinder for both developers and the
 operators who ultimately are going to have to learn to maintain things
 like Zoo Keeper as a result is worth it.

Hi Mike,

I think you are right in bringing up the cost that adding a DLM to the
solution brings to operators, as it is something important to take into
consideration, and I would like to say that Ceilometer is already using
Tooz so operators are already familiar with these DLM, but unfortunately
that would be stretching the truth, since Cinder is present in 73% of
OpenStack production workloads while Ceilometer is only in 33% of them,
so we would be certainly disturbing some operators.

But we must not forget that the only operators that would need to worry
about deploying and maintaining the DLM are those wanting to deploy
Active-Active configurations (for Active-Passive configuration Tooz will
be working with local file locks like we are doing now), and some of
those may think like Duncan does: I already have to administer rabbit,
mysql, backends, horizon, load ballancers, rate limiters...  adding
redis isn't going to make it that much harder.

That's why I don't think this is such a big deal for the vast majority
of operators.

On the developer side I have to disagree, there is no difference between
using Tooz and using current oslo synchronization mechanism for non
Active-Active deployments.

 
 **Key point**: We're not scaling Cinder itself, it's about scaling to
 avoid build up of operations from the storage backend solutions
 themselves.

You must also consider that Active-Active solution will help deployments
where downtime is not an option or have SLAs with uptime or operational
requirements, it's not only about increasing volume of operations and
reducing times.

 
 Whatever people think ZooKeeper scaling level is going to accomplish
 is not even a question. We don't need it, because Cinder isn't as
 complex as people are making it.
 
 I'd like to think the Cinder team is a great in recognizing potential
 cross project initiatives. Look at what Thang Pham has done with
 Nova's version object solution. He made a generic solution into an
 Oslo solution for all, and Cinder is using it. That was awesome, and
 people really appreciated that there was a focus for other projects to
 get better, not just Cinder.

To be fair, Tooz is just one of those cross project initiatives you are
describing, it's a generic solution that can be used in all projects,
not just Ceilometer.

 
 Have people consider Ironic's hash ring solution? The project Akanda
 is now adopting it [1], and I think it might have potential. I'd
 appreciate it if interested parties could have this evaluated before
 the Cinder midcycle sprint next week, to be ready for discussion.
 

I will have a look at the hash ring solution you mention and see if it
makes sense to use it.

And I would really love to see the HA A-A discussion enabled for remote
people, as some of us are interested in the discussion but won't be able
to attend.  In my case problems with living in the Old World  :-(

In a way I have to agree with you that sometimes we make Cinder look
more complex than it really is, and in my case the solution I proposed
in the post was way too complex as it has been pointed out.  I just
tried to solve de A-A problem and fix some other issues like recovering
lost jobs (those waiting for locks) at the same time.

There is an alternative solution I am considering that will be much
simpler and will align with Walter's efforts to remove locks from the
Volume Manager.  I just need to give it a hard think to make sure the
solution has all bases covered.

The main reason why I am suggesting using Tooz and a DLM is because I
think it will allow us to reach Active-Active faster and with less
effort, not because I

Re: [openstack-dev] [all] cross project communication: Return request-id to caller

[Gorka Eguileor]

On Tue, Jul 28, 2015 at 09:48:25AM -0400, Doug Hellmann wrote:
 Excerpts from Gorka Eguileor's message of 2015-07-28 10:37:42 +0200:
  On Fri, Jul 24, 2015 at 10:08:45AM -0400, Doug Hellmann wrote:
   Excerpts from Kekane, Abhishek's message of 2015-07-24 06:33:00 +:
Hi Devs,

X-Openstack-Request-Id. We have analysed python-cinderclient, 
python-glanceclient, python-novaclient, python-keystoneclient and 
python-neutronclient to check the return types.

There are 9 ways return values are returned from python clients:
1. List
2. Dict
3. Resource class object
4. None
5. Tuple
6. Exception
7. Boolean (True/False, for keystoneclient)
8. Generator (for list api's in glanceclient)
9. String (for novaclient)

Out of 9 we have solution for all return types except generator.
In case of glance-client list api's are returning generator which is 
immutable. So it is not possible to return request-id in this case, 
which is a blocker for adopting the solution.

I have added detail analysis for above return types in etherpad [2] as 
solution #3.

If you have any suggestion in case of generation type then please let 
me know.
   
   It should be possible to create a new class to wrap the existing
   generator and implement the iterator protocol [3].
   
   [3] 
   https://docs.python.org/2/reference/expressions.html#generator-iterator-methods
   
   Doug
   
  
  Unless I'm missing something I think we wouldn't even need to create a
  new class that implements the iterator protocol, we can just return a
  generator that generates from the other one.
  
  For example, for each of the requests, if we get the generator in
  variable *result* that returns dictionaries and we want to add *headers*
  to each dictionary:
  
  return (DictWithHeaders(resource, headers) for resource in result)
  
  Wouldn't that work?
 
 That would work, but it wouldn't be consistent with the way I read
 [2] as describing how the other methods are being updated.
 
 For example, a method that now returns a list() will return a
 ListWithHeaders(), and only that returned object will have the
 headers, not its contents. A caller could do:

You are right, it wouldn't be consistent with other methods, and that's
not good. So the new iterator class wrapper seems to be the way to go.


Gorka.

 
   response = client.some_method_returning_a_list()
   print reponse.headers
 
 but could not do
 
   print response[0].headers
 
 and get the same values.
 
 Creating a GeneratorWithHeaders class mirrors that behavior for
 methods that return generators instead of lists.
 
 Doug
 
[2] https://etherpad.openstack.org/p/request-id
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] cross project communication: Return request-id to caller

[Gorka Eguileor]

On Fri, Jul 24, 2015 at 10:08:45AM -0400, Doug Hellmann wrote:
 Excerpts from Kekane, Abhishek's message of 2015-07-24 06:33:00 +:
  Hi Devs,
  
  X-Openstack-Request-Id. We have analysed python-cinderclient, 
  python-glanceclient, python-novaclient, python-keystoneclient and 
  python-neutronclient to check the return types.
  
  There are 9 ways return values are returned from python clients:
  1. List
  2. Dict
  3. Resource class object
  4. None
  5. Tuple
  6. Exception
  7. Boolean (True/False, for keystoneclient)
  8. Generator (for list api's in glanceclient)
  9. String (for novaclient)
  
  Out of 9 we have solution for all return types except generator.
  In case of glance-client list api's are returning generator which is 
  immutable. So it is not possible to return request-id in this case, which 
  is a blocker for adopting the solution.
  
  I have added detail analysis for above return types in etherpad [2] as 
  solution #3.
  
  If you have any suggestion in case of generation type then please let me 
  know.
 
 It should be possible to create a new class to wrap the existing
 generator and implement the iterator protocol [3].
 
 [3] 
 https://docs.python.org/2/reference/expressions.html#generator-iterator-methods
 
 Doug
 

Unless I'm missing something I think we wouldn't even need to create a
new class that implements the iterator protocol, we can just return a
generator that generates from the other one.

For example, for each of the requests, if we get the generator in
variable *result* that returns dictionaries and we want to add *headers*
to each dictionary:

return (DictWithHeaders(resource, headers) for resource in result)

Wouldn't that work?

Cheers,
Gorka.


  
  
  [1] 
  http://eavesdrop.openstack.org/meetings/crossproject/2015/crossproject.2015-07-07-21.01.log.html
  [2] https://etherpad.openstack.org/p/request-id
  
  
  Thanks  Best Regards,
  
  Abhishek Kekane
  
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Cinder] A possible solution for HA Active-Active

[Gorka Eguileor]

Hi all,

I know we've all been looking at the HA Active-Active problem in Cinder
and trying our best to figure out possible solutions to the different
issues, and since current plan is going to take a while (because it
requires that we finish first fixing Cinder-Nova interactions), I've been
looking at alternatives that allow Active-Active configurations without
needing to wait for those changes to take effect.

And I think I have found a possible solution, but since the HA A-A
problem has a lot of moving parts I ended up upgrading my initial
Etherpad notes to a post [1].

Even if we decide that this is not the way to go, which we'll probably
do, I still think that the post brings a little clarity on all the
moving parts of the problem, even some that are not reflected on our
Etherpad [2], and it can help us not miss anything when deciding on a
different solution.

Cheers,
Gorka.

[1]: http://gorka.eguileor.com/a-cinder-road-to-activeactive-ha/
[2]: https://etherpad.openstack.org/p/cinder-active-active-vol-service-issues

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder]Restrict volume creation based on type

[Gorka Eguileor]

On Mon, Jul 13, 2015 at 07:56:36AM +0300, Eduard Matei wrote:
 Hi,
 
 Forgot to mention, indeed we configured extra_specs: volume_backend_name.
 The problem is that a volume of this type can get scheduled on a node where
 c-vol is not configured with this backend.
 
 F.e.: I have 3 storage nodes (c-vol) and two have the driver configured,
 the third one doesn't have it; when i try to create a volume of this type,
 sometimes the c-vol on third node gets the call, and it fails because the
 driver is not configured.
 
 Maybe we didn't configure something properly, i tried looking at c-sch but
 i can't figure out why the third node got chosen.
 
 Thanks,
 
 Eduard

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Hi,

You wouldn't happen to have the same hostname on all those 3 nodes,
right?

I mean, if all your storage nodes are running with something like
localhost.localdomain, then the Scheduler (or the API node for that
matter) won't be able to properly address them individually.

Cheers,
Gorka.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack][cinder]A discussion about quota update lower than current usage.

[Gorka Eguileor]

On Fri, Jul 10, 2015 at 10:28:06AM +0300, Duncan Thomas wrote:
 That is a semantic change to the api that will break anybody who has
 tooling expecting the current behavior. Since there are perfectly sensible
 uses of the current behavior, that is not a good thing.

Hi Duncan,

I don't think that will be the case, if it's an optional argument that
by default preserves current behavior (force = True), then it shouldn't
break anything for all callers that don't use that new option.

And for those that want the new behavior, they can always pass force set
to false.

Cheers,
Gorka.

 On 10 Jul 2015 07:33, hao wang sxmatch1...@gmail.com wrote:
 
  Cinder now doesn't check the existing resource when user lower the quota.
  It's reasonable for admin can adjust the quota limit to lower level than
  current usage.
  But it also bring confusion that I have received to end user, they saw the
  current usage
  was more than limit, but they can't create resources any more.
 
  So there have been 'bug' reported[1] and code patch[2] committed, I knew
  it may be
  inappropriate as 'bug fix', but just want to optimize this API of
  updating quota.
 
  We are proposing to add an option argument which is named 'force' in
  request body.
  Of course the default value is True that means admin can adjust the quota
  lower then
  current usage as same as what we did now. When the force is False, that
  will occur
  a Validation and return 400 Bad Request if the update value is lower than
  current usage.
 
  I wonder to know folks' opinions and suggestions about this change to see
  if this is value to merge this patch.
 
  [1]https://bugs.launchpad.net/neutron/+bug/1304234
  [2]https://review.openstack.org/#/c/197938/
 
  Thanks~
 
  --
 
  Best Wishes For You!
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] python-cinderclient functional tests

[Gorka Eguileor]

On Mon, Jul 06, 2015 at 09:48:27PM +0300, Ivan Kolodyazhny wrote:
 Hi all,
 
 As you may know, we've got experimental job [1] to run functional tests [2]
 for python-cinderclient with devstack setup.
 
 Functional tests for python-cinderclient is very important because it's
 almost the only way to test python-cinderclient with Cinder API. For now,
 we've got only Rally which uses cinderclient to test Cinder. Tempest uses
 own client for all APIs.
 
 Current tests coverage are very low.. That's why I would like to ask
 everyone to contribute to python-cinderclient. I created etherpad [3] with
 current progress. You can find me (e0ne) or any other core in
 #openstack-cinder  in IRC.
 
 
 Aslo, what do you think about moving cinderclient functional tests from
 experimental to non-voting queue to make it more public and run it with
 every patch to python-cinderclient?
 

+1
I think it is a great idea to move it to non-voting.

 
 [1] https://review.openstack.org/#/c/182528/
 [2]
 https://github.com/openstack/python-cinderclient/tree/master/cinderclient/tests/functional
 [3] https://etherpad.openstack.org/p/cinder-client-functional-tests
 
 
 Regards,
 Ivan Kolodyazhny

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Security][Bandit] Bandit gate usage

[Gorka Eguileor]

On Thu, Jul 02, 2015 at 07:09:41PM +, Kelsey, Timothy John wrote:
 Hello Stackers,
 A few intrepid projects have started adopting Bandit, an automatic security 
 linter built by the security project, into their gate tests. This is very 
 rewarding to see for those of us who have worked on the project and people 
 with an interest in securing the OpenStack codebase. The list of (known) 
 adopters so far:
 
 - Keystone
 - Keystone-client
 - Barbican
 - Anchor
 - Sahara
 - Magnum
 
 If you know of, or are involved in a project that’s using Bandit and isn’t on 
 our list then please let us know, it would be great to hear your feedback. If 
 you would like to begin using it then check out our wiki for instructions 
 here [1].  If you have no idea what this Bandit thing is then perhaps this 
 presentation from the Vancouver summit might be interesting to you [2]. A 
 Bandit gate job can be configured either as an experimental or none-voting 
 job, so if your interested in trying it out you can give it a go and decide 
 if its a good fit for your project before fully committing.

Hi,

At Cinder we are adding [1] basic bandit configuration for high and
medium severity results as a tox option, but not running it by default
for now.

Cheers,
Gorka

[1]: https://review.openstack.org/#/c/179568/

 
 Bandit is regularly discussed in the Security Project IRC meetings and 
 feedback is very welcome. If you have questions or suggestions then feel free 
 to drop in or reply here.
 
 [1] https://wiki.openstack.org/wiki/Security/Projects/Bandit
 [2] https://www.youtube.com/watch?v=hxbbpdUdU_k
 
 Many thanks
 
 --
 Tim Kelsey
 OpenStack Security member
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][oslo] Locks for create from volume/snapshot

[Gorka Eguileor]

On Mon, Jun 29, 2015 at 03:45:56PM +0300, Duncan Thomas wrote:
 On 29 June 2015 at 15:23, Dulko, Michal michal.du...@intel.com wrote:
 
   There’s also some similar situations when we actually don’t lock on
  resources. For  example – a cgsnapshot may get deleted while creating a
  consistencygroup from it.
 
 
 
  From my perspective it seems best to have atomic state changes and
  state-based exclusion in API. We would need some kind of
  currently_used_to_create_snapshot/volums/consistencygroups states to
  achieve that. Then we would be also able to return VolumeIsBusy exceptions
  so retrying a request would be on the user side.
 
 
 
 I'd agree, except that gives quite a big behaviour change in the
 tenant-facing API, which will break clients and scripts. Not sure how to
 square that circle... I'd say V3 API except Mike might kill me...

I'd prefer not to add another item to the list of things to get HA, much
less one on the scale of a new version.

As far as I can see, we have 3 cases where we use or need to use locks:

1- Locking multiple writing access to a resource
2- Prevent modification of a resource being used for reading
3- Backend drivers

1- Locking multiple writing access to a resource
These locks can most likely be avoided if we implement atomic state
changes (with compare-and-swap) and use current state to prevent
multiple writes on the same resource, since writes change the status of
the resource.  There's already a spec proposing this [1].


2- Prevent modification of a resource in read use
I only see 2 options here:

- Limit numbers of readers to 1 and use Tooz's Locks as DLM. This would
  be implemented quite easily, although it would not be very efficient.
- Implement shared locks in Tooz or in DB.  One way to implement this in
  the DB would be to add a field with a counter of tasks currently using
  the resource for reading.  Modifications to this counter would use a
  compare and swap to check the status when increasing the counter and
  doing the increase on the DB instead of doing it in the Cinder node.
  Status changes would also work with compare-and-swap and besides
  checking current status for availability it would check the counter to
  be 0.

The drawback of the DB implementation is that an aborted operation would
be locking the resource.  But it could be solved if we use TaskFlow for
operations and on the revert method we decrement the counter.  One big
advantage is that we don't need heartbeats to be periodically sent to
prevent locks from being released and it's easy to pass the lock from
the API to the Volume node.

If we implement this in Tooz we could start implementing it in only 1
driver and recommend only using that until the rest are available.


3- Backend drivers
Depending on the drivers they could not need locks, or they could do with
file locks local to the node (since Cinder would be preventing multiple
write access to the same resource) or they may need a DLM if they need,
for example, to prevent simultaneous operations on the same pool from
different nodes.

For this case Tooz would be the best solution, since drivers should not
access the DB and Tooz allows using file locks as well as distributed
locking.

Cheers,
Gorka


[1]: https://review.openstack.org/#/c/149894/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][stable] Cinder client broken in Juno

[Gorka Eguileor]

On Tue, Jun 23, 2015 at 08:49:55AM -0700, Mike Perez wrote:
 There was a bug raised [1] from some large deployments that the Cinder
 client 1.2.0 and beyond is not working because of version discovery.
 Unfortunately it's not taking into account of deployments that have a
 proxy.

A little bit unrelated, but volume pagination in Cinder client is also
broken due to Version Discovery:
https://bugs.launchpad.net/python-cinderclient/+bug/1453755

 
 Cinder client asks Keystone to find a publicURL based on a version.
 Keystone will gather data from the service catalog and ask Cinder for
 a list of the public endpoints and compare. For the proxy cases,
 Cinder is giving internal URLs back to the proxy and Keystone ends up
 using that instead of the publicURL in the service catalog. As a
 result, clients usually won't be able to use the internal URL and
 rightfully so.
 
 This is all correctly setup on the deployer's side, this an issue with
 the server side code of Cinder.
 
 There is a patch that allows the deployer to specify a configuration
 option public_endpoint [2] which was introduced in a patch in Kilo
 [3]. The problem though is we can't expect people to already be
 running Kilo to take advantage of this, and it leaves deployers
 running stable releases of Juno in the dark with clients upgrading and
 using the latest.
 
 Two options:
 
 1) Revert version discovery which was introduced in Kilo for Cinder client.
 
 2) Grant exception on backporting [4] a patch that helps with this
 problem, and introduces a config option that does not change default
 behavior. I'm also not sure if this should be considered for Icehouse.

+1 to option 2 and I wouldn't be totally against considering it for
Icehouse.

Cheers,
Gorka.
 
 
 [1] - https://launchpad.net/bugs/1464160
 [2] - 
 http://docs.openstack.org/kilo/config-reference/content/cinder-conf-changes-kilo.html
 [3] - https://review.openstack.org/#/c/159374/
 [4] - https://review.openstack.org/#/c/194719/
 
 --
 Mike Perez
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Updates from the Summit

[Gorka Eguileor]

On Wed, May 27, 2015 at 08:47:42AM -0400, Davanum Srinivas wrote:
 Hi Team,
 
 Here are the etherpads from the summit[1].

I remember that in Taskflow's Fishbowl session we discussed not only
pause/yield option but abort/cancel for long running tasks as well, but
reviewing the Etherpad now I don't see it there.

Should I just add it to Ideas for Liberty section or there's a
specific reason why it wasn't included?

Cheers,
Gorka.

 Some highlights are as follows:
 Oslo.messaging : Took status of the existing zmq driver, proposed a
 new driver in parallel to the existing zmq driver. Also looked at
 possibility of using Pika with RabbitMQ. Folks from pivotal promised
 to help with our scenarios as well.
 Oslo.rootwrap : Debated daemon vs a new privileged service. The Nova
 change to add rootwrap as daemon is on hold pending progress on the
 privsep proposal/activity.
 Oslo.versionedobjects : We had a nice presentation from Dan about what
 o.vo can do and a deepdive into what we could do in next release.
 Taskflow : Josh and team came up with several new features and how to
 improve usability
 
 We will also have several new libraries in Liberty (oslo.cache,
 oslo.service, oslo.reports, futurist, automaton etc). We talked about
 our release processes, functional testing, deprecation strategies and
 debated a but about how best to move to async models as well. Please
 see etherpads for detailed information.
 
 thanks,
 dims
 
 [1] https://wiki.openstack.org/wiki/Design_Summit/Liberty/Etherpads#Oslo
 
 -- 
 Davanum Srinivas :: https://twitter.com/dims
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [Nova] Quota delete API behavior and incorrect Quota usage reported after deletion

[Gorka Eguileor]

On Wed, Apr 08, 2015 at 10:13:27PM +0200, Gorka Eguileor wrote:
 Hi all,
 
 This message is in relation with a bug on Quota deletion API that
 affects both Nova [1] and Cinder [2], so we should discuss together
 what's the desired solution as to be consistent in both projects.
 
 Currently Quota deletion API removes not only Quota limits, but also
 Current Quota Usage and Quota Reservations. Which, from an Operator's
 point of view doesn't make that much sense, since they expect
 preservation of Usage and Reservations.
 
 I first created a patch for Cinder [3], then seeing that Nova's issue
 was the same I created one for Nova as well [4], but the solution was
 not unanimously accepted, and it was suggested that a new endpoint
 should be created for this new behavior (only deleting Quota limits).
 
 My reasoning for not creating a new endpoint in the first place and
 changing current endpoint instead, is that I saw this as a bug, not a
 new feature; I believe delete endpoint, like create, is only meant to
 affect Quota limits, as Usage and Reservations are handled by OpenStack
 itself. If you cannot create Quota usage you shouldn't be able to
 manually delete them either.
 
 Some additional topics were discussed on IRC and Gerrit:
 - Shouldn't delete set quota limits to unlimited instead of deleting
   them and thus apply default quota limits?: This is a matter of how the
   Quota delete is understood, as Delete quotas and leave no quota
   limits, not even defaults or just Delete additional quota limits
   that override defaults.
 - What about cascade deleting a tenant? Wouldn't it need to delete Usage
   and Reservations with the same API call?: Since Quota Reservations and
   Usage are handled by OpenStack, once related resources are deleted so
   will be the pertinent Reservations and Usage Quotas.
 
 In the matter of setting Quotas to unlimited on deletion I believe we
 should keep current behavior, which means it would use defaults or any
 other quotas that are in place, for example you delete a User's Quota
 limits, but Tenant's limits would still apply.
 
 As I see it, we should decide if it's OK to change existing endpoint or
 if, as it was suggested, we should create a new endpoint with a more
 pertinent name, like something related to reset quota limits.
 
 I, for one, believe we should change current behavior as it's not doing
 what it's meant to do. But I must admit that my understanding of how
 this endpoint is currently being used and how such a decision affects
 services is limited.
 
 Anyway, I have no problem changing the patches to whatever we decide is
 best.
 
 
 Cheers,
 Gorka.
 
 
 [1]: https://bugs.launchpad.net/nova/+bug/1410032
 [2]: https://bugs.launchpad.net/cinder/+bug/1410034
 [3]: https://review.openstack.org/#/c/162722/
 [4]: https://review.openstack.org/#/c/163423/
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Update on the Cinder side: This was discussed in a meeting [1], we
agreed to change the API and the patch to fix this has already been
merged.

The reasons why we considered that changing the API was acceptable were:
- It was considered to be a bug: Unlike in Nova, API documentation is
  more explicit; stating that after deletion it would return to default
  values, which implies that it's only talking about limits.
- It falls within the API change guidelines [2], like the Bugfixed OK
  example that modifies the counting of hosts.

Nova patch has been updated with the same solution in case it is
considered a valid fix.

Cheers,
Gorka.


[1]: 
http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-04-29-16.00.txt
[2]: https://wiki.openstack.org/wiki/APIChangeGuidelines

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev