Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Suresh Vinapamula]

Hi,

I did some investigation last week, why this was happening before I file
the bug. Here are my observations and I would like to work on it. Any
guidance is appreciated.


Upgrade procedure:
In my setup I have openstack controller on one node, neutron on another
node and rest of the nodes are computes. Assume they are running juno. As
the first step,
a) Bringup up a new openstack node in kilo
b) Migrate database from juno to kilo.
c) Run xxx manage db sync commands.
d) Set upgrade levels to juno
e) Migrate neutron and computes to the newer controller.

second step.
a) Upgrade neutron, and then start compute upgrades one at a time.

third step,
a) Finalize the upgrade by clearing upgrade levels and restarting services.
b) Issue migrate flavor data command.

This procedure is openstack side by side upgraded and neutron and computes
inplace upgraded.

Issue:
VMs spawned after step1 and during step2 are not flavor migrated. So,
migration to liberty was failing.

Investigation:
flavor migrate command filters instance uuids in nova.instance_extra table
whose 'flavor' column is NULL and fill in with the necessary flavor
information and delete those respective rows from the
nova.instance_system_metadata associated with that VM.
For the VMs spawned after the first step and before step2, this flavor
information in nova.instance_extra and also respective flavor information
in nova.instance_system_metadata table.
Since the filter looks for 'flavor ' column as NULL, these instances are
not caught to delete entries in nova.instance_system_metadata. And presence
of this data is failing nova db sync while migrating to liberty.

Possible Solutions:
I feel we should broaden the filter so that instances spawned after step1
and during step 2 are also accounted for data translation. As a quick
verification, I tried this by having no filer and it worked. I know filter
give efficiency, but would it matter in this context or should we broaden
the filter?

Please let me know if I am going in the right direction.

thanks




On Thu, Sep 1, 2016 at 10:28 AM, Matt Riedemann 
wrote:

> On 9/1/2016 12:05 PM, Suresh Vinapamula wrote:
>
>> On 8/31/2016 4:17 PM, Dan Smith wrote:
>>
>> Thanks Dan for your response. While I do run that before I start
>> my
>> move to liberty, what I see is that it doesn't seem to flavor
>> migrate
>> meta data for the VMs that are spawned after controller upgrade
>> from
>> juno to kilo and before all computes upgraded from juno to kilo.
>> The
>> current work around is to delete those VMs that are spawned after
>> controller upgrade and before all computes upgrade, and then
>> initiate
>> liberty upgrade. Then it works fine.
>>
>> I can't think of any reason why that would be, or why it would be a
>> problem. Instances created after the controllers are upgraded should
>> not
>> have old-style flavor info, so they need not be touched by the
>> migration
>> code.
>>
>> Maybe filing a bug is in order describing what you see?
>>
>> --Dan
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.op
>> enstack.org?subject:unsubscribe
>> > >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> Also, are you running with the latest kilo patch update? There were
>> some bug fixes backported after the release from what I remember.
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>> Hi Matt,
>>
>> Thanks for the response. I am currently using 2015.1.2. From the release
>> notes of 2015.1.4 https://wiki.openstack.org/wiki/ReleaseNotes/2015.1.4,
>> I can't correlate any fixes with this issue. Am I missing something?
>>
>> thanks
>>
>> Suresh
>>
>>
> The things I was looking for look like they are in 2015.1.2 so you should
> be OK there for at least known issues.
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Matt Riedemann]

On 9/1/2016 12:05 PM, Suresh Vinapamula wrote:

On 8/31/2016 4:17 PM, Dan Smith wrote:

Thanks Dan for your response. While I do run that before I start my
move to liberty, what I see is that it doesn't seem to flavor migrate
meta data for the VMs that are spawned after controller upgrade from
juno to kilo and before all computes upgraded from juno to kilo. The
current work around is to delete those VMs that are spawned after
controller upgrade and before all computes upgrade, and then initiate
liberty upgrade. Then it works fine.

I can't think of any reason why that would be, or why it would be a
problem. Instances created after the controllers are upgraded should not
have old-style flavor info, so they need not be touched by the migration
code.

Maybe filing a bug is in order describing what you see?

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Also, are you running with the latest kilo patch update? There were
some bug fixes backported after the release from what I remember.

--

Thanks,

Matt Riedemann

Hi Matt,

Thanks for the response. I am currently using 2015.1.2. From the release notes 
of 2015.1.4 https://wiki.openstack.org/wiki/ReleaseNotes/2015.1.4, I can't 
correlate any fixes with this issue. Am I missing something?

thanks

Suresh



The things I was looking for look like they are in 2015.1.2 so you 
should be OK there for at least known issues.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Suresh Vinapamula]

On 8/31/2016 4:17 PM, Dan Smith wrote:

Thanks Dan for your response. While I do run that before I start my
move to liberty, what I see is that it doesn't seem to flavor migrate
meta data for the VMs that are spawned after controller upgrade from
juno to kilo and before all computes upgraded from juno to kilo. The
current work around is to delete those VMs that are spawned after
controller upgrade and before all computes upgrade, and then initiate
liberty upgrade. Then it works fine.

I can't think of any reason why that would be, or why it would be a
problem. Instances created after the controllers are upgraded should not
have old-style flavor info, so they need not be touched by the migration
code.

Maybe filing a bug is in order describing what you see?

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Also, are you running with the latest kilo patch update? There were some bug
fixes backported after the release from what I remember.

--

Thanks,

Matt Riedemann

Hi Matt,

Thanks for the response. I am currently using 2015.1.2. From the
release notes of 2015.1.4
https://wiki.openstack.org/wiki/ReleaseNotes/2015.1.4, I can't
correlate any fixes with this issue. Am I missing something?

thanks

Suresh
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Matt Riedemann]

On 8/31/2016 4:17 PM, Dan Smith wrote:

Thanks Dan for your response. While I do run that before I start my
move to liberty, what I see is that it doesn't seem to flavor migrate
meta data for the VMs that are spawned after controller upgrade from
juno to kilo and before all computes upgraded from juno to kilo. The
current work around is to delete those VMs that are spawned after
controller upgrade and before all computes upgrade, and then initiate
liberty upgrade. Then it works fine.


I can't think of any reason why that would be, or why it would be a
problem. Instances created after the controllers are upgraded should not
have old-style flavor info, so they need not be touched by the migration
code.

Maybe filing a bug is in order describing what you see?

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Also, are you running with the latest kilo patch update? There were some 
bug fixes backported after the release from what I remember.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Suresh Vinapamula]

Sure, will file a bug with my observations.

On Wed, Aug 31, 2016 at 2:17 PM, Dan Smith  wrote:

> > Thanks Dan for your response. While I do run that before I start my
> > move to liberty, what I see is that it doesn't seem to flavor migrate
> > meta data for the VMs that are spawned after controller upgrade from
> > juno to kilo and before all computes upgraded from juno to kilo. The
> > current work around is to delete those VMs that are spawned after
> > controller upgrade and before all computes upgrade, and then initiate
> > liberty upgrade. Then it works fine.
>
> I can't think of any reason why that would be, or why it would be a
> problem. Instances created after the controllers are upgraded should not
> have old-style flavor info, so they need not be touched by the migration
> code.
>
> Maybe filing a bug is in order describing what you see?
>
> --Dan
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Suresh Vinapamula]

>> While migrate_flavor_data seem to flavor migrate meta data of the VMs
>> that were spawned before upgrade procedure, it doesn't seem to flavor
>> migrate for the VMs that were spawned during the upgrade procedure more
>> specifically after openstack controller upgrade and before compute
>> upgrade. Am I missing something here or is it by intention?

>You can run the flavor migration as often as you need, and can certainly
>run it after your last compute is upgraded before you start to move into
>liberty.
>
>--Dan


Thanks Dan for your response. While I do run that before I start my
move to liberty, what I see is that it doesn't seem to flavor migrate
meta data for the VMs that are spawned after controller upgrade from
juno to kilo and before all computes upgraded from juno to kilo. The
current work around is to delete those VMs that are spawned after
controller upgrade and before all computes upgrade, and then initiate
liberty upgrade. Then it works fine.


Suresh
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Dan Smith]

> Thanks Dan for your response. While I do run that before I start my
> move to liberty, what I see is that it doesn't seem to flavor migrate
> meta data for the VMs that are spawned after controller upgrade from
> juno to kilo and before all computes upgraded from juno to kilo. The
> current work around is to delete those VMs that are spawned after
> controller upgrade and before all computes upgrade, and then initiate
> liberty upgrade. Then it works fine.

I can't think of any reason why that would be, or why it would be a
problem. Instances created after the controllers are upgraded should not
have old-style flavor info, so they need not be touched by the migration
code.

Maybe filing a bug is in order describing what you see?

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] migrate_flavor_data doesn't flavor migrate meta data of VMs spawned during upgrade.

[Dan Smith]

> While migrate_flavor_data seem to flavor migrate meta data of the VMs
> that were spawned before upgrade procedure, it doesn't seem to flavor
> migrate for the VMs that were spawned during the upgrade procedure more
> specifically after openstack controller upgrade and before compute
> upgrade. Am I missing something here or is it by intention?

You can run the flavor migration as often as you need, and can certainly
run it after your last compute is upgraded before you start to move into
liberty.

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev