[openstack-dev] [nova] Bug triage and tag owner list

[Timofei Durakov]

Hi team,

I'd like to share a link [1] to Nova Bug Triage wiki page. This page
contains a tag owner list and instructions that allow subscribing not for
all nova bugs, but for a certain tag. It would be great if all tags in a
table have an owner, so if you are interested in some of them, and could
help triaging certain tagged bugs, you are welcome to edit the page and
subscribe to a tag.

Timofey

[1] - https://wiki.openstack.org/wiki/Nova/BugTriage
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Live migration sub-team lead

[Timofei Durakov]

Paul,

thanks for leading live migration sub-team. A lot of work were done in that
area for last year!

Best,
Timofey



On Mon, Oct 3, 2016 at 7:22 PM, Murray, Paul (HP Cloud) <pmur...@hpe.com>
wrote:

> Hi All,
>
> I have had the pleasure of chairing the live migration sub-team for the
> last year. Recently, as some of you will have noticed, my time has been
> stretched to the extent that I have started to neglect the task. Timofei
> Durakov has stood in for me on several occasions recently and has now
> agreed to take over full time. I had planned to bring this up in the
> meeting tomorrow, but yet again I am unable to attend.
>
> So thanks to all in the sub-team for being so self-organising and making
> the meetings so easy. Please turn to Timofei as your first point of contact
> for all things live migration from now on.
>
> Best regards,
> Paul
>
> P.S. I will be in the meetings when I can.
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Timofei Durakov]

Hi team,

I agree that it's kind of strange thing that nova dumps xml definition to
the disk but doesn't use it(at least I do not aware of it).
How the proposed changed would be aligned with other drivers? The worst
case I could imagine here is that libvirt has an xml as a source of truth,
while others use nova for the same purposes. Taking that into account, the
question here would be:  why not to store all required information(e.g.
boot order) in DB instead?

Timofey


On Fri, Sep 30, 2016 at 7:36 PM, Murray, Paul (HP Cloud) 
wrote:

>
>
> From: Matthew Booth
> Reply-To: "openstack-dev@lists.openstack.org"
> Date: Friday, 30 September 2016 at 17:03
> To: "openstack-dev@lists.openstack.org"
> Subject: Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain
> XML canonical
>
> On Fri, Sep 30, 2016 at 4:38 PM, Murray, Paul (HP Cloud) 
> wrote:
>
>>
>>
>>
>>
>>
>> On 27/09/2016, 18:12, "Daniel P. Berrange"  wrote:
>>
>> >On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
>> >> On 09/27/2016 10:17 AM, Matthew Booth wrote:
>> >>
>> >> > I think we should be able to create a domain, but once created we
>> should never
>> >> > redefine a domain. We can do adding and removing devices dynamically
>> using
>> >> > libvirt's apis, secure in the knowledge that libvirt will persist
>> this for us.
>> >> > When we upgrade the host, libvirt can ensure we don't break guests
>> which are on
>> >> > it. Evacuate should be pretty much the only reason to start again.
>> >>
>> >> Sounds interesting.  How would you handle live migration?
>> >>
>> >> Currently we regenerate the XML file on the destination from the nova
>> DB.  I
>> >> guess in your proposal we'd need some way of copying the XML file from
>> the
>> >> source to the dest, and then modifying the appropriate segments to
>> adjust
>> >> things like CPU/NUMA pinning?
>> >
>> >Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
>> >new persistent XML on the target host automatically.
>>
>> We have a problem migrating rescued instances that has a fix in progress
>> based
>> on regenerating the xml on unrescue, see:
>>
>>  https://blueprints.launchpad.net/nova/+spec/live-migrate-re
>> scued-instances
>>
>> That might be another case for generating the xml.
>>
>
> I thought it was a counter-argument (unless I've misunderstood). If you
> migrate the instance as-is without modification, you don't need to worry
> about whether it's currently a rescue instance. This problem goes away.
>
> The major complication I can think of is things which genuinely must
> change during a migration, for example cpu pinning.
>
>
> Rescue currently saves the libvirt xml in a separate file and replaces it
> with new xml to define a vm with a rescue boot disk; to unrescue it
> replaces the libvirt xml used for the rescue with the saved file to go back
> to the original state. On migration that saved xml would be lost because
> its an arbitrary file that is not handled in the migration. The spec above
> relies on the fact that we do not need to save it or copy it because we can
> recreate it from nova. So yes, the migration just works for the rescued vm,
> but when it is unrescued the original libvirt xml would be regenerated.
>
> Paul
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Bug team meeting

[Timofei Durakov]

Hello everyone,

Bug team meeting on Sept. 13 is canceled. So the next time it will be Sept.
20 18:00 UTC.
By that time I'll update agenda [1], everyone is welcome to do the same.

Timofey

[1] - https://wiki.openstack.org/wiki/Meetings/Nova/BugsTeam
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] resignation from bug czar role

[Timofei Durakov]

Hi, folks,

Thanks, Markus, for doing this job! I'm interested in this activity.

Timofey

On Mon, Sep 5, 2016 at 7:20 PM, Sylvain Bauza  wrote:

>
>
> Le 05/09/2016 13:19, Markus Zoeller a écrit :
>
>> TL;DR: bug czar role for Nova is vacant from now on
>>
>>
>> After doing bug triage for ~1 year, which was quiet interesting, it's
>> time for me to move to different topics. My tasks within the company
>> internal team are shifting too. Unfortunately less Nova for me in the
>> next (hopefully short) time. That means I'm resigning from the bug czar
>> role as of now.
>>
>>
>> Observations in this timeframe
>> --
>>
>> * The quality of most of the bug reports could be better. Very often
>> they are not actionable. A bug report which isn't actionable burns
>> resources without any benefit. The pattern I've seen is:
>>  * 1/3 : invalid because they are support requests or a wrong
>> understanding
>>  * 1/3 : could be reasonable but essential information is missing
>>  * 1/3 : sounds reasonable + has a little info, should be looked at
>>Very few follow this template which is shown when you open a new
>> report: https://wiki.openstack.org/wiki/Nova/BugsTeam/BugReportTemplate
>>
>> * We get ~40 new bug reports per week. With the current number of people
>> who do bug triage, the number of overall bug reports doesn't decline. I
>> started collecting data 6 months ago:
>>
>> http://45.55.105.55:3000/dashboard/db/openstack-bugs?from=
>> now-6M=1
>>
>> * I wish the cores would engage more in bug triaging. If one core every
>> other week would do the bug triage for 1 week, a core would have to do
>> that only once per dev cycle. I'm aware of the review backlog though :/
>>
>> * I wish more non-cores would engage more in bug triaging.
>>
>> * We don't have contacts for a lot of areas in Nova:
>>https://wiki.openstack.org/wiki/Nova/BugTriage#Tag_Owner_List
>>
>> * Keeping the bug reports in a consistent state is cumbersome:
>>http://45.55.105.55:8082/bugs-dashboard.html#tabInProgressStale
>>We could introduce more automation here.
>>
>>
>> Things we should continue
>> -
>>
>> * Bug reports older that the oldest supported stable release should be
>>expired. Maybe best when the EOL tag gets applied.
>>
>> https://github.com/openstack-infra/release-tools/blob/master
>> /expire_old_bug_reports.py
>>http://lists.openstack.org/pipermail/openstack-dev/2016-May
>> /095654.html
>>
>> * We never came to a real conclusion how the ops communicated the RFEs
>> to us. The way of using "wishlist" bug reports wasn't successful IMO.
>> The last proposal was to use the ops ML to bring an RFE into some
>> actionable shape and then create a backlog spec out of it.
>>http://lists.openstack.org/pipermail/openstack-dev/2016-Mar
>> ch/089365.html
>>
>>
>>
>> Things we should start
>> --
>>
>> * A cross-project discussion of (easy) ways to collect and send debug
>> data to upstream OpenStack. Almost no bug report in Nova had the result
>> of "sosreport" attached although we ask for that in the report template.
>>
>>
>>
>> Some last words
>> ---
>>
>> * Whoever wants to do the job next, I offer some kind of onboarding.
>>
>> * I'll push a change to remove the IRC meetings in the next few days:
>>http://eavesdrop.openstack.org/#Nova_Bugs_Team_Meeting
>>
>> * The tooling I used will still be available at:
>>https://github.com/markuszoeller/openstack/tree/master/
>> scripts/launchpad
>>
>> * My server which hosts some dashboards will still be available at:
>>http://45.55.105.55:3000/dashboard/db/openstack-bugs
>>http://45.55.105.55:8082/bugs-dashboard.html
>>http://45.55.105.55:8082/bugs-stats.html
>>
>> * I did an evaluation of Storyboard in July 2016 and it looks promising.
>> Give it a shot at: https://storyboard-dev.openstack.org/#!/project/2 If
>> you don't like something there, push a change, it's Python based.
>>
>> * I'll still hang out in the IRC channels, but don't expect much from me.
>>
>>
>> Thanks a lot to the people who helped making Nova a better project by
>> doing bug triage! Special thanks to auggy who put a lot(!) of effort
>> into that.
>>
>> See you (hopefully) in Barcelona!
>>
>
> As said on IRC, hope we'll still see you around, and see you in Barcelona.
> You made a great job !
>
> -Sylvain
>
>
> --
>> Regards,
>> Markus Zoeller (markus_z)
>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 

[openstack-dev] [nova] Live-migration subteam meeting

[Timofei Durakov]

Hello,

Next meeting will be on August 30, 14:00 UTC as usual. Agenda is available
here:
https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration#Agenda_for_next_meeting


If you have topics to discuss, please update it accordingly.

Timofey.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][cinder] Live-migration job status update

[Timofei Durakov]

Hello everyone,

As you may notice, gate-tempest-dsvm-multinode-live-migration job became
voting. It tests live-migration against no shared storage environment, and
uses Ubuntu Xenial. List of tests being executed:

   - tempest.api.compute.admin.test_live_migration.LiveBlockMigra
   tionTestJSON.test_live_block_migration
   - tempest.api.compute.admin.test_live_migration.LiveAutoBlockM
   igrationV225TestJSON.test_live_block_migration
   - tempest.api.compute.admin.test_live_migration.LiveBlockMigra
   tionTestJSON.test_live_block_migration_paused
   - tempest.api.compute.admin.test_live_migration.LiveAutoBlockM
   igrationV225TestJSON.test_live_block_migration_paused

test_volume_backed_live_migration is skipped due to [1].

Next step is to re-enable again testing against NFS shared storage for
ephemerals. There is a bug [2] that affects stability and requires
additional research(possible test could be excluded, as it's done now for
[1]). Also, there is a patch [3] in progress that adds testing of serial
consoles and live migration.
After that, the plan is to re-enable Ceph as a backend and consider of
including job for stable branches either.

Also, volunteers for fixing [1](the reason of [cinder] in a list of tags)
and [2] are required. If anyone is interested, you are very welcome to
participate in fixing that 2.

Thanks,
Timofey.

[1] - https://bugs.launchpad.net/nova/+bug/1524898
[2] - https://bugs.launchpad.net/nova/+bug/1590929
[3] - https://review.openstack.org/#/c/347471/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [infra] Intel NFV CI voting permission

[Timofei Durakov]

Hi everyone,

got a question on that Intel NFV CI: has statistics for stable branches
been checked either?
I have a patch to stable/liberty [1] that failed on all Intel NFV CI jobs 2
times in a row. Who could help me to figure out the root cause for that?

Thanks in advance,
Timofey.

[1] - https://review.openstack.org/#/c/358439/

On Mon, Aug 8, 2016 at 6:27 PM, Ptacek, MichalX 
wrote:

> Change deployed,
> Thanks for your comments,
>
> Michal
>
> -Original Message-
> From: Markus Zoeller [mailto:mzoel...@linux.vnet.ibm.com]
> Sent: Monday, August 08, 2016 3:15 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [nova] [infra] Intel NFV CI voting permission
>
> On 03.08.2016 22:02, Znoinski, Waldemar wrote:
> > Thanks
> > Much appreciated
> >
> > Making use of the opportunity here... what's the next big thing a CI
> > (like one testing NFV) should be doing? (multinode or there's
> > something else more important?)
>
> Two tiny nits on the comment the CI is giving a change:
>
> "Build failed (check pipeline). To recheck leave a comment with
> intel-nfv-ci recheck. For more info go to
> https://wiki.openstack.org/wiki/ThirdPartySystems/Intel-NFV-CI.;
>
> 1) Could you put the recheck command in quotation? Like:
>To recheck leave a comment with 'intel-nfv-ci recheck'.
> 2) The link to the wiki seems to be wrong (hyphen vs. underscore):
>https://wiki.openstack.org/wiki/ThirdPartySystems/Intel_NFV_CI
>
> --
> Regards, Markus Zoeller (markus_z)
>
> >  >-Original Message-
> >  >From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> >  >Sent: Wednesday, August 3, 2016 8:28 PM
> >  >To: openstack-dev@lists.openstack.org
> >  >Subject: Re: [openstack-dev] [nova] [infra] Intel NFV CI voting
> > permission  >  >On 8/3/2016 8:18 AM, Znoinski, Waldemar wrote:
> >  >> [WZ] Hi Matt, do we need, a Gerrit-style, +2 on that from someone?
> >  >> Infra Team doesn't keep these settings in their puppet/config
> > files on git -  >all the Gerrit changes are done via Gerrit GUI so
> > they rely on Cores to add CIs  >to the appropriate ci group, nova-ci in
> this case.
> >  >>
> >  >>
> >  >
> >  >Sorry, I wasn't sure what the next step here was, I guess it was the
> > nova-ci  >membership change, which is done now:
> >  >
> >  >https://review.openstack.org/#/admin/groups/511,members
> >  >
> >  >--
> >  >
> >  >Thanks,
> >  >
> >  >Matt Riedemann
> >  >
> >  >
> >  >__
> >  >
> >  >OpenStack Development Mailing List (not for usage questions)
> >  >Unsubscribe: OpenStack-dev-
> >  >requ...@lists.openstack.org?subject:unsubscribe
> >  >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > --
> > Intel Research and Development Ireland Limited Registered in Ireland
> > Registered Office: Collinstown Industrial Park, Leixlip, County
> > Kildare Registered Number: 308263
> >
> >
> > This e-mail and any attachments may contain confidential material for
> > the sole use of the intended recipient(s). Any review or distribution
> > by others is strictly prohibited. If you are not the intended
> > recipient, please contact the sender and delete all copies.
> >
> >
> > __
> >  OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> --
> Intel Research and Development Ireland Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
>
>
> This e-mail and any attachments may contain confidential material for the
> sole
> use of the intended recipient(s). Any review or distribution by others is
> strictly prohibited. If you are not the intended recipient, please contact
> the
> sender and delete all copies.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [nova] Removal of live_migration_flag and block_migration_flag config options

[Timofei Durakov]

If operator haven't explicitly defined  live_migration_tunnelled param in
nova.conf, after upgrade is done it's default value will be set to False.
If operator set this param explicitly, everything will be unchanged. To
notify about this change I'm proposing to use release notes, as It's
usually done. So there will be no upgrades impact related to this change.

On Tue, Aug 2, 2016 at 10:51 PM, Chris Friesen <chris.frie...@windriver.com>
wrote:

> On 08/02/2016 09:14 AM, Timofei Durakov wrote:
>
>> Hi,
>>
>> Taking into account everything above I'd prefer to see
>> live_migration_tunnelled(that corresponds to VIR_MIGRATE_TUNNELLED)
>> defaulted to
>> False. We just need to make a release note for this change, and on the
>> host
>> startup do LOG.warning to notify the operator that there are no tunnels
>> for
>> live-migration. For me, it will be enough. Then just put [1] on top of it.
>>
>
> How would upgrades work?  Presumably you'd have to get all the existing
> compute nodes able to handle un-tunnelled live migrations, then you'd
> live-migrate from the old compute nodes to the new ones using tunnelled
> migrations (where live migration is possible), but after that everything
> would be un-tunnelled?
>
> Chris
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Removal of live_migration_flag and block_migration_flag config options

[Timofei Durakov]

Hi,

Taking into account everything above I'd prefer to see
live_migration_tunnelled(that corresponds to VIR_MIGRATE_TUNNELLED)
defaulted to False. We just need to make a release note for this change,
and on the host startup do LOG.warning to notify the operator that there
are no tunnels for live-migration. For me, it will be enough. Then just put
[1] on top of it.

Thanks,
Timofey


On Tue, Aug 2, 2016 at 5:36 PM, Koniszewski, Pawel <
pawel.koniszew...@intel.com> wrote:

> In Mitaka development cycle 'live_migration_flag' and
> 'block_migration_flag' have been marked as deprecated for removal. I'm
> working on a patch [1] to remove both of them and want to ask what we
> should do with live_migration_tunnelled logic.
>
> The default configuration of both flags contain VIR_MIGRATE_TUNNELLED
> option. It is there to avoid the need to configure the network to allow
> direct communication between hypervisors. However, tradeoff is that it
> slows down all migrations by up to 80% due to increased number of memory
> copies and single-threaded encryption mechanism in Libvirt. By 80% here I
> mean that transfer between source and destination node is around 2Gb/s on a
> 10Gb network. I believe that this is a configuration issue and people
> deploying OpenStack are not aware that live migrations with this flag will
> not work. I'm not sure that this is something we wanted to achieve. AFAIK
> most operators are turning it OFF in order to make live migration usable.
>
> Going to a new flag that is there to keep possibility to turn tunneling on
> - Live_migration_tunnelled [2] which is a tri-state boolean - None, False,
> True:
>
> * True - means that live migrations will be tunneled through libvirt.
> * False - no tunneling, native hypervisor transport.
> * None - nova will choose default based on, e.g., the availability of
> native encryption support in the hypervisor. (Default value)
>
> Right now we don't have any logic implemented for None value which is a
> default value. So the question here is should I implement logic so that if
> live_migration_tunnelled=None it will still use VIR_MIGRATE_TUNNELLED if
> native encryption is not available? Given the impact of this flag I'm not
> sure that we really want to keep it there. Another option is to change
> default value of live_migration_tunnelled to be True. In both cases we will
> again end up with slower LM and people complaining that LM does not work at
> all in OpenStack.
>
> Thoughts?
>
> [1] https://review.openstack.org/#/c/334860/
> [2]
> https://github.com/openstack/nova/blob/be59c19c969acf6b25b0711f0ebfb26aaed0a171/nova/conf/libvirt.py#L107
>
> Kind Regards,
> Pawel Koniszewski
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest][nova] Microversions support coverage

[Timofei Durakov]

I think we could at least fix-up schema validation.

On Wed, Jul 6, 2016 at 5:51 PM, Matt Riedemann <mrie...@linux.vnet.ibm.com>
wrote:

> On 7/6/2016 9:37 AM, Timofei Durakov wrote:
>
>> Hi,
>>
>> there are several patches in a tempest that improves micro
>> versions coverage for Nova REST API:
>> https://review.openstack.org/#/c/337598
>> https://review.openstack.org/#/c/338248
>> https://review.openstack.org/#/c/287605
>> https://review.openstack.org/#/c/338256
>> https://review.openstack.org/#/c/233176
>> https://review.openstack.org/#/c/327191
>>
>> The purpose of this thread is to align our efforts in merging these and
>> also ask other contributors to join this activity, as it will improve
>> nova tempest coverage.
>>
>>
>> Timofey.
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> Are these all needed in Tempest? We don't need 100% nova microversion REST
> API coverage in Tempest if there are things that we can test inside nova,
> like in the functional tests. The only thing we don't have in nova is the
> response schema validation. For the most part I only ask for a microversion
> test in Tempest when it requires other services like cinder or neutron, or
> it hits new behavior in the virt drivers.
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest][nova] Microversions support coverage

[Timofei Durakov]

Hi,

there are several patches in a tempest that improves micro
versions coverage for Nova REST API:
https://review.openstack.org/#/c/337598
https://review.openstack.org/#/c/338248
https://review.openstack.org/#/c/287605
https://review.openstack.org/#/c/338256
https://review.openstack.org/#/c/233176
https://review.openstack.org/#/c/327191

The purpose of this thread is to align our efforts in merging these and
also ask other contributors to join this activity, as it will improve nova
tempest coverage.


Timofey.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][infra][ci] bulk repeating a test job on a single review in parallel ?

[Timofei Durakov]

This option is already available for live-migration job, as it's hooks are
under nova project.

On Fri, Jul 1, 2016 at 5:13 PM, Jeremy Stanley  wrote:

> Have you considered just writing a throwaway devstack-gate change
> which overrides the gate_hook to run that one suspect Tempest test,
> say, a thousand times in a loop? Would be far more efficient if you
> don't need to be concerned with all the environment setup/teardown
> overhead.
> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][infra][ci] bulk repeating a test job on a single review in parallel ?

[Timofei Durakov]

Hi,

talking about live-migration job. If you are OK to re-run tempest tests
multiple times on single environment you could wrap this line:
https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh#L30
in for-cycle.

Timofey.

On Fri, Jul 1, 2016 at 10:31 AM, Spencer Krum  wrote:

>
> This is some neat work. I like it.
>
> We have a graph of nodepool nodes in use in grafana[1]. Looking at a 30
> day window, its pretty easy to tell that the weekends are low activity.
> Running big tests like this on a Saturday would be great. Of course,
> there won't be much infra support on that day.
>
> What we can do for running just the multinode migration test is create a
> new pipeline(maybe call it heavyload or something) and attach 100 copies
> of the test to that pipeline. Then you only need one gerrit change and
> you issue a 'check heavyload' comment on the review and it will run all
> 100 copies of the test. Gate-tempest-dsvm-multinode-live-migration-1,
> gate-tempest-dsvm-multinode-live-migration-2 ... and so on. I think this
> is better than reusing the experimental pipeline since people will try
> to check experimental on nova from time to time and probably don't
> expect to use 200 virtual machines. It's not a very clean suggestion,
> but it would work. Someone else might have a better idea.
>
>
>
> [1]
> http://grafana.openstack.org/dashboard/db/nodepool?panelId=4=1464741261723=1467333261724
>
> --
>   Spencer Krum
>   n...@spencerkrum.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Let me know if you have an approved spec but unapproved blueprint

[Timofei Durakov]

Hi

https://blueprints.launchpad.net/nova/+spec/remove-compute-compute-communication
- bp is not approved, but the spec is.

Timofey

On Tue, Jun 14, 2016 at 3:47 AM, joehuang  wrote:

> Hi, Matt,
>
> Thank you for the clarification.
>
> Best Regards
> Chaoyi Huang ( Joe Huang )
>
>
> -Original Message-
> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> Sent: Monday, June 13, 2016 9:07 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [nova] Let me know if you have an approved
> spec but unapproved blueprint
>
> On 6/12/2016 7:48 PM, joehuang wrote:
> > Hello,
> >
> > This spec is not approved yet:
> > https://review.openstack.org/#/c/295595/
> >
> > But the BP is approved:
> > https://blueprints.launchpad.net/nova/+spec/expose-quiesce-unquiesce-a
> > pi
> >
> > Don't know how to deal with the spec now. Is this spec killed? Should
> Nova support application level consistency snapshot for disaster recovery
> purpose or not?
> >
> > Best Regards
> > Chaoyi Huang ( Joe Huang )
> >
> > -Original Message-
> > From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> > Sent: Sunday, June 12, 2016 9:08 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: [openstack-dev] [nova] Let me know if you have an approved
> > spec but unapproved blueprint
> >
> > I've come across several changes up for review that are tied to Newton
> blueprints which have specs approved but the blueprints in launchpad are
> not yet approved.
> >
> > If you have a spec that was approved for Newton but your blueprint in
> launchpad isn't approved yet, please ping me (mriedem) in IRC or reply to
> this thread to get it approved and tracked for the Newton release.
> > It's important (at least to me) that we have an accurate representation
> of how much work we're trying to get done this release, especially with
> non-priority feature freeze coming up in three weeks.
> >
>
> Neither the spec nor the blueprint is approved. The blueprint was
> previously approved in mitaka but is not for newton, with reasons in the
> spec review for newton.
>
> At this point we're past non-priority spec approval freeze so this isn't
> going to get in for newton. There are a lot of concerns about this one so
> it's going to be tabled for at least this release, we can revisit in ocata,
> but it adds a lot of complexity and it's more than we're willing to take on
> right now given everything else planned for this release.
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Deprecating the live_migration_flag and block_migration_flag config options

[Timofei Durakov]

I've submitted one more patch with potential fix:

https://review.openstack.org/#/c/326258/

Timofey

On Mon, Jun 6, 2016 at 11:58 PM, Timofei Durakov <tdura...@mirantis.com>
wrote:

> On Mon, Jun 6, 2016 at 11:26 PM, Matt Riedemann <
> mrie...@linux.vnet.ibm.com> wrote:
>
>> On 6/6/2016 12:15 PM, Matt Riedemann wrote:
>>
>>> On 1/8/2016 12:28 PM, Mark McLoughlin wrote:
>>>
>>>> On Fri, 2016-01-08 at 14:11 +, Daniel P. Berrange wrote:
>>>>
>>>>> On Thu, Jan 07, 2016 at 09:07:00PM +, Mark McLoughlin wrote:
>>>>>
>>>>>> On Thu, 2016-01-07 at 12:23 +0100, Sahid Orentino Ferdjaoui
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Jan 04, 2016 at 09:12:06PM +, Mark McLoughlin
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> commit 8ecf93e[1] got me thinking - the live_migration_flag
>>>>>>>> config option unnecessarily allows operators choose arbitrary
>>>>>>>> behavior of the migrateToURI() libvirt call, to the extent
>>>>>>>> that we allow the operator to configure a behavior that can
>>>>>>>> result in data loss[1].
>>>>>>>>
>>>>>>>> I see that danpb recently said something similar:
>>>>>>>>
>>>>>>>> https://review.openstack.org/171098
>>>>>>>>
>>>>>>>> "Honestly, I wish we'd just kill off  'live_migration_flag'
>>>>>>>> and 'block_migration_flag' as config options. We really
>>>>>>>> should not be exposing low level libvirt API flags as admin
>>>>>>>> tunable settings.
>>>>>>>>
>>>>>>>> Nova should really be in charge of picking the correct set of
>>>>>>>> flags for the current libvirt version, and the operation it
>>>>>>>> needs to perform. We might need to add other more sensible
>>>>>>>> config options in their place [..]"
>>>>>>>>
>>>>>>>
>>>>>>> Nova should really handle internal flags and this serie is
>>>>>>> running in the right way.
>>>>>>>
>>>>>>> ...
>>>>>>>>
>>>>>>>
>>>>>>> 4) Add a new config option for tunneled versus native:
>>>>>>>>
>>>>>>>> [libvirt] live_migration_tunneled = true
>>>>>>>>
>>>>>>>> This enables the use of the VIR_MIGRATE_TUNNELLED flag. We
>>>>>>>> have historically defaulted to tunneled mode because it
>>>>>>>> requires the least configuration and is currently the only
>>>>>>>> way to have a secure migration channel.
>>>>>>>>
>>>>>>>> danpb's quote above continues with:
>>>>>>>>
>>>>>>>> "perhaps a "live_migration_secure_channel" to indicate that
>>>>>>>> migration must use encryption, which would imply use of
>>>>>>>> TUNNELLED flag"
>>>>>>>>
>>>>>>>> So we need to discuss whether the config option should
>>>>>>>> express the choice of tunneled vs native, or whether it
>>>>>>>> should express another choice which implies tunneled vs
>>>>>>>> native.
>>>>>>>>
>>>>>>>> https://review.openstack.org/263434
>>>>>>>>
>>>>>>>
>>>>>>> We probably have to consider that operator does not know much
>>>>>>> about internal libvirt flags, so options we are exposing for
>>>>>>> him should reflect benefice of using them. I commented on your
>>>>>>> review we should at least explain benefice of using this option
>>>>>>> whatever the name is.
>>>>>>>
>>>>>>
>>>>>> As predicted, plenty of discussion on this point in the review
>>>>>> :)
>>>>>>
>>>>>> You're right that we don't give the operator any guidance in the
>>>>>> help message about how to choose true or false for this:
>>>>>>
>>

Re: [openstack-dev] [nova][libvirt] Deprecating the live_migration_flag and block_migration_flag config options

[Timofei Durakov]

On Mon, Jun 6, 2016 at 11:26 PM, Matt Riedemann 
wrote:

> On 6/6/2016 12:15 PM, Matt Riedemann wrote:
>
>> On 1/8/2016 12:28 PM, Mark McLoughlin wrote:
>>
>>> On Fri, 2016-01-08 at 14:11 +, Daniel P. Berrange wrote:
>>>
 On Thu, Jan 07, 2016 at 09:07:00PM +, Mark McLoughlin wrote:

> On Thu, 2016-01-07 at 12:23 +0100, Sahid Orentino Ferdjaoui
> wrote:
>
>> On Mon, Jan 04, 2016 at 09:12:06PM +, Mark McLoughlin
>> wrote:
>>
>>> Hi
>>>
>>> commit 8ecf93e[1] got me thinking - the live_migration_flag
>>> config option unnecessarily allows operators choose arbitrary
>>> behavior of the migrateToURI() libvirt call, to the extent
>>> that we allow the operator to configure a behavior that can
>>> result in data loss[1].
>>>
>>> I see that danpb recently said something similar:
>>>
>>> https://review.openstack.org/171098
>>>
>>> "Honestly, I wish we'd just kill off  'live_migration_flag'
>>> and 'block_migration_flag' as config options. We really
>>> should not be exposing low level libvirt API flags as admin
>>> tunable settings.
>>>
>>> Nova should really be in charge of picking the correct set of
>>> flags for the current libvirt version, and the operation it
>>> needs to perform. We might need to add other more sensible
>>> config options in their place [..]"
>>>
>>
>> Nova should really handle internal flags and this serie is
>> running in the right way.
>>
>> ...
>>>
>>
>> 4) Add a new config option for tunneled versus native:
>>>
>>> [libvirt] live_migration_tunneled = true
>>>
>>> This enables the use of the VIR_MIGRATE_TUNNELLED flag. We
>>> have historically defaulted to tunneled mode because it
>>> requires the least configuration and is currently the only
>>> way to have a secure migration channel.
>>>
>>> danpb's quote above continues with:
>>>
>>> "perhaps a "live_migration_secure_channel" to indicate that
>>> migration must use encryption, which would imply use of
>>> TUNNELLED flag"
>>>
>>> So we need to discuss whether the config option should
>>> express the choice of tunneled vs native, or whether it
>>> should express another choice which implies tunneled vs
>>> native.
>>>
>>> https://review.openstack.org/263434
>>>
>>
>> We probably have to consider that operator does not know much
>> about internal libvirt flags, so options we are exposing for
>> him should reflect benefice of using them. I commented on your
>> review we should at least explain benefice of using this option
>> whatever the name is.
>>
>
> As predicted, plenty of discussion on this point in the review
> :)
>
> You're right that we don't give the operator any guidance in the
> help message about how to choose true or false for this:
>
> Whether to use tunneled migration, where migration data is
> transported over the libvirtd connection. If True, we use the
> VIR_MIGRATE_TUNNELLED migration flag
>
> libvirt's own docs on this are here:
>
> https://libvirt.org/migration.html#transport
>
> which emphasizes:
>
> - the data copies involved in tunneling - the extra configuration
> steps required for native - the encryption support you get when
> tunneling
>
> The discussions I've seen on this topic wrt Nova have revolved
> around:
>
> - that tunneling allows for an encrypted transport[1] - that
> qemu's NBD based drive-mirror block migration isn't supported
> using tunneled mode, and that danpb is working on fixing this
> limitation in libvirt - "selective" block migration[2] won't work
> with the fallback qemu block migration support, and so won't
> currently work in tunneled mode
>

 I'm not working on fixing it, but IIRC some other dev had proposed
 patches.


> So, the advise to operators would be:
>
> - You may want to choose tunneled=False for improved block
> migration capabilities, but this limitation will go away in
> future. - You may want to choose tunneled=False if you wish to
> trade and encrypted transport for a (potentially negligible)
> performance improvement.
>
> Does that make sense?
>
> As for how to name the option, and as I said in the review, I
> think it makes sense to be straightforward here and make it
> clearly about choosing to disable libvirt's tunneled transport.
>
> If we name it any other way, I think our explanation for
> operators will immediately jump to explaining (a) that it
> influences the TUNNELLED flag, and (b) the differences between
> the tunneled and native transports. So, if we're going to have to
> talk about tunneled versus native, why obscure that detail?
>

Re: [openstack-dev] [Nova] State machines in Nova

[Timofei Durakov]

>From my sight, I concerned proposed transition from option #1 to option #2.
because it would be quite big change. So I wonder, has any component team
implemented such transition. Open questions:

   - upgrades story potential issues
   - dealing with clients(?)
   - promoting state machine from verification of states to conductor of
   the task(success stories)

Timofey

On Wed, Jun 1, 2016 at 12:51 PM, Miles Gould <mgo...@redhat.com> wrote:

> On 31/05/16 21:03, Timofei Durakov wrote:
>
>> there is blueprint[1] that was approved during Liberty and resubmitted
>> to Newton(with spec[2]).
>> The idea is to define state machines for operations as live-migration,
>> resize, etc. and to deal with them operation states.
>>
>
> +1 to introducing an explicit state machine - IME they make complex logic
> much easier to reason about. However, think carefully about how you'll make
> changes to that state machine later. In Ironic, this is an ongoing problem:
> every time we change the state machine, we have to decide whether to lie to
> older clients (and if so, what lie to tell them), or whether to present
> them with the truth (and if so, how badly they'll break). AIUI this would
> be a much smaller problem if we'd considered this possibility carefully at
> the beginning.
>
> Miles
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] State machines in Nova

[Timofei Durakov]

Hi team,

there is blueprint[1] that was approved during Liberty and resubmitted to
Newton(with spec[2]).
The idea is to define state machines for operations as live-migration,
resize, etc. and to deal with them operation states.
The spec PoC patches are overall good. At the same time I think is will be
good to get agreement on the usage of state-machines in Nova.
There are 2 options:

   - implement proposed change and use state machines to deal with states
   only
   - procs:
 - could be implemented/merged right now
 - cleans up states for migrations
  - cons:
 - state machine only deal with states, and it will be hard to
 build on top of it task API, as bp [1] was designed for another thing.


   - use state machines in Task API(which I'm going to work on during next
   release):
  - procs:
 - Task API will orchestrate and deal with long running tasks
 - usage state-machines could help with actions
 rollbacks/retries/etc.
  - cons:
 - big amount of work
 - requires time.

I'd like to discuss these options in this thread.

Timofey

[1] -
https://blueprints.launchpad.net/openstack/?searchtext=migration-state-machine
[2] - https://review.openstack.org/#/c/320849/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Live Migration: Austin summit update

[Timofei Durakov]

We've tested live migration of instances under load for following features:
 - Auto-converge
 - XBZRLE compression
Tests were done against qemu 2.5. Instances had 2GB ram and stress tool was
used to emulate load:

$ stress -m 1 --vm-bytes 200M --vm-keep

None of them could be called silver-bullet for live-migration process.
Load mentioned above made live-migration process not that stable as
wanted.
I've also planned to tests all features that Daniel mentioned and it would
be intresting
to compare results.

Timofey.

On Wed, May 4, 2016 at 11:48 AM, Daniel P. Berrange 
wrote:

> On Tue, May 03, 2016 at 04:16:43PM -0600, Chris Friesen wrote:
> > On 05/03/2016 03:14 AM, Daniel P. Berrange wrote:
> >
> > >There are currently many options for live migration with QEMU that can
> > >assist in completion
> >
> > 
> >
> > >Given this I've spent the last week creating an automated test harness
> > >for QEMU upstream which triggers migration with an extreme guest CPU
> > >load and measures the performance impact of these features on the guest,
> > >and whether the migration actually completes.
> > >
> > >I hope to be able to publish the results of this investigation this week
> > >which should facilitate us in deciding which is best to use for
> OpenStack.
> > >The spoiler though is that all the options are pretty terrible, except
> for
> > >post-copy.
> >
> > Just to be clear, it's not really CPU load that's the issue though,
> right?
> >
> > Presumably it would be more accurate to say that the issue is the rate at
> > which unique memory pages are being dirtied and the total number of dirty
> > pages relative to your copy bandwidth.
> >
> > This probably doesn't change the results though...at a high enough dirty
> > rate you either pause the VM to keep it from dirtying more memory or you
> > post-copy migrate and dirty the memory on the destination.
>
> Yes that's correct - I should have been more explicit. A high rate of
> dirtying memory implies high CPU load, but high CPU load does not imply
> high rate of dirtying memory. My stress test used for benchmarking is
> producing a high rate of dirtying memory.
>
> Regards,
> Daniel
> --
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org  -o- http://virt-manager.org
> :|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc
> :|
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Should be instance_dir in all nova compute node same ?

[Timofei Durakov]

Hi,

>From the first sight there are no restrictions for instance_path option.
Could you please provide some details about the use case?
I wonder would this functionality be really useful for cloud-operators, or
we just should add description to instance path option, forcing to use the
same path on compute nodes?

Timofey



On Fri, Apr 29, 2016 at 4:47 AM, Eli Qiao  wrote:

> hi team,
>
> Is there any require that all compute node's instance_dir should be same?
>
> I recently get an issue while doing live migration with migrateToURI3
> method of libvirt python interface, if
> source and dest host are configured with difference instance_dir,
> migration will fail since migrateToURI3
> requires a new xml, but we don't modified the instance_dir of dest xml.
>
> I reported a bug on [1]_ , would like to get confirmation before spend
> effort working on it.
>
> [1] https://bugs.launchpad.net/nova/+bug/1576245
>
> Thanks.
>
> --
>
> Best Regards, Eli Qiao (乔立勇)
> Intel OTC China
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][live migration] Feature exposure

[Timofei Durakov]

Hello,

Last week I started thread [1] on openstack-operators to ask Openstack
users about the way to expose features for live-migrations.

So here are results:

*16,7%* - All this features should work under the hood and I don't want to
enable them manually over Nova API
*33,3%* - There should be some abstraction,which allows to decide
Instance importance for me
*33,3%* - I want to get ability to specify all this features for live
migration over NovaAPI
*16,7%* - Other

Timofey

[1] -
http://lists.openstack.org/pipermail/openstack-operators/2016-April/010203.html
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Nova quota statistics counting issue

[Timofei Durakov]

Hi,

I think it would be ok to store persistently quota details on compute side,
as was discussed during mitaka mid-cycle[1] for migrations[2]. So if
compute service fails we could restore state and update quota after compute
restart.

Timofey

[1] - https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking
[2] - https://review.openstack.org/#/c/291161/5/nova/compute/background.py




On Wed, Apr 13, 2016 at 7:27 PM, Dmitry Stepanenko  wrote:

> Hi Team,
>
> I worked on nova quota statistics issue (
> https://bugs.launchpad.net/nova/+bug/1284424) happenning when nova-*
> processes are restarted during removing instances and was able to reproduce
> it. For repro I used devstack and started nova-api and nova-compute in
> separate screen windows. For killing them I used ctrl+c. As I found this
> issue happened if nova-* processes are killed after instance was deleted
> but right before quota commit procedure finishes.
>
> We discussed these results with Markus Zoeller and decided that even
> though killing nova processes is a bit exotic event, this still should be
> fixed because quotas counting affects billing and very important for us.
>
> So, we need to introduce some mechanism that will prevent us from reaching
> inconsistent states in terms of quotas. In other words, this mechanism
> should work in such a way that both instance create/remove operation and
> quota usage recount operation happened or not happened together.
>
> Any ideas how to do that properly?
>
> Kind regards,
> Dmitry
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Dedicated CI job for live-migration testing status update

[Timofei Durakov]

Hello,

Separate job for testing live-migration with different storage
configurations(No shared storage/NFS/Ceph was implemented during Mitaka and
available to be executed using experimental pipeline.

The job is ready but shows same stability as
gate-tempest-dsvm-multinode-full.
Bugs like [1] [2] affects its stability. So the main idea for now it to
make this job running against latest libvirt/qemu versions. Markus Zoeller
and Tony Breeds are working on implementation of devstack plugin for that
[3]. So once plugin ready it will be possible to check. Another option for
this is to use Xenial images in experimental job, which will contain newer
libvirt/qemu version. Infra team already add ability to use 16.04 for
experimental[4], so I'm going to try this approach.

Work items:
- Cover negative testcases for live-migration(to check that all rollback
logic works well) - in progress;
- Check state of VM after migration
- Live migration for instance under workload

Timofey.

[1] - https://bugs.launchpad.net/nova/+bug/1535232
[2] - https://bugs.launchpad.net/nova/+bug/1524898
[3] -
https://review.openstack.org/#/q/project:openstack/devstack-plugin-additional-pkg-repos
[4] - https://review.openstack.org/#/c/302949/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Migration progress

[Timofei Durakov]

Hi,

In case of live-migration reporting I'd rather go with real-time stats,
queried from compute, instead of reporting this data to db first. While
amount of of rpc requests/db updates is relatively small, total number of
such requests depends on amount of active migrations. While realtime data
from compute allows to decrease it. Not every migration will trigger
operator to gather statistics, and each of triggered will require only 2
rpc per request instead of 2 rpc and db write per 3/5/etc. seconds.

Timofey.

On Sun, Feb 7, 2016 at 10:31 PM, Jay Pipes  wrote:

> On 02/04/2016 11:02 PM, Bhandaru, Malini K wrote:
>
>> Another thought, for such ephemeral/changing data, such as progress,
>> why not save the information in the cache (and flush to database at a
>> lower rate), and retrieve for display to active listeners/UI from the
>> cache. Once complete or aborted, of course flush the cache.
>>
>> Also should we provide a "verbose flag", that is only capture
>> progress information when requested? That is when a human user might
>> be issuing the command from the cli or GUI tool.
>>
>
> I agree with you, Malini, on the above suggestion that there is some doubt
> as to the value of saving this temporal data to the database.
>
> Why not just have an on-demand model that simply routes the request for
> progress information directly to the compute node and sends the progress
> amount back directly to the nova-api service instead of going to the
> database at all?
>
> Another alternative would be to use a push model instead of a poll model,
> but that would require a pretty significant change to the code...
>
> Best,
> -jay
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Timofei Durakov]

Hi,
some work items:
http://lists.openstack.org/pipermail/openstack-dev/2015-September/073965.html
- ci coverage for live-migration
https://blueprints.launchpad.net/nova/+spec/split-different-live-migration-types
- compute + drivers code cleanup

On Fri, Sep 18, 2015 at 6:47 PM, John Garbutt  wrote:

> On 18 September 2015 at 16:23, Daniel P. Berrange 
> wrote:
> > On Fri, Sep 18, 2015 at 11:53:05AM +, Murray, Paul (HP Cloud) wrote:
> >> Hi All,
> >>
> >> There are various efforts going on around live migration at the moment:
> >> fixing up CI, bug fixes, additions to cover more corner cases, proposals
> >> for new operations
> >>
> >> Generally live migration could do with a little TLC (see: [1]), so I am
> >> going to suggest we give some of that care in the next cycle.
> >>
> >> Please respond to this post if you have an interest in this and what you
> >> would like to see done. Include anything you are already getting on with
> >> so we get a clear picture. If there is enough interest I'll put this
> >> together as a proposal for a work stream. Something along the lines of
> >> "robustify live migration".
> >
> >
> >
> > Testing. Testing. Testing.
>
> +1 for Testing
>
> The "CI for reliable live-migration" thread was covering some of the
> details on the multi-host CI options.
>
> Thanks,
> johnthetubaugy
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] CI for reliable live-migration

[Timofei Durakov]

Hello,
Update for gate-tempest-dsvm-multinode-full job.
Here is top 12 failing tests in weekly period:
tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_manual_to_auto:
14
tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_auto_to_manual:
14
tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_resize_server_confirm:
12
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert:
12
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm:
12
tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration_paused:
12
tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_verify_resize_state:
12
tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_list_migrations_in_flavor_resize_situation:
12
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm_from_stopped:
12
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern:
10
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern:
10
tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration:
10


Full list of failing tests: http://xsnippet.org/360947/


On Fri, Aug 28, 2015 at 12:14 AM, Kraminsky, Arkadiy <
arkadiy.kramin...@hp.com> wrote:

> Hello,
>
> I'm a new developer on the Openstack project and am in the process of
> creating live migration CI for HP's 3PAR and Lefthand backends. I noticed
> you guys are looking for someone to pick up Joe Gordon's change for volume
> backed live migration tests and we can sure use something like this. I can
> take a look into the change, and see what I can do. :)
>
> Thanks,
>
> Arkadiy Kraminsky
> 
> From: Joe Gordon [joe.gord...@gmail.com]
> Sent: Wednesday, August 26, 2015 9:26 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [nova] CI for reliable live-migration
>
>
>
> On Wed, Aug 26, 2015 at 8:18 AM, Matt Riedemann <
> mrie...@linux.vnet.ibm.com<mailto:mrie...@linux.vnet.ibm.com>> wrote:
>
>
> On 8/26/2015 3:21 AM, Timofei Durakov wrote:
> Hello,
>
> Here is the situation: nova has live-migration feature but doesn't have
> ci job to cover it by functional tests, only
> gate-tempest-dsvm-multinode-full(non-voting, btw), which covers
> block-migration only.
> The problem here is, that live-migration could be different, depending
> on how instance was booted(volume-backed/ephemeral), how environment is
> configured(is shared instance directory(NFS, for example), or RBD used
> to store ephemeral disk), or for example user don't have that and is
> going to use --block-migrate flag. To claim that we have reliable
> live-migration in nova, we should check it at least on envs with rbd or
> nfs as more popular than envs without shared storages at all.
> Here is the steps for that:
>
>  1. make  gate-tempest-dsvm-multinode-full voting, as it looks OK for
> block-migration testing purposes;
>
> When we are ready to make multinode voting we should remove the equivalent
> single node job.
>
>
> If it's been stable for awhile then I'd be OK with making it voting on
> nova changes, I agree it's important to have at least *something* that
> gates on multi-node testing for nova since we seem to break this a few
> times per release.
>
> Last I checked it isn't as stable is single node yet:
> http://jogo.github.io/gate/multinode [0].  The data going into graphite
> is a bit noisy so this may be a red herring, but at the very least it needs
> to be investigated. When I was last looking into this there were at least
> two known bugs:
>
> https://bugs.launchpad.net/nova/+bug/1445569
> <https://bugs.launchpad.net/nova/+bug/1445569>
> https://bugs.launchpad.net/nova/+bug/1462305
>
>
> [0]
> http://graphite.openstack.org/graph/?from=-36hours=500=now=800=ff=00=100=0=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.{SUCCESS,FAILURE})),%275hours%27),%20%27gate-tempest-dsvm-full%27),%27orange%27)=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.{SUCCESS,FAILURE})),%275hours%27),%20%27gate-tempest-dsvm-multinode-full%27),%27brown%27)=Check%20Failure%20Rates%20(36%20hours)&_t=0.48646087432280183
> <
> http://graphite.openstack.org/graph/?from=-36hours=500=now=800=ff=00=

Re: [openstack-dev] [nova] CI for reliable live-migration

[Timofei Durakov]

Update:

1. Job fails from time to time, I'm collecting statistics to understand
whether it is valid fails or some races, etc.
2. This sounds good:

 jogo has had a patch up for this for awhile:
 https://review.openstack.org/#/c/165233/

3. It's required more research:

 We already have a voting ceph job for nova - can we turn that into a
 multi-node testing job and run live migration with shared storage using
 that?

4.  I think no: there is a branch in execution flow that could be checked,
when we have shared instance path only.

 Can't we use a multi-node ceph job (#3) for this?


On Wed, Aug 26, 2015 at 6:18 PM, Matt Riedemann mrie...@linux.vnet.ibm.com
wrote:



 On 8/26/2015 3:21 AM, Timofei Durakov wrote:

 Hello,

 Here is the situation: nova has live-migration feature but doesn't have
 ci job to cover it by functional tests, only
 gate-tempest-dsvm-multinode-full(non-voting, btw), which covers
 block-migration only.
 The problem here is, that live-migration could be different, depending
 on how instance was booted(volume-backed/ephemeral), how environment is
 configured(is shared instance directory(NFS, for example), or RBD used
 to store ephemeral disk), or for example user don't have that and is
 going to use --block-migrate flag. To claim that we have reliable
 live-migration in nova, we should check it at least on envs with rbd or
 nfs as more popular than envs without shared storages at all.
 Here is the steps for that:

  1. make  gate-tempest-dsvm-multinode-full voting, as it looks OK for
 block-migration testing purposes;


 If it's been stable for awhile then I'd be OK with making it voting on
 nova changes, I agree it's important to have at least *something* that
 gates on multi-node testing for nova since we seem to break this a few
 times per release.

  2. contribute to tempest to cover volume-backed instances live-migration;


 jogo has had a patch up for this for awhile:

 https://review.openstack.org/#/c/165233/

 Since it's not full time on openstack anymore I assume some help there in
 picking up the change would be appreciated.

  3. make another job with rbd for storing ephemerals, it also requires
 changing tempest config;


 We already have a voting ceph job for nova - can we turn that into a
 multi-node testing job and run live migration with shared storage using
 that?

  4. make job with nfs for ephemerals.


 Can't we use a multi-node ceph job (#3) for this?


 These steps should help us to improve current situation with
 live-migration.

 --
 Timofey.



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 --

 Thanks,

 Matt Riedemann


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] CI for reliable live-migration

[Timofei Durakov]

Hello,

Here is the situation: nova has live-migration feature but doesn't have ci
job to cover it by functional tests, only
gate-tempest-dsvm-multinode-full(non-voting,
btw), which covers block-migration only.
The problem here is, that live-migration could be different, depending on
how instance was booted(volume-backed/ephemeral), how environment is
configured(is shared instance directory(NFS, for example), or RBD used to
store ephemeral disk), or for example user don't have that and is going to
use --block-migrate flag. To claim that we have reliable live-migration in
nova, we should check it at least on envs with rbd or nfs as more popular
than envs without shared storages at all.
Here is the steps for that:

   1. make  gate-tempest-dsvm-multinode-full voting, as it looks OK for
   block-migration testing purposes;
   2. contribute to tempest to cover volume-backed instances live-migration;
   3. make another job with rbd for storing ephemerals, it also requires
   changing tempest config;
   4. make job with nfs for ephemerals.

These steps should help us to improve current situation with
live-migration.

--
Timofey.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Silent nova fails

[Timofei Durakov]

Hello,

In current design there are places when nova fails while executing users
CLI commands, but no error messages, except some logs in nova-compute,
produced [1] . The problem is that there is no response from compute node
to conductor, as RPC cast is used.

To fix this nova should make a synchronous call before operation itself to
verify that it is valid. E.g. here is my patch that fixes this problem in
resize operation [2]

So, I would like to get feedback about such hypervisor checks before
operations. Nova already makes these checks during live-migration process:
conductor calls compute manager[3], which also consults with driver[4]. And
as for me I think we should use such logic in resize operation.

Timofey.

[1] https://bugs.launchpad.net/nova/+bug/1455460

[2] https://review.openstack.org/195088

[3]
https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L144

[4]
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5157
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] [stable] Freze exception

[Timofei Durakov]

Hello

I'd like to ask for a freeze exception for the follow bug fix:

*https://review.openstack.org/#/c/198340/
https://review.openstack.org/#/c/198340/*

bug: *https://bugs.launchpad.net/nova/+bug/197
https://bugs.launchpad.net/nova/+bug/197*

merged bug fix in master: *https://review.openstack.org/#/c/173913/
https://review.openstack.org/#/c/173913/*

During live migration _post_live_migration and
post_live_migration_at_destination_method are executed simultaneously,
because second one is called over rpc.cast.
In _post_live_migration method there was setup_network_on_host call
with teardown=True, which expects new host in instances table db
field.

This update could be happened later, as it executes on destination
node in second method. To guarantee execution order
setup_network_on_host call, which cleans
dhcp-hostfile is moved to destination node.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev