Re: [Openstack-operators] Need feedback for nova aborting cold migration function
We are using multiple storage backend / topology on our side ranging from ScaleIO to CEPH passing by local compute host storage (were we need cold storage) and VNX, I have to said that CEPH is our best bet. Since we use it we clearly reduced our outages, allowed our user advanced features such as live-migration, boot from volumes and on top of that a better and more reliable performance. Yet we still need to get live and cold migration the same features set as our users/customers are really expecting us to provide a seamless experience between options. I can’t really speak out about real numbers but I’m within the video game industry if that help to drive support and traction/interest. Thanks for the survey btw. Kind regards, Gaël. Le mer. 23 mai 2018 à 23:36, a écrit : > Hi Matt, > > > > We store the service logs which are created by VM on that storage. > > > > I don't mean to be glib, but have you considered maybe not doing that? > > The load issue on storage is due to the way we deploy our business > softwares on VM. > The best way is introducing a new storage and separate the SAN, but we > cannot change our deployment method due to it's cost and other limitations. > On a long-term, our operation team will change the deployment method to > better one to resolve this problem. > > On the other hand, we would like to build a tool to support VM migration > that is unaware of which migration method is used for VM migration (Cold or > Live). Feature parity wise, if live migration supports cancel feature, then > we think that cold migration must support it as well. > > -- > Yukinori Sagara > Platform Engineering Department, NTT DATA Corp. > > > 差出人: Matt Riedemann > 送信日時: 2018年5月18日 1:39 > 宛先: openstack-operators@lists.openstack.org > 件名: Re: [Openstack-operators] Need feedback for nova aborting cold > migration function > > On 5/15/2018 3:48 AM, saga...@nttdata.co.jp wrote: > > We store the service logs which are created by VM on that storage. > > I don't mean to be glib, but have you considered maybe not doing that? > > -- > > Thanks, > > Matt > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Need feedback for nova aborting cold migration function
Hi Matt, > > We store the service logs which are created by VM on that storage. > > I don't mean to be glib, but have you considered maybe not doing that? The load issue on storage is due to the way we deploy our business softwares on VM. The best way is introducing a new storage and separate the SAN, but we cannot change our deployment method due to it's cost and other limitations. On a long-term, our operation team will change the deployment method to better one to resolve this problem. On the other hand, we would like to build a tool to support VM migration that is unaware of which migration method is used for VM migration (Cold or Live). Feature parity wise, if live migration supports cancel feature, then we think that cold migration must support it as well. -- Yukinori Sagara Platform Engineering Department, NTT DATA Corp. 差出人: Matt Riedemann 送信日時: 2018年5月18日 1:39 宛先: openstack-operators@lists.openstack.org 件名: Re: [Openstack-operators] Need feedback for nova aborting cold migration function On 5/15/2018 3:48 AM, saga...@nttdata.co.jp wrote: > We store the service logs which are created by VM on that storage. I don't mean to be glib, but have you considered maybe not doing that? -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Need feedback for nova aborting cold migration function
On 5/15/2018 3:48 AM, saga...@nttdata.co.jp wrote: We store the service logs which are created by VM on that storage. I don't mean to be glib, but have you considered maybe not doing that? -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Need feedback for nova aborting cold migration function
Hi Matt, > On 5/9/2018 9:33 PM, saga...@nttdata.co.jp wrote: > > Operation planning of cold migration is difficult because cold migration > > time will vary drastically as it also depends on the load on storage > > servers at that point of time. If cold migration task stalls for any > > unknown reasons, operators may decide to cancel it manually. > > What storage backend are you using? What are some reasons that it has > stalled in the past? Our storage backend is EMC VNX, and we have not shared the instance-store storage among compute nodes. The storage is also accessed by external system. We store the service logs which are created by VM on that storage. Our system needs to backup those logs by transferring to other storage. Those logs sometimes becomes very large, and the load of storage also becomes high. In those situation, migrating the VM takes more time than expected in advance, so we would like to cancel some migration task on the way if maintenance time being close to the end. > On 5/9/2018 9:33 PM, saga...@nttdata.co.jp wrote: > > We always do the maintenance work on midnight during limited time-slot to > > minimize impact to our users. > > Also, why are you doing maintenance with cold migration? Why not do live > migration for your maintenance (which already supports the abort function). We would like to migrate stopped servers as it is. As the reason above, we think we can operate the system more flexible if we able to cancel cold-migration as live-migration can. -- Yukinori Sagara Platform Engineering Department, NTT DATA Corp. 差出人: Matt Riedemann 送信日時: 2018年5月10日 22:59 宛先: openstack-operators@lists.openstack.org 件名: Re: [Openstack-operators] Need feedback for nova aborting cold migration function On 5/9/2018 9:33 PM, saga...@nttdata.co.jp wrote: > We always do the maintenance work on midnight during limited time-slot to > minimize impact to our users. Also, why are you doing maintenance with cold migration? Why not do live migration for your maintenance (which already supports the abort function). -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Need feedback for nova aborting cold migration function
On 5/9/2018 9:33 PM, saga...@nttdata.co.jp wrote: We always do the maintenance work on midnight during limited time-slot to minimize impact to our users. Also, why are you doing maintenance with cold migration? Why not do live migration for your maintenance (which already supports the abort function). -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Need feedback for nova aborting cold migration function
On 5/9/2018 9:33 PM, saga...@nttdata.co.jp wrote: Operation planning of cold migration is difficult because cold migration time will vary drastically as it also depends on the load on storage servers at that point of time. If cold migration task stalls for any unknown reasons, operators may decide to cancel it manually. What storage backend are you using? What are some reasons that it has stalled in the past? -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Need feedback for nova aborting cold migration function
Flint and Yukinori, Thank you for your replies! On 2018/05/10 11:33, saga...@nttdata.co.jp wrote: Hi Takashi, and guys, We are operating large telco enterprise cloud. We always do the maintenance work on midnight during limited time-slot to minimize impact to our users. Operation planning of cold migration is difficult because cold migration time will vary drastically as it also depends on the load on storage servers at that point of time. If cold migration task stalls for any unknown reasons, operators may decide to cancel it manually. This requires several manual steps to be carried out for recovering from such situation such as kill the copy process, reset-state, stop, and start the VM. If we have the ability to cancel cold migration, we can resume our service safely even though the migration is not complete in the stipulated maintenance time window. As of today, we can solve the above issue by following manual procedure to recover instances from cold migration failure but we still need to follow these steps every time. We can build our own tool to automate this process but we will need to maintain it by ourselves as this feature is not supported by any OpenStack distribution. If Nova supports function to cancel cold migration, it’s definitely going to help us to bring instances back from cold migration failure thus improving service availability to our end users. Secondly, we don’t need to worry about maintaining procedure manual or proprietary tool by ourselves which will be a huge win for us. We are definitely interested in this function and we would love to see it in the next coming release. Thank you for your hard work. -- Yukinori Sagara Platform Engineering Department, NTT DATA Corp. Hi everyone, I'm going to add the aborting cold migration function [1] in nova. I would like to ask operators' feedback on this. The cold migration is an administrator operation by default. If administrators perform cold migration and it is stalled out, users cannot do their operations (e.g. starting the VM). In that case, if administrators can abort the cold migration by using this function, it enables users to operate their VMs. If you are a person like the following, would you reply to this mail? * Those who need this function * Those who will use this function if it is implemented * Those who think that it is better to have this function * Those who are interested in this function [1] https://review.openstack.org/#/c/334732/ Regards, Takashi Natsume NTT Software Innovation Center E-mail: natsume.takashi at lab.ntt.co.jp Regards, Takashi Natsume NTT Software Innovation Center E-mail: natsume.taka...@lab.ntt.co.jp ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] Need feedback for nova aborting cold migration function
Hi Takashi, and guys, We are operating large telco enterprise cloud. We always do the maintenance work on midnight during limited time-slot to minimize impact to our users. Operation planning of cold migration is difficult because cold migration time will vary drastically as it also depends on the load on storage servers at that point of time. If cold migration task stalls for any unknown reasons, operators may decide to cancel it manually. This requires several manual steps to be carried out for recovering from such situation such as kill the copy process, reset-state, stop, and start the VM. If we have the ability to cancel cold migration, we can resume our service safely even though the migration is not complete in the stipulated maintenance time window. As of today, we can solve the above issue by following manual procedure to recover instances from cold migration failure but we still need to follow these steps every time. We can build our own tool to automate this process but we will need to maintain it by ourselves as this feature is not supported by any OpenStack distribution. If Nova supports function to cancel cold migration, it’s definitely going to help us to bring instances back from cold migration failure thus improving service availability to our end users. Secondly, we don’t need to worry about maintaining procedure manual or proprietary tool by ourselves which will be a huge win for us. We are definitely interested in this function and we would love to see it in the next coming release. Thank you for your hard work. -- Yukinori Sagara Platform Engineering Department, NTT DATA Corp. > Hi everyone, > > I'm going to add the aborting cold migration function [1] in nova. > I would like to ask operators' feedback on this. > > The cold migration is an administrator operation by default. > If administrators perform cold migration and it is stalled out, > users cannot do their operations (e.g. starting the VM). > > In that case, if administrators can abort the cold migration by using > this function, > it enables users to operate their VMs. > > If you are a person like the following, would you reply to this mail? > > * Those who need this function > * Those who will use this function if it is implemented > * Those who think that it is better to have this function > * Those who are interested in this function > > [1] https://review.openstack.org/#/c/334732/ > > Regards, > Takashi Natsume > NTT Software Innovation Center > E-mail: natsume.takashi at lab.ntt.co.jp ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Need feedback for nova aborting cold migration function
As an operator dealing with platforms that do cold migration I would like to be able to abort and rollback the process. That would give us a better service quality and availability. We do have no choices but to use cold migration on some of our remote sites as they don’t get a unified storage such as CEPH for cost management. Those remote sites have to growth and gain traction before being budgeted for a truly powerful distributed storage backend. Due to such limitations I would love to be able to reduce the time our customers are impacted by such move while doing maintenance or any other jobs requiring us to do a migration. Thanks for the hard work on this topic! Le mar. 1 mai 2018 à 03:03, Takashi Natsume a écrit : > Hi everyone, > > I'm going to add the aborting cold migration function [1] in nova. > I would like to ask operators' feedback on this. > > The cold migration is an administrator operation by default. > If administrators perform cold migration and it is stalled out, > users cannot do their operations (e.g. starting the VM). > > In that case, if administrators can abort the cold migration by using > this function, > it enables users to operate their VMs. > > If you are a person like the following, would you reply to this mail? > > * Those who need this function > * Those who will use this function if it is implemented > * Those who think that it is better to have this function > * Those who are interested in this function > > [1] https://review.openstack.org/#/c/334732/ > > Regards, > Takashi Natsume > NTT Software Innovation Center > E-mail: natsume.taka...@lab.ntt.co.jp > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] Need feedback for nova aborting cold migration function
Hi everyone, I'm going to add the aborting cold migration function [1] in nova. I would like to ask operators' feedback on this. The cold migration is an administrator operation by default. If administrators perform cold migration and it is stalled out, users cannot do their operations (e.g. starting the VM). In that case, if administrators can abort the cold migration by using this function, it enables users to operate their VMs. If you are a person like the following, would you reply to this mail? * Those who need this function * Those who will use this function if it is implemented * Those who think that it is better to have this function * Those who are interested in this function [1] https://review.openstack.org/#/c/334732/ Regards, Takashi Natsume NTT Software Innovation Center E-mail: natsume.taka...@lab.ntt.co.jp ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
