Re: [openstack-dev] [cinder] [nova] Problem of Volume(in-use) Live Migration with ceph backend
Great, Jon. Thanks for your reply. I am looking forward to your report.
Cheers,
Boxiang
On 10/23/2018 22:01，Jon Bernard wrote：
* melanie witt wrote:
On Mon, 22 Oct 2018 11:45:55 +0800 (GMT+08:00), Boxiang Zhu wrote:
I created a new vm and a new volume with type 'ceph'[So that the volume
will be created on one of two hosts. I assume that the volume created on
host dev@rbd-1#ceph this time]. Next step is to attach the volume to the
vm. At last I want to migrate the volume from host dev@rbd-1#ceph to
host dev@rbd-2#ceph, but it failed with the exception
'NotImplementedError(_("Swap only supports host devices")'.
So that, my real problem is that is there any work to migrate
volume(*in-use*)(*ceph rbd*) from one host(pool) to another host(pool)
in the same ceph cluster?
The difference between the spec[2] with my scope is only one is
*available*(the spec) and another is *in-use*(my scope).
[1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/
[2] https://review.openstack.org/#/c/296150
Ah, I think I understand now, thank you for providing all of those details.
And I think you explained it in your first email, that cinder supports
migration of ceph volumes if they are 'available' but not if they are
'in-use'. Apologies that I didn't get your meaning the first time.
I see now the code you were referring to is this [3]:
if volume.status not in ('available', 'retyping', 'maintenance'):
LOG.debug('Only available volumes can be migrated using backend '
'assisted migration. Falling back to generic migration.')
return refuse_to_migrate
So because your volume is not 'available', 'retyping', or 'maintenance',
it's falling back to generic migration, which will end up with an error in
nova because the source_path is not set in the volume config.
Can anyone from the cinder team chime in about whether the ceph volume
migration could be expanded to allow migration of 'in-use' volumes? Is there
a reason not to allow migration of 'in-use' volumes?
Generally speaking, Nova must facilitate the migration of a live (or
in-use) volume. A volume attached to a running instance requires code
in the I/O path to correctly route traffic to the correct location - so
Cinder must refuse (or defer) a migrate operation if the volume is
attached. Until somewhat recently Qemu and Libvirt did not support the
migration to non-block (RBD) targets which is the reason for lack of
support. I believe we now have all of the pieces to perform this
operation successfully, but I suspect it will require a setup with
correct versions of all the related software. I will try to verify this
during the current release cycle and report back.
--
Jon
[3]
https://github.com/openstack/cinder/blob/c42fdc470223d27850627fd4fc9d8cb15f2941f8/cinder/volume/drivers/rbd.py#L1618-L1621
Cheers,
-melanie
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] [nova] Problem of Volume(in-use) Live Migration with ceph backend
Jay and Melanie, It's my fault to let you misunderstand the problem. I should
describe my problem more clearly. My problem is not to migrate volumes between
two ceph clusters.
I have two clusters, one is openstack cluster(allinone env, hostname is dev)
and another is ceph cluster. Omit the integrated configurations for openstack
and ceph.[1] The special config of cinder.conf is as followed:
[DEFAULT]
enabled_backends = rbd-1,rbd-2
..
[rbd-1]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph
rbd_pool = volumes001
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = true
rbd_max_clone_depth = 2
rbd_store_chunk_size = 4
rados_connect_timeout = 5
rbd_user = cinder
rbd_secret_uuid = 86d3922a-b471-4dc1-bb89-b46ab7024e81
[rbd-2]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph
rbd_pool = volumes002
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = true
rbd_max_clone_depth = 2
rbd_store_chunk_size = 4
rados_connect_timeout = 5
rbd_user = cinder
rbd_secret_uuid = 86d3922a-b471-4dc1-bb89-b46ab7024e81
There will be two hosts named dev@rbd-1#ceph and dev@rbd-2#ceph.
Then I create a volume type named 'ceph' with the command 'cinder type-create
ceph' and add extra_spec 'volume_backend_name=ceph' for it with the command
'cinder type-key set volume_backend_name=ceph'.
I created a new vm and a new volume with type 'ceph'[So that the volume will be
created on one of two hosts. I assume that the volume created on host
dev@rbd-1#ceph this time]. Next step is to attach the volume to the vm. At last
I want to migrate the volume from host dev@rbd-1#ceph to host dev@rbd-2#ceph,
but it failed with the exception 'NotImplementedError(_("Swap only supports
host devices")'.
So that, my real problem is that is there any work to migrate
volume(in-use)(ceph rbd) from one host(pool) to another host(pool) in the same
ceph cluster?
The difference between the spec[2] with my scope is only one is available(the
spec) and another is in-use(my scope).
[1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/
[2] https://review.openstack.org/#/c/296150
Cheers,
Boxiang
On 10/21/2018 23:19，Jay S. Bryant wrote：
Boxiang,
I have not herd any discussion of extending this functionality for Ceph to work
between different Ceph Clusters. I wasn't aware, however, that the existing
spec was limited to one Ceph cluster. So, that is good to know.
I would recommend reaching out to Jon Bernard or Eric Harney for guidance on
how to proceed. They work closely with the Ceph driver and could provide
insight.
Jay
On 10/19/2018 10:21 AM, Boxiang Zhu wrote:
Hi melanie, thanks for your reply.
The version of my cinder and nova is Rocky. The scope of the cinder spec[1]
is only for available volume migration between two pools from the same ceph
cluster.
If the volume is in-use status[2], it will call the generic migration function.
So that as you
describe it, on the nova side, it raises NotImplementedError(_("Swap only
supports host devices").
The get_config of net volume[3] has not source_path.
So does anyone try to succeed to migrate volume(in-use) with ceph backend or is
anyone doing something of it?
[1] https://review.openstack.org/#/c/296150
[2] https://review.openstack.org/#/c/256091/23/cinder/volume/drivers/rbd.py
[3]
https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/volume/net.py#L101
Cheers,
Boxiang
On 10/19/2018 22:39，melanie witt wrote：
On Fri, 19 Oct 2018 11:33:52 +0800 (GMT+08:00), Boxiang Zhu wrote:
When I use the LVM backend to create the volume, then attach it to a vm.
I can migrate the volume(in-use) from one host to another. The nova
libvirt will call the 'rebase' to finish it. But if using ceph backend,
it raises exception 'Swap only supports host devices'. So now it does
not support to migrate volume(in-use). Does anyone do this work now? Or
Is there any way to let me migrate volume(in-use) with ceph backend?
What version of cinder and nova are you using?
I found this question/answer on ask.openstack.org:
https://ask.openstack.org/en/question/112954/volume-migration-fails-notimplementederror-swap-only-supports-host-devices/
and it looks like there was some work done on the cinder side [1] to
enable migration of in-use volumes with ceph semi-recently (Queens).
On the nova side, the code looks for the source_path in the volume
config, and if there is not one present, it raises
NotImplementedError(_("Swap only supports host devices"). So in your
environment, the volume configs must be missing a source_path.
If you are using at least Queens version, then there must be something
additional missing that we would need to do to make the migration work.
[1] https://blueprints.launchpad.net/cinder/+spec/ceph-volume-migrate
Cheers,
-melanie
Re: [openstack-dev] [cinder] [nova] Problem of Volume(in-use) Live Migration with ceph backend
Hi melanie, thanks for your reply.
The version of my cinder and nova is Rocky. The scope of the cinder spec[1]
is only for available volume migration between two pools from the same ceph
cluster.
If the volume is in-use status[2], it will call the generic migration function.
So that as you
describe it, on the nova side, it raises NotImplementedError(_("Swap only
supports host devices").
The get_config of net volume[3] has not source_path.
So does anyone try to succeed to migrate volume(in-use) with ceph backend or is
anyone doing something of it?
[1] https://review.openstack.org/#/c/296150
[2] https://review.openstack.org/#/c/256091/23/cinder/volume/drivers/rbd.py
[3]
https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/volume/net.py#L101
Cheers,
Boxiang
On 10/19/2018 22:39，melanie witt wrote：
On Fri, 19 Oct 2018 11:33:52 +0800 (GMT+08:00), Boxiang Zhu wrote:
When I use the LVM backend to create the volume, then attach it to a vm.
I can migrate the volume(in-use) from one host to another. The nova
libvirt will call the 'rebase' to finish it. But if using ceph backend,
it raises exception 'Swap only supports host devices'. So now it does
not support to migrate volume(in-use). Does anyone do this work now? Or
Is there any way to let me migrate volume(in-use) with ceph backend?
What version of cinder and nova are you using?
I found this question/answer on ask.openstack.org:
https://ask.openstack.org/en/question/112954/volume-migration-fails-notimplementederror-swap-only-supports-host-devices/
and it looks like there was some work done on the cinder side [1] to
enable migration of in-use volumes with ceph semi-recently (Queens).
On the nova side, the code looks for the source_path in the volume
config, and if there is not one present, it raises
NotImplementedError(_("Swap only supports host devices"). So in your
environment, the volume configs must be missing a source_path.
If you are using at least Queens version, then there must be something
additional missing that we would need to do to make the migration work.
[1] https://blueprints.launchpad.net/cinder/+spec/ceph-volume-migrate
Cheers,
-melanie
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder] [nova] Problem of Volume(in-use) Live Migration with ceph backend
Hi folks, When I use the LVM backend to create the volume, then attach it to a vm. I can migrate the volume(in-use) from one host to another. The nova libvirt will call the 'rebase' to finish it. But if using ceph backend, it raises exception 'Swap only supports host devices'. So now it does not support to migrate volume(in-use). Does anyone do this work now? Or Is there any way to let me migrate volume(in-use) with ceph backend? Cheers, Boxiang __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
