Re: [ceph-users] QEMU maps RBD but can't read them
On Tue, Jun 12, 2018 at 4:53 AM, Wladimir Mutel wrote: > Jason Dillaman wrote: > >>> One more question, how should I set profile 'rbd-read-only' >>> properly >>> ? I tried to set is for 'client.iso' on both 'iso' and 'jerasure21' >>> pools, >>> and this did not work. Set profile on both pools to 'rbd', it worked. But >>> I >>> don't want my iso imaged to be accidentally modified by virtual guests. >>> Can >>> this be solved with Ceph auth, or in some other way ? (in fact, I look >>> for >>> Ceph equivalent of 'chattr +i') > > >> QEMU doesn't currently handle the case for opening RBD images in >> read-only mode, so if you attempt to use 'profile rbd-read-only', I >> suspect attempting to open the image will fail. You could perhaps take >> a middle ground and just apply 'profile rbd-read-only pool=jerasure21' >> to protect the contents of the image. > > > For QEMU I found that profile 'rbd-read-only' currently does not > work. So, I use 'profile rbd' for both replicated and erasure pools, and > hope that 'readonly' configuration in QEMU disk would help. > In my past experience I found that running 'kvm ... -cdrom > something.iso' sometimes would modify that .iso-file, so I had to set > immutable attribute on the FS level. I opened a tracker ticket [1] to submit a patch to QEMU to support read-only images in the RBD block driver. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [1] http://tracker.ceph.com/issues/24506 -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
Jason Dillaman wrote: One more question, how should I set profile 'rbd-read-only' properly ? I tried to set is for 'client.iso' on both 'iso' and 'jerasure21' pools, and this did not work. Set profile on both pools to 'rbd', it worked. But I don't want my iso imaged to be accidentally modified by virtual guests. Can this be solved with Ceph auth, or in some other way ? (in fact, I look for Ceph equivalent of 'chattr +i') QEMU doesn't currently handle the case for opening RBD images in read-only mode, so if you attempt to use 'profile rbd-read-only', I suspect attempting to open the image will fail. You could perhaps take a middle ground and just apply 'profile rbd-read-only pool=jerasure21' to protect the contents of the image. For QEMU I found that profile 'rbd-read-only' currently does not work. So, I use 'profile rbd' for both replicated and erasure pools, and hope that 'readonly' configuration in QEMU disk would help. In my past experience I found that running 'kvm ... -cdrom something.iso' sometimes would modify that .iso-file, so I had to set immutable attribute on the FS level. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
On Wed, Jun 6, 2018 at 4:48 PM, Wladimir Mutel wrote: > Jason Dillaman wrote: > The caps for those users looks correct for Luminous and later clusters. Any chance you are using data pools with the images? It's just odd that you have enough permissions to open the RBD image but cannot read its data objects. > > >>> Yes, I use erasure-pool as data-pool for these images >>> (to save on replication overhead). >>> Should I add it to the [osd] profile list ? > > >> Indeed, that's the problem since the libvirt and/or iso user doesn't >> have access to the data-pool. > > > This really helped, thanks ! > > client.iso > key: AQBp...gA== > caps: [mon] profile rbd > caps: [osd] profile rbd pool=iso, profile rbd pool=jerasure21 > client.libvirt > key: AQBt...IA== > caps: [mon] profile rbd > caps: [osd] profile rbd pool=libvirt, profile rbd pool=jerasure21 > > Now I can boot the VM from the .iso image and install Windows. > > One more question, how should I set profile 'rbd-read-only' properly > ? I tried to set is for 'client.iso' on both 'iso' and 'jerasure21' pools, > and this did not work. Set profile on both pools to 'rbd', it worked. But I > don't want my iso imaged to be accidentally modified by virtual guests. Can > this be solved with Ceph auth, or in some other way ? (in fact, I look for > Ceph equivalent of 'chattr +i') > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com QEMU doesn't currently handle the case for opening RBD images in read-only mode, so if you attempt to use 'profile rbd-read-only', I suspect attempting to open the image will fail. You could perhaps take a middle ground and just apply 'profile rbd-read-only pool=jerasure21' to protect the contents of the image. -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
Jason Dillaman wrote: The caps for those users looks correct for Luminous and later clusters. Any chance you are using data pools with the images? It's just odd that you have enough permissions to open the RBD image but cannot read its data objects. Yes, I use erasure-pool as data-pool for these images (to save on replication overhead). Should I add it to the [osd] profile list ? Indeed, that's the problem since the libvirt and/or iso user doesn't have access to the data-pool. This really helped, thanks ! client.iso key: AQBp...gA== caps: [mon] profile rbd caps: [osd] profile rbd pool=iso, profile rbd pool=jerasure21 client.libvirt key: AQBt...IA== caps: [mon] profile rbd caps: [osd] profile rbd pool=libvirt, profile rbd pool=jerasure21 Now I can boot the VM from the .iso image and install Windows. One more question, how should I set profile 'rbd-read-only' properly ? I tried to set is for 'client.iso' on both 'iso' and 'jerasure21' pools, and this did not work. Set profile on both pools to 'rbd', it worked. But I don't want my iso imaged to be accidentally modified by virtual guests. Can this be solved with Ceph auth, or in some other way ? (in fact, I look for Ceph equivalent of 'chattr +i') ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
On Wed, Jun 6, 2018 at 3:02 PM, Wladimir Mutel wrote: > Jason Dillaman wrote: >> >> The caps for those users looks correct for Luminous and later >> clusters. Any chance you are using data pools with the images? It's >> just odd that you have enough permissions to open the RBD image but >> cannot read its data objects. > > > Yes, I use erasure-pool as data-pool for these images > (to save on replication overhead). > Should I add it to the [osd] profile list ? Indeed, that's the problem since the libvirt and/or iso user doesn't have access to the data-pool. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
Jason Dillaman wrote: The caps for those users looks correct for Luminous and later clusters. Any chance you are using data pools with the images? It's just odd that you have enough permissions to open the RBD image but cannot read its data objects. Yes, I use erasure-pool as data-pool for these images (to save on replication overhead). Should I add it to the [osd] profile list ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
The caps for those users looks correct for Luminous and later clusters. Any chance you are using data pools with the images? It's just odd that you have enough permissions to open the RBD image but cannot read its data objects. On Wed, Jun 6, 2018 at 2:46 PM, Wladimir Mutel wrote: > Jason Dillaman wrote: >> >> Can you run "rbd --id libvirt --pool libvirt win206-test-3tb > temprary file name>" w/o error? It sounds like your CephX caps for >> client.libvirt are not permitting read access to the image data >> objects. > > > I tried to run 'rbd export' with these params, > but it said it was unable to find a keyring. > Is keyring file mandatory for every client ? > > 'ceph auth ls' shows these accounts with seemingly-proper > permissions : > > client.iso > key: AQBp...gA== > caps: [mon] profile rbd > caps: [osd] profile rbd pool=iso > client.libvirt > key: AQBt...IA== > caps: [mon] profile rbd > caps: [osd] profile rbd pool=libvirt > > And these same keys are listed in /etc/libvirt/secrets : > > /etc/libvirt/secrets# ls | while read a ; do echo $a : $(cat $a) ; done > ac1d8d7b-d243-4474-841d-91c26fd93a14.base64 : AQBt...IA== > > ac1d8d7b-d243-4474-841d-91c26fd93a14.xml : private='yes'> ac1d8d7b-d243-4474-841d-91c26fd93a14 > CEPH passphrase example > ceph_example > > cf00c7e4-740a-4935-9d7c-223d3c81871f.base64 : AQBp...gA== > > cf00c7e4-740a-4935-9d7c-223d3c81871f.xml : private='yes'> cf00c7e4-740a-4935-9d7c-223d3c81871f > CEPH ISO pool > ceph_iso > > I just thought this should be enough. no ? > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
Jason Dillaman wrote: Can you run "rbd --id libvirt --pool libvirt win206-test-3tb " w/o error? It sounds like your CephX caps for client.libvirt are not permitting read access to the image data objects. I tried to run 'rbd export' with these params, but it said it was unable to find a keyring. Is keyring file mandatory for every client ? 'ceph auth ls' shows these accounts with seemingly-proper permissions : client.iso key: AQBp...gA== caps: [mon] profile rbd caps: [osd] profile rbd pool=iso client.libvirt key: AQBt...IA== caps: [mon] profile rbd caps: [osd] profile rbd pool=libvirt And these same keys are listed in /etc/libvirt/secrets : /etc/libvirt/secrets# ls | while read a ; do echo $a : $(cat $a) ; done ac1d8d7b-d243-4474-841d-91c26fd93a14.base64 : AQBt...IA== ac1d8d7b-d243-4474-841d-91c26fd93a14.xml : private='yes'> ac1d8d7b-d243-4474-841d-91c26fd93a14 CEPH passphrase example ceph_example cf00c7e4-740a-4935-9d7c-223d3c81871f.base64 : AQBp...gA== cf00c7e4-740a-4935-9d7c-223d3c81871f.xml : private='yes'> cf00c7e4-740a-4935-9d7c-223d3c81871f CEPH ISO pool ceph_iso I just thought this should be enough. no ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] QEMU maps RBD but can't read them
Can you run "rbd --id libvirt --pool libvirt win206-test-3tb " w/o error? It sounds like your CephX caps for client.libvirt are not permitting read access to the image data objects. On Wed, Jun 6, 2018 at 2:18 PM, Wladimir Mutel wrote: > > Dear all, > > I installed QEMU, libvirtd and its RBD plugins and now trying > to make QEMU use my Ceph storage. I created 'iso' pool > and imported Windows installation image there (rbd import). > Also I created 'libvirt' pool and there, created 2.7-TB image > for Windows installation. I created client.iso and > client.libvirt accounts for Ceph authentication, > and configured their secrets for pool access in virsh > (as told in http://docs.ceph.com/docs/master/rbd/libvirt/ ). > Then I started pools and checked that I can list their contents > from virsh. Then I created a VM with dummy HDD and optical > drive, and edited them using 'virsh edit' : > > > > > > > > > > > > > > > > > > > >name='iso/SW_DVD9_Win_Server_STD_CORE_2016_64Bit_Russian_-4_DC_STD_MLF_X21-70539.ISO'> > > > > > > > > > Now I see this in the systemd journalctl : > > чер 06 16:24:12 p10s qemu-system-x86_64[4907]: 2018-06-06 16:24:12.147 > 7f40f37fe700 -1 librbd::io::ObjectRequest: 0x7f40d4010500 > handle_read_object: failed to read from object: (1) Operation not permitted > > What should I check and where ? > I can map the same RBD using rbd-nbd and read sectors > from the mapped device. If I map using kernel RBD driver > (I know this is not recommended to do on the same host), > I get : > > чер 06 16:27:54 p10s kernel: rbd: image > SW_DVD9_Win_Server_STD_CORE_2016_64Bit_Russian_-4_DC_STD_MLF_X21-70539.ISO: > image uses unsupported features: 0x38 > > and > > RBD image feature set mismatch. You can disable features unsupported by the > kernel with "rbd feature disable > iso/SW_DVD9_Win_Server_STD_CORE_2016_64Bit_Russian_-4_DC_STD_MLF_X21-70539.ISO > object-map fast-diff deep-flatten". > > Probably I need to change some attributes for the RBD > to be usable with QEMU. Please give some hints. > Thank you in advance. > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
