Re: [ceph-users] Qemu RBD image usage

[Marc Roos]

 
This should get you started with using rbd.


  
  

  
  



  
  
  WDC
  WD40EFRX-68WT0N0
  
  
  





cat > secret.xml <


client.rbd.vps secret


EOF

virsh secret-define --file secret.xml

virsh secret-set-value --secret  --base64 `ceph auth get-key 
client.rbd.vps 2>/dev/null`



-Original Message-
To: ceph-users@lists.ceph.com
Cc: d...@ceph.io
Subject: [ceph-users] Qemu RBD image usage

Hi all,
   I want to attach another RBD image into the Qemu VM to be used as 
disk.
   However, it always failed.  The VM definiation xml is attached.
   Could anyone tell me where I did wrong?
   || nstcc3@nstcloudcc3:~$ sudo virsh start ubuntu_18_04_mysql 
--console
   || error: Failed to start domain ubuntu_18_04_mysql
   || error: internal error: process exited while connecting to monitor:
   || 2019-12-09T16:24:30.284454Z qemu-system-x86_64: -drive
   || 
file=rbd:rwl_mysql/mysql_image:auth_supported=none:mon_host=nstcloudcc4\
:6789,format=raw,if=none,id=drive-virtio-disk1:
   || error connecting: Operation not supported


   The cluster info is below:
   || ceph@nstcloudcc3:~$ ceph --version
   || ceph version 14.0.0-16935-g9b6ef711f3 
(9b6ef711f3a40898756457cb287bf291f45943f0) octopus (dev)
   || ceph@nstcloudcc3:~$ ceph -s
   ||   cluster:
   || id: e31502ff-1fb4-40b7-89a8-2b85a77a3b09
   || health: HEALTH_OK
   ||  
   ||   services:
   || mon: 1 daemons, quorum nstcloudcc4 (age 2h)
   || mgr: nstcloudcc4(active, since 2h)
   || osd: 4 osds: 4 up (since 2h), 4 in (since 2h)
   ||  
   ||   data:
   || pools:   1 pools, 128 pgs
   || objects: 6 objects, 6.3 KiB
   || usage:   4.0 GiB used, 7.3 TiB / 7.3 TiB avail
   || pgs: 128 active+clean
   ||  
   || ceph@nstcloudcc3:~$
   || ceph@nstcloudcc3:~$ rbd info rwl_mysql/mysql_image
   || rbd image 'mysql_image':
   || size 100 GiB in 25600 objects
   || order 22 (4 MiB objects)
   || snapshot_count: 0
   || id: 110feda39b1c
   || block_name_prefix: rbd_data.110feda39b1c
   || format: 2
   || features: layering, exclusive-lock, object-map, fast-diff, 
deep-flatten
   || op_features: 
   || flags: 
   || create_timestamp: Mon Dec  9 23:48:17 2019
   || access_timestamp: Mon Dec  9 23:48:17 2019
   || modify_timestamp: Mon Dec  9 23:48:17 2019

B.R.
Changcheng


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Qemu RBD image usage

[Liu, Changcheng]

Hi all,
   I want to attach another RBD image into the Qemu VM to be used as disk.
   However, it always failed.  The VM definiation xml is attached.
   Could anyone tell me where I did wrong?
   || nstcc3@nstcloudcc3:~$ sudo virsh start ubuntu_18_04_mysql --console
   || error: Failed to start domain ubuntu_18_04_mysql
   || error: internal error: process exited while connecting to monitor:
   || 2019-12-09T16:24:30.284454Z qemu-system-x86_64: -drive
   || 
file=rbd:rwl_mysql/mysql_image:auth_supported=none:mon_host=nstcloudcc4\:6789,format=raw,if=none,id=drive-virtio-disk1:
   || error connecting: Operation not supported


   The cluster info is below:
   || ceph@nstcloudcc3:~$ ceph --version
   || ceph version 14.0.0-16935-g9b6ef711f3 
(9b6ef711f3a40898756457cb287bf291f45943f0) octopus (dev)
   || ceph@nstcloudcc3:~$ ceph -s
   ||   cluster:
   || id: e31502ff-1fb4-40b7-89a8-2b85a77a3b09
   || health: HEALTH_OK
   ||  
   ||   services:
   || mon: 1 daemons, quorum nstcloudcc4 (age 2h)
   || mgr: nstcloudcc4(active, since 2h)
   || osd: 4 osds: 4 up (since 2h), 4 in (since 2h)
   ||  
   ||   data:
   || pools:   1 pools, 128 pgs
   || objects: 6 objects, 6.3 KiB
   || usage:   4.0 GiB used, 7.3 TiB / 7.3 TiB avail
   || pgs: 128 active+clean
   ||  
   || ceph@nstcloudcc3:~$
   || ceph@nstcloudcc3:~$ rbd info rwl_mysql/mysql_image
   || rbd image 'mysql_image':
   || size 100 GiB in 25600 objects
   || order 22 (4 MiB objects)
   || snapshot_count: 0
   || id: 110feda39b1c
   || block_name_prefix: rbd_data.110feda39b1c
   || format: 2
   || features: layering, exclusive-lock, object-map, fast-diff, 
deep-flatten
   || op_features: 
   || flags: 
   || create_timestamp: Mon Dec  9 23:48:17 2019
   || access_timestamp: Mon Dec  9 23:48:17 2019
   || modify_timestamp: Mon Dec  9 23:48:17 2019

B.R.
Changcheng


ubuntu_18_04_mysql.xml
Description: XML document
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] qemu/rbd: threads vs native, performance tuning

[Elias Abacioglu]

Hi,

I was reading this thread:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008486.html

And I am trying to get better performance in my virtual machines.
These are my RBD settings:
"rbd_cache": "true",
"rbd_cache_block_writes_upfront": "false",
"rbd_cache_max_dirty": "25165824",
"rbd_cache_max_dirty_age": "1.00",
"rbd_cache_max_dirty_object": "0",
"rbd_cache_size": "33554432",
"rbd_cache_target_dirty": "16777216",
"rbd_cache_writethrough_until_flush": "true",

I decided to test native mode and ran fio like this inside a VM:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G
--readwrite=randrw --rwmixread=75

I tested these two setups in qemu.



I ran fio a couple times to have a little variance and here is the results:

 READ: io=3071.7MB, aggrb=96718KB/s, minb=96718KB/s, maxb=96718KB/s,
mint=32521msec, maxt=32521msec
WRITE: io=1024.4MB, aggrb=32253KB/s, minb=32253KB/s, maxb=32253KB/s,
mint=32521msec, maxt=32521msec
 READ: io=3071.7MB, aggrb=96451KB/s, minb=96451KB/s, maxb=96451KB/s,
mint=32611msec, maxt=32611msec
WRITE: io=1024.4MB, aggrb=32164KB/s, minb=32164KB/s, maxb=32164KB/s,
mint=32611msec, maxt=32611msec
 READ: io=3071.7MB, aggrb=93763KB/s, minb=93763KB/s, maxb=93763KB/s,
mint=33546msec, maxt=33546msec
WRITE: io=1024.4MB, aggrb=31267KB/s, minb=31267KB/s, maxb=31267KB/s,
mint=33546msec, maxt=33546msec
---

DISK = [ driver = "raw" , cache = "directsync" , discard = "unmap" , io
= "native" ]
 READ: io=3071.7MB, aggrb=68771KB/s, minb=68771KB/s, maxb=68771KB/s,
mint=45737msec, maxt=45737msec
WRITE: io=1024.4MB, aggrb=22933KB/s, minb=22933KB/s, maxb=22933KB/s,
mint=45737msec, maxt=45737msec
 READ: io=3071.7MB, aggrb=67794KB/s, minb=67794KB/s, maxb=67794KB/s,
mint=46396msec, maxt=46396msec
WRITE: io=1024.4MB, aggrb=22607KB/s, minb=22607KB/s, maxb=22607KB/s,
mint=46396msec, maxt=46396msec
 READ: io=3071.7MB, aggrb=67536KB/s, minb=67536KB/s, maxb=67536KB/s,
mint=46573msec, maxt=46573msec
WRITE: io=1024.4MB, aggrb=22521KB/s, minb=22521KB/s, maxb=22521KB/s,
mint=46573msec, maxt=46573msec

So native is around 30-40% faster than threads according to this.
But I have a few questions now.
1. Is it safe to run cache='directsync' io='native' documentation refers to
writeback/threads?
2. How can I get even better performance? These benchmarks are from a pool
with 11 NVME Bluestore OSDs, 2x10Gb NIC. It feels pretty slow IMO.

Thanks,
Elias
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd and ceph striping

[Jason Dillaman]

On Thu, Oct 20, 2016 at 1:51 AM, Ahmed Mostafa
 wrote:
> different OSDs

PGs -- but more or less correct since the OSDs will process requests
for a particular PG sequentially and not in parallel.

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd and ceph striping

[Ahmed Mostafa]

Does this also mean that strip count can be thought of as the number of
parrallel writes to different objects at different OSDs ?

Thank you

On Thursday, 20 October 2016, Jason Dillaman  wrote:

> librbd (used by QEMU to provide RBD-backed disks) uses librados and
> provides the necessary handling for striping across multiple backing
> objects. When you don't specify "fancy" striping options via
> "--stripe-count" and "--stripe-unit", it essentially defaults to
> stripe count of 1 and stripe unit of the object size (defaults to
> 4MB).
>
> The use-case for fancy striping settings for an RBD image are images
> that have lots of small, sequential IO. The rationale for that is that
> normally these small, sequential IOs will continue to hit the same PG
> until the object boundary is crossed. However, if you were to use a
> small stripe unit that matched your normal IO size (or a small
> multiple thereof), your small, sequential IO requests would be sent to
>  PGs -- spreading the load.
>
> On Wed, Oct 19, 2016 at 12:32 PM, Ahmed Mostafa
> > wrote:
> > Hello
> >
> > From the documentation i understand that clients that uses librados must
> > perform striping for themselves, but i do not understand how could this
> be
> > if we have striping options in ceph ? i mean i can create rbd images that
> > has configuration for striping, count and unite size.
> >
> > So my question is, if i created an RBD image that have striping enabled
> and
> > configured, will that make a difference with qemu-rbd ? by difference i
> mean
> > enhancing performance of my virtual machines i/o and allowing utilizing
> the
> > cluster resources
> >
> > Thank you
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
> --
> Jason
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd and ceph striping

[Jason Dillaman]

librbd (used by QEMU to provide RBD-backed disks) uses librados and
provides the necessary handling for striping across multiple backing
objects. When you don't specify "fancy" striping options via
"--stripe-count" and "--stripe-unit", it essentially defaults to
stripe count of 1 and stripe unit of the object size (defaults to
4MB).

The use-case for fancy striping settings for an RBD image are images
that have lots of small, sequential IO. The rationale for that is that
normally these small, sequential IOs will continue to hit the same PG
until the object boundary is crossed. However, if you were to use a
small stripe unit that matched your normal IO size (or a small
multiple thereof), your small, sequential IO requests would be sent to
 PGs -- spreading the load.

On Wed, Oct 19, 2016 at 12:32 PM, Ahmed Mostafa
 wrote:
> Hello
>
> From the documentation i understand that clients that uses librados must
> perform striping for themselves, but i do not understand how could this be
> if we have striping options in ceph ? i mean i can create rbd images that
> has configuration for striping, count and unite size.
>
> So my question is, if i created an RBD image that have striping enabled and
> configured, will that make a difference with qemu-rbd ? by difference i mean
> enhancing performance of my virtual machines i/o and allowing utilizing the
> cluster resources
>
> Thank you
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] qemu-rbd and ceph striping

[Ahmed Mostafa]

Hello

>From the documentation i understand that clients that uses librados must
perform striping for themselves, but i do not understand how could this be
if we have striping options in ceph ? i mean i can create rbd images that
has configuration for striping, count and unite size.

So my question is, if i created an RBD image that have striping enabled and
configured, will that make a difference with qemu-rbd ? by difference i
mean enhancing performance of my virtual machines i/o and allowing
utilizing the cluster resources

Thank you
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Qemu+RBD recommended cache mode and AIO settings

[Ilya Dryomov]

On Tue, Mar 22, 2016 at 4:48 PM, Jason Dillaman  wrote:
>> Hi Jason,
>>
>> Le 22/03/2016 14:12, Jason Dillaman a écrit :
>> >
>> > We actually recommend that OpenStack be configured to use writeback cache
>> > [1].  If the guest OS is properly issuing flush requests, the cache will
>> > still provide crash-consistency.  By default, the cache will automatically
>> > start up in writethrough mode (when configured for writeback) until the
>> > first OS flush is received.
>> >
>>
>> Phew, that was the good reasoning then, thank you for your confirmation. :)
>>
>> >> I interpret native as kernel-managed I/O, and as the RBD through librbd
>> >> isn't exposed as a block device on the hypervisor, I configured threads
>> >> I/O for all our guest VMs.
>> >
>> > While I have nothing to empirically back up the following statement, I
>> > would actually recommend "native".  When set to "threads", QEMU will use a
>> > dispatch thread to invoke librbd IO operations instead of passing the IO
>> > request "directly" from the guest OS.  librbd itself already has its own
>> > IO dispatch thread with is enabled by default (via the
>> > rbd_non_blocking_aio config option), so adding an extra IO dispatching
>> > layer will just add additional latency / thread context switching.
>> >
>>
>> Well, if only that would be possible...
>> Here's the error message from libvirt when starting a VM with
>> native+writeback:
>>
>> """
>> native I/O needs either no disk cache or directsync cache mode, QEMU
>> will fallback to aio=threads
>> """
>>
>
> Learn something new everyday: looking at QEMU's internals, that flag actually 
> only makes a difference for local IO backends (files and devices).  
> Therefore, no need to set it for librbd volumes.

And libvirt's error message is misleading.  What they are after is
O_DIRECT modes, i.e. cache=none or cache=directsync.  cache=none != "no
disk cache"...

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Qemu+RBD recommended cache mode and AIO settings

[Jason Dillaman]

> Hi Jason,
> 
> Le 22/03/2016 14:12, Jason Dillaman a écrit :
> >
> > We actually recommend that OpenStack be configured to use writeback cache
> > [1].  If the guest OS is properly issuing flush requests, the cache will
> > still provide crash-consistency.  By default, the cache will automatically
> > start up in writethrough mode (when configured for writeback) until the
> > first OS flush is received.
> >
> 
> Phew, that was the good reasoning then, thank you for your confirmation. :)
> 
> >> I interpret native as kernel-managed I/O, and as the RBD through librbd
> >> isn't exposed as a block device on the hypervisor, I configured threads
> >> I/O for all our guest VMs.
> >
> > While I have nothing to empirically back up the following statement, I
> > would actually recommend "native".  When set to "threads", QEMU will use a
> > dispatch thread to invoke librbd IO operations instead of passing the IO
> > request "directly" from the guest OS.  librbd itself already has its own
> > IO dispatch thread with is enabled by default (via the
> > rbd_non_blocking_aio config option), so adding an extra IO dispatching
> > layer will just add additional latency / thread context switching.
> >
> 
> Well, if only that would be possible...
> Here's the error message from libvirt when starting a VM with
> native+writeback:
> 
> """
> native I/O needs either no disk cache or directsync cache mode, QEMU
> will fallback to aio=threads
> """
> 

Learn something new everyday: looking at QEMU's internals, that flag actually 
only makes a difference for local IO backends (files and devices).  Therefore, 
no need to set it for librbd volumes.

> >>> librbd has a setting called 'rbd_op_threads' which seems to be related to
> >>> AIO.
> >>> When does this kick in?
> >
> > This is related to the librbd IO dispatch thread pool.  Keep it at the
> > default value of "1" as higher settings will prevent IO flushes from
> > operating correctly.
> >
> >>>
> >>> Yes, a lot of questions where the internet gives a lot of answers.
> >>>
> >>> Some feedback would be nice!
> >>>
> >>> Thanks,
> >>>
> >>> Wido
> >>> ___
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >
> > [1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/#configuring-nova
> >
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 

Jason Dillaman 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Qemu+RBD recommended cache mode and AIO settings

[Loris Cuoghi]

Hi Jason,

Le 22/03/2016 14:12, Jason Dillaman a écrit :


We actually recommend that OpenStack be configured to use writeback cache [1].  
If the guest OS is properly issuing flush requests, the cache will still 
provide crash-consistency.  By default, the cache will automatically start up 
in writethrough mode (when configured for writeback) until the first OS flush 
is received.



Phew, that was the good reasoning then, thank you for your confirmation. :)


I interpret native as kernel-managed I/O, and as the RBD through librbd
isn't exposed as a block device on the hypervisor, I configured threads
I/O for all our guest VMs.


While I have nothing to empirically back up the following statement, I would actually recommend 
"native".  When set to "threads", QEMU will use a dispatch thread to invoke librbd IO 
operations instead of passing the IO request "directly" from the guest OS.  librbd itself already 
has its own IO dispatch thread with is enabled by default (via the rbd_non_blocking_aio config option), so 
adding an extra IO dispatching layer will just add additional latency / thread context switching.



Well, if only that would be possible...
Here's the error message from libvirt when starting a VM with 
native+writeback:


"""
native I/O needs either no disk cache or directsync cache mode, QEMU 
will fallback to aio=threads

"""


librbd has a setting called 'rbd_op_threads' which seems to be related to
AIO.
When does this kick in?


This is related to the librbd IO dispatch thread pool.  Keep it at the default value of 
"1" as higher settings will prevent IO flushes from operating correctly.



Yes, a lot of questions where the internet gives a lot of answers.

Some feedback would be nice!

Thanks,

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/#configuring-nova



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Qemu+RBD recommended cache mode and AIO settings

[Jason Dillaman]

> > I've been looking on the internet regarding two settings which might
> > influence
> > performance with librbd.
> >
> > When attaching a disk with Qemu you can set a few things:
> > - cache
> > - aio
> >
> > The default for libvirt (in both CloudStack and OpenStack) for 'cache' is
> > 'none'. Is that still the recommend value combined with librbd
> > (write)cache?
> >
> 
> We've been using "writeback" since end of last year, looking for an
> explicit writeback policy taking advantage of the librbd cache, but we
> haven't got any problem with "none" before that.
> 

We actually recommend that OpenStack be configured to use writeback cache [1].  
If the guest OS is properly issuing flush requests, the cache will still 
provide crash-consistency.  By default, the cache will automatically start up 
in writethrough mode (when configured for writeback) until the first OS flush 
is received.

> > In libvirt you can set 'io' to:
> > - native
> > - threads
> >
> > This translates to the 'aio' flags to Qemu. What is recommended here? I
> > found:
> > - io=native for block device based VMs
> > - io=threads for file-based VMs
> >
> > This seems to suggest that 'native' should be used for librbd. Is that
> > still
> > correct?
> >
> 
> I interpret native as kernel-managed I/O, and as the RBD through librbd
> isn't exposed as a block device on the hypervisor, I configured threads
> I/O for all our guest VMs.

While I have nothing to empirically back up the following statement, I would 
actually recommend "native".  When set to "threads", QEMU will use a dispatch 
thread to invoke librbd IO operations instead of passing the IO request 
"directly" from the guest OS.  librbd itself already has its own IO dispatch 
thread with is enabled by default (via the rbd_non_blocking_aio config option), 
so adding an extra IO dispatching layer will just add additional latency / 
thread context switching.

> > librbd has a setting called 'rbd_op_threads' which seems to be related to
> > AIO.
> > When does this kick in?

This is related to the librbd IO dispatch thread pool.  Keep it at the default 
value of "1" as higher settings will prevent IO flushes from operating 
correctly.

> >
> > Yes, a lot of questions where the internet gives a lot of answers.
> >
> > Some feedback would be nice!
> >
> > Thanks,
> >
> > Wido
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

[1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/#configuring-nova


-- 

Jason Dillaman 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Qemu+RBD recommended cache mode and AIO settings

[Loris Cuoghi]

Hi Wido,

Le 22/03/2016 13:52, Wido den Hollander a écrit :

Hi,

I've been looking on the internet regarding two settings which might influence
performance with librbd.

When attaching a disk with Qemu you can set a few things:
- cache
- aio

The default for libvirt (in both CloudStack and OpenStack) for 'cache' is
'none'. Is that still the recommend value combined with librbd (write)cache?



We've been using "writeback" since end of last year, looking for an 
explicit writeback policy taking advantage of the librbd cache, but we 
haven't got any problem with "none" before that.



In libvirt you can set 'io' to:
- native
- threads

This translates to the 'aio' flags to Qemu. What is recommended here? I found:
- io=native for block device based VMs
- io=threads for file-based VMs

This seems to suggest that 'native' should be used for librbd. Is that still
correct?



I interpret native as kernel-managed I/O, and as the RBD through librbd 
isn't exposed as a block device on the hypervisor, I configured threads 
I/O for all our guest VMs.



librbd has a setting called 'rbd_op_threads' which seems to be related to AIO.
When does this kick in?

Yes, a lot of questions where the internet gives a lot of answers.

Some feedback would be nice!

Thanks,

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Qemu+RBD recommended cache mode and AIO settings

[Wido den Hollander]

Hi,

I've been looking on the internet regarding two settings which might influence
performance with librbd.

When attaching a disk with Qemu you can set a few things:
- cache
- aio

The default for libvirt (in both CloudStack and OpenStack) for 'cache' is
'none'. Is that still the recommend value combined with librbd (write)cache?

In libvirt you can set 'io' to:
- native
- threads

This translates to the 'aio' flags to Qemu. What is recommended here? I found:
- io=native for block device based VMs
- io=threads for file-based VMs

This seems to suggest that 'native' should be used for librbd. Is that still
correct?

librbd has a setting called 'rbd_op_threads' which seems to be related to AIO.
When does this kick in?

Yes, a lot of questions where the internet gives a lot of answers.

Some feedback would be nice!

Thanks,

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu + rbd block driver with cache=writeback, is live migration safe ?

[Stefan Priebe - Profihost AG]

Am 13.04.2014 um 06:39 schrieb Alexandre DERUMIER aderum...@odiso.com:

 And I read:
 https://www.mail-archive.com/ceph-users@lists.ceph.com/msg06890.html
 
 Niw don't get me wrong, erring on the side of safety is quite sensible,
 but the impact of having no caching by qemu is in the order of a 2
 magnitudes easily.
 
 
 Thanks for link reference !

Works for me too. Had already done 100'drets of migrations.

Greets,
Stefan 

 
 
 
 
 - Mail original - 
 
 De: Christian Balzer ch...@gol.com 
 À: Alex Crow ac...@integrafin.co.uk 
 Cc: ceph-users@lists.ceph.com 
 Envoyé: Samedi 12 Avril 2014 17:56:07 
 Objet: Re: [ceph-users] qemu + rbd block driver with cache=writeback, is live 
 migration safe ? 
 
 
 Hello, 
 
 On Sat, 12 Apr 2014 16:26:40 +0100 Alex Crow wrote: 
 
 Hi. 
 
 I've read in many places that you should never use writeback on any kind 
 of shared storage. Caching is better dealt with on the storage side 
 anyway as you have hopefully provided resilience there. In fact if your 
 SAN/NAS is good enough it's supposed to be best to use none as the 
 caching algo. 
 
 If you need caching on the hypervisor side it would probably better to 
 use something like bcache/dmcache etc.
 And I read: 
 https://www.mail-archive.com/ceph-users@lists.ceph.com/msg06890.html 
 
 Niw don't get me wrong, erring on the side of safety is quite sensible, 
 but the impact of having no caching by qemu is in the order of a 2 
 magnitudes easily. 
 
 Beers, 
 
 Christian 
 
 Cheers 
 
 Alex 
 
 
 On 12/04/14 16:01, Alexandre DERUMIER wrote: 
 Hello, 
 
 I known that qemu live migration with disk with cache=writeback are 
 not safe with storage like nfs,iscsi... 
 
 Is it also true with rbd ? 
 
 
 If yes, it is possible to disable manually writeback online with qmp ? 
 
 Best Regards, 
 
 Alexandre 
 ___ 
 ceph-users mailing list 
 ceph-users@lists.ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___ 
 ceph-users mailing list 
 ceph-users@lists.ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 -- 
 Christian Balzer Network/Systems Engineer 
 ch...@gol.com Global OnLine Japan/Fusion Communications 
 http://www.gol.com/ 
 ___ 
 ceph-users mailing list 
 ceph-users@lists.ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] qemu + rbd block driver with cache=writeback, is live migration safe ?

[Alexandre DERUMIER]

Hello,

I known that qemu live migration with disk with cache=writeback are not safe 
with storage like nfs,iscsi...

Is it also true with rbd ?


If yes, it is possible to disable manually writeback online with qmp ?

Best Regards,

Alexandre
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu + rbd block driver with cache=writeback, is live migration safe ?

[Alex Crow]

Hi.

I've read in many places that you should never use writeback on any kind 
of shared storage. Caching is better dealt with on the storage side 
anyway as you have hopefully provided resilience there. In fact if your 
SAN/NAS is good enough it's supposed to be best to use none as the 
caching algo.


If you need caching on the hypervisor side it would probably better to 
use something like bcache/dmcache etc.


Cheers

Alex


On 12/04/14 16:01, Alexandre DERUMIER wrote:

Hello,

I known that qemu live migration with disk with cache=writeback are not safe 
with storage like nfs,iscsi...

Is it also true with rbd ?


If yes, it is possible to disable manually writeback online with qmp ?

Best Regards,

Alexandre
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu + rbd block driver with cache=writeback, is live migration safe ?

[Christian Balzer]

Hello,

On Sat, 12 Apr 2014 16:26:40 +0100 Alex Crow wrote:

 Hi.
 
 I've read in many places that you should never use writeback on any kind 
 of shared storage. Caching is better dealt with on the storage side 
 anyway as you have hopefully provided resilience there. In fact if your 
 SAN/NAS is good enough it's supposed to be best to use none as the 
 caching algo.
 
 If you need caching on the hypervisor side it would probably better to 
 use something like bcache/dmcache etc.

And I read:
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg06890.html

Niw don't get me wrong, erring on the side of safety is quite sensible,
but the impact of having no caching by qemu is in the order of a 2
magnitudes easily.

Beers,

Christian

 Cheers
 
 Alex
 
 
 On 12/04/14 16:01, Alexandre DERUMIER wrote:
  Hello,
 
  I known that qemu live migration with disk with cache=writeback are
  not safe with storage like nfs,iscsi...
 
  Is it also true with rbd ?
 
 
  If yes, it is possible to disable manually writeback online with qmp ?
 
  Best Regards,
 
  Alexandre
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu + rbd block driver with cache=writeback, is live migration safe ?

[Alexandre DERUMIER]

And I read:
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg06890.html

Niw don't get me wrong, erring on the side of safety is quite sensible,
but the impact of having no caching by qemu is in the order of a 2
magnitudes easily.


Thanks for link reference !




- Mail original - 

De: Christian Balzer ch...@gol.com 
À: Alex Crow ac...@integrafin.co.uk 
Cc: ceph-users@lists.ceph.com 
Envoyé: Samedi 12 Avril 2014 17:56:07 
Objet: Re: [ceph-users] qemu + rbd block driver with cache=writeback, is live 
migration safe ? 


Hello, 

On Sat, 12 Apr 2014 16:26:40 +0100 Alex Crow wrote: 

 Hi. 
 
 I've read in many places that you should never use writeback on any kind 
 of shared storage. Caching is better dealt with on the storage side 
 anyway as you have hopefully provided resilience there. In fact if your 
 SAN/NAS is good enough it's supposed to be best to use none as the 
 caching algo. 
 
 If you need caching on the hypervisor side it would probably better to 
 use something like bcache/dmcache etc. 
 
And I read: 
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg06890.html 

Niw don't get me wrong, erring on the side of safety is quite sensible, 
but the impact of having no caching by qemu is in the order of a 2 
magnitudes easily. 

Beers, 

Christian 

 Cheers 
 
 Alex 
 
 
 On 12/04/14 16:01, Alexandre DERUMIER wrote: 
  Hello, 
  
  I known that qemu live migration with disk with cache=writeback are 
  not safe with storage like nfs,iscsi... 
  
  Is it also true with rbd ? 
  
  
  If yes, it is possible to disable manually writeback online with qmp ? 
  
  Best Regards, 
  
  Alexandre 
  ___ 
  ceph-users mailing list 
  ceph-users@lists.ceph.com 
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
  
 
 ___ 
 ceph-users mailing list 
 ceph-users@lists.ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 


-- 
Christian Balzer Network/Systems Engineer 
ch...@gol.com Global OnLine Japan/Fusion Communications 
http://www.gol.com/ 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd

[Sebastien Han]

There is a RBD engine for FIO, have a look at 
http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html

 
Sébastien Han 
Cloud Engineer 

Always give 100%. Unless you're giving blood.” 

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

On 12 Mar 2014, at 04:25, Kyle Bader kyle.ba...@gmail.com wrote:

 I tried rbd-fuse and it's throughput using fio is approx. 1/4 that of the 
 kernel client.
 
 Can you please let me know how to setup RBD backend for FIO? I'm assuming 
 this RBD backend is also based on librbd?
 
 You will probably have to build fio from source since the rbd engine is new:
 
 https://github.com/axboe/fio
 
 Assuming you already have a cluster and a client configured this
 should do the trick:
 
 https://github.com/axboe/fio/blob/master/examples/rbd.fio
 
 -- 
 
 Kyle
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] qemu-rbd

[Sushma Gurram]

Hi,

I'm trying to follow the instructions for QEMU rbd installation at 
http://ceph.com/docs/master/rbd/qemu-rbd/

I tried to write a raw qemu image to ceph cluster using the following command
qemu-img convert -f raw -O raw ../linux-0.2.img rbd:data/linux

OSD seems to be working, but it seems to take forever. However, an rbd image 
seems to be created in the cluster's data pool.
When I use the newly created rbd image, it goes a little further in boot and 
hangs. Looks like the entire image was not written.

Could you please let me know how I could go about debugging this issue?

Thanks,
Sushma



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd

[Gregory Farnum]

On Tue, Mar 11, 2014 at 1:38 PM, Sushma Gurram
sushma.gur...@sandisk.com wrote:
 Hi,



 I'm trying to follow the instructions for QEMU rbd installation at
 http://ceph.com/docs/master/rbd/qemu-rbd/



 I tried to write a raw qemu image to ceph cluster using the following
 command

 qemu-img convert -f raw -O raw ../linux-0.2.img rbd:data/linux



 OSD seems to be working, but it seems to take forever. However, an rbd image
 seems to be created in the cluster's data pool.

Okay, so it's creating the image in the data pool because that's what
you told it to. Are you saying it seems hung, or just slow?


 When I use the newly created rbd image, it goes a little further in boot and
 hangs. Looks like the entire image was not written.

Did you wait for qemu-img to finish?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd

[Sushma Gurram]

It seems good with master branch. Sorry about the confusion.

On a side note, is it possible to create/access the block device using librbd 
and run fio on it?

Thanks,
Sushma

-Original Message-
From: Gregory Farnum [mailto:g...@inktank.com]
Sent: Tuesday, March 11, 2014 2:00 PM
To: Sushma Gurram
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] qemu-rbd

On Tue, Mar 11, 2014 at 1:38 PM, Sushma Gurram sushma.gur...@sandisk.com 
wrote:
 Hi,



 I'm trying to follow the instructions for QEMU rbd installation at
 http://ceph.com/docs/master/rbd/qemu-rbd/



 I tried to write a raw qemu image to ceph cluster using the following
 command

 qemu-img convert -f raw -O raw ../linux-0.2.img rbd:data/linux



 OSD seems to be working, but it seems to take forever. However, an rbd
 image seems to be created in the cluster's data pool.

Okay, so it's creating the image in the data pool because that's what you told 
it to. Are you saying it seems hung, or just slow?


 When I use the newly created rbd image, it goes a little further in
 boot and hangs. Looks like the entire image was not written.

Did you wait for qemu-img to finish?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com




PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd

[Gregory Farnum]

On Tue, Mar 11, 2014 at 2:24 PM, Sushma Gurram
sushma.gur...@sandisk.com wrote:
 It seems good with master branch. Sorry about the confusion.

 On a side note, is it possible to create/access the block device using librbd 
 and run fio on it?

...yes? librbd is the userspace library that QEMU is using to access
it to begin with; same with rbd-fuse. There's also the new RBD backend
for FIO, but I'm not sure what's involved in setting that up. Or did
you mean something else?
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-rbd

[Kyle Bader]

 I tried rbd-fuse and it's throughput using fio is approx. 1/4 that of the 
 kernel client.

 Can you please let me know how to setup RBD backend for FIO? I'm assuming 
 this RBD backend is also based on librbd?

You will probably have to build fio from source since the rbd engine is new:

https://github.com/axboe/fio

Assuming you already have a cluster and a client configured this
should do the trick:

https://github.com/axboe/fio/blob/master/examples/rbd.fio

-- 

Kyle
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com