From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Jake
Young
Sent: 06 March 2015 12:52
To: Nick Fisk
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] tgt and krbd
On Thursday, March 5, 2015, Nick Fisk <n...@fisk.me.uk> wrote:
Hi All,
Just a heads up after a day’s experimentation.
I believe tgt with its default settings has a small write cache when exporting
a kernel mapped RBD. Doing some write tests I saw 4 times the write throughput
when using tgt aio + krbd compared to tgt with the builtin librbd.
After running the following command against the LUN, which apparently disables
write cache, Performance dropped back to what I am seeing using tgt+librbd and
also the same as fio.
tgtadm --op update --mode logicalunit --tid 2 --lun 3 -P
mode_page=8:0:18:0x10:0:0xff:0xff:0:0:0xff:0xff:0xff:0xff:0x80:0x14:0:0:0:0:0:0
From that I can only deduce that using tgt + krbd in its default state is not
100% safe to use, especially in an HA environment.
Nick
Hey Nick,
tgt actually does not have any caches. No read, no write. tgt's design is to
passthrough all commands to the backend as efficiently as possible.
http://lists.wpkg.org/pipermail/stgt/2013-May/005788.html
The configuration parameters just inform the initiators whether the backend
storage has a cache. Clearly this makes a big difference for you. What
initiator are you using with this test?
Maybe the kernel is doing the caching. What tuning parameters do you have on
the krbd disk?
It could be that using aio is much more efficient. Maybe built in lib rbd isn't
doing aio?
Jake
Hi Jake,
Hmm that’s interesting, it’s definitely effecting write behaviour though.
I was running iometer doing single io depth writes in a windows VM on ESXi
using its software initiator, which as far as I’m aware should be sending sync
writes for each request.
I saw in iostat on the tgt server that my 128kb writes were being coalesced
into ~1024kb writes, which would explain the performance increase. So something
somewhere is doing caching, albeit on a small scale.
The krbd disk was all using default settings. I know the RBD support for tgt
is using the librbd sync writes which I suppose might explain the default
difference, but this should be the expected behaviour.
Nick
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
