Hi,
I tried experimenting with RBD striping feature:
rbd image 'volume-bd873c3f-c8c7-4270-81f8-951f65fc860c':
size 50 GiB in 12800 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 187607b9ebf28a
block_name_prefix: rbd_data.187607b9ebf28a
format: 2
features: layering, striping,
Hi!
> Btw, the first interesting find: I enabled 'rbd_balance_parent_reads' on
> the clients, and single-thread reads now scale much better, I routinely get
> similar readings from a single disk doing 4k reads with 1 thread:
It seems to me that this function should not give any gain in "real"
On 10/6/21 09:23, Zakhar Kirpichenko wrote:
Hi,
Indeed, that's a lot of CPU and RAM, the idea was to put sufficient
resources in case we want to expand the nodes with more storage and do
EC. I guess having excessive resources shouldn't hurt performance? :-)
That was also my take. Untill an
On 10/5/21 17:06, Zakhar Kirpichenko wrote:
Hi,
I built a CEPH 16.2.x cluster with relatively fast and modern hardware, and
its performance is kind of disappointing. I would very much appreciate an
advice and/or pointers :-)
The hardware is 3 x Supermicro SSG-6029P nodes, each equipped with:
These are valid points, thank you for the input!
/Z
On Wed, Oct 6, 2021 at 11:39 AM Stefan Kooman wrote:
> On 10/6/21 09:23, Zakhar Kirpichenko wrote:
> > Hi,
> >
> > Indeed, that's a lot of CPU and RAM, the idea was to put sufficient
> > resources in case we want to expand the nodes with more
That's true, but we do intend to scale this cluster as necessary. Some new
nodes are already being prepared, the only thing I'm really worried about
is further expansion: we're now getting 3+ months of lead time on some
components and vendors suggest that it will get worse :-(
/Z
On Wed, Oct 6,
I've initially disabled power-saving features, which nicely improved the
network latency.
Btw, the first interesting find: I enabled 'rbd_balance_parent_reads' on
the clients, and single-thread reads now scale much better, I routinely get
similar readings from a single disk doing 4k reads with 1
> I guess having excessive resources shouldn't hurt performance? :-)
You’d think so — but I’ve seen a situation where it seemed to.
Dedicated mon nodes with dual CPUs far in excess of what they needed. C-state
flapping appeared to negatively impact the NIC driver and network (and mon)
Hi,
Indeed, that's a lot of CPU and RAM, the idea was to put sufficient
resources in case we want to expand the nodes with more storage and do EC.
I guess having excessive resources shouldn't hurt performance? :-)
/Z
On Wed, Oct 6, 2021 at 9:26 AM Stefan Kooman wrote:
> On 10/5/21 17:06,
Hi,
Thanks for this!
As I mentioned in my original message, the latency is rather low, under
0.15 ms RTT. HDD write caches are disabled (I disabled them when setting
the cluster up and verified just now with sdparm).
/Z
On Wed, Oct 6, 2021 at 9:18 AM Christian Wuerdig <
Den ons 6 okt. 2021 kl 08:19 skrev Christian Wuerdig
:
> It's been pointed out in the past that disabling the HDD write cache can
> actually improve latency quite substantially (e.g.
> https://ceph-users.ceph.narkive.com/UU9QMu9W/disabling-write-cache-on-sata-hdds-reduces-write-latency-7-times)
>
Hm, generally ceph is mostly latency sensitive which would more translate
into IOPs limits rather than bandwidth. In a single threaded write scenario
your throughput is limited by the latency of the write path which is
generally network + OSD write path + disk. People have managed to get write
Actually recent versions of virtio-blk support discard as well. In general,
I found that virtio-scsi isn't worth it. I appreciate your input though,
thanks!
/Z
On Wed, 6 Oct 2021, 09:03 Janne Johansson, wrote:
> Den ons 6 okt. 2021 kl 06:37 skrev Zakhar Kirpichenko :
> > Got it. I don't have
Den ons 6 okt. 2021 kl 06:37 skrev Zakhar Kirpichenko :
> Got it. I don't have any specific throttling set up for RBD-backed storage.
> I also previously tested several different backends and found that virtio
> consistently produced better performance than virtio-scsi in different
> scenarios,
Got it. I don't have any specific throttling set up for RBD-backed storage.
I also previously tested several different backends and found that virtio
consistently produced better performance than virtio-scsi in different
scenarios, thus my VMs run virtio.
/Z
On Wed, Oct 6, 2021 at 7:10 AM
To be clear, I’m suspecting explicit throttling as described here:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-blockio-techniques
not impact from virtualization as such,
Hi!
The clients are KVM VMs, there's QEMU/libvirt impact for sure. I will test
with a baremetal client and see whether it performs much better.
/Z
On Wed, 6 Oct 2021, 01:29 Anthony D'Atri, wrote:
> The lead PG handling ops isn’t a factor, with RBD your volumes touch
> dozens / hundreds of
Hi!
I can post the crush map tomorrow morning, but it definitely isn't
targeting the NVME drives.
I'm having a performance issue specifically with the HDD-backed pool, where
each OSD is an NVME-backed WAL/DB + HDD-backed storage.
/Z
On Tue, 5 Oct 2021, 22:43 Tor Martin Ølberg, wrote:
> Hi
I'm not sure, fio might be showing some bogus values in the summary, I'll
check the readings again tomorrow.
Another thing I noticed is that writes seem bandwidth-limited and don't
scale well with block size and/or number of threads. I.e. one clients
writes at about the same speed regardless of
Hi Zakhar,
Out of curiosity, what does your crushmap look like? Probably a long shot
but are you sure your crush map is targeting the NVME's for the rados bench
you are performing?
Tor Martin Ølberg
On Tue, Oct 5, 2021 at 9:31 PM Christian Wuerdig <
christian.wuer...@gmail.com> wrote:
> Maybe
Maybe some info is missing but 7k write IOPs at 4k block size seem fairly
decent (as you also state) - the bandwidth automatically follows from that
so not sure what you're expecting?
I am a bit puzzled though - by my math 7k IOPS at 4k should only be
27MiB/sec - not sure how the 120MiB/sec was
Hi Marc,
Many thanks for your comment! As I mentioned, rados bench results are more
or less acceptable and explainable. RBD clients writing at ~120 MB/st tops
(regardless of the number of threads or block size btw) and reading ~50
MB/s in a single thread (I managed to read over 500 MB/s using 16
You are aware of this:
https://yourcmc.ru/wiki/Ceph_performance
I am having these results with ssd and 2.2GHz xeon and no cpu
state/freq/cpugovernor optimalization, so your results with hdd look quite ok
to me.
[@c01 ~]# rados -p rbd.ssd bench 30 write
Maintaining 16 concurrent writes of
23 matches
Mail list logo
