Hi,
i think i have a similar problem with my Octopus cluster.
$ ceph osd df | grep ssd
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
31 ssd 0.36400 1.00000 373 GiB 289 GiB 258 GiB 91 MiB 31 GiB 84 GiB 77.46 1.20
110 up ===> not rebalanced, is it normal?
46 ssd 0.36400 1.00000 373 GiB 234 GiB 209 GiB 303 KiB 25 GiB 138 GiB 62.86
0.97 88 up
47 ssd 0.36400 1.00000 373 GiB 222 GiB 198 GiB 1.9 MiB 24 GiB 151 GiB 59.51
0.92 94 up
30 ssd 0.18199 1.00000 186 GiB 111 GiB 99 GiB 16 MiB 13 GiB 75 GiB 59.76 0.92
47 up ===> smaller disk to ignore.
32 ssd 0.36400 1.00000 373 GiB 210 GiB 187 GiB 51 MiB 23 GiB 163 GiB 56.26 0.87
86 up
33 ssd 0.36400 1.00000 373 GiB 206 GiB 184 GiB 79 MiB 22 GiB 167 GiB 55.31 0.85
87 up
osd.31 have 110 PGs (and VAR=1.20).
So i have activated the ceph balancer in upmap mode (on all my pools) but no
pg_upmap_items have been created for PGs on SSD disks.
I have 2 pools (size=3) primary hold on these 6 OSDs with the following hybrid
crush rule:
rule replicated_ibo_one_copy_fast {
id 6
type replicated
min_size 2
max_size 3
step take default class ssd
step chooseleaf firstn -1 type host
step emit
step take ibo
step chooseleaf firstn 1 type host
step emit
}
"ibo" is a distant rack with hosts containing only 8TB hdd disks (for
resilience only, no performance needed).
Thanks for reading.
David.
----- Le 10 Déc 21, à 14:11, Erik Lindahl erik.lind...@gmail.com a écrit :
> Hi,
>
> We are experimenting with using manually created crush maps to pick one SSD
> as primary and and two HDD devices. Since all our HDDs have the DB & WAL on
> NVMe drives, this gives us a nice combination of pretty good write
> performance, and great read performance while keeping costs manageable for
> hundreds of TB of storage.
>
> We have 16 nodes with ~300 HDDs and four separate nodes with 64 7.6TB SSDs.
>
> However, we're noticing that the usage on the SSDs isn't very balanced at
> all - it's ranging from 26% to 52% for some reason (The balancer is active
> and seems to be happy).
>
>
> I suspect this might have to do with the placement groups now being mixed
> (i.e., each pg uses 1x SSD and 2x HDD). Is there anything we can do about
> this to achieve balanced SSD usage automatically?
>
> I've included the crush map below, just in case we/I screwed up something
> there instead :-)
>
>
> Cheers,
>
> Erik
>
>
> {
> "rule_id": 11,
> "rule_name": "1ssd_2hdd",
> "ruleset": 11,
> "type": 1,
> "min_size": 1,
> "max_size": 10,
> "steps": [
> {
> "op": "take",
> "item": -52,
> "item_name": "default~ssd"
> },
> {
> "op": "chooseleaf_firstn",
> "num": 1,
> "type": "host"
> },
> {
> "op": "emit"
> },
> {
> "op": "take",
> "item": -24,
> "item_name": "default~hdd"
> },
> {
> "op": "chooseleaf_firstn",
> "num": -1,
> "type": "host"
> },
> {
> "op": "emit"
> }
> ]
> }
>
> --
> Erik Lindahl <erik.lind...@gmail.com>
> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
