Home directories probably means lots of small objects. Default minimum allocation size of BlueStore on HDD is 64 kiB, so there's a lot of overhead for everything smaller;
Details: google bluestore min alloc size, can only be changed during OSD creation Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Dec 6, 2019 at 12:57 PM Jochen Schulz <sch...@math.uni-goettingen.de> wrote: > Hi! > > Thank you! > The output of both commands are below. > I still dont understand why there are 21T used data (because 5.5T*3 = > 16.5T != 21T) and why there seems to be only 4.5 T MAX AVAIL, but the > osd output tells we have 25T free space. > > > $ sudo ceph df > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 45 TiB 24 TiB 21 TiB 21 TiB 46.33 > ssd 596 GiB 524 GiB 1.7 GiB 72 GiB 12.09 > TOTAL 46 TiB 25 TiB 21 TiB 21 TiB 45.89 > > POOLS: > POOL ID STORED OBJECTS USED %USED > MAX AVAIL > images 8 149 GiB 38.30k 354 GiB 2.52 > 4.5 TiB > cephfs_data 9 5.5 TiB 26.61M 20 TiB 60.36 > 4.5 TiB > cephfs_metadata 10 12 GiB 3.17M 13 GiB 2.57 > 164 GiB > > > $ sudo ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL > %USE VAR PGS STATUS > 0 hdd 0.89000 1.00000 931 GiB 456 GiB 453 GiB 136 MiB 3.5 GiB 475 > GiB 49.01 1.07 103 up > 1 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 100 MiB 3.9 GiB 436 > GiB 53.14 1.16 89 up > 4 hdd 0.89000 1.00000 931 GiB 345 GiB 342 GiB 108 MiB 2.9 GiB 586 > GiB 37.05 0.81 87 up > 5 hdd 0.89000 1.00000 931 GiB 521 GiB 517 GiB 108 MiB 4.1 GiB 410 > GiB 55.96 1.22 98 up > 6 hdd 0.89000 1.00000 931 GiB 367 GiB 364 GiB 95 MiB 3.2 GiB 564 > GiB 39.44 0.86 95 up > 7 hdd 0.89000 1.00000 931 GiB 540 GiB 536 GiB 77 MiB 3.7 GiB 392 > GiB 57.96 1.26 111 up > 20 hdd 0.89000 1.00000 931 GiB 382 GiB 378 GiB 60 MiB 3.3 GiB 550 > GiB 40.96 0.89 85 up > 23 hdd 1.81929 1.00000 1.8 TiB 706 GiB 701 GiB 113 MiB 4.9 GiB 1.1 > TiB 37.92 0.83 182 up > 44 hdd 0.89000 1.00000 931 GiB 468 GiB 465 GiB 34 MiB 3.3 GiB 463 > GiB 50.29 1.10 93 up > 45 hdd 1.78999 1.00000 1.8 TiB 882 GiB 875 GiB 138 MiB 6.3 GiB 981 > GiB 47.33 1.03 179 up > 46 hdd 1.78999 1.00000 1.8 TiB 910 GiB 903 GiB 127 MiB 6.4 GiB 953 > GiB 48.83 1.06 192 up > 22 ssd 0.11639 1.00000 119 GiB 15 GiB 357 MiB 12 GiB 2.8 GiB 104 > GiB 12.61 0.27 315 up > 12 hdd 0.89000 1.00000 931 GiB 499 GiB 494 GiB 64 MiB 4.5 GiB 432 > GiB 53.57 1.17 116 up > 13 hdd 0.89000 1.00000 931 GiB 536 GiB 532 GiB 48 MiB 4.4 GiB 395 > GiB 57.59 1.26 109 up > 30 hdd 0.89000 1.00000 931 GiB 510 GiB 506 GiB 33 MiB 3.9 GiB 421 > GiB 54.80 1.19 100 up > 32 hdd 0.89000 1.00000 931 GiB 495 GiB 491 GiB 56 MiB 4.1 GiB 436 > GiB 53.17 1.16 101 up > 33 hdd 0.89000 1.00000 931 GiB 333 GiB 330 GiB 56 MiB 3.1 GiB 598 > GiB 35.80 0.78 82 up > 15 ssd 0.11639 1.00000 119 GiB 14 GiB 336 MiB 11 GiB 2.9 GiB 105 > GiB 12.13 0.26 305 up > 17 hdd 0.89000 1.00000 931 GiB 577 GiB 573 GiB 77 MiB 4.4 GiB 354 > GiB 61.99 1.35 97 up > 18 hdd 0.89000 1.00000 931 GiB 413 GiB 409 GiB 70 MiB 4.0 GiB 518 > GiB 44.34 0.97 95 up > 19 hdd 1.81879 1.00000 1.8 TiB 895 GiB 889 GiB 144 MiB 5.6 GiB 967 > GiB 48.06 1.05 184 up > 21 hdd 0.89000 1.00000 931 GiB 360 GiB 357 GiB 60 MiB 3.4 GiB 570 > GiB 38.72 0.84 100 up > 31 hdd 0.90909 1.00000 931 GiB 508 GiB 505 GiB 80 MiB 3.5 GiB 423 > GiB 54.58 1.19 102 up > 25 ssd 0.11639 1.00000 119 GiB 14 GiB 339 MiB 11 GiB 2.7 GiB 105 > GiB 11.86 0.26 310 up > 8 hdd 0.89000 1.00000 931 GiB 359 GiB 356 GiB 72 MiB 3.1 GiB 572 > GiB 38.55 0.84 80 up > 9 hdd 0.89000 1.00000 931 GiB 376 GiB 373 GiB 42 MiB 3.0 GiB 555 > GiB 40.39 0.88 87 up > 24 hdd 0.89000 1.00000 931 GiB 342 GiB 339 GiB 70 MiB 2.8 GiB 590 > GiB 36.69 0.80 78 up > 26 hdd 1.78999 1.00000 1.8 TiB 921 GiB 915 GiB 129 MiB 6.1 GiB 942 > GiB 49.45 1.08 177 up > 27 hdd 1.78999 1.00000 1.8 TiB 891 GiB 885 GiB 125 MiB 5.7 GiB 972 > GiB 47.82 1.04 208 up > 35 hdd 1.81929 1.00000 1.8 TiB 819 GiB 814 GiB 110 MiB 5.3 GiB 1.0 > TiB 43.99 0.96 184 up > 29 ssd 0.11638 1.00000 119 GiB 15 GiB 339 MiB 11 GiB 2.9 GiB 105 > GiB 12.25 0.27 311 up > 14 hdd 1.81929 1.00000 1.8 TiB 889 GiB 884 GiB 1.9 MiB 4.3 GiB 974 > GiB 47.70 1.04 162 up > 28 hdd 1.81929 1.00000 1.8 TiB 609 GiB 606 GiB 3.4 MiB 2.9 GiB 1.2 > TiB 32.67 0.71 169 up > 36 hdd 1.36429 1.00000 1.4 TiB 591 GiB 589 GiB 2.0 MiB 2.9 GiB 806 > GiB 42.34 0.92 135 up > 37 hdd 1.81929 1.00000 1.8 TiB 840 GiB 836 GiB 2.5 MiB 4.1 GiB 1023 > GiB 45.12 0.98 177 up > 38 hdd 1.81929 1.00000 1.8 TiB 914 GiB 909 GiB 2.2 MiB 4.4 GiB 949 > GiB 49.05 1.07 182 up > 2 hdd 0.89000 1.00000 931 GiB 449 GiB 445 GiB 60 MiB 3.4 GiB 482 > GiB 48.19 1.05 89 up > 3 hdd 0.89000 1.00000 931 GiB 333 GiB 329 GiB 75 MiB 3.0 GiB 598 > GiB 35.73 0.78 86 up > 10 hdd 0.89000 1.00000 931 GiB 416 GiB 412 GiB 54 MiB 3.8 GiB 515 > GiB 44.68 0.97 100 up > 11 hdd 1.81879 1.00000 1.8 TiB 803 GiB 798 GiB 70 MiB 5.3 GiB 1.0 > TiB 43.14 0.94 188 up > 34 hdd 0.90909 1.00000 931 GiB 605 GiB 600 GiB 64 MiB 4.2 GiB 326 > GiB 64.95 1.42 106 up > 16 ssd 0.11639 1.00000 119 GiB 14 GiB 322 MiB 10 GiB 3.1 GiB 105 > GiB 11.59 0.25 295 up > TOTAL 46 TiB 21 TiB 21 TiB 59 GiB 165 GiB 25 > TiB 45.89 > MIN/MAX VAR: 0.25/1.42 STDDEV: 13.76 > > > > > On 06.12.19 12:27, Aleksey Gutikov wrote: > > On 6.12.19 13:29, Jochen Schulz wrote: > >> Hi! > >> > >> We have a ceph cluster with 42 OSD in production as a server providing > >> mainly home-directories of users. Ceph is 14.2.4 nautilus. > >> > >> We have 3 pools. One images (for rbd images) a cephfs_metadata and a > >> cephfs_data pool. > >> > >> Our raw data is about 5.6T. All pools have replica size 3 and there are > >> only very little snapshots in the rbd images pool, the cephfspool doesnt > >> use snapshots. > >> > >> How is it possible that the status tells us, that 21T/46T is used, > >> because thats much more than 3 times the raw size. > >> > >> Also, to make that more confusing, there as at least half of the cluster > >> free, and we get pg backfill_toofull after we added some OSDs lately. > >> The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T > >> free. > >> > >> The autoscale module seems to calculate the 20T times 3 for the space > >> needed and thus has wrong numbers (see below). > >> > >> Status of the cluster is added below too. > >> > >> how can these size/capacity numbers be explained? > >> and, would be there a recommendation to change something? > >> > >> Thank you in advance! > >> > >> best > >> Jochen > >> > >> > >> # ceph -s > >> > >> cluster: > >> id: 2b16167f-3f33-4580-a0e9-7a71978f403d > >> health: HEALTH_ERR > >> Degraded data redundancy (low space): 1 pg backfill_toofull > >> 1 subtrees have overcommitted pool target_size_bytes > >> 1 subtrees have overcommitted pool target_size_ratio > >> 2 pools have too many placement groups > >> > >> services: > >> mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d) > >> mgr: earl(active, since 24h), standbys: assam > >> mds: cephfs:1 {0=assam=up:active} 1 up:standby > >> osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped > >> pgs > >> > >> data: > >> pools: 3 pools, 2048 pgs > >> objects: 29.80M objects, 5.6 TiB > >> usage: 21 TiB used, 25 TiB / 46 TiB avail > >> pgs: 1164396/89411013 objects misplaced (1.302%) > >> 2018 active+clean > >> 22 active+remapped+backfill_wait > >> 7 active+remapped+backfilling > >> 1 active+remapped+backfill_wait+backfill_toofull > >> > >> io: > >> client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr > >> recovery: 9.2 MiB/s, 41 objects/s > >> > >> > >> # ceph osd pool autoscale-status > >> POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO > >> TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > >> images 354.2G 3.0 46100G 0.0231 > >> 1.0 1024 32 warn > >> cephfs_metadata 13260M 3.0 595.7G 0.0652 > >> 1.0 512 8 warn > >> cephfs_data 20802G 3.0 46100G 1.3537 > >> 1.0 512 warn > >> > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > > > > Please, provide output of ceph df and ceph osd df - that should explain > > both questions (21T and 82%). > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io >
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
