--- Begin Message ---
On December 1, 2022 8:37:36 AM GMT+01:00, Rainer Krienke
<[email protected]> wrote:
>Hello,
>
>I run a a hyperconverged pve cluster (V7.2) with 11 nodes. Each node has 8 4TB
>disks. pve and ceph are installed an running.
What's the intended use for these? And what disks are they!
>
>Next I wanted to create some ceph-pools with each 512 pgs. Since I want to use
>erasure coding (5+3) when creating a pool one rbd pool for metadata and the
>data pool are created. I used pveceph pool this command:
>
>pveceph pool create px-e --erasure-coding k=5,m=3 --pg_autoscale_mode off
>--pg_num 512 --pg_num_min 128
>
It's __512 * 8 / num_osds__ to get the rough amount of PGs a OSD will be
associated with.
And use 4+3, as erasure profiles with a power of two perform better.
Also the m is the amount off independent OSDs you can loose before loosing
data. Is 3 your intent?
And last but not least 5+3 will involve always 8 OSDs for a read/write. Plus
objects are split, the size of the actual chunk matters much when HDDs are used.
>I was able to create two pools in this way but the third pveceph call threw
>this error:
>
>"got unexpected control message: TASK ERROR: error with 'osd pool create':
>mon_command failed - pg_num 512 size 8 would mean 22148 total pgs, which
>exceeds max 22000 (mon_max_pg_per_osd 250 * num_in_osds 88)"
>
>What I do not understand now are the calculations behind the scenes for the
>calculated total pg number of 22148. But how is this total number "22148"
>calculated?
>
>I already reduced the number of pgs for the metadata pool of each ec-pool and
>so I was able to create 4 pools in this way. But just for fun I now tried to
>create ec-pool number 5 and I see the message from above.
>
>Here are the pools created by now (scraped from ceph osd pool
>autoscale-status):
>Pool: Size: Bias: PG_NUM:
>rbd 4599 1.0 32
>px-a-data 528.2G 1.0 512
>px-a-metadata 838.1k 1.0 128
>px-b-data 0 1.0 512
>px-b-metadata 19 1.0 128
>px-c-data 0 1.0 512
>px-c-metadata 19 1.0 128
>px-d-data 0 1.0 512
>px-d-metadata 0 1.0 128
>
>So the total number of pgs for all pools is currently 2592 which is far from
>22148 pgs?
>
>Any ideas?
>Thanks Rainer
Cheers,
Alwin
Hi Rainer,
--- End Message ---
_______________________________________________
pve-user mailing list
[email protected]
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
