Arguably the error should say something like
" TASK ERROR: error with 'osd pool create': mon_command failed - pg_num 512
size 8 would result in 251 cumulative PGs per OSD (22148 total PG replicas on
88 ‘in’ OSDs), which exceeds the mon_max_pg_per_osd value of 250.
IMHO this is sort of mixing up max_pgs_per_osd with mon_max_pool_pg_num.
Here’s the code
auto max_pgs = max_pgs_per_osd * num_osds;
if (projected > max_pgs) {
if (pool >= 0) {
*ss << "pool id " << pool;
}
*ss << " pg_num " << pg_num << " size " << size
<< " would mean " << projected
<< " total pgs, which exceeds max " << max_pgs
<< " (mon_max_pg_per_osd " << max_pgs_per_osd
<< " * num_in_osds " << num_osds << ")";
return -ERANGE;
}
IMHO this is sort of confusing max_pgs_per_osd with mon_max_pool_pg_num.
Here’s a PR with a suggested cleanup for clarity:
https://github.com/ceph/ceph/pull/49180
> On Dec 1, 2022, at 13:00, Eugen Block <ebl...@nde.ag> wrote:
>
> Hi,
>
> you need to take into account the number of replicas as well. With 88 OSDs
> and the default max PGs per OSD of 250 you get the mentioned 22000 PGs
> (including replica): 88 x 250 = 22000.
> With EC pools each chunk counts as one replica. So you should consider
> shrinking your pools or let autoscaler do that for you.
>
> Regards
> Eugen
>
> Zitat von Rainer Krienke <krie...@uni-koblenz.de>:
>
>> Hello,
>>
>> I run a a hyperconverged pve cluster (V7.2) with 11 nodes. Each node has 8
>> 4TB disks. pve and ceph are installed an running.
>>
>> I wanted to create some ceph-pools with each 512 pgs. Since I want to use
>> erasure coding (5+3) when creating a pool one rbd pool for metadata and the
>> data pool are created. I used pveceph pool this command:
>>
>> pveceph pool create px-e --erasure-coding k=5,m=3 --pg_autoscale_mode off
>> --pg_num 512 --pg_num_min 128
>>
>> I was able to create two pools in this way but the third pveceph call threw
>> this error:
>>
>> "got unexpected control message: TASK ERROR: error with 'osd pool create':
>> mon_command failed - pg_num 512 size 8 would mean 22148 total pgs, which
>> exceeds max 22000 (mon_max_pg_per_osd 250 * num_in_osds 88)"
>>
>> I also tried the direct way to create a new pool using:
>> ceph osd pool create <pool> 512 128 erasure <profile> but the error message
>> below remains.
>>
>> What I do not understand now are the calculations behind the scenes for the
>> calculated total pg number of 22148. How is this total number "22148"
>> calculated?
>>
>> I already reduced the number of pgs for the metadata pool of each ec-pool
>> and so I was able to create 4 pools in this way. But just for fun I now
>> tried to create ec-pool number 5 and I see the message from above again.
>>
>> Here are the pools created by now (scraped from ceph osd pool
>> autoscale-status):
>> Pool: Size: Bias: PG_NUM:
>> rbd 4599 1.0 32
>> px-a-data 528.2G 1.0 512
>> px-a-metadata 838.1k 1.0 128
>> px-b-data 0 1.0 512
>> px-b-metadata 19 1.0 128
>> px-c-data 0 1.0 512
>> px-c-metadata 19 1.0 128
>> px-d-data 0 1.0 512
>> px-d-metadata 0 1.0 128
>>
>> So the total number of pgs for all pools is currently 2592 which is far from
>> 22148 pgs?
>>
>> Any ideas?
>> Thanks Rainer
>> --
>> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
>> 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
>> PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
>> 1001312
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
