Re: [ceph-users] Balancer: uneven OSDs
How many pools do you have? What does your CRUSH map look like? Wild guess: it's related to your tiny tiny disks (10 GiB) and the distribution you are seeing in df is due to uneven db/metadata allocations. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, May 29, 2019 at 1:24 AM Tarek Zegar wrote: > I enabled the balancer plugin and even tried to manually invoke it but it > won't allow any changes. Looking at ceph osd df, it's not even at all. > Thoughts? > > root@hostadmin:~# ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 1 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 > 3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156 > 6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144 > 0 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 > 5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159 > 7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141 > 2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90 > 4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99 > 8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111 > TOTAL 90 GiB 53 GiB 37 GiB 72.93 > MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67 > > > root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd > osdmaptool: osdmap file 'om' > writing upmap command output to: out.txt > checking for upmap cleanups > upmap, max-count 100, max* deviation 0.01 <---really? It's not even close > to 1% across the drives* > limiting to pools rbd (1) > *no upmaps proposed* > > > ceph balancer optimize myplan > Error EALREADY: Unable to find further optimization,or distribution is > already perfect > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Balancer: uneven OSDs
These OSDs are far too small at only 10GiB for the balancer to try and do any work. It's not uncommon for metadata like OSDMaps to exceed that size in error states and in any real deployment a single PG will be at least that large. There are probably parameters you can tweak to try and make it work, but I wouldn't bother since the behavior will be nothing like what you'd see in anything of size. -Greg On Wed, May 29, 2019 at 8:52 AM Tarek Zegar wrote: > > Can anyone help with this? Why can't I optimize this cluster, the pg counts > and data distribution is way off. > __ > > I enabled the balancer plugin and even tried to manually invoke it but it > won't allow any changes. Looking at ceph osd df, it's not even at all. > Thoughts? > > root@hostadmin:~# ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 1 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 > 3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156 > 6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144 > 0 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 > 5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159 > 7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141 > 2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90 > 4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99 > 8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111 > TOTAL 90 GiB 53 GiB 37 GiB 72.93 > MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67 > > > root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd > osdmaptool: osdmap file 'om' > writing upmap command output to: out.txt > checking for upmap cleanups > upmap, max-count 100, max deviation 0.01 <---really? It's not even close to > 1% across the drives > limiting to pools rbd (1) > no upmaps proposed > > > ceph balancer optimize myplan > Error EALREADY: Unable to find further optimization,or distribution is > already perfect > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Balancer: uneven OSDs
54.287 7f40d747a700 4 mgr[py] Loaded module_config entry > mgr/balancer/active:1 > 2019-05-29 17:06:54.287 7f40d747a700 4 mgr[py] Loaded module_config entry > mgr/balancer/max_misplaced:.50 > 2019-05-29 17:06:54.287 7f40d747a700 4 mgr[py] Loaded module_config entry > mgr/balancer/mode:upmap > 2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/active > 2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/begin_time > 2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/end_time > 2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/sleep_interval > *2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] Optimize plan > auto_2019-05-29_17:06:54* > 2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/mode > 2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/max_misplaced > 2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] Mode upmap, max > misplaced 0.50 > 2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] do_upmap > 2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/upmap_max_iterations > 2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key: > mgr/balancer/upmap_max_deviation > 2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] pools ['rbd'] > *2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] prepared 0/10 changes* > > > Inactive hide details for Oliver Freyermuth ---05/29/2019 11:59:39 AM---Hi > Tarek, Am 29.05.19 um 18:49 schrieb Tarek Zegar:Oliver Freyermuth > ---05/29/2019 11:59:39 AM---Hi Tarek, Am 29.05.19 um 18:49 schrieb Tarek > Zegar: > > From: Oliver Freyermuth > To: Tarek Zegar > Cc: ceph-users@lists.ceph.com > Date: 05/29/2019 11:59 AM > Subject: [EXTERNAL] Re: [ceph-users] Balancer: uneven OSDs > > -- > > > > Hi Tarek, > > Am 29.05.19 um 18:49 schrieb Tarek Zegar: >> Hi Oliver, >> >> Thank you for the response, I did ensure that min-client-compact-level is >> indeed Luminous (see below). I have no kernel mapped rbd clients. Ceph >> versions reports mimic. Also below is the output of ceph balancer status. >> One thing to note, I did enable the balancer after I already filled the >> cluster, not from the onset. I had hoped that it wouldn't matter, though >> your comment "if the compat-level is too old for upmap, you'll only find a >> small warning about that in the logfiles" leaves me to believe that it will >> *not* work in doing it this way, please confirm and let me know what message >> to look for in /var/log/ceph. > > it should also work well on existing clusters - we have also used it on a > Luminous cluster after it was already half-filled, and it worked well - > that's what it was made for ;-). > The only issue we encountered was that the client-compat-level needed to be > set to Luminous before enabling the balancer plugin, but since you can always > disable and re-enable a plugin, > this is not a "blocker". > > Do you see anything in the logs of the active mgr when disabling and > re-enabling the balancer plugin? > That's how we initially found the message that we needed to raise the > client-compat-level. > > Cheers, > Oliver > >> >> Thank you! >> >> root@hostadmin:~# ceph balancer status
Re: [ceph-users] Balancer: uneven OSDs
Hi Oliver
Here is the output of the active mgr log after I toggled balancer off / on,
I grep'd out only "balancer" as it was far to verbose (see below). When I
look at ceph osd df I see it optimized :)
I would like to understand two things however, why is "prepared 0/10
changes" zero if it actually did something, what in the log can I look for
before I toggled that said basically "hey balancer isn't going to work
because I still think min-client-compact-level < luminous"
Thanks for helping me in getting this working!
root@hostmonitor1:/var/log/ceph# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
1 hdd 0.0098000 B 0 B 0 B 00 0
3 hdd 0.00980 1.0 10 GiB 5.3 GiB 4.7 GiB 53.25 0.97 150
6 hdd 0.00980 1.0 10 GiB 5.6 GiB 4.4 GiB 56.07 1.03 150
0 hdd 0.0098000 B 0 B 0 B 00 0
5 hdd 0.00980 1.0 10 GiB 5.7 GiB 4.3 GiB 56.97 1.04 151
7 hdd 0.00980 1.0 10 GiB 5.2 GiB 4.8 GiB 52.35 0.96 149
2 hdd 0.0098000 B 0 B 0 B 00 0
4 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.25 1.01 150
8 hdd 0.00980 1.0 10 GiB 5.4 GiB 4.6 GiB 54.07 0.99 150
TOTAL 70 GiB 34 GiB 36 GiB 54.66
MIN/MAX VAR: 0.96/1.04 STDDEV: 1.60
2019-05-29 17:06:49.324 7f40ce42a700 0 log_channel(audit) log [DBG] :
from='client.11262 192.168.0.12:0/4104979884' entity='client.admin' cmd=
[{"prefix": "balancer off", "target": ["mgr", ""]}]: dispatch
2019-05-29 17:06:49.324 7f40ce42a700 1 mgr.server handle_command
pyc_prefix: 'balancer status'
2019-05-29 17:06:49.324 7f40ce42a700 1 mgr.server handle_command
pyc_prefix: 'balancer mode'
2019-05-29 17:06:49.324 7f40ce42a700 1 mgr.server handle_command
pyc_prefix: 'balancer on'
2019-05-29 17:06:49.324 7f40ce42a700 1 mgr.server handle_command
pyc_prefix: 'balancer off'
2019-05-29 17:06:49.324 7f40cec2b700 1 mgr[balancer] Handling command:
'{'prefix': 'balancer off', 'target': ['mgr', '']}'
2019-05-29 17:06:49.388 7f40d747a700 4 mgr[py] Loaded module_config entry
mgr/balancer/max_misplaced:.50
2019-05-29 17:06:49.388 7f40d747a700 4 mgr[py] Loaded module_config entry
mgr/balancer/mode:upmap
2019-05-29 17:06:49.539 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/active
2019-05-29 17:06:49.539 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/begin_time
2019-05-29 17:06:49.539 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/end_time
2019-05-29 17:06:49.539 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/sleep_interval
2019-05-29 17:06:54.279 7f40ce42a700 4 mgr.server handle_command
prefix=balancer on
2019-05-29 17:06:54.279 7f40ce42a700 0 log_channel(audit) log [DBG] :
from='client.11268 192.168.0.12:0/1339099349' entity='client.admin' cmd=
[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch
2019-05-29 17:06:54.279 7f40ce42a700 1 mgr.server handle_command
pyc_prefix: 'balancer status'
2019-05-29 17:06:54.279 7f40ce42a700 1 mgr.server handle_command
pyc_prefix: 'balancer mode'
2019-05-29 17:06:54.279 7f40ce42a700 1 mgr.server handle_command
pyc_prefix: 'balancer on'
2019-05-29 17:06:54.279 7f40cec2b700 1 mgr[balancer] Handling command:
'{'prefix': 'balancer on', 'target': ['mgr', '']}'
2019-05-29 17:06:54.287 7f40d747a700 4 mgr[py] Loaded module_config entry
mgr/balancer/active:1
2019-05-29 17:06:54.287 7f40d747a700 4 mgr[py] Loaded module_config entry
mgr/balancer/max_misplaced:.50
2019-05-29 17:06:54.287 7f40d747a700 4 mgr[py] Loaded module_config entry
mgr/balancer/mode:upmap
2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/active
2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/begin_time
2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/end_time
2019-05-29 17:06:54.299 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/sleep_interval
2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] Optimize plan
auto_2019-05-29_17:06:54
2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/mode
2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/max_misplaced
2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] Mode upmap, max
misplaced 0.50
2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr[balancer] do_upmap
2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_iterations
2019-05-29 17:06:54.327 7f40cd3e8700 4 mgr get_config get_config key:
mgr/balancer/upmap_max_deviation
2019-05-29
Re: [ceph-users] Balancer: uneven OSDs
Hi Tarek,
Am 29.05.19 um 18:49 schrieb Tarek Zegar:
> Hi Oliver,
>
> Thank you for the response, I did ensure that min-client-compact-level is
> indeed Luminous (see below). I have no kernel mapped rbd clients. Ceph
> versions reports mimic. Also below is the output of ceph balancer status. One
> thing to note, I did enable the balancer after I already filled the cluster,
> not from the onset. I had hoped that it wouldn't matter, though your comment
> "if the compat-level is too old for upmap, you'll only find a small warning
> about that in the logfiles" leaves me to believe that it will *not* work in
> doing it this way, please confirm and let me know what message to look for in
> /var/log/ceph.
it should also work well on existing clusters - we have also used it on a
Luminous cluster after it was already half-filled, and it worked well - that's
what it was made for ;-).
The only issue we encountered was that the client-compat-level needed to be set
to Luminous before enabling the balancer plugin, but since you can always
disable and re-enable a plugin,
this is not a "blocker".
Do you see anything in the logs of the active mgr when disabling and
re-enabling the balancer plugin?
That's how we initially found the message that we needed to raise the
client-compat-level.
Cheers,
Oliver
>
> Thank you!
>
> root@hostadmin:~# ceph balancer status
> {
> "active": true,
> "plans": [],
> "mode": "upmap"
> }
>
>
>
> root@hostadmin:~# ceph features
> {
> "mon": [
> {
> "features": "0x3ffddff8ffacfffb",
> "release": "luminous",
> "num": 3
> }
> ],
> "osd": [
> {
> "features": "0x3ffddff8ffacfffb",
> "release": "luminous",
> "num": 7
> }
> ],
> "client": [
> {
> "features": "0x3ffddff8ffacfffb",
> "release": "luminous",
> "num": 1
> }
> ],
> "mgr": [
> {
> "features": "0x3ffddff8ffacfffb",
> "release": "luminous",
> "num": 3
> }
> ]
> }
>
>
>
>
> Inactive hide details for Oliver Freyermuth ---05/29/2019 11:13:51 AM---Hi
> Tarek, what's the output of "ceph balancer status"?Oliver Freyermuth
> ---05/29/2019 11:13:51 AM---Hi Tarek, what's the output of "ceph balancer
> status"?
>
> From: Oliver Freyermuth
> To: ceph-users@lists.ceph.com
> Date: 05/29/2019 11:13 AM
> Subject: [EXTERNAL] Re: [ceph-users] Balancer: uneven OSDs
> Sent by: "ceph-users"
>
> --
>
>
>
> Hi Tarek,
>
> what's the output of "ceph balancer status"?
> In case you are using "upmap" mode, you must make sure to have a
> min-client-compat-level of at least Luminous:
> http://docs.ceph.com/docs/mimic/rados/operations/upmap/
> Of course, please be aware that your clients must be recent enough
> (especially for kernel clients).
>
> Sadly, if the compat-level is too old for upmap, you'll only find a small
> warning about that in the logfiles,
> but no error on terminal when activating the balancer or any other kind of
> erroneous / health condition.
>
> Cheers,
> Oliver
>
> Am 29.05.19 um 17:52 schrieb Tarek Zegar:
>> Can anyone help with this? Why can't I optimize this cluster, the pg counts
>> and data distribution is way off.
>> __
>>
>> I enabled the balancer plugin and even tried to manually invoke it but it
>> won't allow any changes. Looking at ceph osd df, it's not even at all.
>> Thoughts?
&g
Re: [ceph-users] Balancer: uneven OSDs
Hi Oliver,
Thank you for the response, I did ensure that min-client-compact-level is
indeed Luminous (see below). I have no kernel mapped rbd clients. Ceph
versions reports mimic. Also below is the output of ceph balancer status.
One thing to note, I did enable the balancer after I already filled the
cluster, not from the onset. I had hoped that it wouldn't matter, though
your comment "if the compat-level is too old for upmap, you'll only find a
small warning about that in the logfiles" leaves me to believe that it will
*not* work in doing it this way, please confirm and let me know what
message to look for in /var/log/ceph.
Thank you!
root@hostadmin:~# ceph balancer status
{
"active": true,
"plans": [],
"mode": "upmap"
}
root@hostadmin:~# ceph features
{
"mon": [
{
"features": "0x3ffddff8ffacfffb",
"release": "luminous",
"num": 3
}
],
"osd": [
{
"features": "0x3ffddff8ffacfffb",
"release": "luminous",
"num": 7
}
],
"client": [
{
"features": "0x3ffddff8ffacfffb",
"release": "luminous",
"num": 1
}
],
"mgr": [
{
"features": "0x3ffddff8ffacfffb",
"release": "luminous",
"num": 3
}
]
}
From: Oliver Freyermuth
To: ceph-users@lists.ceph.com
Date: 05/29/2019 11:13 AM
Subject:[EXTERNAL] Re: [ceph-users] Balancer: uneven OSDs
Sent by:"ceph-users"
Hi Tarek,
what's the output of "ceph balancer status"?
In case you are using "upmap" mode, you must make sure to have a
min-client-compat-level of at least Luminous:
http://docs.ceph.com/docs/mimic/rados/operations/upmap/
Of course, please be aware that your clients must be recent enough
(especially for kernel clients).
Sadly, if the compat-level is too old for upmap, you'll only find a small
warning about that in the logfiles,
but no error on terminal when activating the balancer or any other kind of
erroneous / health condition.
Cheers,
Oliver
Am 29.05.19 um 17:52 schrieb Tarek Zegar:
> Can anyone help with this? Why can't I optimize this cluster, the pg
counts and data distribution is way off.
> __
>
> I enabled the balancer plugin and even tried to manually invoke it but it
won't allow any changes. Looking at ceph osd df, it's not even at all.
Thoughts?
>
> root@hostadmin:~# ceph osd df
> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
> 1 hdd 0.00980 0 0 B 0 B 0 B 0 0 0
> 3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156
> 6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144
> 0 hdd 0.00980 0 0 B 0 B 0 B 0 0 0
> 5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159
> 7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141
> 2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90
> 4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99
> 8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111
> TOTAL 90 GiB 53 GiB 37 GiB 72.93
> MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67
>
>
> root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd
> osdmaptool: osdmap file 'om'
> writing upmap command output to: out.txt
> checking for upmap cleanups
> upmap, max-count 100, max*deviation 0.01 <---really? It's not even close
to 1% across the drives*
> limiting to pools rbd (1)
> *no upmaps proposed*
>
>
> ceph balancer optimize myplan
> Error EALREADY: Unable to find further optimization,or distribution is
already perfect
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
(See attached file: smime.p7s)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
smime.p7s
Description: Binary data
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Balancer: uneven OSDs
Hi Tarek, what's the output of "ceph balancer status"? In case you are using "upmap" mode, you must make sure to have a min-client-compat-level of at least Luminous: http://docs.ceph.com/docs/mimic/rados/operations/upmap/ Of course, please be aware that your clients must be recent enough (especially for kernel clients). Sadly, if the compat-level is too old for upmap, you'll only find a small warning about that in the logfiles, but no error on terminal when activating the balancer or any other kind of erroneous / health condition. Cheers, Oliver Am 29.05.19 um 17:52 schrieb Tarek Zegar: Can anyone help with this? Why can't I optimize this cluster, the pg counts and data distribution is way off. __ I enabled the balancer plugin and even tried to manually invoke it but it won't allow any changes. Looking at ceph osd df, it's not even at all. Thoughts? root@hostadmin:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 1 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156 6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144 0 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159 7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141 2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90 4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99 8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111 TOTAL 90 GiB 53 GiB 37 GiB 72.93 MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67 root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd osdmaptool: osdmap file 'om' writing upmap command output to: out.txt checking for upmap cleanups upmap, max-count 100, max*deviation 0.01 <---really? It's not even close to 1% across the drives* limiting to pools rbd (1) *no upmaps proposed* ceph balancer optimize myplan Error EALREADY: Unable to find further optimization,or distribution is already perfect ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Balancer: uneven OSDs
I had this with balancer active and "crush-compat" MIN/MAX VAR: 0.43/1.59 STDDEV: 10.81 And by increasing the pg of some pools (from 8 to 64) and deleting empty pools, I went to this MIN/MAX VAR: 0.59/1.28 STDDEV: 6.83 (Do not want to go to this upmap yet) -Original Message- From: Tarek Zegar [mailto:tze...@us.ibm.com] Sent: woensdag 29 mei 2019 17:52 To: ceph-users Subject: *SPAM***** [ceph-users] Balancer: uneven OSDs Can anyone help with this? Why can't I optimize this cluster, the pg counts and data distribution is way off. __ I enabled the balancer plugin and even tried to manually invoke it but it won't allow any changes. Looking at ceph osd df, it's not even at all. Thoughts? root@hostadmin:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 1 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156 6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144 0 hdd 0.00980 0 0 B 0 B 0 B 0 0 0 5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159 7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141 2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90 4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99 8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111 TOTAL 90 GiB 53 GiB 37 GiB 72.93 MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67 root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd osdmaptool: osdmap file 'om' writing upmap command output to: out.txt checking for upmap cleanups upmap, max-count 100, max deviation 0.01 <---really? It's not even close to 1% across the drives limiting to pools rbd (1) no upmaps proposed ceph balancer optimize myplan Error EALREADY: Unable to find further optimization,or distribution is already perfect ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Balancer: uneven OSDs
Can anyone help with this? Why can't I optimize this cluster, the pg counts and data distribution is way off. __ I enabled the balancer plugin and even tried to manually invoke it but it won't allow any changes. Looking at ceph osd df, it's not even at all. Thoughts? root@hostadmin:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL%USE VAR PGS 1 hdd 0.0098000 B 0 B 0 B 00 0 3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156 6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144 0 hdd 0.0098000 B 0 B 0 B 00 0 5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159 7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141 2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90 4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99 8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111 TOTAL 90 GiB 53 GiB 37 GiB 72.93 MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67 root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd osdmaptool: osdmap file 'om' writing upmap command output to: out.txt checking for upmap cleanups upmap, max-count 100, max deviation 0.01 <---really? It's not even close to 1% across the drives limiting to pools rbd (1) no upmaps proposed ceph balancer optimize myplan Error EALREADY: Unable to find further optimization,or distribution is already perfect ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Balancer: uneven OSDs
I enabled the balancer plugin and even tried to manually invoke it but it won't allow any changes. Looking at ceph osd df, it's not even at all. Thoughts? root@hostadmin:~# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL%USE VAR PGS 1 hdd 0.0098000 B 0 B 0 B 00 0 3 hdd 0.00980 1.0 10 GiB 8.3 GiB 1.7 GiB 82.83 1.14 156 6 hdd 0.00980 1.0 10 GiB 8.4 GiB 1.6 GiB 83.77 1.15 144 0 hdd 0.0098000 B 0 B 0 B 00 0 5 hdd 0.00980 1.0 10 GiB 9.0 GiB 1021 MiB 90.03 1.23 159 7 hdd 0.00980 1.0 10 GiB 7.7 GiB 2.3 GiB 76.57 1.05 141 2 hdd 0.00980 1.0 10 GiB 5.5 GiB 4.5 GiB 55.42 0.76 90 4 hdd 0.00980 1.0 10 GiB 5.9 GiB 4.1 GiB 58.78 0.81 99 8 hdd 0.00980 1.0 10 GiB 6.3 GiB 3.7 GiB 63.12 0.87 111 TOTAL 90 GiB 53 GiB 37 GiB 72.93 MIN/MAX VAR: 0.76/1.23 STDDEV: 12.67 root@hostadmin:~# osdmaptool om --upmap out.txt --upmap-pool rbd osdmaptool: osdmap file 'om' writing upmap command output to: out.txt checking for upmap cleanups upmap, max-count 100, max deviation 0.01 <---really? It's not even close to 1% across the drives limiting to pools rbd (1) no upmaps proposed ceph balancer optimize myplan Error EALREADY: Unable to find further optimization,or distribution is already perfect ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
