Hi,
Problem: The balancer still refuses to run and distribution is not what it was
on Luminous.
I presume a flag somewhere wasn't unset whilst the autoscaling was reducing
PGs? I wasted a lot of time this last week after having enabled
pg_autoscale_mode on pools in a cluster. I had to manually reweight some OSDs
that were near full and the pg_num changes would constantly degrade redundancy
so the balancer wouldn't run. I eventually disabled the autoscaling mode on
pools and manually set pg_num on pools. The cluster has finished moving
everything around and everything is back to full health again.
I was planning on setting the reweight on the OSDs I'd changed back to 1 and
then immediately running a manual balancer plan. I could also run a manual
balancing plan first, adjust the reweight a little bit and repeat this cycle
until all OSDs again have a reweight of 1.
Commands:
ceph balancer eval
ceph balancer optimize myplan
ceph balancer eval myplan
ceph balancer show myplan
ceph balancer execute myplan
ceph balancer reset
Output:
current cluster score 0.018841 (lower is better)
Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is
decreasing, or distribution is already perfect
Error EINVAL: option "myplan" not a plan or a pool
Error ENOENT: plan myplan not found
Error ENOENT: plan myplan not found
Balancer mode:
[admin@kvm7e ~]# ceph balancer status
{
"active": true,
"plans": [],
"mode": "upmap"
}
Current distribution on Ceph Nautilus:
[admin@kvm7e ~]# ceph osd df | sort -g -k 5
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS
100 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 6.8 MiB 3.1 GiB 509 GiB
72.68 0.97 56 up
101 hdd 1.81929 0.89999 1.8 TiB 1.3 TiB 1.3 TiB 5.2 MiB 3.0 GiB 523 GiB
71.92 0.96 56 up
200 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 5.8 MiB 3.3 GiB 373 GiB
79.99 1.07 62 up
201 hdd 1.81929 0.84999 1.8 TiB 1.5 TiB 1.5 TiB 5.3 MiB 3.4 GiB 298 GiB
83.98 1.13 65 up
300 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 5.8 MiB 3.2 GiB 311 GiB
83.31 1.12 65 up
301 hdd 1.81929 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 6.2 MiB 3.2 GiB 385 GiB
79.31 1.06 61 up
400 hdd 1.81929 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 4.4 MiB 2.8 GiB 446 GiB
76.06 1.02 61 up
401 hdd 1.81929 0.84999 1.8 TiB 1.4 TiB 1.4 TiB 4.7 MiB 3.2 GiB 413 GiB
77.82 1.04 62 up
500 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 3.9 MiB 2.7 GiB 546 GiB
70.69 0.95 56 up
501 hdd 1.81929 0.89999 1.8 TiB 1.3 TiB 1.3 TiB 4.1 MiB 3.1 GiB 520 GiB
72.07 0.97 56 up
102 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 13 MiB 8.0 GiB 1.5 TiB
72.76 0.98 179 up
103 hdd 5.45789 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 17 MiB 10 GiB 1.2 TiB
78.54 1.05 185 up
202 hdd 5.45789 0.89999 5.5 TiB 3.9 TiB 3.9 TiB 7.9 MiB 8.3 GiB 1.6 TiB
71.08 0.95 170 up
203 hdd 5.45789 0.89999 5.5 TiB 4.0 TiB 4.0 TiB 11 MiB 9.0 GiB 1.4 TiB
73.44 0.98 174 up
302 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 13 MiB 9.3 GiB 1.3 TiB
77.05 1.03 185 up
303 hdd 5.45789 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 13 MiB 10 GiB 1.1 TiB
78.99 1.06 191 up
402 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 18 MiB 8.9 GiB 1.2 TiB
77.44 1.04 186 up
403 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 13 MiB 8.5 GiB 1.2 TiB
77.46 1.04 185 up
502 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 18 MiB 8.9 GiB 1.2 TiB
77.85 1.04 186 up
503 hdd 5.45789 1.00000 5.5 TiB 4.1 TiB 4.1 TiB 13 MiB 9.6 GiB 1.3 TiB
75.32 1.01 181 up
104 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 22 MiB 14 GiB 2.1 TiB
76.43 1.02 305 up
105 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 23 MiB 14 GiB 2.1 TiB
77.34 1.04 311 up
204 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 31 MiB 15 GiB 2.1 TiB
77.17 1.03 311 up
205 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 20 MiB 14 GiB 2.1 TiB
76.82 1.03 312 up
304 hdd 9.09560 0.89999 9.1 TiB 6.4 TiB 6.4 TiB 18 MiB 13 GiB 2.7 TiB
70.45 0.94 283 up
305 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 17 MiB 15 GiB 2.1 TiB
76.93 1.03 311 up
404 hdd 9.09560 0.89999 9.1 TiB 6.6 TiB 6.6 TiB 18 MiB 14 GiB 2.5 TiB
72.58 0.97 288 up
405 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 22 MiB 14 GiB 2.1 TiB
77.20 1.03 311 up
504 hdd 9.09560 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 30 MiB 14 GiB 2.0 TiB
77.68 1.04 311 up
505 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.8 TiB 24 MiB 14 GiB 2.2 TiB
75.37 1.01 304 up
10100 ssd 0.37700 1.00000 386 GiB 68 GiB 67 GiB 4 KiB 1024 MiB 318 GiB
17.65 0.24 35 up
10101 ssd 0.37700 1.00000 386 GiB 68 GiB 67 GiB 24 KiB 1024 MiB 318 GiB
17.73 0.24 36 up
10200 ssd 0.37700 1.00000 386 GiB 65 GiB 64 GiB 28 KiB 1024 MiB 321 GiB
16.82 0.23 40 up
10201 ssd 0.37700 1.00000 386 GiB 57 GiB 56 GiB 20 KiB 1024 MiB 329 GiB
14.68 0.20 37 up
10500 ssd 0.37700 1.00000 386 GiB 63 GiB 62 GiB 12 KiB 1024 MiB 323 GiB
16.29 0.22 38 up
10501 ssd 0.37700 1.00000 386 GiB 77 GiB 76 GiB 41 KiB 1024 MiB 309 GiB
19.98 0.27 40 up
10300 ssd 0.40630 1.00000 416 GiB 68 GiB 67 GiB 23 MiB 1001 MiB 348 GiB
16.42 0.22 38 up
10301 ssd 0.40630 1.00000 416 GiB 71 GiB 70 GiB 28 MiB 996 MiB 345 GiB
17.06 0.23 39 up
10400 ssd 0.40630 1.00000 416 GiB 67 GiB 66 GiB 31 MiB 993 MiB 349 GiB
16.14 0.22 40 up
10401 ssd 0.40630 1.00000 416 GiB 68 GiB 67 GiB 29 MiB 995 MiB 348 GiB
16.38 0.22 41 up
TOTAL 168 TiB 125 TiB 125 TiB 524 MiB 271 GiB 43 TiB
74.60
Previous distribution, also on Ceph Nautilus, shortly after upgrading from
Luminous:
[admin@kvm7e ~]# ceph osd df | sort -g -k 5
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS
100 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 9.2 MiB 2.5 GiB 490 GiB
73.71 1.01 59 up
101 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.2 MiB 2.5 GiB 489 GiB
73.77 1.01 58 up
200 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.1 MiB 2.5 GiB 492 GiB
73.61 1.01 59 up
201 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.2 MiB 2.5 GiB 491 GiB
73.65 1.01 58 up
300 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 6.1 MiB 2.5 GiB 490 GiB
73.68 1.01 58 up
301 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 13 MiB 2.5 GiB 490 GiB
73.72 1.01 59 up
400 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.6 MiB 2.5 GiB 489 GiB
73.75 1.01 59 up
401 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.7 MiB 2.5 GiB 493 GiB
73.56 1.01 59 up
500 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.8 MiB 2.5 GiB 491 GiB
73.66 1.01 59 up
501 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 10 MiB 2.5 GiB 491 GiB
73.63 1.01 58 up
102 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4 TiB
73.63 1.01 175 up
103 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 20 MiB 7.4 GiB 1.4 TiB
73.49 1.00 176 up
202 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4 TiB
73.60 1.01 175 up
203 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 18 MiB 7.4 GiB 1.4 TiB
73.51 1.00 175 up
302 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 13 MiB 7.4 GiB 1.4 TiB
73.57 1.01 174 up
303 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 22 MiB 7.4 GiB 1.4 TiB
73.56 1.01 177 up
402 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 25 MiB 7.4 GiB 1.4 TiB
73.51 1.00 176 up
403 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 18 MiB 7.4 GiB 1.4 TiB
73.59 1.01 175 up
502 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 22 MiB 7.4 GiB 1.4 TiB
73.62 1.01 174 up
503 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4 TiB
73.54 1.01 177 up
104 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 28 MiB 13 GiB 2.2 TiB
75.68 1.03 304 up
105 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 23 MiB 13 GiB 2.2 TiB
75.63 1.03 301 up
204 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB
75.65 1.03 301 up
205 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 41 MiB 13 GiB 2.2 TiB
75.71 1.03 302 up
304 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 42 MiB 13 GiB 2.2 TiB
75.68 1.03 302 up
305 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB
75.67 1.03 304 up
404 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB
75.68 1.03 298 up
405 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2 TiB
75.69 1.03 301 up
504 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 38 MiB 13 GiB 2.2 TiB
75.67 1.03 301 up
505 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 42 MiB 13 GiB 2.2 TiB
75.67 1.03 300 up
10100 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 28 MiB 996 MiB 376 GiB
9.60 0.13 35 up
10101 ssd 0.40630 1.00000 416 GiB 44 GiB 43 GiB 20 MiB 1004 MiB 372 GiB
10.57 0.14 37 up
10200 ssd 0.40630 1.00000 416 GiB 32 GiB 31 GiB 15 MiB 1009 MiB 384 GiB
7.80 0.11 36 up
10201 ssd 0.40630 1.00000 416 GiB 32 GiB 31 GiB 16 MiB 1008 MiB 384 GiB
7.67 0.10 37 up
10300 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 17 MiB 1007 MiB 376 GiB
9.71 0.13 36 up
10301 ssd 0.40630 1.00000 416 GiB 43 GiB 42 GiB 18 MiB 1006 MiB 373 GiB
10.31 0.14 37 up
10400 ssd 0.40630 1.00000 416 GiB 34 GiB 33 GiB 17 MiB 1007 MiB 382 GiB
8.14 0.11 35 up
10401 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 19 MiB 1005 MiB 376 GiB
9.64 0.13 36 up
10500 ssd 0.40630 1.00000 416 GiB 38 GiB 37 GiB 18 MiB 1006 MiB 378 GiB
9.11 0.12 37 up
10501 ssd 0.40630 1.00000 416 GiB 46 GiB 45 GiB 18 MiB 1006 MiB 370 GiB
10.98 0.15 37 up
TOTAL 168 TiB 123 TiB 123 TiB 839 MiB 234 GiB 45 TiB
73.16
Additional cluster information:
[admin@kvm7e ~]# ceph -s
cluster:
id: cf07a431-7d5c-45df-ab7d-7c92ea7b5cd1
health: HEALTH_WARN
15 pgs not scrubbed in time
services:
mon: 3 daemons, quorum kvm7a,kvm7b,kvm7e (age 19h)
mgr: kvm7e(active, since 2d), standbys: kvm7a, kvm7b
mds: cephfs:1 {0=kvm7e=up:active} 2 up:standby
osd: 40 osds: 40 up (since 3h), 40 in (since 2d)
data:
pools: 7 pools, 1251 pgs
objects: 18.85M objects, 72 TiB
usage: 125 TiB used, 43 TiB / 168 TiB avail
pgs: 1250 active+clean
1 active+clean+scrubbing
io:
client: 23 MiB/s rd, 22 MiB/s wr, 193 op/s rd, 12 op/s wr
cache: 13 MiB/s flush, 2 op/s promote
[admin@kvm7e ~]# ceph osd dump
epoch 13680
fsid cf07a431-7d5c-45df-ab7d-7c92ea7b5cd1
created 2019-06-07 11:49:30.007615
modified 2019-08-03 20:23:01.282690
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 203
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client luminous
require_osd_release nautilus
pool 1 'rbd_ssd' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 12003 lfor 0/9743/11876 flags
hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps [1~3]
pool 5 'cephfs_data' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 5 pgp_num 5 last_change 13397 lfor 0/9750/9748 flags hashpspool
stripe_width 0 application cephfs
pool 6 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 4 pgp_num 4 last_change 13397 lfor 0/9735/9733 flags hashpspool
stripe_width 0 application cephfs
pool 7 'rbd_hdd' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 13448 lfor 369/10607/13428 flags
hashpspool,selfmanaged_snaps tiers 9 read_tier 9 write_tier 9 stripe_width 0
application rbd
removed_snaps [1~3]
pool 8 'ec_hdd' erasure size 5 min_size 4 crush_rule 3 object_hash rjenkins
pg_num 1050 pgp_num 1050 last_change 13397 lfor 384/12035/12033 flags
hashpspool,ec_overwrites,selfmanaged_snaps tiers 10 read_tier 10 write_tier 10
stripe_width 12288 application rbd
removed_snaps [1~3]
pool 9 'rbd_hdd_cache' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 32 pgp_num 32 last_change 13483 lfor 369/9733/13462 flags
hashpspool,incomplete_clones,selfmanaged_snaps tier_of 7 cache_mode writeback
target_bytes 137438953472 hit_set bloom{false_positive_probability: 0.05,
target_size: 0, seed: 0} 14400s x12 decay_rate 0 search_last_n 0
min_read_recency_for_promote 2 min_write_recency_for_promote 2 stripe_width 0
removed_snaps [1~3]
pool 10 'ec_hdd_cache' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 32 pgp_num 32 last_change 12035 lfor 384/384/384 flags
hashpspool,incomplete_clones,selfmanaged_snaps tier_of 8 cache_mode writeback
target_bytes 137438953472 hit_set bloom{false_positive_probability: 0.05,
target_size: 0, seed: 0} 14400s x12 decay_rate 0 search_last_n 0
min_read_recency_for_promote 2 min_write_recency_for_promote 2 stripe_width 0
removed_snaps [1~3]
max_osd 10502
osd.100 up in weight 1 up_from 13526 up_thru 13677 down_at 13520
last_clean_interval [10434,13519)
[v2:10.250.1.2:6858/214607,v1:10.250.1.2:6859/214607]
[v2:10.250.1.2:6860/214607,v1:10.250.1.2:6861/214607] exists,up
11cc23aa-021c-4145-b051-9492572ce41b
osd.101 up in weight 0.899994 up_from 13517 up_thru 13677 down_at 13509
last_clean_interval [10434,13508)
[v2:10.250.1.2:6837/198048,v1:10.250.1.2:6839/198048]
[v2:10.250.1.2:6841/198048,v1:10.250.1.2:6843/198048] exists,up
bb75d594-c2ee-4606-a101-bdac0990031c
osd.102 up in weight 1 up_from 13499 up_thru 13677 down_at 13494
last_clean_interval [10438,13493)
[v2:10.250.1.2:6850/61445,v1:10.250.1.2:6851/61445]
[v2:10.250.1.2:6852/61445,v1:10.250.1.2:6853/61445] exists,up
03b5b5bf-b2ae-49dd-9244-daed6a0fdfed
osd.103 up in weight 1 up_from 13529 up_thru 13677 down_at 13524
last_clean_interval [11526,13523)
[v2:10.250.1.2:6834/251484,v1:10.250.1.2:6835/251484]
[v2:10.250.1.2:6836/251484,v1:10.250.1.2:6838/251484] exists,up
98346e6a-6081-4c22-93cc-685d920315ae
osd.104 up in weight 1 up_from 13513 up_thru 13677 down_at 13502
last_clean_interval [10446,13501)
[v2:10.250.1.2:6803/180986,v1:10.250.1.2:6806/180986]
[v2:10.250.1.2:6808/180986,v1:10.250.1.2:6810/180986] exists,up
23ff8534-df68-4c94-b7f1-d6a3b03ec267
osd.105 up in weight 1 up_from 13504 up_thru 13677 down_at 13496
last_clean_interval [11944,13495)
[v2:10.250.1.2:6824/114849,v1:10.250.1.2:6826/114849]
[v2:10.250.1.2:6828/114849,v1:10.250.1.2:6829/114849] exists,up
29dbbe08-735c-4094-80df-f93f3f37ec57
osd.200 up in weight 1 up_from 13562 up_thru 13677 down_at 13555
last_clean_interval [10683,13554)
[v2:10.250.1.3:6819/2991067,v1:10.250.1.3:6822/2991067]
[v2:10.250.1.3:6824/2991067,v1:10.250.1.3:6827/2991067] exists,up
71d27d56-59bf-4b89-ab19-a43602fee8cf
osd.201 up in weight 0.849994 up_from 13552 up_thru 13677 down_at 13543
last_clean_interval [10679,13542)
[v2:10.250.1.3:6818/2962845,v1:10.250.1.3:6821/2962845]
[v2:10.250.1.3:6825/2962845,v1:10.250.1.3:6828/2962845] exists,up
3865fc54-bc1a-406c-b4da-40420979f1e9
osd.202 up in weight 0.899994 up_from 13539 up_thru 13677 down_at 13533
last_clean_interval [10689,13532)
[v2:10.250.1.3:6851/2932338,v1:10.250.1.3:6853/2932338]
[v2:10.250.1.3:6855/2932338,v1:10.250.1.3:6857/2932338] exists,up
f0da6bae-bc91-4e49-a97c-1e3b0a8edc8b
osd.203 up in weight 0.899994 up_from 13564 up_thru 13677 down_at 13560
last_clean_interval [12286,13559)
[v2:10.250.1.3:6842/3004897,v1:10.250.1.3:6843/3004897]
[v2:10.250.1.3:6844/3004897,v1:10.250.1.3:6845/3004897] exists,up
49f58c54-c420-4e80-a837-bbd7213d37d3
osd.204 up in weight 1 up_from 13557 up_thru 13677 down_at 13550
last_clean_interval [10691,13549)
[v2:10.250.1.3:6820/2985153,v1:10.250.1.3:6823/2985153]
[v2:10.250.1.3:6826/2985153,v1:10.250.1.3:6830/2985153] exists,up
1f45ea2a-ce81-4b58-ba44-3f23849a0d33
osd.205 up in weight 1 up_from 13547 up_thru 13677 down_at 13537
last_clean_interval [10694,13536)
[v2:10.250.1.3:6850/2956087,v1:10.250.1.3:6852/2956087]
[v2:10.250.1.3:6854/2956087,v1:10.250.1.3:6856/2956087] exists,up
b6af027b-e2a1-4682-a348-95d62064dd7c
osd.300 up in weight 1 up_from 13595 up_thru 13677 down_at 13589
last_clean_interval [3035,13588)
[v2:10.250.1.4:6843/3569006,v1:10.250.1.4:6845/3569006]
[v2:10.250.1.4:6847/3569006,v1:10.250.1.4:6848/3569006] exists,up
f97770f2-8370-4c50-9f5a-661bad636581
osd.301 up in weight 1 up_from 13597 up_thru 13677 down_at 13593
last_clean_interval [3032,13592)
[v2:10.250.1.4:6840/3570563,v1:10.250.1.4:6841/3570563]
[v2:10.250.1.4:6842/3570563,v1:10.250.1.4:6844/3570563] exists,up
6a36c28a-98fc-4feb-9151-f8ced7934e11
osd.302 up in weight 1 up_from 13591 up_thru 13677 down_at 13585
last_clean_interval [3049,13584)
[v2:10.250.1.4:6856/3567489,v1:10.250.1.4:6857/3567489]
[v2:10.250.1.4:6858/3567489,v1:10.250.1.4:6859/3567489] exists,up
97a86edc-67ae-46db-bfc6-eb84cd1ddfb8
osd.303 up in weight 1 up_from 13579 up_thru 13677 down_at 13573
last_clean_interval [3038,13572)
[v2:10.250.1.4:6818/3549971,v1:10.250.1.4:6819/3549971]
[v2:10.250.1.4:6820/3549971,v1:10.250.1.4:6821/3549971] exists,up
05ee7eb9-01bf-480a-bd0a-21dbdcd49b9c
osd.304 up in weight 0.899994 up_from 13583 up_thru 13677 down_at 13577
last_clean_interval [3045,13576)
[v2:10.250.1.4:6826/3556874,v1:10.250.1.4:6827/3556874]
[v2:10.250.1.4:6828/3556874,v1:10.250.1.4:6829/3556874] exists,up
d1ffa811-5205-4d1e-919f-a15c0f0354af
osd.305 up in weight 1 up_from 13587 up_thru 13677 down_at 13581
last_clean_interval [3043,13580)
[v2:10.250.1.4:6800/3563705,v1:10.250.1.4:6801/3563705]
[v2:10.250.1.4:6834/3563705,v1:10.250.1.4:6835/3563705] exists,up
6b3ce147-4d11-45c7-b39d-9ae6f8c3652d
osd.400 up in weight 1 up_from 13619 up_thru 13677 down_at 13612
last_clean_interval [2999,13611)
[v2:10.250.1.5:6800/3440750,v1:10.250.1.5:6801/3440750]
[v2:10.250.1.5:6802/3440750,v1:10.250.1.5:6803/3440750] exists,up
93c4fb8a-7cac-4c35-8a4e-60f6129b2624
osd.401 up in weight 0.849994 up_from 13627 up_thru 13677 down_at 13621
last_clean_interval [3001,13620)
[v2:10.250.1.5:6816/3451440,v1:10.250.1.5:6817/3451440]
[v2:10.250.1.5:6818/3451440,v1:10.250.1.5:6819/3451440] exists,up
0415bcaa-2531-49f6-bd53-3c9b059c2cca
osd.402 up in weight 1 up_from 13642 up_thru 13677 down_at 13629
last_clean_interval [3005,13628)
[v2:10.250.1.5:6808/3456506,v1:10.250.1.5:6809/3456506]
[v2:10.250.1.5:6810/3456506,v1:10.250.1.5:6811/3456506] exists,up
fe1771d3-5990-43ad-9e50-d91f8b526bc4
osd.403 up in weight 1 up_from 13614 up_thru 13677 down_at 13601
last_clean_interval [3019,13600)
[v2:10.250.1.5:6822/3438702,v1:10.250.1.5:6825/3438702]
[v2:10.250.1.5:6828/3438702,v1:10.250.1.5:6831/3438702] exists,up
92f38a49-ae2d-41cf-a3a2-ce4d05f90254
osd.404 up in weight 0.899994 up_from 13645 up_thru 13677 down_at 13640
last_clean_interval [3014,13639)
[v2:10.250.1.5:6823/3465084,v1:10.250.1.5:6829/3465084]
[v2:10.250.1.5:6851/3465084,v1:10.250.1.5:6852/3465084] exists,up
cbc6cb36-1c0e-4621-83dc-2a5f54b0fe87
osd.405 up in weight 1 up_from 13623 up_thru 13677 down_at 13617
last_clean_interval [3010,13616)
[v2:10.250.1.5:6848/3449867,v1:10.250.1.5:6849/3449867]
[v2:10.250.1.5:6855/3449867,v1:10.250.1.5:6856/3449867] exists,up
f80757ab-358c-4c90-ae80-f15a21e869d0
osd.500 up in weight 1 up_from 13669 up_thru 13669 down_at 13663
last_clean_interval [2969,13662)
[v2:10.250.1.6:6810/3764835,v1:10.250.1.6:6812/3764835]
[v2:10.250.1.6:6814/3764835,v1:10.250.1.6:6816/3764835] exists,up
bb06e92f-f816-4679-a3b7-bb1280842fe4
osd.501 up in weight 0.899994 up_from 13651 up_thru 13651 down_at 13647
last_clean_interval [2970,13646)
[v2:10.250.1.6:6858/3751653,v1:10.250.1.6:6859/3751653]
[v2:10.250.1.6:6860/3751653,v1:10.250.1.6:6861/3751653] exists,up
80031729-eefa-4ae2-9dd2-672802fcf7f4
osd.502 up in weight 1 up_from 13677 up_thru 13677 down_at 13667
last_clean_interval [2982,13666)
[v2:10.250.1.6:6839/3767566,v1:10.250.1.6:6842/3767566]
[v2:10.250.1.6:6845/3767566,v1:10.250.1.6:6848/3767566] exists,up
68edb6b0-ef54-41dd-85fa-ae7bf266acdf
osd.503 up in weight 1 up_from 13656 up_thru 13656 down_at 13649
last_clean_interval [2972,13648)
[v2:10.250.1.6:6811/3754044,v1:10.250.1.6:6813/3754044]
[v2:10.250.1.6:6815/3754044,v1:10.250.1.6:6817/3754044] exists,up
cf84dd7b-8113-4178-a027-3d9b38aca683
osd.504 up in weight 1 up_from 13665 up_thru 13665 down_at 13659
last_clean_interval [2975,13658)
[v2:10.250.1.6:6834/3763908,v1:10.250.1.6:6835/3763908]
[v2:10.250.1.6:6836/3763908,v1:10.250.1.6:6837/3763908] exists,up
ea02de0f-0bdf-4ea1-8101-84e7a3282087
osd.505 up in weight 1 up_from 13661 up_thru 13661 down_at 13653
last_clean_interval [2976,13652)
[v2:10.250.1.6:6840/3759237,v1:10.250.1.6:6843/3759237]
[v2:10.250.1.6:6846/3759237,v1:10.250.1.6:6849/3759237] exists,up
8f5e335e-d2a2-4899-a026-0b0c25378edc
osd.10100 up in weight 1 up_from 13522 up_thru 13678 down_at 13515
last_clean_interval [10432,13514)
[v2:10.250.1.2:6802/199825,v1:10.250.1.2:6804/199825]
[v2:10.250.1.2:6805/199825,v1:10.250.1.2:6807/199825] exists,up
dc24ae90-6ac9-4414-a7ee-086cbb5de4b5
osd.10101 up in weight 1 up_from 13511 up_thru 13675 down_at 13507
last_clean_interval [10429,13506)
[v2:10.250.1.2:6814/182479,v1:10.250.1.2:6816/182479]
[v2:10.250.1.2:6818/182479,v1:10.250.1.2:6820/182479] exists,up
733b43e5-988c-4f1d-9161-b9b55e7db98e
osd.10200 up in weight 1 up_from 13535 up_thru 13678 down_at 13531
last_clean_interval [10681,13530)
[v2:10.250.1.3:6803/2919718,v1:10.250.1.3:6805/2919718]
[v2:10.250.1.3:6807/2919718,v1:10.250.1.3:6809/2919718] exists,up
7fbb3d00-6333-4353-afc5-f11bb2a1ce7a
osd.10201 up in weight 1 up_from 13545 up_thru 13678 down_at 13541
last_clean_interval [10699,13540)
[v2:10.250.1.3:6802/2956573,v1:10.250.1.3:6804/2956573]
[v2:10.250.1.3:6806/2956573,v1:10.250.1.3:6808/2956573] exists,up
07b8ec5e-36d5-4739-ae4c-9698e524ba69
osd.10300 up in weight 1 up_from 13575 up_thru 13678 down_at 13569
last_clean_interval [3030,13568)
[v2:10.250.1.4:6810/3545209,v1:10.250.1.4:6811/3545209]
[v2:10.250.1.4:6812/3545209,v1:10.250.1.4:6813/3545209] exists,up
016ec42d-08ee-4f59-afdd-c48ad0c046a7
osd.10301 up in weight 1 up_from 13571 up_thru 13678 down_at 13566
last_clean_interval [3030,13565)
[v2:10.250.1.4:6802/3544816,v1:10.250.1.4:6803/3544816]
[v2:10.250.1.4:6804/3544816,v1:10.250.1.4:6805/3544816] exists,up
f364315a-92b1-4fc6-b6cd-235fc6a87186
osd.10400 up in weight 1 up_from 13603 up_thru 13678 down_at 13599
last_clean_interval [2997,13598)
[v2:10.250.1.5:6830/3434175,v1:10.250.1.5:6834/3434175]
[v2:10.250.1.5:6837/3434175,v1:10.250.1.5:6839/3434175] exists,up
1f7c6e5e-2bb7-4e95-8da0-2dfc5d285a17
osd.10401 up in weight 1 up_from 13631 up_thru 13678 down_at 13625
last_clean_interval [2997,13624)
[v2:10.250.1.5:6821/3451855,v1:10.250.1.5:6826/3451855]
[v2:10.250.1.5:6833/3451855,v1:10.250.1.5:6836/3451855] exists,up
9ecd21dc-c610-4cf1-a270-f407eedcf85d
osd.10500 up in weight 1 up_from 13675 up_thru 13675 down_at 13671
last_clean_interval [10932,13670)
[v2:10.250.1.6:6802/3767787,v1:10.250.1.6:6803/3767787]
[v2:10.250.1.6:6804/3767787,v1:10.250.1.6:6805/3767787] exists,up
ddff6c5c-2d93-48ba-9662-8a02ba2fab49
osd.10501 up in weight 1 up_from 13678 up_thru 13678 down_at 13673
last_clean_interval [10937,13672)
[v2:10.250.1.6:6826/3768089,v1:10.250.1.6:6827/3768089]
[v2:10.250.1.6:6828/3768089,v1:10.250.1.6:6829/3768089] exists,up
7a6df9ba-e51d-4817-bc4c-c695bcf748c2
pg_upmap_items 1.c [10400,10401]
pg_upmap_items 1.29 [10400,10401]
pg_upmap_items 6.1 [503,502]
pg_upmap_items 7.0 [305,304]
pg_upmap_items 7.2 [102,103,305,304]
pg_upmap_items 7.3 [305,302]
pg_upmap_items 7.4 [305,302]
pg_upmap_items 7.8 [305,302]
pg_upmap_items 7.a [305,304]
pg_upmap_items 7.c [305,304]
pg_upmap_items 7.d [505,502]
pg_upmap_items 7.15 [204,203]
pg_upmap_items 7.16 [305,302]
pg_upmap_items 7.17 [202,203]
pg_upmap_items 7.1a [503,502]
pg_upmap_items 7.1b [104,102]
pg_upmap_items 7.27 [305,300]
pg_upmap_items 7.2f [102,101]
pg_upmap_items 7.30 [402,400]
pg_upmap_items 7.38 [405,404]
pg_upmap_items 8.0 [404,400]
pg_upmap_items 8.5 [302,305,400,402]
pg_upmap_items 8.6 [200,204]
pg_upmap_items 8.a [303,305,404,403]
pg_upmap_items 8.c [501,505]
pg_upmap_items 8.d [404,405]
pg_upmap_items 8.10 [304,305,202,205]
pg_upmap_items 8.11 [200,204]
pg_upmap_items 8.12 [302,305]
pg_upmap_items 8.17 [103,102]
pg_upmap_items 8.18 [304,305]
pg_upmap_items 8.19 [101,104]
pg_upmap_items 8.1b [501,503]
pg_upmap_items 8.25 [304,302]
pg_upmap_items 8.26 [201,204]
pg_upmap_items 8.28 [304,305,203,205]
pg_upmap_items 8.38 [501,503]
pg_upmap_items 8.45 [202,204]
pg_upmap_items 8.46 [304,305]
pg_upmap_items 8.48 [404,403]
pg_upmap_items 8.49 [404,403]
pg_upmap_items 8.51 [103,102]
pg_upmap_items 8.52 [202,205]
pg_upmap_items 8.54 [404,403]
pg_upmap_items 8.5a [201,205,404,405]
pg_upmap_items 8.5d [404,403]
pg_upmap_items 8.60 [202,205]
pg_upmap_items 8.63 [404,403]
pg_upmap_items 8.6a [304,302]
pg_upmap_items 8.71 [202,205]
pg_upmap_items 8.7d [501,503]
pg_upmap_items 8.85 [103,102]
pg_upmap_items 8.8b [304,305]
pg_upmap_items 8.91 [304,305]
pg_upmap_items 8.92 [303,301]
pg_upmap_items 8.97 [404,403]
pg_upmap_items 8.a0 [404,403]
pg_upmap_items 8.a1 [501,503]
pg_upmap_items 8.a3 [404,403]
pg_upmap_items 8.a6 [404,403]
pg_upmap_items 8.a9 [404,403]
pg_upmap_items 8.aa [404,403]
pg_upmap_items 8.ac [304,302]
pg_upmap_items 8.ad [304,302]
pg_upmap_items 8.b2 [404,403]
pg_upmap_items 8.b9 [304,305,202,205]
pg_upmap_items 8.c0 [201,205]
pg_upmap_items 8.c2 [501,503]
pg_upmap_items 8.c3 [202,205]
pg_upmap_items 8.c7 [304,305,201,205]
pg_upmap_items 8.d6 [202,205]
pg_upmap_items 8.d7 [203,204]
pg_upmap_items 8.da [202,205]
pg_upmap_items 8.de [404,403]
pg_upmap_items 8.e7 [304,305]
pg_upmap_items 8.e8 [404,405]
pg_upmap_items 8.f9 [501,503]
pg_upmap_items 8.106 [303,305]
pg_upmap_items 8.10f [504,503]
pg_upmap_items 8.118 [404,400]
pg_upmap_items 8.11a [202,205]
pg_upmap_items 8.127 [404,403]
pg_upmap_items 8.12a [501,503]
pg_upmap_items 8.132 [304,305]
pg_upmap_items 8.141 [304,302]
pg_upmap_items 8.142 [404,405]
pg_upmap_items 8.145 [300,305]
pg_upmap_items 8.147 [201,204]
pg_upmap_items 8.14e [504,503]
pg_upmap_items 8.150 [303,302]
pg_upmap_items 8.151 [203,205,404,405]
pg_upmap_items 8.155 [501,503]
pg_upmap_items 8.15d [202,205]
pg_upmap_items 8.165 [404,403]
pg_upmap_items 8.166 [303,305]
pg_upmap_items 8.170 [404,403]
pg_upmap_items 8.171 [304,305]
pg_upmap_items 8.17e [304,305]
pg_upmap_items 8.17f [203,204]
pg_upmap_items 8.182 [404,400]
pg_upmap_items 8.18c [501,503]
pg_upmap_items 8.190 [202,205]
pg_upmap_items 8.191 [504,503]
pg_upmap_items 8.195 [202,204]
pg_upmap_items 8.196 [304,305,201,204]
pg_upmap_items 8.19f [304,305]
pg_upmap_items 8.1a2 [202,204]
pg_upmap_items 8.1a8 [501,500]
pg_upmap_items 8.1af [103,102,504,503,404,403]
pg_upmap_items 8.1b5 [404,403]
pg_upmap_items 8.1bb [404,403]
pg_upmap_items 8.1c2 [404,403]
pg_upmap_items 8.1c4 [504,503]
pg_upmap_items 8.1c7 [202,205]
pg_upmap_items 8.1ca [404,403]
pg_upmap_items 8.1cd [202,205]
pg_upmap_items 8.1cf [304,305]
pg_upmap_items 8.1d1 [404,405]
pg_upmap_items 8.1d4 [202,205]
pg_upmap_items 8.1df [504,503,203,205]
pg_upmap_items 8.1e2 [404,403]
pg_upmap_items 8.1e3 [501,505]
pg_upmap_items 8.1e5 [303,305]
pg_upmap_items 8.1ec [404,403]
pg_upmap_items 8.1f2 [501,500]
pg_upmap_items 8.1f3 [404,403]
pg_upmap_items 8.1fb [304,305]
pg_upmap_items 8.205 [404,403]
pg_upmap_items 8.20d [404,405]
pg_upmap_items 8.20f [304,301]
pg_upmap_items 8.215 [404,403]
pg_upmap_items 8.216 [203,205,404,400]
pg_upmap_items 8.21b [404,403]
pg_upmap_items 8.224 [404,403]
pg_upmap_items 8.230 [202,205]
pg_upmap_items 8.237 [201,205,404,400]
pg_upmap_items 8.241 [501,503]
pg_upmap_items 8.246 [103,102]
pg_upmap_items 8.24b [304,305,202,204]
pg_upmap_items 8.24f [404,403]
pg_upmap_items 8.254 [304,305]
pg_upmap_items 8.25c [404,400]
pg_upmap_items 8.269 [304,305]
pg_upmap_items 8.272 [404,400]
pg_upmap_items 8.275 [504,503,404,405]
pg_upmap_items 8.279 [404,403]
pg_upmap_items 8.27e [303,301]
pg_upmap_items 8.280 [202,205]
pg_upmap_items 8.289 [501,505]
pg_upmap_items 8.290 [501,500]
pg_upmap_items 8.299 [501,505]
pg_upmap_items 8.29c [404,403]
pg_upmap_items 8.29e [401,400]
pg_upmap_items 8.2a1 [404,403]
pg_upmap_items 8.2a3 [501,500]
pg_upmap_items 8.2a4 [404,405]
pg_upmap_items 8.2a5 [404,403]
pg_upmap_items 8.2a6 [501,503]
pg_upmap_items 8.2a8 [203,205]
pg_upmap_items 8.2b0 [304,305,404,403]
pg_upmap_items 8.2b2 [201,205]
pg_upmap_items 8.2b8 [404,403]
pg_upmap_items 8.2b9 [501,500]
pg_upmap_items 8.2be [303,305]
pg_upmap_items 8.2c4 [304,305]
pg_upmap_items 8.2ca [501,503]
pg_upmap_items 8.2d5 [203,204]
pg_upmap_items 8.2d7 [103,102,404,400]
pg_upmap_items 8.2da [304,302]
pg_upmap_items 8.2dc [103,102]
pg_upmap_items 8.2dd [304,305]
pg_upmap_items 8.2e3 [203,205]
pg_upmap_items 8.2ed [203,205]
pg_upmap_items 8.2ef [404,403]
pg_upmap_items 8.300 [303,302]
pg_upmap_items 8.303 [404,403,304,305]
pg_upmap_items 8.304 [203,205]
pg_upmap_items 8.307 [304,305]
pg_upmap_items 8.309 [202,205]
pg_upmap_items 8.30a [201,204]
pg_upmap_items 8.312 [304,302]
pg_upmap_items 8.319 [501,500]
pg_upmap_items 8.31e [304,305]
pg_upmap_items 8.324 [303,301]
pg_upmap_items 8.336 [404,403]
pg_upmap_items 8.338 [303,305]
pg_upmap_items 8.339 [303,302]
pg_upmap_items 8.33a [404,400]
pg_upmap_items 8.33c [304,305]
pg_upmap_items 8.344 [404,403]
pg_upmap_items 8.346 [201,204]
pg_upmap_items 8.349 [504,503]
pg_upmap_items 8.350 [501,500]
pg_upmap_items 8.356 [101,102]
pg_upmap_items 8.358 [404,405]
pg_upmap_items 8.363 [103,102]
pg_upmap_items 8.364 [404,403]
pg_upmap_items 8.366 [404,403]
pg_upmap_items 8.369 [304,305]
pg_upmap_items 8.36b [103,102]
pg_upmap_items 8.373 [404,403]
pg_upmap_items 8.383 [404,403]
pg_upmap_items 8.39d [203,205]
pg_upmap_items 8.3a3 [103,102]
pg_upmap_items 8.3a6 [304,305]
pg_upmap_items 8.3ab [304,305]
pg_upmap_items 8.3af [304,305]
pg_upmap_items 8.3b3 [404,405]
pg_upmap_items 8.3b4 [303,305]
pg_upmap_items 8.3b7 [404,403]
pg_upmap_items 8.3b9 [404,403]
pg_upmap_items 8.3ba [404,403,201,205]
pg_upmap_items 8.3bd [404,405]
pg_upmap_items 8.3c0 [304,305]
pg_upmap_items 8.3c3 [404,403]
pg_upmap_items 8.3ca [404,403]
pg_upmap_items 8.3cf [404,405]
pg_upmap_items 8.3d0 [404,405]
pg_upmap_items 8.3da [404,403]
pg_upmap_items 8.3e4 [404,405]
pg_upmap_items 8.3ea [404,405]
pg_upmap_items 8.3ec [203,205]
pg_upmap_items 8.3f3 [501,505]
pg_upmap_items 8.3f7 [304,305]
pg_upmap_items 8.3fb [404,405]
pg_upmap_items 8.3fc [304,305]
pg_upmap_items 8.400 [105,102,404,403]
pg_upmap_items 8.409 [404,403]
pg_upmap_items 8.40b [103,102,404,405]
pg_upmap_items 8.40c [404,400]
pg_upmap_items 8.410 [404,403]
pg_upmap_items 8.411 [404,405]
pg_upmap_items 8.417 [404,403]
pg_upmap_items 8.418 [404,403]
pg_upmap_items 9.2 [10401,10400]
pg_upmap_items 9.9 [10200,10201]
Regards
David Herselman
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
