The OSDs are all there.
$ sudo ceph osd stat
osdmap e60609: 72 osds: 72 up, 72 in
an I have attached the result of ceph osd tree, and ceph osd dump commands.
I got some extra info about the network problem. A faulty network device has
flooded the network eating up all the bandwidth so the OSDs were not able to
properly communicate with each other. This has lasted for almost 1 day.
Thank you,
Laszlo
On 10.03.2017 12:19, Brad Hubbard wrote:
To me it looks like someone may have done an "rm" on these OSDs but
not removed them from the crushmap. This does not happen
automatically.
Do these OSDs show up in "ceph osd tree" and "ceph osd dump" ? If so,
paste the output.
Without knowing what exactly happened here it may be difficult to work
out how to proceed.
In order to go clean the primary needs to communicate with multiple
OSDs, some of which are marked DNE and seem to be uncontactable.
This seems to be more than a network issue (unless the outage is still
happening).
http://docs.ceph.com/docs/master/rados/operations/pg-states/?highlight=incomplete
On Fri, Mar 10, 2017 at 6:09 PM, Laszlo Budai <las...@componentsoft.eu> wrote:
Hello,
I was informed that due to a networking issue the ceph cluster network was
affected. There was a huge packet loss, and network interfaces were
flipping. That's all I got.
This outage has lasted a longer period of time. So I assume that some OSD
may have been considered dead and the data from them has been moved away to
other PGs (this is what ceph is supposed to do if I'm correct). Probably
that was the point when the listed PGs have appeared into the picture.
From the query we can see this for one of those OSDs:
{
"peer": "14",
"pgid": "3.367",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": "[]",
"history": {
"epoch_created": 4,
"last_epoch_started": 54899,
"last_epoch_clean": 55143,
"last_epoch_split": 0,
"same_up_since": 60603,
"same_interval_since": 60603,
"same_primary_since": 60593,
"last_scrub": "2852'33528",
"last_scrub_stamp": "2017-02-26 02:36:55.210150",
"last_deep_scrub": "2852'16480",
"last_deep_scrub_stamp": "2017-02-21 00:14:08.866448",
"last_clean_scrub_stamp": "2017-02-26 02:36:55.210150"
},
"stats": {
"version": "0'0",
"reported_seq": "14",
"reported_epoch": "59779",
"state": "down+peering",
"last_fresh": "2017-02-27 16:30:16.230519",
"last_change": "2017-02-27 16:30:15.267995",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "2017-02-27 16:30:16.230519",
"last_undegraded": "2017-02-27 16:30:16.230519",
"last_fullsized": "2017-02-27 16:30:16.230519",
"mapping_epoch": 60601,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 4,
"last_epoch_clean": 55143,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "2852'33528",
"last_scrub_stamp": "2017-02-26 02:36:55.210150",
"last_deep_scrub": "2852'16480",
"last_deep_scrub_stamp": "2017-02-21 00:14:08.866448",
"last_clean_scrub_stamp": "2017-02-26 02:36:55.210150",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": "0",
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0
},
"up": [
28,
35,
2
],
"acting": [
28,
35,
2
],
"blocked_by": [],
"up_primary": 28,
"acting_primary": 28
},
"empty": 1,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"current_last_stamp": "0.000000",
"current_info": {
"begin": "0.000000",
"end": "0.000000",
"version": "0'0",
"using_gmt": "1"
},
"history": []
}
},
Where can I read more about the meaning of each parameter, some of them have
quite self explanatory names, but not all (or probably we need a deeper
knowledge to understand them).
Isn't there any parameter that would say when was that OSD assigned to the
given PG? Also the stat_sum shows 0 for all its parameters. Why is it
blocking then?
Is there a way to tell the PG to forget about that OSD?
Thank you,
Laszlo
On 10.03.2017 03:05, Brad Hubbard wrote:
Can you explain more about what happened?
The query shows progress is blocked by the following OSDs.
"blocked_by": [
14,
17,
51,
58,
63,
64,
68,
70
],
Some of these OSDs are marked as "dne" (Does Not Exist).
peer": "17",
"dne": 1,
"peer": "51",
"dne": 1,
"peer": "58",
"dne": 1,
"peer": "64",
"dne": 1,
"peer": "70",
"dne": 1,
Can we get a complete background here please?
On Thu, Mar 9, 2017 at 10:53 PM, Laszlo Budai <las...@componentsoft.eu>
wrote:
Hello,
After a major network outage our ceph cluster ended up with an inactive
PG:
# ceph health detail
HEALTH_WARN 1 pgs incomplete; 1 pgs stuck inactive; 1 pgs stuck unclean;
1
requests are blocked > 32 sec; 1 osds have slow requests
pg 3.367 is stuck inactive for 912263.766607, current state incomplete,
last
acting [28,35,2]
pg 3.367 is stuck unclean for 912263.766688, current state incomplete,
last
acting [28,35,2]
pg 3.367 is incomplete, acting [28,35,2]
1 ops are blocked > 268435 sec
1 ops are blocked > 268435 sec on osd.28
1 osds have slow requests
# ceph -s
cluster 6713d1b8-83da-11e6-aa79-525400d98c5a
health HEALTH_WARN
1 pgs incomplete
1 pgs stuck inactive
1 pgs stuck unclean
1 requests are blocked > 32 sec
monmap e3: 3 mons at
{tv-dl360-1=10.12.193.73:6789/0,tv-dl360-2=10.12.193.74:6789/0,tv-dl360-3=10.12.193.75:6789/0}
election epoch 72, quorum 0,1,2
tv-dl360-1,tv-dl360-2,tv-dl360-3
osdmap e60609: 72 osds: 72 up, 72 in
pgmap v3670252: 4864 pgs, 11 pools, 134 GB data, 23778 objects
490 GB used, 130 TB / 130 TB avail
4863 active+clean
1 incomplete
client io 0 B/s rd, 38465 B/s wr, 2 op/s
ceph pg repair doesn't change anything. What should I try to recover it?
Attached is the result of ceph pg query on the problem PG.
Thank you,
Laszlo
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
epoch 60609
fsid 6713d1b8-83da-11e6-aa79-525400d98c5a
created 2016-10-17 11:28:35.501106
modified 2017-03-09 13:32:30.302087
flags
pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0
pool 1 'images' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 256 pgp_num 256 last_change 2783 flags hashpspool stripe_width 0
removed_snaps [1~9,c~c2,cf~280]
pool 2 'instances' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 2048 pgp_num 2048 last_change 241 flags hashpspool stripe_width 0
removed_snaps [1~3]
pool 3 'volumes' replicated size 3 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 2048 pgp_num 2048 last_change 59778 flags hashpspool
stripe_width 0
removed_snaps [1~13d]
pool 4 '.rgw.root' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 32 pgp_num 32 last_change 266 owner 18446744073709551615 flags
hashpspool stripe_width 0
pool 5 '.rgw.control' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 32 pgp_num 32 last_change 270 flags hashpspool stripe_width 0
pool 6 '.rgw' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins
pg_num 32 pgp_num 32 last_change 272 flags hashpspool stripe_width 0
pool 7 '.rgw.gc' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 32 pgp_num 32 last_change 272 flags hashpspool stripe_width 0
pool 8 '.users.uid' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 32 pgp_num 32 last_change 279 flags hashpspool stripe_width 0
pool 9 '.rgw.buckets.index' replicated size 3 min_size 2 crush_ruleset 0
object_hash rjenkins pg_num 32 pgp_num 32 last_change 285 flags hashpspool
stripe_width 0
pool 10 '.rgw.buckets' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 256 pgp_num 256 last_change 2846 flags hashpspool stripe_width 0
max_osd 72
osd.0 up in weight 1 up_from 60563 up_thru 60607 down_at 60556
last_clean_interval [59469,60562) 10.12.193.71:6826/2955
192.168.91.71:6808/1002955 192.168.91.71:6809/1002955 10.12.193.71:6816/1002955
exists,up bdf8c61d-83d8-4833-990c-3a5c0be404cf
osd.1 up in weight 1 up_from 60557 up_thru 60607 down_at 60556
last_clean_interval [59449,60556) 10.12.193.68:6815/1325
192.168.91.68:6801/1001325 192.168.91.68:6802/1001325 10.12.193.68:6805/1001325
exists,up a6acb440-3d13-48f9-abee-c67913201a2b
osd.2 up in weight 1 up_from 60607 up_thru 60607 down_at 60605
last_clean_interval [60603,60604) 10.12.193.72:6801/7265
192.168.91.72:6806/7265 192.168.91.72:6807/7265 10.12.193.72:6807/7265
exists,up 6a4db644-22c2-4a18-83d9-3c39c0a72966
osd.3 up in weight 1 up_from 60537 up_thru 60607 down_at 60534
last_clean_interval [60376,60536) 10.12.193.70:6822/2913
192.168.91.70:6832/1002913 192.168.91.70:6833/1002913 10.12.193.70:6828/1002913
exists,up 1d282d68-5b07-43b1-8fa0-58c5444ae50f
osd.4 up in weight 1 up_from 60535 up_thru 60607 down_at 60534
last_clean_interval [59849,60534) 10.12.193.69:6810/21561
192.168.91.69:6826/1021561 192.168.91.69:6827/1021561 10.12.193.69:6825/1021561
exists,up 395394bc-91eb-4812-964b-329256b6eb67
osd.5 up in weight 1 up_from 60542 up_thru 60607 down_at 60534
last_clean_interval [59442,60541) 10.12.193.67:6812/16762
192.168.91.67:6832/1016762 192.168.91.67:6833/1016762 10.12.193.67:6828/1016762
exists,up b5c925f3-17f7-4db8-ba0c-5badf702c0d0
osd.6 up in weight 1 up_from 60538 up_thru 60607 down_at 60534
last_clean_interval [59469,60537) 10.12.193.71:6803/2184
192.168.91.71:6828/1002184 192.168.91.71:6829/1002184 10.12.193.71:6802/1002184
exists,up 9427c51d-1303-40a8-952a-b1ebcaf20fda
osd.7 up in weight 1 up_from 60537 up_thru 60605 down_at 60534
last_clean_interval [59448,60535) 10.12.193.68:6824/1683
192.168.91.68:6828/1001683 192.168.91.68:6829/1001683 10.12.193.68:6812/1001683
exists,up ab9bfd4e-df2d-4fbd-bdb4-f8d2dd5154e7
osd.8 up in weight 1 up_from 60560 up_thru 60593 down_at 60556
last_clean_interval [59476,60559) 10.12.193.72:6826/6096
192.168.91.72:6803/1006096 192.168.91.72:6805/1006096 10.12.193.72:6804/1006096
exists,up 50e37b81-6eaf-4b0c-81a8-93cabc2a4e9d
osd.9 up in weight 1 up_from 60542 up_thru 60607 down_at 60534
last_clean_interval [60386,60541) 10.12.193.70:6820/2864
192.168.91.70:6808/1002864 192.168.91.70:6835/1002864 10.12.193.70:6830/1002864
exists,up eb1d674b-a09a-4f57-807a-409ea2e3c597
osd.10 up in weight 1 up_from 60554 up_thru 60607 down_at 60553
last_clean_interval [59850,60553) 10.12.193.69:6812/21670
192.168.91.69:6822/1021670 192.168.91.69:6823/1021670 10.12.193.69:6823/1021670
exists,up 961437ac-0605-4e7a-bb47-65157c160241
osd.11 up in weight 1 up_from 60537 up_thru 60607 down_at 60534
last_clean_interval [59443,60535) 10.12.193.67:6820/17254
192.168.91.67:6830/1017254 192.168.91.67:6831/1017254 10.12.193.67:6827/1017254
exists,up f17163d6-683f-486d-bd96-35b6615ecfd2
osd.12 up in weight 1 up_from 60538 up_thru 60593 down_at 60534
last_clean_interval [59470,60537) 10.12.193.71:6830/3144
192.168.91.71:6830/1003144 192.168.91.71:6832/1003144 10.12.193.71:6808/1003144
exists,up e7f2c05e-de6e-4805-aac5-d8ced7888679
osd.13 up in weight 1 up_from 60537 up_thru 60607 down_at 60534
last_clean_interval [59449,60536) 10.12.193.68:6800/1062
192.168.91.68:6812/1001062 192.168.91.68:6834/1001062 10.12.193.68:6822/1001062
exists,up b9b40bc3-8ece-4b7a-aadf-95cd8a5a48de
osd.14 up in weight 1 up_from 60537 up_thru 60593 down_at 60534
last_clean_interval [59475,60536) 10.12.193.72:6808/5027
192.168.91.72:6831/1005027 192.168.91.72:6832/1005027 10.12.193.72:6822/1005027
exists,up bb437866-ead4-4663-b346-d0be19c6f6c1
osd.15 up in weight 1 up_from 60535 up_thru 60605 down_at 60534
last_clean_interval [60377,60534) 10.12.193.70:6816/2611
192.168.91.70:6824/1002611 192.168.91.70:6827/1002611 10.12.193.70:6825/1002611
exists,up 5aa55888-4c82-4718-872b-aa6559dc485c
osd.16 up in weight 1 up_from 60559 up_thru 60607 down_at 60556
last_clean_interval [59848,60557) 10.12.193.69:6802/21074
192.168.91.69:6800/1021074 192.168.91.69:6801/1021074 10.12.193.69:6801/1021074
exists,up d2b28ddc-285e-49a9-8bf8-8d4472faa617
osd.17 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [59442,60535) 10.12.193.67:6814/16881
192.168.91.67:6826/1016881 192.168.91.67:6827/1016881 10.12.193.67:6825/1016881
exists,up 2b67b3d7-1274-4ce1-a9d3-6cc557d93254
osd.18 up in weight 1 up_from 60537 up_thru 60607 down_at 60534
last_clean_interval [59469,60536) 10.12.193.71:6801/2188
192.168.91.71:6824/1002188 192.168.91.71:6825/1002188 10.12.193.71:6804/1002188
exists,up cef750c2-c5f0-49f8-b794-d3d4deb0274d
osd.19 up in weight 1 up_from 60542 up_thru 60607 down_at 60534
last_clean_interval [59449,60541) 10.12.193.68:6828/1860
192.168.91.68:6835/1001860 192.168.91.68:6836/1001860 10.12.193.68:6830/1001860
exists,up bac0a6ea-89aa-43a7-a682-8f22da6f78c1
osd.20 up in weight 1 up_from 60536 up_thru 60593 down_at 60534
last_clean_interval [59475,60535) 10.12.193.72:6813/5331
192.168.91.72:6808/1005331 192.168.91.72:6815/1005331 10.12.193.72:6806/1005331
exists,up 04cceaba-ca17-482e-bc59-40ce6a203738
osd.21 up in weight 1 up_from 60545 up_thru 60607 down_at 60534
last_clean_interval [60367,60542) 10.12.193.70:6808/2199
192.168.91.70:6822/1002199 192.168.91.70:6834/1002199 10.12.193.70:6829/1002199
exists,up 7808e117-397b-472c-9670-4bdc483b6ec5
osd.22 up in weight 1 up_from 60548 up_thru 60607 down_at 60534
last_clean_interval [59851,60547) 10.12.193.69:6822/22136
192.168.91.69:6814/1022136 192.168.91.69:6815/1022136 10.12.193.69:6815/1022136
exists,up 449f717a-f9ec-4933-aaff-1d401ccfa957
osd.23 up in weight 1 up_from 60558 up_thru 60607 down_at 60556
last_clean_interval [59443,60557) 10.12.193.67:6818/17196
192.168.91.67:6800/1017196 192.168.91.67:6801/1017196 10.12.193.67:6801/1017196
exists,up e5e0c270-d5be-4ec9-a1b3-0fd02da51dda
osd.24 up in weight 1 up_from 60540 up_thru 60607 down_at 60534
last_clean_interval [59469,60539) 10.12.193.71:6822/2742
192.168.91.71:6836/1002742 192.168.91.71:6837/1002742 10.12.193.71:6812/1002742
exists,up afdca50a-8974-408c-841d-56f57a2acc9b
osd.25 up in weight 1 up_from 60547 up_thru 60607 down_at 60534
last_clean_interval [59449,60546) 10.12.193.68:6802/1052
192.168.91.68:6837/1001052 192.168.91.68:6838/1001052 10.12.193.68:6831/1001052
exists,up 23c66476-daa8-4cdc-bbfd-e58c4b38078c
osd.26 up in weight 1 up_from 60560 up_thru 60589 down_at 60556
last_clean_interval [59475,60559) 10.12.193.72:6805/4904
192.168.91.72:6809/1004904 192.168.91.72:6810/1004904 10.12.193.72:6809/1004904
exists,up 210e9831-b5ed-4c27-8ff1-89708fcff0e0
osd.27 up in weight 1 up_from 60554 up_thru 60607 down_at 60553
last_clean_interval [60373,60553) 10.12.193.70:6812/2407
192.168.91.70:6810/1002407 192.168.91.70:6841/1002407 10.12.193.70:6807/1002407
exists,up dfb407c8-9bd7-4ed9-bb82-3358b6d3dc5f
osd.28 up in weight 1 up_from 60593 up_thru 60607 down_at 60583
last_clean_interval [60576,60582) 10.12.193.69:6804/11178
192.168.91.69:6803/11178 192.168.91.69:6806/11178 10.12.193.69:6805/11178
exists,up c2adca59-6629-4ed8-bfe9-c43ce3a6d4b6
osd.29 up in weight 1 up_from 60552 up_thru 60607 down_at 60550
last_clean_interval [59441,60551) 10.12.193.67:6802/16057
192.168.91.67:6836/1016057 192.168.91.67:6837/1016057 10.12.193.67:6830/1016057
exists,up 4c88f2ec-faa2-4258-ae38-92b655b8b021
osd.30 up in weight 1 up_from 60542 up_thru 60607 down_at 60534
last_clean_interval [59470,60541) 10.12.193.71:6828/3057
192.168.91.71:6838/1003057 192.168.91.71:6839/1003057 10.12.193.71:6818/1003057
exists,up 2d3b1780-c0c5-44a6-a7d6-9495275a607d
osd.31 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [59448,60535) 10.12.193.68:6821/1581
192.168.91.68:6814/1001581 192.168.91.68:6816/1001581 10.12.193.68:6810/1001581
exists,up 13f17fba-b4af-433d-aea2-8439335f50dc
osd.32 up in weight 1 up_from 60537 up_thru 60593 down_at 60534
last_clean_interval [59476,60535) 10.12.193.72:6819/5743
192.168.91.72:6828/1005743 192.168.91.72:6830/1005743 10.12.193.72:6828/1005743
exists,up 91bf05a0-2218-4a3f-9c09-eaf88f1bd05f
osd.33 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [60363,60535) 10.12.193.70:6804/1995
192.168.91.70:6830/1001995 192.168.91.70:6831/1001995 10.12.193.70:6827/1001995
exists,up ec30c2a5-d225-4c99-b58f-9b5339a697c4
osd.34 up in weight 1 up_from 60543 up_thru 60607 down_at 60534
last_clean_interval [59849,60541) 10.12.193.69:6806/21324
192.168.91.69:6834/1021324 192.168.91.69:6835/1021324 10.12.193.69:6829/1021324
exists,up 1bfd424c-ddb3-4309-9f60-981c85224e37
osd.35 up in weight 1 up_from 60589 up_thru 60607 down_at 60585
last_clean_interval [60568,60584) 10.12.193.67:6805/11183
192.168.91.67:6804/11183 192.168.91.67:6805/11183 10.12.193.67:6806/11183
exists,up c612be90-28eb-42d7-93f2-a6a9bb6d8f17
osd.36 up in weight 1 up_from 60537 up_thru 60593 down_at 60534
last_clean_interval [59469,60536) 10.12.193.71:6817/2535
192.168.91.71:6826/1002535 192.168.91.71:6827/1002535 10.12.193.71:6800/1002535
exists,up ce8e5531-eec7-4dd0-a0bb-86f3b5699454
osd.37 up in weight 1 up_from 60537 up_thru 60607 down_at 60534
last_clean_interval [59448,60536) 10.12.193.68:6809/1128
192.168.91.68:6830/1001128 192.168.91.68:6831/1001128 10.12.193.68:6816/1001128
exists,up 389d67d0-4732-4045-bafb-21dc5bde81b4
osd.38 up in weight 1 up_from 60537 up_thru 60593 down_at 60534
last_clean_interval [59476,60536) 10.12.193.72:6811/5295
192.168.91.72:6819/1005295 192.168.91.72:6826/1005295 10.12.193.72:6810/1005295
exists,up a688aab8-6efa-49ef-bc6f-f97786df3898
osd.39 up in weight 1 up_from 60542 up_thru 60607 down_at 60534
last_clean_interval [60369,60541) 10.12.193.70:6806/2094
192.168.91.70:6818/1002094 192.168.91.70:6836/1002094 10.12.193.70:6833/1002094
exists,up 57f495b6-412b-482f-852d-e0ddfdac3e34
osd.40 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [59850,60535) 10.12.193.69:6816/21940
192.168.91.69:6832/1021940 192.168.91.69:6833/1021940 10.12.193.69:6828/1021940
exists,up 08f745e0-b1d7-4e45-98a9-ebed7333fbb9
osd.41 up in weight 1 up_from 60557 up_thru 60607 down_at 60556
last_clean_interval [59443,60556) 10.12.193.67:6810/16634
192.168.91.67:6802/1016634 192.168.91.67:6803/1016634 10.12.193.67:6803/1016634
exists,up fb6f0ac3-d2f8-405c-8652-16f9e1077385
osd.42 up in weight 1 up_from 60560 up_thru 60607 down_at 60556
last_clean_interval [59468,60559) 10.12.193.71:6807/2186
192.168.91.71:6802/1002186 192.168.91.71:6803/1002186 10.12.193.71:6806/1002186
exists,up fa1ceca4-80a2-49a7-94cf-32728e8a49c9
osd.43 up in weight 1 up_from 60558 up_thru 60607 down_at 60556
last_clean_interval [59449,60557) 10.12.193.68:6803/1054
192.168.91.68:6810/1001054 192.168.91.68:6811/1001054 10.12.193.68:6811/1001054
exists,up db466c69-8f2d-4f57-99f9-361c871eb249
osd.44 up in weight 1 up_from 60542 up_thru 60593 down_at 60534
last_clean_interval [59474,60541) 10.12.193.72:6803/4855
192.168.91.72:6836/1004855 192.168.91.72:6837/1004855 10.12.193.72:6830/1004855
exists,up 177a09a0-fe35-45b2-8fb2-137cebe58b36
osd.45 up in weight 1 up_from 60537 up_thru 60607 down_at 60534
last_clean_interval [60364,60536) 10.12.193.70:6802/1928
192.168.91.70:6828/1001928 192.168.91.70:6829/1001928 10.12.193.70:6826/1001928
exists,up a38276ab-6362-43d8-9643-946ae3c31568
osd.46 up in weight 1 up_from 60548 up_thru 60607 down_at 60534
last_clean_interval [59850,60547) 10.12.193.69:6814/21858
192.168.91.69:6804/1021858 192.168.91.69:6837/1021858 10.12.193.69:6830/1021858
exists,up b59d67b2-e9ea-457b-9ee8-dd1f28453c65
osd.47 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [59441,60535) 10.12.193.67:6800/15914
192.168.91.67:6828/1015914 192.168.91.67:6829/1015914 10.12.193.67:6826/1015914
exists,up bd5541a9-7458-44d0-9fa8-5b319e59ad43
osd.48 up in weight 1 up_from 60539 up_thru 60607 down_at 60534
last_clean_interval [59469,60537) 10.12.193.71:6820/2629
192.168.91.71:6831/1002629 192.168.91.71:6833/1002629 10.12.193.71:6810/1002629
exists,up 74ee6c8a-6db8-4232-902e-b5e2d2a95a51
osd.49 up in weight 1 up_from 60560 up_thru 60607 down_at 60556
last_clean_interval [59449,60559) 10.12.193.68:6801/1059
192.168.91.68:6813/1001059 192.168.91.68:6818/1001059 10.12.193.68:6814/1001059
exists,up 73bf5889-e373-4d2b-9838-21df4890709c
osd.50 up in weight 1 up_from 60542 up_thru 60593 down_at 60534
last_clean_interval [59476,60541) 10.12.193.72:6817/5626
192.168.91.72:6840/1005626 192.168.91.72:6841/1005626 10.12.193.72:6831/1005626
exists,up ada64b06-b1dc-4f34-b375-489836aee510
osd.51 up in weight 1 up_from 60543 up_thru 60607 down_at 60534
last_clean_interval [60376,60542) 10.12.193.70:6810/2346
192.168.91.70:6839/1002346 192.168.91.70:6840/1002346 10.12.193.70:6809/1002346
exists,up 5f1b86a2-d0c3-493a-8bcf-dcbf4a054f47
osd.52 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [59851,60535) 10.12.193.69:6820/22132
192.168.91.69:6830/1022132 192.168.91.69:6831/1022132 10.12.193.69:6827/1022132
exists,up c23ed2fb-4080-4bff-ae70-2cd43bc59544
osd.53 up in weight 1 up_from 60559 up_thru 60607 down_at 60556
last_clean_interval [59442,60558) 10.12.193.67:6808/16500
192.168.91.67:6814/1016500 192.168.91.67:6815/1016500 10.12.193.67:6815/1016500
exists,up d4b3f0af-949e-4fcb-a7a1-32d96afc3305
osd.54 up in weight 1 up_from 60559 up_thru 60607 down_at 60556
last_clean_interval [59469,60558) 10.12.193.71:6813/2253
192.168.91.71:6800/1002253 192.168.91.71:6801/1002253 10.12.193.71:6805/1002253
exists,up ea4077dd-f6d6-4843-bdf3-9cb2b6b13bee
osd.55 up in weight 1 up_from 60558 up_thru 60607 down_at 60556
last_clean_interval [59450,60557) 10.12.193.68:6826/1773
192.168.91.68:6807/1001773 192.168.91.68:6809/1001773 10.12.193.68:6808/1001773
exists,up 042c79f0-4a83-410c-a501-f522f62bf5dc
osd.56 up in weight 1 up_from 60542 up_thru 60593 down_at 60534
last_clean_interval [59476,60541) 10.12.193.72:6815/5524
192.168.91.72:6834/1005524 192.168.91.72:6835/1005524 10.12.193.72:6829/1005524
exists,up 01a00dce-0bd0-4068-bf5b-b93521a68208
osd.57 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [60369,60535) 10.12.193.70:6814/2509
192.168.91.70:6825/1002509 192.168.91.70:6826/1002509 10.12.193.70:6824/1002509
exists,up 05fbdc7d-002e-4f4d-b068-77253dfa4595
osd.58 up in weight 1 up_from 60559 up_thru 60607 down_at 60556
last_clean_interval [59850,60558) 10.12.193.69:6808/21451
192.168.91.69:6802/1021451 192.168.91.69:6805/1021451 10.12.193.69:6803/1021451
exists,up b2ed6957-9824-4f7c-8a25-bc8c1a64bfd0
osd.59 up in weight 1 up_from 60558 up_thru 60593 down_at 60556
last_clean_interval [59443,60557) 10.12.193.67:6816/16986
192.168.91.67:6806/1016986 192.168.91.67:6807/1016986 10.12.193.67:6807/1016986
exists,up 9579da6f-8b46-4465-b6b5-e3676f4dc3dc
osd.60 up in weight 1 up_from 60562 up_thru 60607 down_at 60556
last_clean_interval [59469,60561) 10.12.193.71:6815/2382
192.168.91.71:6804/1002382 192.168.91.71:6810/1002382 10.12.193.71:6819/1002382
exists,up e0f0bb3a-5dda-47d4-a6a9-21418cc24c17
osd.61 up in weight 1 up_from 60535 up_thru 60607 down_at 60534
last_clean_interval [59448,60534) 10.12.193.68:6819/1501
192.168.91.68:6800/1001501 192.168.91.68:6806/1001501 10.12.193.68:6804/1001501
exists,up dd60e730-9bab-42ce-a5d6-9db6d25006ac
osd.62 up in weight 1 up_from 60536 up_thru 60593 down_at 60534
last_clean_interval [59476,60535) 10.12.193.72:6824/6054
192.168.91.72:6800/1006054 192.168.91.72:6804/1006054 10.12.193.72:6800/1006054
exists,up 910f92c4-45bb-4d7f-984d-9b1f8109a963
osd.63 up in weight 1 up_from 60543 up_thru 60607 down_at 60534
last_clean_interval [60311,60542) 10.12.193.70:6800/965
192.168.91.70:6837/1000965 192.168.91.70:6838/1000965 10.12.193.70:6832/1000965
exists,up 4af592fc-029a-4afe-be31-c2d6b728ddbb
osd.64 up in weight 1 up_from 60535 up_thru 60607 down_at 60534
last_clean_interval [59769,60534) 10.12.193.69:6800/18864
192.168.91.69:6824/1018864 192.168.91.69:6825/1018864 10.12.193.69:6824/1018864
exists,up 37451657-c9f7-4d05-a47f-c6ba35f47b26
osd.65 up in weight 1 up_from 60560 up_thru 60607 down_at 60556
last_clean_interval [59442,60559) 10.12.193.67:6804/16207
192.168.91.67:6812/1016207 192.168.91.67:6813/1016207 10.12.193.67:6813/1016207
exists,up 32aa2e4f-8a00-4320-bf2f-6c107b36adee
osd.66 up in weight 1 up_from 60539 up_thru 60607 down_at 60534
last_clean_interval [59469,60538) 10.12.193.71:6824/2844
192.168.91.71:6834/1002844 192.168.91.71:6835/1002844 10.12.193.71:6811/1002844
exists,up 55771d00-0ca9-4453-a3af-e1e72520ef87
osd.67 up in weight 1 up_from 60537 up_thru 60607 down_at 60534
last_clean_interval [59449,60536) 10.12.193.68:6813/1252
192.168.91.68:6832/1001252 192.168.91.68:6833/1001252 10.12.193.68:6818/1001252
exists,up b1b29cab-7e44-4651-b142-dd542f8855f4
osd.68 up in weight 1 up_from 60558 up_thru 60589 down_at 60556
last_clean_interval [59477,60556) 10.12.193.72:6821/5915
192.168.91.72:6801/1005915 192.168.91.72:6802/1005915 10.12.193.72:6802/1005915
exists,up 3d5924a7-1c74-48c1-9fae-62825a66df52
osd.69 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [59851,60535) 10.12.193.69:6818/22106
192.168.91.69:6828/1022106 192.168.91.69:6829/1022106 10.12.193.69:6826/1022106
exists,up a7ecc3a7-ecaa-428e-8317-14db07825f9e
osd.70 up in weight 1 up_from 60542 up_thru 60607 down_at 60534
last_clean_interval [60371,60541) 10.12.193.70:6818/2686
192.168.91.70:6800/1002686 192.168.91.70:6806/1002686 10.12.193.70:6831/1002686
exists,up aadd793b-7a53-49ea-948a-0a6f20481f0c
osd.71 up in weight 1 up_from 60536 up_thru 60607 down_at 60534
last_clean_interval [59443,60535) 10.12.193.67:6822/17348
192.168.91.67:6824/1017348 192.168.91.67:6825/1017348 10.12.193.67:6824/1017348
exists,up 0b37aeb9-a92f-449c-8e3c-99fbcf5bd8b8
$ sudo ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 131.03943 root default
-2 21.83990 host tv-c2-al02
0 1.81999 osd.0 up 1.00000 1.00000
6 1.81999 osd.6 up 1.00000 1.00000
12 1.81999 osd.12 up 1.00000 1.00000
18 1.81999 osd.18 up 1.00000 1.00000
24 1.81999 osd.24 up 1.00000 1.00000
30 1.81999 osd.30 up 1.00000 1.00000
36 1.81999 osd.36 up 1.00000 1.00000
42 1.81999 osd.42 up 1.00000 1.00000
48 1.81999 osd.48 up 1.00000 1.00000
54 1.81999 osd.54 up 1.00000 1.00000
60 1.81999 osd.60 up 1.00000 1.00000
66 1.81999 osd.66 up 1.00000 1.00000
-3 21.83990 host tv-c1-al02
1 1.81999 osd.1 up 1.00000 1.00000
7 1.81999 osd.7 up 1.00000 1.00000
13 1.81999 osd.13 up 1.00000 1.00000
19 1.81999 osd.19 up 1.00000 1.00000
25 1.81999 osd.25 up 1.00000 1.00000
31 1.81999 osd.31 up 1.00000 1.00000
37 1.81999 osd.37 up 1.00000 1.00000
43 1.81999 osd.43 up 1.00000 1.00000
49 1.81999 osd.49 up 1.00000 1.00000
55 1.81999 osd.55 up 1.00000 1.00000
61 1.81999 osd.61 up 1.00000 1.00000
67 1.81999 osd.67 up 1.00000 1.00000
-4 21.83990 host tv-c2-al03
2 1.81999 osd.2 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000
14 1.81999 osd.14 up 1.00000 1.00000
20 1.81999 osd.20 up 1.00000 1.00000
26 1.81999 osd.26 up 1.00000 1.00000
32 1.81999 osd.32 up 1.00000 1.00000
38 1.81999 osd.38 up 1.00000 1.00000
44 1.81999 osd.44 up 1.00000 1.00000
50 1.81999 osd.50 up 1.00000 1.00000
56 1.81999 osd.56 up 1.00000 1.00000
62 1.81999 osd.62 up 1.00000 1.00000
68 1.81999 osd.68 up 1.00000 1.00000
-5 21.83990 host tv-c2-al01
3 1.81999 osd.3 up 1.00000 1.00000
9 1.81999 osd.9 up 1.00000 1.00000
15 1.81999 osd.15 up 1.00000 1.00000
21 1.81999 osd.21 up 1.00000 1.00000
27 1.81999 osd.27 up 1.00000 1.00000
33 1.81999 osd.33 up 1.00000 1.00000
39 1.81999 osd.39 up 1.00000 1.00000
45 1.81999 osd.45 up 1.00000 1.00000
51 1.81999 osd.51 up 1.00000 1.00000
57 1.81999 osd.57 up 1.00000 1.00000
63 1.81999 osd.63 up 1.00000 1.00000
70 1.81999 osd.70 up 1.00000 1.00000
-6 21.83990 host tv-c1-al03
4 1.81999 osd.4 up 1.00000 1.00000
10 1.81999 osd.10 up 1.00000 1.00000
16 1.81999 osd.16 up 1.00000 1.00000
22 1.81999 osd.22 up 1.00000 1.00000
28 1.81999 osd.28 up 1.00000 1.00000
34 1.81999 osd.34 up 1.00000 1.00000
40 1.81999 osd.40 up 1.00000 1.00000
46 1.81999 osd.46 up 1.00000 1.00000
52 1.81999 osd.52 up 1.00000 1.00000
58 1.81999 osd.58 up 1.00000 1.00000
64 1.81999 osd.64 up 1.00000 1.00000
69 1.81999 osd.69 up 1.00000 1.00000
-7 21.83990 host tv-c1-al01
5 1.81999 osd.5 up 1.00000 1.00000
11 1.81999 osd.11 up 1.00000 1.00000
17 1.81999 osd.17 up 1.00000 1.00000
23 1.81999 osd.23 up 1.00000 1.00000
29 1.81999 osd.29 up 1.00000 1.00000
35 1.81999 osd.35 up 1.00000 1.00000
41 1.81999 osd.41 up 1.00000 1.00000
47 1.81999 osd.47 up 1.00000 1.00000
53 1.81999 osd.53 up 1.00000 1.00000
59 1.81999 osd.59 up 1.00000 1.00000
65 1.81999 osd.65 up 1.00000 1.00000
71 1.81999 osd.71 up 1.00000 1.00000
$
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
