Hi Pavin, Happy New Year!
Many thanks for the commands. I managed to get the cluster into green status with the repeer command. Still had some slow MDS ops, I decided to purge a few older backups in a repository that's backed by one of the cephfs volumes. This resolved all slow OPS -- there must have been an inconsistent file causing an issue, not a big deal as it's a backup (likely manifested itself when cluster was rebuilding / rebalancing). I'm still having an issue in getting RGWs to be manageable / detected by the cluster (any ideas appreciated): - Increased debug logging to 5/5 across debug_rgw,debug_rgw_datacache, debug_rgw_sync - Redeployed RGW service (removed and recreated service); - All GWs show similar log entries -- There's a failed init realm ID error (could that be it?): 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 deferred set uid:gid to 167:167 (ceph:ceph) 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process radosgw, pid 2 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 framework: beast 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 0 framework conf key: port, val: 80 2022-12-31T20:02:55.570+0000 7fea6d76a5c0 1 radosgw_Main not setting numa affinity 2022-12-31T20:02:55.574+0000 7fea6d76a5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 2022-12-31T20:02:55.574+0000 7fea6d76a5c0 1 D3N datacache enabled: 0 2022-12-31T20:02:55.590+0000 7fea6d76a5c0 4 rgw main: RGWPeriod::init failed to init realm id : (2) No such file or directory 2022-12-31T20:02:55.686+0000 7fea6d76a5c0 4 rgw main: Realm: () 2022-12-31T20:02:55.686+0000 7fea6d76a5c0 4 rgw main: ZoneGroup: default (45518452-8aa6-41b4-99f0-059b255c31cd) 2022-12-31T20:02:55.686+0000 7fea6d76a5c0 4 rgw main: Zone: default (743ea532-f5bc-4cca-891b-c27a586d5129) 2022-12-31T20:02:56.010+0000 7fea6d76a5c0 2 all 8 watchers are set, enabling cache 2022-12-31T20:02:56.026+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:02:57.170+0000 7fea4eab1700 2 garbage collection: garbage collection: start 2022-12-31T20:02:57.170+0000 7fea4e2b0700 2 rgw object expirer Worker thread: object expiration: start 2022-12-31T20:02:57.170+0000 7fea48aa5700 5 lifecycle: schedule life cycle next start time: Sun Jan 1 00:00:00 2023 2022-12-31T20:02:57.170+0000 7fea46aa1700 5 lifecycle: schedule life cycle next start time: Sun Jan 1 00:00:00 2023 2022-12-31T20:02:57.170+0000 7fea44a9d700 5 lifecycle: schedule life cycle next start time: Sun Jan 1 00:00:00 2023 2022-12-31T20:02:59.186+0000 7fea4eab1700 2 garbage collection: garbage collection: stop 2022-12-31T20:03:18.029+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:03:40.029+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:04:02.028+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:04:24.027+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:04:46.031+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:05:08.030+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:05:30.030+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:05:52.029+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:06:14.029+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:06:36.028+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:06:58.032+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:07:20.031+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:07:42.031+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:08:04.030+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:08:26.029+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:08:48.029+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:09:10.032+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:09:32.028+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-12-31T20:09:54.027+0000 7fea52ab9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start On Sat, Dec 31, 2022 at 1:22 AM Pavin Joseph <m...@pavinjoseph.com> wrote: > Hey there, > > Sorry for the late reply. > If the pg issue isn't solved yet, could you run these: > > ceph pg repeer <pgid> > ceph pg repair <pgid> > > Pavin. > > On 29-Dec-22 4:08 AM, Deep Dish wrote: > > Hi Pavin, > > > > The following are additional developments.. There's one PG that's > > stuck and unable to recover. I've attached relevant ceph -s / health > > detail and pg stat outputs below. > > > > - There were some remaining lock files as suggested in /var/run/ceph/ > > pertaining to rgw. I removed the service, deleted any stale lock files > > and redeployed the RGWs. All started with the common log entries across > > all: > > > > 7ff5d9aaf5c0 0 deferred set uid:gid to 167:167 (ceph:ceph) > > > > 7ff5d9aaf5c0 0 ceph version 17.2.5 > > (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process > > radosgw, pid 2 > > > > 7ff5d9aaf5c0 0 framework: beast > > > > 7ff5d9aaf5c0 0 framework conf key: port, val: 80 > > > > 7ff5d9aaf5c0 1 radosgw_Main not setting numa affinity > > > > 7ff5d9aaf5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 > > > > 7ff5d9aaf5c0 1 D3N datacache enabled: 0 > > > > No additional log entries are recorded since starting them post > > re-deployment as per above. > > > > The cluster settled, there is no recovery activity. There is one pg > that's > > stuck and I have a hunch that it's impacting MDS and RGW processes as > > stated in the thread. PG is stuck as as active+remapped+backfilling: > > > > > > > > data: > > > > volumes: 2/2 healthy > > > > pools: 16 pools, 1504 pgs > > > > objects: 24.49M objects, 79 TiB > > > > usage: 119 TiB used, 390 TiB / 508 TiB avail > > > > pgs: 65210/146755179 objects misplaced (0.044%) > > > > 1503 active+clean > > > > 1 active+remapped+backfilling > > > > > > > > progress: > > > > Global Recovery Event (6h) > > > > [===========================.] (remaining: 73s) > > > > # ceph health detail > > > > HEALTH_WARN 1 MDSs report slow metadata IOs; 1 pgs not deep-scrubbed in > > time; 1 pgs not scrubbed in time > > > > [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs > > > > mds.fs01.ceph02mon02.wicrdz(mds.0): 5 slow metadata IOs are blocked > > > > 30 secs, oldest blocked for 74436 secs > > > > [WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time > > > > pg 14.ff not deep-scrubbed since 2022-12-14T19:35:51.893008+0000 > > > > [WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time > > > > pg 14.ff not scrubbed since 2022-12-17T06:33:40.577932+0000 > > > > > > > > From the following pg query: > > > > - "pgid": "14.ffs0" is stuck as peering (osd 5) > > - "pgid": "14.ffs4" is stuck as unknown (osd 18) > > - "pgid": "14.ffs5" is stuck as unknown (osd 24) > > - "pgid": "14.ffs3" is stuck as unknown (osd 42) > > - "pgid": "14.ffs2" is stick as unknown (osd 58) > > - "pgid": "14.ffs1" is marked as active+clean (osd 36) > > > > # ceph pg 14.ff query > > > > { > > > > "snap_trimq": "[]", > > > > "snap_trimq_len": 0, > > > > "state": "active+remapped+backfilling", > > > > "epoch": 19594, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "backfill_targets": [ > > > > "5(0)", > > > > "18(4)", > > > > "24(5)", > > > > "42(3)", > > > > "58(2)" > > > > ], > > > > "acting_recovery_backfill": [ > > > > "5(0)", > > > > "5(2)", > > > > "15(4)", > > > > "18(4)", > > > > "24(5)", > > > > "26(3)", > > > > "36(1)", > > > > "42(3)", > > > > "46(5)", > > > > "50(0)", > > > > "58(2)" > > > > ], > > > > "info": { > > > > "pgid": "14.ffs0", > > > > "last_update": "19550'35077", > > > > "last_complete": "19550'35077", > > > > "log_tail": "13761'32157", > > > > "last_user_version": 35077, > > > > "last_backfill": "MAX", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "19550'35077", > > > > "reported_seq": 396919, > > > > "reported_epoch": 19594, > > > > "state": "active+remapped+backfilling", > > > > "last_fresh": "2022-12-28T22:03:20.278478+0000", > > > > "last_change": "2022-12-26T21:27:51.600940+0000", > > > > "last_active": "2022-12-28T22:03:20.278478+0000", > > > > "last_peered": "2022-12-28T22:03:20.278478+0000", > > > > "last_clean": "2022-12-26T21:27:45.471954+0000", > > > > "last_became_active": "2022-12-26T21:27:51.085966+0000", > > > > "last_became_peered": "2022-12-26T21:27:51.085966+0000", > > > > "last_unstale": "2022-12-28T22:03:20.278478+0000", > > > > "last_undegraded": "2022-12-28T22:03:20.278478+0000", > > > > "last_fullsized": "2022-12-28T22:03:20.278478+0000", > > > > "mapping_epoch": 16615, > > > > "log_start": "13761'32157", > > > > "ondisk_log_start": "13761'32157", > > > > "created": 4537, > > > > "last_epoch_clean": 14655, > > > > "parent": "0.0", > > > > "parent_split_bits": 8, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "objects_scrubbed": 16227, > > > > "log_size": 2920, > > > > "ondisk_log_size": 2920, > > > > "stats_invalid": true, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 14, > > > > "scrub_schedule": "queued for deep scrub", > > > > "scrub_duration": 13.320415128, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 56709530650, > > > > "num_objects": 13548, > > > > "num_object_clones": 0, > > > > "num_object_copies": 81288, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 0, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 65210, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 13548, > > > > "num_whiteouts": 0, > > > > "num_read": 67760, > > > > "num_read_kb": 177798674, > > > > "num_write": 21231, > > > > "num_write_kb": 70024901, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 34481, > > > > "num_bytes_recovered": 144364295675, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "avail_no_missing": [ > > > > "50(0)", > > > > "5(2)", > > > > "15(4)", > > > > "26(3)", > > > > "36(1)", > > > > "46(5)" > > > > ], > > > > "object_location_counts": [ > > > > { > > > > "shards": "5(2),15(4),26(3),36(1),46(5),50(0)", > > > > "objects": 13548 > > > > } > > > > ], > > > > "blocked_by": [], > > > > "up_primary": 5, > > > > "acting_primary": 50, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 0, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > "peer_info": [ > > > > { > > > > "peer": "5(0)", > > > > "pgid": "14.ffs0", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 0, > > > > "last_backfill": "14:ff09a915:::10000001249.00000353:head", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "0'0", > > > > "reported_seq": 2, > > > > "reported_epoch": 16614, > > > > "state": "peering", > > > > "last_fresh": "2022-12-26T21:27:47.993646+0000", > > > > "last_change": "2022-12-26T21:27:47.769141+0000", > > > > "last_active": "0.000000", > > > > "last_peered": "0.000000", > > > > "last_clean": "0.000000", > > > > "last_became_active": "0.000000", > > > > "last_became_peered": "0.000000", > > > > "last_unstale": "2022-12-26T21:27:47.993646+0000", > > > > "last_undegraded": "2022-12-26T21:27:47.993646+0000", > > > > "last_fullsized": "2022-12-26T21:27:47.993646+0000", > > > > "mapping_epoch": 16615, > > > > "log_start": "0'0", > > > > "ondisk_log_start": "0'0", > > > > "created": 4537, > > > > "last_epoch_clean": 14655, > > > > "parent": "0.0", > > > > "parent_split_bits": 0, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "objects_scrubbed": 0, > > > > "log_size": 0, > > > > "ondisk_log_size": 0, > > > > "stats_invalid": false, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 0, > > > > "scrub_schedule": "queued for deep scrub", > > > > "scrub_duration": 0, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 2115497390, > > > > "num_objects": 506, > > > > "num_object_clones": 0, > > > > "num_object_copies": 0, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 13042, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 506, > > > > "num_whiteouts": 0, > > > > "num_read": 0, > > > > "num_read_kb": 0, > > > > "num_write": 42, > > > > "num_write_kb": 172032, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 0, > > > > "num_bytes_recovered": 0, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [ > > > > 24 > > > > ], > > > > "up_primary": 5, > > > > "acting_primary": 50, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 1, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "5(2)", > > > > "pgid": "14.ffs2", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 33714, > > > > "last_backfill": "MAX", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "16465'33714", > > > > "reported_seq": 388679, > > > > "reported_epoch": 16465, > > > > "state": "active+clean", > > > > "last_fresh": "2022-12-26T16:10:12.977587+0000", > > > > "last_change": "2022-12-26T02:28:03.455705+0000", > > > > "last_active": "2022-12-26T16:10:12.977587+0000", > > > > "last_peered": "2022-12-26T16:10:12.977587+0000", > > > > "last_clean": "2022-12-26T16:10:12.977587+0000", > > > > "last_became_active": "2022-12-26T02:28:03.455192+0000", > > > > "last_became_peered": "2022-12-26T02:28:03.455192+0000", > > > > "last_unstale": "2022-12-26T16:10:12.977587+0000", > > > > "last_undegraded": "2022-12-26T16:10:12.977587+0000", > > > > "last_fullsized": "2022-12-26T16:10:12.977587+0000", > > > > "mapping_epoch": 16615, > > > > "log_start": "12957'31152", > > > > "ondisk_log_start": "12957'31152", > > > > "created": 4537, > > > > "last_epoch_clean": 14655, > > > > "parent": "0.0", > > > > "parent_split_bits": 8, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "objects_scrubbed": 16227, > > > > "log_size": 2562, > > > > "ondisk_log_size": 2562, > > > > "stats_invalid": true, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 14, > > > > "scrub_schedule": "queued for deep scrub", > > > > "scrub_duration": 13.320415128, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 51491812456, > > > > "num_objects": 12301, > > > > "num_object_clones": 0, > > > > "num_object_copies": 73806, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 0, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 12301, > > > > "num_whiteouts": 0, > > > > "num_read": 66867, > > > > "num_read_kb": 177079789, > > > > "num_write": 19946, > > > > "num_write_kb": 64929471, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 34017, > > > > "num_bytes_recovered": 142424959053, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": 5, > > > > "acting_primary": 50, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 0, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "15(4)", > > > > "pgid": "14.ffs4", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 33714, > > > > "last_backfill": "MAX", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "16465'33714", > > > > "reported_seq": 388679, > > > > "reported_epoch": 16465, > > > > "state": "active+clean", > > > > "last_fresh": "2022-12-26T16:10:12.977587+0000", > > > > "last_change": "2022-12-26T02:28:03.455705+0000", > > > > "last_active": "2022-12-26T16:10:12.977587+0000", > > > > "last_peered": "2022-12-26T16:10:12.977587+0000", > > > > "last_clean": "2022-12-26T16:10:12.977587+0000", > > > > "last_became_active": "2022-12-26T02:28:03.455192+0000", > > > > "last_became_peered": "2022-12-26T02:28:03.455192+0000", > > > > "last_unstale": "2022-12-26T16:10:12.977587+0000", > > > > "last_undegraded": "2022-12-26T16:10:12.977587+0000", > > > > "last_fullsized": "2022-12-26T16:10:12.977587+0000", > > > > "mapping_epoch": 16615, > > > > "log_start": "12957'31152", > > > > "ondisk_log_start": "12957'31152", > > > > "created": 4537, > > > > "last_epoch_clean": 14655, > > > > "parent": "0.0", > > > > "parent_split_bits": 8, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "objects_scrubbed": 16227, > > > > "log_size": 2562, > > > > "ondisk_log_size": 2562, > > > > "stats_invalid": true, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 14, > > > > "scrub_schedule": "queued for deep scrub", > > > > "scrub_duration": 13.320415128, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 51491812456, > > > > "num_objects": 12301, > > > > "num_object_clones": 0, > > > > "num_object_copies": 73806, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 0, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 12301, > > > > "num_whiteouts": 0, > > > > "num_read": 66867, > > > > "num_read_kb": 177079789, > > > > "num_write": 19946, > > > > "num_write_kb": 64929471, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 34017, > > > > "num_bytes_recovered": 142424959053, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": 5, > > > > "acting_primary": 50, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 0, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "18(4)", > > > > "pgid": "14.ffs4", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 0, > > > > "last_backfill": "14:ff09a915:::10000001249.00000353:head", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "0'0", > > > > "reported_seq": 0, > > > > "reported_epoch": 0, > > > > "state": "unknown", > > > > "last_fresh": "0.000000", > > > > "last_change": "0.000000", > > > > "last_active": "0.000000", > > > > "last_peered": "0.000000", > > > > "last_clean": "0.000000", > > > > "last_became_active": "0.000000", > > > > "last_became_peered": "0.000000", > > > > "last_unstale": "0.000000", > > > > "last_undegraded": "0.000000", > > > > "last_fullsized": "0.000000", > > > > "mapping_epoch": 0, > > > > "log_start": "0'0", > > > > "ondisk_log_start": "0'0", > > > > "created": 0, > > > > "last_epoch_clean": 0, > > > > "parent": "0.0", > > > > "parent_split_bits": 0, > > > > "last_scrub": "0'0", > > > > "last_scrub_stamp": "0.000000", > > > > "last_deep_scrub": "0'0", > > > > "last_deep_scrub_stamp": "0.000000", > > > > "last_clean_scrub_stamp": "0.000000", > > > > "objects_scrubbed": 0, > > > > "log_size": 0, > > > > "ondisk_log_size": 0, > > > > "stats_invalid": false, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 0, > > > > "scrub_schedule": "--", > > > > "scrub_duration": 0, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 2115497390, > > > > "num_objects": 506, > > > > "num_object_clones": 0, > > > > "num_object_copies": 0, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 13042, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 506, > > > > "num_whiteouts": 0, > > > > "num_read": 0, > > > > "num_read_kb": 0, > > > > "num_write": 42, > > > > "num_write_kb": 172032, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 0, > > > > "num_bytes_recovered": 0, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [], > > > > "acting": [], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": -1, > > > > "acting_primary": -1, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 1, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "24(5)", > > > > "pgid": "14.ffs5", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 0, > > > > "last_backfill": "14:ff09a915:::10000001249.00000353:head", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "0'0", > > > > "reported_seq": 0, > > > > "reported_epoch": 0, > > > > "state": "unknown", > > > > "last_fresh": "0.000000", > > > > "last_change": "0.000000", > > > > "last_active": "0.000000", > > > > "last_peered": "0.000000", > > > > "last_clean": "0.000000", > > > > "last_became_active": "0.000000", > > > > "last_became_peered": "0.000000", > > > > "last_unstale": "0.000000", > > > > "last_undegraded": "0.000000", > > > > "last_fullsized": "0.000000", > > > > "mapping_epoch": 0, > > > > "log_start": "0'0", > > > > "ondisk_log_start": "0'0", > > > > "created": 0, > > > > "last_epoch_clean": 0, > > > > "parent": "0.0", > > > > "parent_split_bits": 0, > > > > "last_scrub": "0'0", > > > > "last_scrub_stamp": "0.000000", > > > > "last_deep_scrub": "0'0", > > > > "last_deep_scrub_stamp": "0.000000", > > > > "last_clean_scrub_stamp": "0.000000", > > > > "objects_scrubbed": 0, > > > > "log_size": 0, > > > > "ondisk_log_size": 0, > > > > "stats_invalid": false, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 0, > > > > "scrub_schedule": "--", > > > > "scrub_duration": 0, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 2115497390, > > > > "num_objects": 506, > > > > "num_object_clones": 0, > > > > "num_object_copies": 0, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 13042, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 506, > > > > "num_whiteouts": 0, > > > > "num_read": 0, > > > > "num_read_kb": 0, > > > > "num_write": 42, > > > > "num_write_kb": 172032, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 0, > > > > "num_bytes_recovered": 0, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [], > > > > "acting": [], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": -1, > > > > "acting_primary": -1, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 1, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "26(3)", > > > > "pgid": "14.ffs3", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 33714, > > > > "last_backfill": "MAX", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "16465'33714", > > > > "reported_seq": 388679, > > > > "reported_epoch": 16465, > > > > "state": "active+clean", > > > > "last_fresh": "2022-12-26T16:10:12.977587+0000", > > > > "last_change": "2022-12-26T02:28:03.455705+0000", > > > > "last_active": "2022-12-26T16:10:12.977587+0000", > > > > "last_peered": "2022-12-26T16:10:12.977587+0000", > > > > "last_clean": "2022-12-26T16:10:12.977587+0000", > > > > "last_became_active": "2022-12-26T02:28:03.455192+0000", > > > > "last_became_peered": "2022-12-26T02:28:03.455192+0000", > > > > "last_unstale": "2022-12-26T16:10:12.977587+0000", > > > > "last_undegraded": "2022-12-26T16:10:12.977587+0000", > > > > "last_fullsized": "2022-12-26T16:10:12.977587+0000", > > > > "mapping_epoch": 16615, > > > > "log_start": "12957'31152", > > > > "ondisk_log_start": "12957'31152", > > > > "created": 4537, > > > > "last_epoch_clean": 14655, > > > > "parent": "0.0", > > > > "parent_split_bits": 8, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "objects_scrubbed": 16227, > > > > "log_size": 2562, > > > > "ondisk_log_size": 2562, > > > > "stats_invalid": true, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 14, > > > > "scrub_schedule": "queued for deep scrub", > > > > "scrub_duration": 13.320415128, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 51491812456, > > > > "num_objects": 12301, > > > > "num_object_clones": 0, > > > > "num_object_copies": 73806, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 0, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 12301, > > > > "num_whiteouts": 0, > > > > "num_read": 66867, > > > > "num_read_kb": 177079789, > > > > "num_write": 19946, > > > > "num_write_kb": 64929471, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 34017, > > > > "num_bytes_recovered": 142424959053, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": 5, > > > > "acting_primary": 50, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 0, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "36(1)", > > > > "pgid": "14.ffs1", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 33714, > > > > "last_backfill": "MAX", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "16465'33714", > > > > "reported_seq": 388679, > > > > "reported_epoch": 16465, > > > > "state": "active+clean", > > > > "last_fresh": "2022-12-26T16:10:12.977587+0000", > > > > "last_change": "2022-12-26T02:28:03.455705+0000", > > > > "last_active": "2022-12-26T16:10:12.977587+0000", > > > > "last_peered": "2022-12-26T16:10:12.977587+0000", > > > > "last_clean": "2022-12-26T16:10:12.977587+0000", > > > > "last_became_active": "2022-12-26T02:28:03.455192+0000", > > > > "last_became_peered": "2022-12-26T02:28:03.455192+0000", > > > > "last_unstale": "2022-12-26T16:10:12.977587+0000", > > > > "last_undegraded": "2022-12-26T16:10:12.977587+0000", > > > > "last_fullsized": "2022-12-26T16:10:12.977587+0000", > > > > "mapping_epoch": 16615, > > > > "log_start": "12957'31152", > > > > "ondisk_log_start": "12957'31152", > > > > "created": 4537, > > > > "last_epoch_clean": 14655, > > > > "parent": "0.0", > > > > "parent_split_bits": 8, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "objects_scrubbed": 16227, > > > > "log_size": 2562, > > > > "ondisk_log_size": 2562, > > > > "stats_invalid": true, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 14, > > > > "scrub_schedule": "queued for deep scrub", > > > > "scrub_duration": 13.320415128, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 51491812456, > > > > "num_objects": 12301, > > > > "num_object_clones": 0, > > > > "num_object_copies": 73806, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 0, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 12301, > > > > "num_whiteouts": 0, > > > > "num_read": 66867, > > > > "num_read_kb": 177079789, > > > > "num_write": 19946, > > > > "num_write_kb": 64929471, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 34017, > > > > "num_bytes_recovered": 142424959053, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": 5, > > > > "acting_primary": 50, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 0, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "42(3)", > > > > "pgid": "14.ffs3", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 0, > > > > "last_backfill": "14:ff09a915:::10000001249.00000353:head", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "0'0", > > > > "reported_seq": 0, > > > > "reported_epoch": 0, > > > > "state": "unknown", > > > > "last_fresh": "0.000000", > > > > "last_change": "0.000000", > > > > "last_active": "0.000000", > > > > "last_peered": "0.000000", > > > > "last_clean": "0.000000", > > > > "last_became_active": "0.000000", > > > > "last_became_peered": "0.000000", > > > > "last_unstale": "0.000000", > > > > "last_undegraded": "0.000000", > > > > "last_fullsized": "0.000000", > > > > "mapping_epoch": 0, > > > > "log_start": "0'0", > > > > "ondisk_log_start": "0'0", > > > > "created": 0, > > > > "last_epoch_clean": 0, > > > > "parent": "0.0", > > > > "parent_split_bits": 0, > > > > "last_scrub": "0'0", > > > > "last_scrub_stamp": "0.000000", > > > > "last_deep_scrub": "0'0", > > > > "last_deep_scrub_stamp": "0.000000", > > > > "last_clean_scrub_stamp": "0.000000", > > > > "objects_scrubbed": 0, > > > > "log_size": 0, > > > > "ondisk_log_size": 0, > > > > "stats_invalid": false, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 0, > > > > "scrub_schedule": "--", > > > > "scrub_duration": 0, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 2115497390, > > > > "num_objects": 506, > > > > "num_object_clones": 0, > > > > "num_object_copies": 0, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 13042, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 506, > > > > "num_whiteouts": 0, > > > > "num_read": 0, > > > > "num_read_kb": 0, > > > > "num_write": 42, > > > > "num_write_kb": 172032, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 0, > > > > "num_bytes_recovered": 0, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [], > > > > "acting": [], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": -1, > > > > "acting_primary": -1, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 1, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "46(5)", > > > > "pgid": "14.ffs5", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 33714, > > > > "last_backfill": "MAX", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "16465'33714", > > > > "reported_seq": 388679, > > > > "reported_epoch": 16465, > > > > "state": "active+clean", > > > > "last_fresh": "2022-12-26T16:10:12.977587+0000", > > > > "last_change": "2022-12-26T02:28:03.455705+0000", > > > > "last_active": "2022-12-26T16:10:12.977587+0000", > > > > "last_peered": "2022-12-26T16:10:12.977587+0000", > > > > "last_clean": "2022-12-26T16:10:12.977587+0000", > > > > "last_became_active": "2022-12-26T02:28:03.455192+0000", > > > > "last_became_peered": "2022-12-26T02:28:03.455192+0000", > > > > "last_unstale": "2022-12-26T16:10:12.977587+0000", > > > > "last_undegraded": "2022-12-26T16:10:12.977587+0000", > > > > "last_fullsized": "2022-12-26T16:10:12.977587+0000", > > > > "mapping_epoch": 16615, > > > > "log_start": "12957'31152", > > > > "ondisk_log_start": "12957'31152", > > > > "created": 4537, > > > > "last_epoch_clean": 14655, > > > > "parent": "0.0", > > > > "parent_split_bits": 8, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "objects_scrubbed": 16227, > > > > "log_size": 2562, > > > > "ondisk_log_size": 2562, > > > > "stats_invalid": true, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 14, > > > > "scrub_schedule": "queued for deep scrub", > > > > "scrub_duration": 13.320415128, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 51491812456, > > > > "num_objects": 12301, > > > > "num_object_clones": 0, > > > > "num_object_copies": 73806, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 0, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 12301, > > > > "num_whiteouts": 0, > > > > "num_read": 66867, > > > > "num_read_kb": 177079789, > > > > "num_write": 19946, > > > > "num_write_kb": 64929471, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 34017, > > > > "num_bytes_recovered": 142424959053, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [ > > > > 5, > > > > 36, > > > > 58, > > > > 42, > > > > 18, > > > > 24 > > > > ], > > > > "acting": [ > > > > 50, > > > > 36, > > > > 5, > > > > 26, > > > > 15, > > > > 46 > > > > ], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": 5, > > > > "acting_primary": 50, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 0, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > }, > > > > { > > > > "peer": "58(2)", > > > > "pgid": "14.ffs2", > > > > "last_update": "19550'35077", > > > > "last_complete": "17223'34381", > > > > "log_tail": "12957'31152", > > > > "last_user_version": 0, > > > > "last_backfill": "14:ff09a915:::10000001249.00000353:head", > > > > "purged_snaps": [], > > > > "history": { > > > > "epoch_created": 4537, > > > > "epoch_pool_created": 2032, > > > > "last_epoch_started": 16616, > > > > "last_interval_started": 16615, > > > > "last_epoch_clean": 14655, > > > > "last_interval_clean": 14654, > > > > "last_epoch_split": 4537, > > > > "last_epoch_marked_full": 0, > > > > "same_up_since": 16613, > > > > "same_interval_since": 16615, > > > > "same_primary_since": 16615, > > > > "last_scrub": "3817'25569", > > > > "last_scrub_stamp": "2022-12-17T06:33:40.577932+0000", > > > > "last_deep_scrub": "3756'21592", > > > > "last_deep_scrub_stamp": > "2022-12-14T19:35:51.893008+0000", > > > > "last_clean_scrub_stamp": > "2022-12-17T06:33:40.577932+0000", > > > > "prior_readable_until_ub": 0 > > > > }, > > > > "stats": { > > > > "version": "0'0", > > > > "reported_seq": 0, > > > > "reported_epoch": 0, > > > > "state": "unknown", > > > > "last_fresh": "0.000000", > > > > "last_change": "0.000000", > > > > "last_active": "0.000000", > > > > "last_peered": "0.000000", > > > > "last_clean": "0.000000", > > > > "last_became_active": "0.000000", > > > > "last_became_peered": "0.000000", > > > > "last_unstale": "0.000000", > > > > "last_undegraded": "0.000000", > > > > "last_fullsized": "0.000000", > > > > "mapping_epoch": 0, > > > > "log_start": "0'0", > > > > "ondisk_log_start": "0'0", > > > > "created": 0, > > > > "last_epoch_clean": 0, > > > > "parent": "0.0", > > > > "parent_split_bits": 0, > > > > "last_scrub": "0'0", > > > > "last_scrub_stamp": "0.000000", > > > > "last_deep_scrub": "0'0", > > > > "last_deep_scrub_stamp": "0.000000", > > > > "last_clean_scrub_stamp": "0.000000", > > > > "objects_scrubbed": 0, > > > > "log_size": 0, > > > > "ondisk_log_size": 0, > > > > "stats_invalid": false, > > > > "dirty_stats_invalid": false, > > > > "omap_stats_invalid": false, > > > > "hitset_stats_invalid": false, > > > > "hitset_bytes_stats_invalid": false, > > > > "pin_stats_invalid": false, > > > > "manifest_stats_invalid": false, > > > > "snaptrimq_len": 0, > > > > "last_scrub_duration": 0, > > > > "scrub_schedule": "--", > > > > "scrub_duration": 0, > > > > "objects_trimmed": 0, > > > > "snaptrim_duration": 0, > > > > "stat_sum": { > > > > "num_bytes": 2115497390, > > > > "num_objects": 506, > > > > "num_object_clones": 0, > > > > "num_object_copies": 0, > > > > "num_objects_missing_on_primary": 0, > > > > "num_objects_missing": 13042, > > > > "num_objects_degraded": 0, > > > > "num_objects_misplaced": 0, > > > > "num_objects_unfound": 0, > > > > "num_objects_dirty": 506, > > > > "num_whiteouts": 0, > > > > "num_read": 0, > > > > "num_read_kb": 0, > > > > "num_write": 42, > > > > "num_write_kb": 172032, > > > > "num_scrub_errors": 0, > > > > "num_shallow_scrub_errors": 0, > > > > "num_deep_scrub_errors": 0, > > > > "num_objects_recovered": 0, > > > > "num_bytes_recovered": 0, > > > > "num_keys_recovered": 0, > > > > "num_objects_omap": 0, > > > > "num_objects_hit_set_archive": 0, > > > > "num_bytes_hit_set_archive": 0, > > > > "num_flush": 0, > > > > "num_flush_kb": 0, > > > > "num_evict": 0, > > > > "num_evict_kb": 0, > > > > "num_promote": 0, > > > > "num_flush_mode_high": 0, > > > > "num_flush_mode_low": 0, > > > > "num_evict_mode_some": 0, > > > > "num_evict_mode_full": 0, > > > > "num_objects_pinned": 0, > > > > "num_legacy_snapsets": 0, > > > > "num_large_omap_objects": 0, > > > > "num_objects_manifest": 0, > > > > "num_omap_bytes": 0, > > > > "num_omap_keys": 0, > > > > "num_objects_repaired": 0 > > > > }, > > > > "up": [], > > > > "acting": [], > > > > "avail_no_missing": [], > > > > "object_location_counts": [], > > > > "blocked_by": [], > > > > "up_primary": -1, > > > > "acting_primary": -1, > > > > "purged_snaps": [] > > > > }, > > > > "empty": 0, > > > > "dne": 0, > > > > "incomplete": 1, > > > > "last_epoch_started": 16616, > > > > "hit_set_history": { > > > > "current_last_update": "0'0", > > > > "history": [] > > > > } > > > > } > > > > ], > > > > "recovery_state": [ > > > > { > > > > "name": "Started/Primary/Active", > > > > "enter_time": "2022-12-26T21:27:49.970477+0000", > > > > "might_have_unfound": [], > > > > "recovery_progress": { > > > > "backfill_targets": [ > > > > "5(0)", > > > > "18(4)", > > > > "24(5)", > > > > "42(3)", > > > > "58(2)" > > > > ], > > > > "waiting_on_backfill": [], > > > > "last_backfill_started": > > "14:ff09b920:::10000005377.00000209:head", > > > > "backfill_info": { > > > > "begin": "14:ff09bed4:::10000001a7f.0000182d:head", > > > > "end": "14:ff0a8fd7:::1000000072c.000001a0:head", > > > > "objects": [ > > > > { > > > > "object": > > "14:ff09bed4:::10000001a7f.0000182d:head", > > > > "version": "3801'24969" > > > > }, > > > > { > > > > "object": > > "14:ff09c414:::10000004daf.00001063:head", > > > > "version": "6812'28471" > > > > }, > > > > { > > > > "object": > > "14:ff09c530:::10000003957.00000189:head", > > > > "version": "9167'29571" > > > > }, > > > > { > > > > "object": > > "14:ff09c607:::100000011db.00000977:head", > > > > "version": "3703'16620" > > > > }, > > > > { > > > > "object": > > "14:ff09c922:::10000001230.000001e7:head", > > > > "version": "3728'18321" > > > > }, > > > > { > > > > "object": > > "14:ff09cb55:::1000000454e.000009a3:head", > > > > "version": "3706'17783" > > > > }, > > > > { > > > > "object": > > "14:ff09ce4d:::10000001316.00000154:head", > > > > "version": "3798'23751" > > > > }, > > > > { > > > > "object": > > "14:ff09d097:::10000003c4f.00000085:head", > > > > "version": "3822'26172" > > > > }, > > > > { > > > > "object": > > "14:ff09d449:::1000000188a.0000034a:head", > > > > "version": "4627'27150" > > > > }, > > > > { > > > > "object": > > "14:ff09d6b4:::10000003c4a.00000b0f:head", > > > > "version": "3822'26170" > > > > }, > > > > { > > > > "object": > > "14:ff09de4e:::10000001b63.000004c5:head", > > > > "version": "3608'11396" > > > > }, > > > > { > > > > "object": > > "14:ff09e1e3:::10000002a02.000016cb:head", > > > > "version": "12960'31316" > > > > }, > > > > { > > > > "object": > > "14:ff09f059:::10000002c09.00000095:head", > > > > "version": "13226'31609" > > > > }, > > > > { > > > > "object": > > "14:ff09f069:::1000000279e.0000012f:head", > > > > "version": "13274'31844" > > > > }, > > > > { > > > > "object": > > "14:ff0a0029:::10000001cd0.000000a2:head", > > > > "version": "3702'16325" > > > > }, > > > > { > > > > "object": > > "14:ff0a083b:::100000016c6.0000013e:head", > > > > "version": "3497'8495" > > > > }, > > > > { > > > > "object": > > "14:ff0a0d58:::1000000468d.00001cba:head", > > > > "version": "9168'29618" > > > > }, > > > > { > > > > "object": > > "14:ff0a0dfd:::10000000540.0000064a:head", > > > > "version": "3493'8171" > > > > }, > > > > { > > > > "object": > > "14:ff0a1149:::10000005049.00000b52:head", > > > > "version": "3529'10013" > > > > }, > > > > { > > > > "object": > > "14:ff0a1cdb:::100000009df.0000afec:head", > > > > "version": "2106'380" > > > > }, > > > > { > > > > "object": > > "14:ff0a1e1a:::1000000390b.00000106:head", > > > > "version": "6812'28456" > > > > }, > > > > { > > > > "object": > > "14:ff0a2180:::10000003868.000000b9:head", > > > > "version": "3706'17681" > > > > }, > > > > { > > > > "object": > > "14:ff0a2438:::1000000133e.00000024:head", > > > > "version": "3798'24057" > > > > }, > > > > { > > > > "object": > > "14:ff0a280e:::10000002ccc.000004af:head", > > > > "version": "15259'33150" > > > > }, > > > > { > > > > "object": > > "14:ff0a2f43:::10000003d40.00000665:head", > > > > "version": "10508'29957" > > > > }, > > > > { > > > > "object": > > "14:ff0a315f:::100000011db.00000caa:head", > > > > "version": "3703'16628" > > > > }, > > > > { > > > > "object": > > "14:ff0a3347:::10000004576.00001fb9:head", > > > > "version": "3732'19654" > > > > }, > > > > { > > > > "object": > > "14:ff0a3948:::100000032eb.0000011a:head", > > > > "version": "3612'11760" > > > > }, > > > > { > > > > "object": > > "14:ff0a3af1:::1000000275d.0000027a:head", > > > > "version": "12960'31292" > > > > }, > > > > { > > > > "object": > > "14:ff0a4073:::100000018fe.0000355d:head", > > > > "version": "3498'8949" > > > > }, > > > > { > > > > "object": > > "14:ff0a44db:::10000001a11.00000203:head", > > > > "version": "3755'21336" > > > > }, > > > > { > > > > "object": > > "14:ff0a4888:::1000000246e.0000360a:head", > > > > "version": "13794'32296" > > > > }, > > > > { > > > > "object": > > "14:ff0a4e86:::100000054b1.000019c1:head", > > > > "version": "12890'30650" > > > > }, > > > > { > > > > "object": > > "14:ff0a51c6:::10000003970.0000096e:head", > > > > "version": "10636'30139" > > > > }, > > > > { > > > > "object": > > "14:ff0a5302:::10000001871.000018eb:head", > > > > "version": "3817'25516" > > > > }, > > > > { > > > > "object": > > "14:ff0a56ba:::10000002848.00000266:head", > > > > "version": "13855'32461" > > > > }, > > > > { > > > > "object": > > "14:ff0a61f3:::10000000759.00000030:head", > > > > "version": "3703'17434" > > > > }, > > > > { > > > > "object": > > "14:ff0a68e3:::10000001c17.0000012c:head", > > > > "version": "3653'13629" > > > > }, > > > > { > > > > "object": > > "14:ff0a722a:::10000003e74.00000091:head", > > > > "version": "3778'22970" > > > > }, > > > > { > > > > "object": > > "14:ff0a7557:::100000044fe.00001664:head", > > > > "version": "3660'14671" > > > > }, > > > > { > > > > "object": > > "14:ff0a8f78:::10000001821.00000844:head", > > > > "version": "3797'23629" > > > > }, > > > > { > > > > "object": > > "14:ff0a8f8f:::10000001bea.00000202:head", > > > > "version": "3653'13526" > > > > } > > > > ] > > > > }, > > > > "peer_backfill_info": [ > > > > "5(0)", > > > > { > > > > "begin": "MAX", > > > > "end": "MAX", > > > > "objects": [] > > > > }, > > > > "18(4)", > > > > { > > > > "begin": "MAX", > > > > "end": "MAX", > > > > "objects": [] > > > > }, > > > > "24(5)", > > > > { > > > > "begin": "MAX", > > > > "end": "MAX", > > > > "objects": [] > > > > }, > > > > "42(3)", > > > > { > > > > "begin": "MAX", > > > > "end": "MAX", > > > > "objects": [] > > > > }, > > > > "58(2)", > > > > { > > > > "begin": "MAX", > > > > "end": "MAX", > > > > "objects": [] > > > > } > > > > ], > > > > "backfills_in_flight": [ > > > > "14:ff09b0d9:::100000033c2.00000146:head", > > > > "14:ff09b6e5:::10000004b37.00000161:head", > > > > "14:ff09b7b4:::1000000278f.00000374:head", > > > > "14:ff09b920:::10000005377.00000209:head" > > > > ], > > > > "recovering": [ > > > > "14:ff09b0d9:::100000033c2.00000146:head", > > > > "14:ff09b6e5:::10000004b37.00000161:head", > > > > "14:ff09b7b4:::1000000278f.00000374:head", > > > > "14:ff09b920:::10000005377.00000209:head" > > > > ], > > > > "pg_backend": { > > > > "recovery_ops": [ > > > > { > > > > "hoid": > > "14:ff09b0d9:::100000033c2.00000146:head", > > > > "v": "3677'15440", > > > > "missing_on": > "5(0),18(4),24(5),42(3),58(2)", > > > > "missing_on_shards": "0,2,3,4,5", > > > > "recovery_info": > > "ObjectRecoveryInfo(14:ff09b0d9:::100000033c2.00000146:head@3677'15440, > > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, > > object_exist: 1)", > > > > "recovery_progress": > > "ObjectRecoveryProgress(!first, data_recovered_to:4194304, > > data_complete:true, omap_recovered_to:, omap_complete:true, > error:false)", > > > > "state": "WRITING", > > > > "waiting_on_pushes": "5(0)", > > > > "extent_requested": "0,8388608" > > > > }, > > > > { > > > > "hoid": > > "14:ff09b6e5:::10000004b37.00000161:head", > > > > "v": "3549'10993", > > > > "missing_on": > "5(0),18(4),24(5),42(3),58(2)", > > > > "missing_on_shards": "0,2,3,4,5", > > > > "recovery_info": > > "ObjectRecoveryInfo(14:ff09b6e5:::10000004b37.00000161:head@3549'10993, > > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, > > object_exist: 1)", > > > > "recovery_progress": > > "ObjectRecoveryProgress(!first, data_recovered_to:4194304, > > data_complete:true, omap_recovered_to:, omap_complete:true, > error:false)", > > > > "state": "WRITING", > > > > "waiting_on_pushes": "5(0)", > > > > "extent_requested": "0,8388608" > > > > }, > > > > { > > > > "hoid": > > "14:ff09b7b4:::1000000278f.00000374:head", > > > > "v": "13272'31836", > > > > "missing_on": > "5(0),18(4),24(5),42(3),58(2)", > > > > "missing_on_shards": "0,2,3,4,5", > > > > "recovery_info": > > "ObjectRecoveryInfo(14:ff09b7b4:::1000000278f.00000374:head@13272'31836, > > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, > > object_exist: 1)", > > > > "recovery_progress": > > "ObjectRecoveryProgress(!first, data_recovered_to:4194304, > > data_complete:true, omap_recovered_to:, omap_complete:true, > error:false)", > > > > "state": "WRITING", > > > > "waiting_on_pushes": "5(0)", > > > > "extent_requested": "0,8388608" > > > > }, > > > > { > > > > "hoid": > > "14:ff09b920:::10000005377.00000209:head", > > > > "v": "10440'29880", > > > > "missing_on": > "5(0),18(4),24(5),42(3),58(2)", > > > > "missing_on_shards": "0,2,3,4,5", > > > > "recovery_info": > > "ObjectRecoveryInfo(14:ff09b920:::10000005377.00000209:head@10440'29880, > > size: 4194304, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, > > object_exist: 1)", > > > > "recovery_progress": > > "ObjectRecoveryProgress(!first, data_recovered_to:4194304, > > data_complete:true, omap_recovered_to:, omap_complete:true, > error:false)", > > > > "state": "WRITING", > > > > "waiting_on_pushes": "5(0)", > > > > "extent_requested": "0,8388608" > > > > } > > > > ], > > > > "read_ops": [] > > > > } > > > > } > > > > }, > > > > { > > > > "name": "Started", > > > > "enter_time": "2022-12-26T21:27:48.908226+0000" > > > > } > > > > ], > > > > "scrubber": { > > > > "active": false, > > > > "must_scrub": true, > > > > "must_deep_scrub": true, > > > > "must_repair": false, > > > > "need_auto": false, > > > > "scrub_reg_stamp": "1.000000", > > > > "schedule": "queued for deep scrub" > > > > }, > > > > "agent_state": {} > > > > } > > > > > > > > On Wed, Dec 28, 2022 at 6:46 AM Pavin Joseph <m...@pavinjoseph.com> wrote: > > > >> 1. This is a guess, but check /var/[lib|run]/ceph for any lock files. > >> 2. This is more straightforward to fix, add faster WAL/Block device/LV > >> for each OSD or create a fast storage pool just for metadata. Also, > >> experiment with MDS cache size/trim [0] settings. > >> > >> [0]: https://docs.ceph.com/en/latest/cephfs/cache-configuration/ > >> > >> On 28-Dec-22 7:23 AM, Deep Dish wrote: > >>> Got logging enabled as per > >>> https://ceph.io/en/news/blog/2022/centralized_logging/. My embedded > >>> grafana doesn't come up in the dashboard, but at least I have log > (files) > >>> on my nodes. Interesting. > >>> > >>> Two issues plaguing my cluster: > >>> > >>> 1 - RGWs not manageable > >>> 2 - MDS_SLOW_METADATA_IO warning (impact to cephfs) > >>> > >>> Issue 1: > >>> > >>> I have 4x RGWs deployed. All started / processes running. They all > >>> report similar log entries: > >>> > >>> 7fcc32b6a5c0 0 deferred set uid:gid to 167:167 (ceph:ceph) > >>> > >>> 7fcc32b6a5c0 0 ceph version 17.2.5 > >>> (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process > >>> radosgw, pid 2 > >>> > >>> 7fcc32b6a5c0 0 framework: beast > >>> > >>> 7fcc32b6a5c0 0 framework conf key: port, val: 80 > >>> > >>> 7fcc32b6a5c0 1 radosgw_Main not setting numa affinity > >>> > >>> 7fcc32b6a5c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 > >>> > >>> 7fcc32b6a5c0 1 D3N datacache enabled: 0 > >>> > >>> 7fcc0869a700 0 INFO: RGWReshardLock::lock found lock on > >> reshard.0000000011 > >>> to be held by another RGW process; skipping for now > >>> > >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on > >> lc.1, > >>> sleep 5, try again > >>> > >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on > >> lc.3, > >>> sleep 5, try again > >>> > >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0dea5700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> > >>> 7fcc0bea1700 0 lifecycle: RGWLC::process() failed to acquire lock on > >>> lc.16, sleep 5, try again > >>> (repeating) > >>> > >>> Seems like a stale lock, not previously cleaned up when the cluster was > >>> busy recovering and rebalancing. > >>> > >>> Issue 2: > >>> > >>> ceph health detail: > >>> > >>> [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs > >>> > >>> mds.fs01.ceph02mon03.rjcxat(mds.0): 8 slow metadata IOs are > blocked > >>> > >>> 30 secs, oldest blocked for 39485 secs > >>> > >>> Log entries from ceph02mon03 MDS host: > >>> > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131271 from mon.4 > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131272 from mon.4 > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131273 from mon.4 > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131274 from mon.4 > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131275 from mon.4 > >>> 7fe36c6b8700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 > >>> included below; oldest blocked for > 33.126589 secs > >>> 7fe36c6b8700 0 log_channel(cluster) log [WRN] : slow request > 33.126588 > >>> seconds old, received at 2022-12-27T19:45:45.952225+0000: > >>> client_request(client.55009:99980 create > >>> #0x10000000bc2/vzdump-qemu-30003-2022_12_27-14_43_43.log > >>> 2022-12-27T19:45:45.948045+0000 caller_uid=0, caller_gid=0{}) currently > >>> submit entry: journal_and_reply > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131276 from mon.4 > >>> 7fe36c6b8700 0 log_channel(cluster) log [WRN] : 1 slow requests, 0 > >>> included below; oldest blocked for > 38.126737 secs > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131277 from mon.4 > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131278 from mon.4 > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131279 from mon.4 > >>> 7fe36debb700 1 mds.fs01.ceph02mon03.rjcxat Updating MDS map to > version > >>> 131280 from mon.4 > >>> > >>> > >>> I suspect that the file in the log above int's the culprit. How can I > >> get > >>> to the root cause of MDS slowdowns? > >>> > >>> > >>> On Tue, Dec 27, 2022 at 3:32 PM Pavin Joseph <m...@pavinjoseph.com> > wrote: > >>> > >>>> Interesting, the logs show the crash module [0] itself has crashed. > >>>> Something sent it a SIGINT or SIGTERM and the module didn't handle it > >>>> correctly due to what seems like a bug in the code. > >>>> > >>>> I haven't experienced the crash module itself crashing yet (in Quincy) > >>>> because nothing sent a SIG[INT|TERM] to it yet. > >>>> > >>>> So I'd continue investigating into why these signals were sent to the > >>>> crash module. > >>>> > >>>> To fix the crash module from crashing, go to "/usr/bin/ceph-crash" and > >>>> edit the handler function on line 82 like so: > >>>> > >>>> def handler(signum, frame): > >>>> print('*** Interrupted with signal %d ***' % signum) > >>>> signame = signal.Signals(signum).name > >>>> print(f'Signal handler called with signal {signame} ({signum})') > >>>> print(frame) > >>>> sys.exit(0) > >>>> > >>>> --- > >>>> > >>>> Once the crash module is working, perhaps you could run a "ceph crash > >> ls" > >>>> > >>>> Regarding podman logs, perhaps try this [1]. > >>>> > >>>> [0]: https://docs.ceph.com/en/quincy/mgr/crash/ > >>>> [1]: https://docs.podman.io/en/latest/markdown/podman-logs.1.html > >>>> > >>>> On 27-Dec-22 11:59 PM, Deep Dish wrote: > >>>>> HI Pavin, > >>>>> > >>>>> Thanks for the reply. I'm a bit at a loss honestly as this worked > >>>>> perfectly without any issue up until the rebalance of the cluster. > >>>>> Orchestrator is great. Aside from this (which I suspect is not > >>>>> orchestrator related), I haven't had any issues. > >>>>> > >>>>> In terms of logs, I'm not sure where to start looking in this new > >>>>> containerized environment as they pertain to individual ceph > processes > >>>> -- I > >>>>> assumed everything would be centrally collected within orch. > >>>>> > >>>>> Connecting into the podman container of a RGW, there are no logs in > >>>>> /var/log/ceph aside from ceph-volume. My ceph.conf is minimal with > >> only > >>>>> monitors defined. The only log I'm able to pull is as follows: > >>>>> > >>>>> # podman logs 35d4ac5445ca > >>>>> > >>>>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s > >>>>> > >>>>> Traceback (most recent call last): > >>>>> > >>>>> File "/usr/bin/ceph-crash", line 113, in <module> > >>>>> > >>>>> main() > >>>>> > >>>>> File "/usr/bin/ceph-crash", line 109, in main > >>>>> > >>>>> time.sleep(args.delay * 60) > >>>>> > >>>>> TypeError: handler() takes 1 positional argument but 2 were given > >>>>> > >>>>> INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s > >>>>> > >>>>> > >>>>> > >>>>> Looks like the RGW daemon is crashing. How do I get logs to > persist? > >>>> I > >>>>> suspect I won't be able to use orchestrator to push down the config, > >> and > >>>>> would have to manipulate within the container image itself. > >>>>> > >>>>> I also attempted to redeply the RGW containers without success. > >>>>> > >>>>> On Tue, Dec 27, 2022 at 10:39 AM Pavin Joseph <m...@pavinjoseph.com> > >>>> wrote: > >>>>> > >>>>>> Here's the first things I'd check in your situation: > >>>>>> > >>>>>> 1. Logs > >>>>>> 2. Is the RGW HTTP server running on its port? > >>>>>> 3. Re-check config including authentication. > >>>>>> > >>>>>> ceph orch is too new and didn't pass muster in our own internal > >> testing. > >>>>>> You're braver than most for using it in production. > >>>>>> > >>>>>> Pavin. > >>>>>> > >>>>>> On 27-Dec-22 8:48 PM, Deep Dish wrote: > >>>>>>> Quick update: > >>>>>>> > >>>>>>> - I followed documentation, and ran the following: > >>>>>>> > >>>>>>> # ceph dashboard set-rgw-credentials > >>>>>>> > >>>>>>> Error EINVAL: No RGW credentials found, please consult the > >>>> documentation > >>>>>> on > >>>>>>> how to enable RGW for the dashboard. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> - I see dashboard credentials configured (all this was working fine > >>>>>> before): > >>>>>>> > >>>>>>> > >>>>>>> # ceph dashboard get-rgw-api-access-key > >>>>>>> > >>>>>>> P?????????????????G (? commented out) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Seems to me like my RGW config is non-existent / corrupted for some > >>>>>>> reason. When trying to curl a RGW directly I get a "connection > >>>> refused". > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Dec 27, 2022 at 9:41 AM Deep Dish <deeepd...@gmail.com> > >> wrote: > >>>>>>> > >>>>>>>> I built a net-new Quincy cluster (17.2.5) using ceph orch as > >> follows: > >>>>>>>> > >>>>>>>> 2x mgrs > >>>>>>>> 4x rgw > >>>>>>>> 5x mon > >>>>>>>> 4x rgw > >>>>>>>> 5x mds > >>>>>>>> 6x osd hosts w/ 10 drives each --> will be growing to 7 osd hosts > in > >>>> the > >>>>>>>> coming days. > >>>>>>>> > >>>>>>>> I migrated all data from my legacy nautilus cluster (via > rbd-mirror, > >>>>>>>> rclone for s3 buckets, etc.). All moved over successfully without > >>>>>> issue. > >>>>>>>> > >>>>>>>> The cluster went through a series of rebalancing events (adding > >>>>>> capacity, > >>>>>>>> osd nodes, changing fault domain for EC volumes). > >>>>>>>> > >>>>>>>> It's settled now, however throughout the process all of my RGW > nodes > >>>> are > >>>>>>>> no longer part of the cluster -- meaning ceph doesn't recognize / > >>>> detect > >>>>>>>> them, despite containers, networking, etc. all being setup > >> correctly. > >>>>>>>> This also means I'm unable to manage any RGW functions (via the > >>>>>> dashboard > >>>>>>>> or cli). As an example via cli (within Cephadm shell): > >>>>>>>> > >>>>>>>> # radosgw-admin pools list > >>>>>>>> > >>>>>>>> could not list placement set: (2) No such file or directory > >>>>>>>> > >>>>>>>> I have data in buckets, how can I get my RGWs to return online? > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> ceph-users mailing list -- ceph-users@ceph.io > >>>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io > >>>>>> > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- ceph-users@ceph.io > >>>>> To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users@ceph.io > >>> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io