Yes, these 8 PGs have been in this 'remapped' state for quite awhile. I don't know why CRUSH has not seen fit to designate new OSDs for them so that acting and up match.

For the error in question - ceph upgrade is saying that only 1 PG would become offline if OSD(s) were stopped. So if these 8 pgs were causing the problem, I thought it would tell me specifically.

Greg, is there a way I could check if crush is failing to map properly and figure out why? Because HEALTH_OK is shown even with these 8 active+clean+remapped PGs I assumed it was normal/okay for it to be in this state.

PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN
7.11 42 0 0 42 0 1.76E+08 0 0 631 631 active+clean+remapped 2022-02-10T05:16:21.791091+0000 8564'631 10088:11277 [15,7] 15 [15,7,11] 15 8564'631 2022-02-10T05:16:21.791028+0000 8564'631 2022-02-08T17:38:26.806576+0000 0
9.17 23 0 0 23 0 88554155 0 0 2700 2700 active+clean+remapped 2022-02-09T22:40:19.229658+0000 9668'2700 10088:15778 [22,9] 22 [22,9,2] 22 9668'2700 2022-02-09T22:40:19.229581+0000 9668'2700 2022-02-06T13:09:04.264912+0000 0
11.10 3 0 0 3 0 9752576 0 0 6323 6323 active+clean+remapped 2022-02-10T16:56:10.410048+0000 6255'6323 10088:23237 [0,19] 0 [0,19,2] 0 6255'6323 2022-02-10T16:56:10.409954+0000 6255'6323 2022-02-05T18:08:35.490642+0000 0
11.19 2 0 0 2 0 4194304 0 0 10008 10008 active+clean+remapped 2022-02-09T21:52:33.190075+0000 9862'14908 10088:38973 [19,9] 19 [19,9,12] 19 9862'14908 2022-02-09T21:52:33.190002+0000 8852'14906 2022-02-04T21:34:27.141103+0000 0
11.1a 2 0 0 2 0 4194323 0 0 8522 8522 active+clean+remapped 2022-02-10T10:08:29.451623+0000 5721'8522 10088:29920 [12,24] 12 [12,24,28] 12 5721'8522 2022-02-10T10:08:29.451543+0000 5721'8522 2022-02-09T04:45:34.096178+0000 0
7.1a 67 0 0 67 0 2.81E+08 0 0 1040 1040 active+clean+remapped 2022-02-09T18:39:53.571433+0000 8537'1040 10088:13580 [20,3] 20 [20,3,28] 20 8537'1040 2022-02-09T18:39:53.571328+0000 8537'1040 2022-02-09T18:39:53.571328+0000 0
7.e 63 0 0 63 0 2.6E+08 0 0 591 591 active+clean+remapped 2022-02-10T11:40:11.560673+0000 8442'591 10088:11607 [25,3] 25 [25,3,33] 25 8442'591 2022-02-10T11:40:11.560567+0000 8442'591 2022-02-10T11:40:11.560567+0000 0
9.d 29 0 0 29 0 1.17E+08 0 0 2448 2448 active+clean+remapped 2022-02-10T14:22:42.203264+0000 9784'2448 10088:16349 [22,2] 22 [22,2,8] 22 9784'2448 2022-02-10T14:22:42.203183+0000 9784'2448 2022-02-06T15:38:36.389808+0000 0

Zach 


On 2022-02-10 1:43 PM, gfar...@redhat.com wrote:
?Up? is the set of OSDs which are alive from the calculated crush mapping. ?Acting? includes those extras which have been added in to bring the PG up to proper size. So the PG does have 3 live OSDs serving it.

But perhaps the safety check *is* looking at up instead of acting? That seems like a plausible bug. (Also, if crush is failing to map properly, that?s not a great sign for your cluster health or design.)

On Thu, Feb 10, 2022 at 11:26 AM ? ?? <huw...@outlook.com> wrote:
I believe this is the reason.

I mean number of OSDs in the ?up? set should be at least 1 greater than the min_size for the upgrade to proceed. Or once any OSD is stopped, it can drop below min_size, and prevent the pg from becoming active. So just cleanup the misplaced and the upgrade should proceed automatically.

But I?m a little confused. I think if you have only 2 up OSD in a replicate x3 pool, it should in degraded state, and should give you a HEALTH_WARN.

? 2022?2?11??03:06?Zach Heise (SSCC) <he...@ssc.wisc.edu> ???

?

Hi Weiwen, thanks for replying.

All of my replicated pools, including the newest ssdpool I made most recently, have a min_size of 2. My other two EC pools have a min_size of 3.

Looking at pg dump output again, it does look like the two EC pools have exactly 4 OSDs listed in the "Acting" column, and everything else has 3 OSDs in Acting. So that's as it should be, I believe?

I do have some 'misplaced' objects on 8 different PGs (the active+clean+remapped ones listed in my original ceph -s output), that only have 2 "up" OSDs listed, but in the "Acting" columns each have 3 OSDs as they should. Apparently these 231 misplaced objects aren't enough to cause ceph to drop out of HEALTH_OK status.

Zach


On 2022-02-10 12:41 PM, huw...@outlook.com<mailto:huw...@outlook.com> wrote:

Hi Zach,

How about your min_size setting? Have you checked the number of OSDs in the acting set of every PG is at least 1 greater than the min_size of the corresponding pool?

Weiwen Hu



? 2022?2?10??05:02?Zach Heise (SSCC) <he...@ssc.wisc.edu><mailto:he...@ssc.wisc.edu> ???

?Hello,

ceph health detail says my 5-node cluster is healthy, yet when I ran ceph orch upgrade start --ceph-version 16.2.7 everything seemed to go fine until we got to the OSD section, now for the past hour, every 15 seconds a new log entry of  'Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline)' appears in the logs.

ceph pg dump_stuck (unclean, degraded, etc) shows "ok" for everything too. Yet somehow 1 PG is (apparently) holding up all the OSD upgrades and not letting the process finish. Should I stop the upgrade and try it again? (I haven't done that before so was just nervous to try it). Any other ideas?

 cluster:
   id:     9aa000e8-b999-11eb-82f2-ecf4bbcc0ac0
   health: HEALTH_OK
  services:
   mon: 4 daemons, quorum ceph05,ceph04,ceph01,ceph03 (age 92m)
   mgr: ceph03.futetp(active, since 97m), standbys: ceph01.fblojp
   mds: 1/1 daemons up, 1 hot standby
   osd: 33 osds: 33 up (since 2h), 33 in (since 4h); 9 remapped pgs
  data:
   volumes: 1/1 healthy
   pools:   7 pools, 193 pgs
   objects: 3.72k objects, 14 GiB
   usage:   43 GiB used, 64 TiB / 64 TiB avail
   pgs:     231/11170 objects misplaced (2.068%)
            185 active+clean
            8   active+clean+remapped
  io:
   client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
  progress:
   Upgrade to 16.2.7 (5m)
     [=====.......................] (remaining: 24m)

--
Zach

_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to