Hi,

It would be good to have the full output. Does iostat show the backing
device performing I/O? Additionally, what does ceph -s show for cluster
state? Also, can you check the logs on that OSD, and see if anything looks
abnormal?

David

On Thu, Dec 2, 2021 at 1:20 PM Zach Heise (SSCC) <he...@ssc.wisc.edu> wrote:

> Good morning David,
>
> Assuming you need/want to see the data about the other 31 OSDs, 14 is
> showing:
> ID
> CLASS
> WEIGHT
> REWEIGHT
> SIZE
> RAW USE
> DATA
> OMAP
> META
> AVAIL
> %USE
> VAR
> PGS
> STATUS
> 14
> hdd
> 2.72899
> 0
> 0 B
> 0 B
> 0 B
> 0 B
> 0 B
> 0 B
> 0
> 0
> 1
> up
>
> Zach
>
> On 2021-12-01 5:20 PM, David Orman wrote:
>
> What's "ceph osd df" show?
>
> On Wed, Dec 1, 2021 at 2:20 PM Zach Heise (SSCC) <he...@ssc.wisc.edu>
> wrote:
>
>> I wanted to swap out on existing OSD, preserve the number, and then
>> remove the HDD that had it (osd.14 in this case) and give the ID of 14 to a
>> new SSD that would be taking its place in the same node. First time ever
>> doing this, so not sure what to expect.
>>
>> I followed the instructions here
>> <https://docs.ceph.com/en/latest/cephadm/services/osd/#remove-an-osd>,
>> using the --replace flag.
>>
>> However, I'm a bit concerned that the operation is taking so long in my
>> test cluster. Out of 70TB in the cluster, only 40GB were in use. This is a
>> relatively large OSD in comparison to others in the cluster (2.7TB versus
>> 300GB for most other OSDs) and yet it's been 36 hours with the following
>> status:
>>
>> ceph04.ssc.wisc.edu> ceph orch osd rm status
>> OSD_ID  HOST                 STATE     PG_COUNT  REPLACE  FORCE  
>> DRAIN_STARTED_AT
>> 14      ceph04.ssc.wisc.edu  draining  1         True     True   2021-11-30 
>> 15:22:23.469150+00:00
>>
>>
>> Another note: I don't know why it has the "force = true" set; the command
>> that I ran was just Ceph orch osd rm 14 --replace, without specifying
>> --force. Hopefully not a big deal but still strange.
>>
>> At this point is there any way to tell if it's still actually doing
>> something, or perhaps it is hung? if it is hung, what would be the
>> 'recommended' way to proceed? I know that I could just manually eject the
>> HDD from the chassis and run the "ceph osd crush remove osd.14" command and
>> then manually delete the auth keys, etc, but the documentation seems to
>> state that this shouldn't be necessary if a ceph OSD replacement goes
>> properly.
>>
>>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to