[ceph-users] Removing failing OSD with cephadm?

[Matt Larson]

I have an OSD that is causing slow ops, and appears to be backed by a
failing drive according to smartctl outputs.  I am using cephadm, and
wondering what is the best way to remove this drive from the cluster and
proper steps to replace the disk?

Mark the osd.35 as out.

`sudo ceph osd out osd.35`

Then mark osd.35 as down.

`sudo ceph osd down osd.35`

 The OSD is marked as out, but it does come back up after a couple of
seconds.  I do not know if that is a problem or to just let the drive stay
online as long as it lasts during the removal from the cluster.

 After the recovery completes, I would then `destroy` the osd:

`ceph osd destroy {id} --yes-i-really-mean-it`

(https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/)

Besides checking steps above, my question now is ..* If the drive is acting
very slow and causing slow ops, should I be trying to shut down its OSD
and keep it down? There is an example to stop the OSD on the server using
systemctl, outside of cephadm:*

ssh {osd-host}sudo systemctl stop ceph-osd@{osd-num}


Thanks,
  Matt

-- 
Matt Larson, PhD
Madison, WI  53705 U.S.A.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW cannot list or create openidconnect providers

[mat]

Hi Pritha,

The caps were set correctly. I actually discovered that the SHA1 hash in my 
ThumbprintList was wrong. I had to attach the python debugger to the find the 
real-issue because boto3 seems to suppress the error returned from the iam api. 
The radosgw response is pretty explicit about the real issue. Replacing the 
thumbprint with a valid one fixed my problem.

I followed these instructions to get a valid thumbprint from my cert: 
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
 

V/r,
-= Mat
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW Service SSL HAProxy.cfg

[Jimmy Spets]

The config file for HAProxy is generated by Ceph and I think it should include 
"ssl verify none" on each backed line as the config use plain ip:port notation.
What I wonder is if my yaml config for the RGW and Ingress miss something or if 
it is a bug in the HAProxy config file generator.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

[Konstantin Shalygin]

> On 17 Feb 2023, at 23:20, Anthony D'Atri  wrote:
> 
> 
> 
>> * if rebalance will starts due EDAC or SFP degradation, is faster to fix the 
>> issue via DC engineers and put node back to work
> 
> A judicious mon_osd_down_out_subtree_limit setting can also do this by not 
> rebalancing when an entire node is detected down. 

Yes. But in this case when single disk dead, it's may be not actually dead, the 
examples:

* disk just stuck - reboot or/and physical inject_insert return in to live
* disk read errors - such errors lead to OSD down, but after OSD restart is 
just works normal (Pending Sectors -> Reallocates)

The fill of single 16TB OSD may be a 7-10 days. And it's may be fixed with 
10-20 minutes with duty engineer

> 
>> * noout prevents unwanted OSD's fills and the run out of space => outage of 
>> services
> 
> Do you run your clusters very full?

We provide public services. This means client can rent 1000 disks x 1000GB via 
one terraform command, at 02:00 Saturday night. Just physically impossible to 
add nodes at this case. Any movement without upmap is highly undesirable



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

[Konstantin Shalygin]

> On 17 Feb 2023, at 23:20, Anthony D'Atri  wrote:
> 
> 
> 
>> * if rebalance will starts due EDAC or SFP degradation, is faster to fix the 
>> issue via DC engineers and put node back to work
> 
> A judicious mon_osd_down_out_subtree_limit setting can also do this by not 
> rebalancing when an entire node is detected down. 

Yes. But in this case when single disk dead, it's may be not actually dead, the 
examples:

* disk just stuck - reboot or/and physical inject_insert return in to live
* disk read errors - such errors lead to OSD down, but after OSD restart is 
just works normal (Pending Sectors -> Reallocates)

The fill of single 16TB OSD may be a 7-10 days. And it's may be fixed with 
10-20 minutes with duty engineer

> 
>> * noout prevents unwanted OSD's fills and the run out of space => outage of 
>> services
> 
> Do you run your clusters very full?

We provide public services. This means client can rent 1000 disks x 1000GB via 
one terraform command, at 02:00 Saturday night. Just physically impossible to 
add nodes at this case. Any movement without upmap is highly undesirable



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Next quincy release (17.2.6)

[Laura Flores]

And make sure the PR is passing all required checks and approved.

On Fri, Feb 17, 2023 at 10:25 AM Yuri Weinstein  wrote:

> Hello
>
> We are planning to start QE validation release next week.
> If you have PRs that are to be part of it, please let us know by
> adding "needs-qa" for 'quincy' milestone ASAP.
>
> Thx
> YuriW
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
> --

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage

Red Hat Inc. 

Chicago, IL

lflo...@redhat.com
M: +17087388804
@RedHat    Red Hat
  Red Hat


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Next quincy release (17.2.6)

[Yuri Weinstein]

Hello

We are planning to start QE validation release next week.
If you have PRs that are to be part of it, please let us know by
adding "needs-qa" for 'quincy' milestone ASAP.

Thx
YuriW
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph noout vs ceph norebalance, which is better for minor maintenance

[Anthony D'Atri]

> * if rebalance will starts due EDAC or SFP degradation, is faster to fix the 
> issue via DC engineers and put node back to work

A judicious mon_osd_down_out_subtree_limit setting can also do this by not 
rebalancing when an entire node is detected down. 

> * noout prevents unwanted OSD's fills and the run out of space => outage of 
> services

Do you run your clusters very full?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] bluefs_db_type

[Stolte, Felix]

Hey guys,

most of my osds have HDD for block and SSD for db. But according to "ceph osd 
metadata"  bluefs_db_type = hdd and bluefs_db_rotational = 1.

lsblk -o name, rota reveals the following (sdb is db device for 3 hdds):

sdb 
 0
├─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--b164ba4c--48c9--41a0--8b5e--ae3a6a23a22c
1
├─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--1c7aa9a1--791d--4ed6--8049--9fba8d5ac828
1
└─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--ec92f9c6--d651--46ed--b6cd--4cf37c8ce284
1

I am pretty sure sdb had rota 0 during osd deployment (still has, but lvm 
volumes don’t).

Question 1:
Is the output of osd metadata for bluefs_db_type and bluefs_db_rotational 
relevant for how the osd process is treating the disks? (or does it just 
reflect the value of /sys/block//queue/rotational?)

Question 2:
How can I verify that the osd process is treating the db device as an ssd? If i 
remember correctly, the osd process is using different parameters if db is on 
SSD instead of HDD.


Best regards
Felix


-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io