[ceph-users] Re: Ceph osd df tree takes a long time to respond
Thanks, I just found out the issue https://tracker.ceph.com/issues/43317 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: wrong public_ip after blackout / poweroutage
Hi,
this only a theory, not a proven answer or something. But the
orchestrator does automatically reconfigure daemons depending on the
circumstances. So my theory is, some of the OSD nodes didn't respond
via public network anymore, so ceph tried to use the cluster network
as a fallback. The other way around is more common: if you don't have
a cluster network configured at all, you see logs stating "falling
back to public interface" (or similar). If the orchestrator did
reconfigure the daemons, it would have been logged in the active mgr.
And the result would be a different ceph.conf for the daemons in
/var/lib/ceph/{FSID}/osd.{OSD_ID}/config. If you still have the mgr
logs from after the outage you might find some clues.
Regards,
Eugen
Zitat von mailing-lists :
Dear Cephers,
after a succession of unfortunate events, we have suffered a
complete datacenter blackout today.
Ceph _nearly_ perfectly came back up. The Health was OK and all
services were online, but we were having weird problems. Weird as
in, we could sometimes map rbds and sometimes not, and sometimes we
could use the cephfs and sometimes we could not...
Turns out, some osds (id say 5%) came back with the cluster_ip
address as their public_ip and thus were not reachable.
I do not see any pattern, why some osds are faulty and others are
not. The fault is spread over nearly all nodes. This is an example:
osd.45 up in weight 1 up_from 184143 up_thru 184164 down_at
184142 last_clean_interval [182655,184103)
[v2:192.168.222.20:6834/1536394698,v1:192.168.222.20:6842/1536394698]
[v2:192.168.222.20:6848/1536394698,v1:192.168.222.20:6853/1536394698]
exists,up 002326c9
This should have a public_ip in the first brackets []. Our
cluster-network is 192.168.222.0/24, which is of course only
available on the ceph internal switch.
Simply restarting the osds that were affected solved this problem...
So I am not really asking for your help troubleshooting this; I
would just like to understand if there is a reasonable explanation.
My guess would be some kind of race-condition when the interfaces
came up, but then again, why on ~5% of all osds? ... Anyways im
tired, I hope that this mail is somewhat understandable.
We are running Ceph 17.2.7 with cephadm docker deployment.
If you have any ideas for the cause of this, please let me know. I
have not seen this issue when I'm gracefully rebooting the nodes.
Best
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Full list of metrics provided by ceph exporter daemon
>
> That's not the answer I expected to see :( Are you suggesting enabling all
> possible daemons of Ceph just to obtain a full list of metrics?
Depends if what you're after is potential metrics or ones currently surfaced, I
assumed you meant the latter.
> Isn't there any list in ceph repo or in documentation with all metrics
> exposed?
Use the source, Luke? I don't know of a documented list.
> Another question: ceph_rgw_qactive metric changed its instance_id from
> numeric format to a letter format. Is there any pull request for that in
> Ceph or it is Rook initiative? Example:
I believe recently there was a restructuring of some of the RGW metrics.
>
> metrics from prometheus module:
> ceph_rgw_qactive{instance_id="4134960"} 0.0
> ceph_rgw_qactive{instance_id="4493203"} 0.0
>
> metrics from ceph-exporter:
> ceph_rgw_qactive{instance_id="a"} 0 ceph_rgw_qactive{instance_id="a"} 0
>
>
> чт, 20 июн. 2024 г. в 20:09, Anthony D'Atri :
>
>> curl http://endpoint:port/metrics
>>
>>> On Jun 20, 2024, at 10:15, Peter Razumovsky
>> wrote:
>>>
>>> Hello!
>>>
>>> I'm using Ceph Reef with Rook v1.13 and want to find somewhere a full
>> list
>>> of metrics exported by brand new ceph exporter daemon. We found that some
>>> metrics have been changed after moving from prometheus module metrics to
>> a
>>> separate daemon.
>>>
>>> We used described method [1] to give us a time to mitigate all risks
>> during
>>> transfer to a new daemon. Now we want to obtain a full list of
>>> ceph-exporter metrics to compare old and new metrics and to mitigate
>>> potential risks of losing monitoring data and breaking alerting systems.
>> We
>>> found some examples of metrics [2] but it is not complete so we will
>>> appreciate it if someone points us to the full list.
>>>
>>> [1]
>>>
>> https://docs.ceph.com/en/latest/mgr/prometheus/#ceph-daemon-performance-counters-metrics
>>>
>>> [2] https://docs.ceph.com/en/latest/monitoring/
>>>
>>> --
>>> Best regards,
>>> Peter Razumovsky
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
>
> --
> Best regards,
> Peter Razumovsky
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Full list of metrics provided by ceph exporter daemon
That's not the answer I expected to see :( Are you suggesting enabling all
possible daemons of Ceph just to obtain a full list of metrics? Isn't there
any list in ceph repo or in documentation with all metrics exposed?
Another question: ceph_rgw_qactive metric changed its instance_id from
numeric format to a letter format. Is there any pull request for that in
Ceph or it is Rook initiative? Example:
metrics from prometheus module:
ceph_rgw_qactive{instance_id="4134960"} 0.0
ceph_rgw_qactive{instance_id="4493203"} 0.0
metrics from ceph-exporter:
ceph_rgw_qactive{instance_id="a"} 0 ceph_rgw_qactive{instance_id="a"} 0
чт, 20 июн. 2024 г. в 20:09, Anthony D'Atri :
> curl http://endpoint:port/metrics
>
> > On Jun 20, 2024, at 10:15, Peter Razumovsky
> wrote:
> >
> > Hello!
> >
> > I'm using Ceph Reef with Rook v1.13 and want to find somewhere a full
> list
> > of metrics exported by brand new ceph exporter daemon. We found that some
> > metrics have been changed after moving from prometheus module metrics to
> a
> > separate daemon.
> >
> > We used described method [1] to give us a time to mitigate all risks
> during
> > transfer to a new daemon. Now we want to obtain a full list of
> > ceph-exporter metrics to compare old and new metrics and to mitigate
> > potential risks of losing monitoring data and breaking alerting systems.
> We
> > found some examples of metrics [2] but it is not complete so we will
> > appreciate it if someone points us to the full list.
> >
> > [1]
> >
> https://docs.ceph.com/en/latest/mgr/prometheus/#ceph-daemon-performance-counters-metrics
> >
> > [2] https://docs.ceph.com/en/latest/monitoring/
> >
> > --
> > Best regards,
> > Peter Razumovsky
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
--
Best regards,
Peter Razumovsky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Full list of metrics provided by ceph exporter daemon
curl http://endpoint:port/metrics > On Jun 20, 2024, at 10:15, Peter Razumovsky wrote: > > Hello! > > I'm using Ceph Reef with Rook v1.13 and want to find somewhere a full list > of metrics exported by brand new ceph exporter daemon. We found that some > metrics have been changed after moving from prometheus module metrics to a > separate daemon. > > We used described method [1] to give us a time to mitigate all risks during > transfer to a new daemon. Now we want to obtain a full list of > ceph-exporter metrics to compare old and new metrics and to mitigate > potential risks of losing monitoring data and breaking alerting systems. We > found some examples of metrics [2] but it is not complete so we will > appreciate it if someone points us to the full list. > > [1] > https://docs.ceph.com/en/latest/mgr/prometheus/#ceph-daemon-performance-counters-metrics > > [2] https://docs.ceph.com/en/latest/monitoring/ > > -- > Best regards, > Peter Razumovsky > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Daily slow ops around the same time on different osds
Hi, around the same time sounds like deep-scrubbing. Did you verify if those OSDs from the mentioned pool were scrubbed around that time? The primary OSDs for the PGs would log that. Is that pool that heavily used? Or could there be one failing disk? Zitat von "Szabo, Istvan (Agoda)" : Hi, For some reason daily I receive a slow ops which affect the rgw.log pool the most (which is pretty small, 32pg / 57k objects / 9.5GB data) Could be the issue related to rocksdb tasks? Some log lines: ... 2024-06-18T14:17:43.849+0700 7fd8002ea700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fd8002ea700' had timed out after 15 2024-06-18T14:17:50.865+0700 7fd81a31e700 -1 osd.461 472771 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.3349483990.0:462716190 22.11 22:8deed178:::meta.log.08e85e9f-c16e-43f0-b88d-362f3b7ced2d.15:head [call log.list in=69b] snapc 0=[] ondisk+read+known_if_redirected e472771) 2024-06-18T14:17:50.865+0700 7fd81a31e700 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'dc.rgw.log' : 1 ]) 2024-06-18T14:19:36.040+0700 7fd81e5b4700 1 heartbeat_map is_healthy 'OSD::osd_op_tp ... 2024-06-18T14:19:51.908+0700 7fd81a31e700 -1 osd.461 472771 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.3371539359.0:5095510 22.11 22:8df1b9e1:::datalog.sync-status.shard.61c9d940-fde4-4bed-9389-edc8d7741817.111: head [call lock.lock in=64b] snapc 0=[] ondisk+write+known_if_redirected e472771) 2024-06-18T14:19:51.908+0700 7fd81a31e700 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'delayed' : 2 ] most affected pool [ 'dc.rgw.log' : 2 ]) 2024-06-18T14:19:52.899+0700 7fd81a31e700 -1 osd.461 472771 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.3371539359.0:5095510 22.11 22:8df1b9e1:::datalog.sync-status.shard.61c9d940-fde4-4bed-9389-edc8d7741817.111: I found this thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/FR5V466HBXGRVL3Z3RAFKUPQ2FGK2T53/ but I think it is not relevant for me. We use only SSDs without separated rocksdb so we don' have spillover. CPU is not overloaded (80% idle) however it's interesting that in vmstat the "r" is pretty big which indicates waiting list for cpu: procs ---memory-- ---swap-- -io -system-- --cpu- r b swpd free buff cache si sobibo in cs us sy id wa st 12 0 0 6205288 329803136 1563384400 2149 18940 0 5 3 92 0 0 17 0 0 6156780 329827840 1563725200 125584 250416 646098 1965051 10 8 82 0 0 21 0 0 6154636 329849504 1563611200 99320 245024 493324 1682377 7 6 87 0 0 16 0 0 6144256 329869664 1563692400 87484 301968 623345 1993374 8 7 84 0 0 19 0 0 6057012 329890080 1563793200 160444 303664 549194 1820562 8 6 85 0 0 Any idea what this could indicate on octopus 15.2.17 with ubuntu 20.04? ty This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Replacing SSD disk(metadata, rocksdb) which are associated with HDDs(osd block)
Hi, - Is it common practice to configure SSDs with block.db and associate them with five HDD disks to store OSD blocks when using eight SSDs and forty HDDs? yes, it is common practice. Or would it be better to only store the rgw index on SSDs? I am also curious about the difference in performance between these configurations. With performance the response is usually "it depends". If performance is an issue, why not go all flash? - If SSDs are configured with block.db as described above, will it be necessary to reinstall the five associated OSDs (HDDs) if one SSD fails ? Yes, most likely all OSDs depending on that single SSD will have to be redeployed. There's a chance you might be able to rescue some of the rocksDB LVs (pvmove) Regards, Eugen Zitat von TaekSoo Lim : Hi all, I have configured a ceph cluster to be used as an object store with a combination of SSDs and HDDs, where the block.db is stored on LVM on SSDs and the OSD block is stored on HDDs. I have set up one SSD for storing metadata (rocksDB), and five HDDs are associated with it to store the OSD blocks. During a disk replacement test, if one SSD that stores the block.db fails, all five associated OSDs go down. When replacing and recovering the failed SSD, it seems necessary to reconfigure the five OSDs, which takes too long to recover data and has significant performance impact. So here are two questions: - Is it common practice to configure SSDs with block.db and associate them with five HDD disks to store OSD blocks when using eight SSDs and forty HDDs? Or would it be better to only store the rgw index on SSDs? I am also curious about the difference in performance between these configurations. - If SSDs are configured with block.db as described above, will it be necessary to reinstall the five associated OSDs (HDDs) if one SSD fails ? Also, is there a way to reconstruct the metadata on the newly replaced SSD from the remaining five intact HDDs? As a novice in ceph clusters, I seek advice from experienced users. Thank you. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Full list of metrics provided by ceph exporter daemon
Hello! I'm using Ceph Reef with Rook v1.13 and want to find somewhere a full list of metrics exported by brand new ceph exporter daemon. We found that some metrics have been changed after moving from prometheus module metrics to a separate daemon. We used described method [1] to give us a time to mitigate all risks during transfer to a new daemon. Now we want to obtain a full list of ceph-exporter metrics to compare old and new metrics and to mitigate potential risks of losing monitoring data and breaking alerting systems. We found some examples of metrics [2] but it is not complete so we will appreciate it if someone points us to the full list. [1] https://docs.ceph.com/en/latest/mgr/prometheus/#ceph-daemon-performance-counters-metrics [2] https://docs.ceph.com/en/latest/monitoring/ -- Best regards, Peter Razumovsky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
