[ceph-users] osd removal leaves 'stray daemon'
Hello, I have a question about osd removal/replacement: I just removed an osd where the disk was still running but had read errors, leading to failed deep scrubs - as the intent is to replace this as soon as we manage to get a spare I removed it with the '--replace' flag: # ceph orch osd rm 224 --replace After all placement groups are evacuated I now have 1 osd down/out and showing as 'destroyed': # ceph osd tree ID CLASS WEIGHT TYPE NAMESTATUS REWEIGHT PRI-AFF (...) 214hdd14.55269 osd.214 up 1.0 1.0 224hdd14.55269 osd.224 destroyed 0 1.0 234hdd14.55269 osd.234 up 1.0 1.0 (...) All as expected - but now the health check complains that the (destroyed) osd is not managed: # ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.224 on host ceph19 not managed by cephadm Is this expected behaviour and I have to live with the yellow check until we get a replacement disk and recreate the osd or did something not finish correctly? Regards, Holger -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naund...@rz.uni-kiel.de ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: osd removal leaves 'stray daemon'
Hello,
a mgr failover did not change the situation - the osd still shows up in
the 'ceph node ls' - I assume that this is more or less 'working as
intended' as I did ask for the OSD to be kept in the CRUSH map to be
replacd later - but as we are still not so experienced with Ceph here I
wanted to get some input from other sites.
Regards,
Holger
On 30.11.22 16:28, Adam King wrote:
I typically don't see this when I do OSD replacement. If you do a mgr
failover ("ceph mgr fail") and wait a few minutes does this still show
up? The stray daemon/host warning is roughly equivalent to comparing the
daemons in `ceph node ls` and `ceph orch ps` and seeing if there's
anything in the former but not the latter. Sometimes I have seen the mgr
will have some out of data info in the node ls and a failover will
refresh it.
On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <mailto:naund...@rz.uni-kiel.de>> wrote:
Hello,
I have a question about osd removal/replacement:
I just removed an osd where the disk was still running but had read
errors, leading to failed deep scrubs - as the intent is to replace
this
as soon as we manage to get a spare I removed it with the
'--replace' flag:
# ceph orch osd rm 224 --replace
After all placement groups are evacuated I now have 1 osd down/out
and showing as 'destroyed':
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
(...)
214 hdd 14.55269 osd.214 up 1.0 1.0
224 hdd 14.55269 osd.224 destroyed 0 1.0
234 hdd 14.55269 osd.234 up 1.0 1.0
(...)
All as expected - but now the health check complains that the
(destroyed) osd is not managed:
# ceph health detail
HEALTH_WARN 1 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm
stray daemon osd.224 on host ceph19 not managed by cephadm
Is this expected behaviour and I have to live with the yellow check
until we get a replacement disk and recreate the osd or did something
not finish correctly?
Regards,
Holger
--
Dr. Holger Naundorf
Christian-Albrechts-Universität zu Kiel
Rechenzentrum / HPC / Server und Storage
Tel: +49 431 880-1990
Fax: +49 431 880-1523
naund...@rz.uni-kiel.de <mailto:naund...@rz.uni-kiel.de>
___
ceph-users mailing list -- ceph-users@ceph.io
<mailto:ceph-users@ceph.io>
To unsubscribe send an email to ceph-users-le...@ceph.io
<mailto:ceph-users-le...@ceph.io>
--
Dr. Holger Naundorf
Christian-Albrechts-Universität zu Kiel
Rechenzentrum / HPC / Server und Storage
Tel: +49 431 880-1990
Fax: +49 431 880-1523
naund...@rz.uni-kiel.de
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] cephfs snap-mirror stalled
Hello, we have set up a snap-mirror for a directory on one of our clusters - running ceph version ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) to get mirrorred our other cluster - running ceph version ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable) The initial setup went ok, when the first snapshot was created data started to flow at a decent (for our HW) rate of 100-200MB/s. As the directory contains ~200TB this was expected to take some time - but now the process has stalled completely after ~100TB were mirrored and ~7d running. Up to now I do not have any hints why it has stopped - I do not see any error messages from the cephfs-mirror daemon. Can the small version mismatch be a problem? Any hints where to look to find out what has got stuck are welcome. Regards, Holger -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naund...@rz.uni-kiel.de ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: cephfs snap-mirror stalled
On 06.12.22 09:54, Venky Shankar wrote: Hi Holger, On Tue, Dec 6, 2022 at 1:42 PM Holger Naundorf wrote: Hello, we have set up a snap-mirror for a directory on one of our clusters - running ceph version ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) to get mirrorred our other cluster - running ceph version ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable) The initial setup went ok, when the first snapshot was created data started to flow at a decent (for our HW) rate of 100-200MB/s. As the directory contains ~200TB this was expected to take some time - but now the process has stalled completely after ~100TB were mirrored and ~7d running. Up to now I do not have any hints why it has stopped - I do not see any error messages from the cephfs-mirror daemon. Can the small version mismatch be a problem? Any hints where to look to find out what has got stuck are welcome. I'd look at the mirror daemon logs for any errors to start with. You might want to crank up the log level for debugging (debug cephfs_mirror=20). Even on max debug I do not see anything which looks like an error - but as this is the first time I try to dig into any cephfs-mirror logs I might not notice (as long as it is not red and flashing). The Log basically this type of sequence, repeating forever: (...) cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::Mirror update_fs_mirrors cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror update (0x556fe3a7f130) after 2 seconds cephfs::mirror::Watcher handle_notify: notify_id=751516198184655, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) run: trying to pick from 1 directories cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) pick_directory cephfs::mirror::Watcher handle_notify: notify_id=751516198184656, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::Mirror update_fs_mirrors cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror update (0x556fe3a7fc70) after 2 seconds cephfs::mirror::Watcher handle_notify: notify_id=751516198184657, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify (...) Regards, Holger -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naund...@rz.uni-kiel.de ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naund...@rz.uni-kiel.de ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: cephfs snap-mirror stalled
On 06.12.22 14:17, Venky Shankar wrote: On Tue, Dec 6, 2022 at 6:34 PM Holger Naundorf wrote: On 06.12.22 09:54, Venky Shankar wrote: Hi Holger, On Tue, Dec 6, 2022 at 1:42 PM Holger Naundorf wrote: Hello, we have set up a snap-mirror for a directory on one of our clusters - running ceph version ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) to get mirrorred our other cluster - running ceph version ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable) The initial setup went ok, when the first snapshot was created data started to flow at a decent (for our HW) rate of 100-200MB/s. As the directory contains ~200TB this was expected to take some time - but now the process has stalled completely after ~100TB were mirrored and ~7d running. Up to now I do not have any hints why it has stopped - I do not see any error messages from the cephfs-mirror daemon. Can the small version mismatch be a problem? Any hints where to look to find out what has got stuck are welcome. I'd look at the mirror daemon logs for any errors to start with. You might want to crank up the log level for debugging (debug cephfs_mirror=20). Even on max debug I do not see anything which looks like an error - but as this is the first time I try to dig into any cephfs-mirror logs I might not notice (as long as it is not red and flashing). The Log basically this type of sequence, repeating forever: (...) cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::Mirror update_fs_mirrors cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror update (0x556fe3a7f130) after 2 seconds cephfs::mirror::Watcher handle_notify: notify_id=751516198184655, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) run: trying to pick from 1 directories cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) pick_directory cephfs::mirror::Watcher handle_notify: notify_id=751516198184656, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::Mirror update_fs_mirrors cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror update (0x556fe3a7fc70) after 2 seconds cephfs::mirror::Watcher handle_notify: notify_id=751516198184657, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify (...) Basically, the interesting bit is not captured since it probably happened sometime back. Could you please set the following: debug cephfs_mirror = 20 debug client = 20 and restart the mirror daemon? The daemon would start synchronizing again. When synchronizing stalls, please share the daemon logs. If the log is huge, you could upload them via ceph-post-file. If I set debug_client to 20 'huge' is an understatement. I now have three huge logfiles - one pair with debug_mirror set to 20 capturing the restart and the point where the sync stalls again and one with both mirror and client debug at 20 capturing the restart - but as this setting created ~10GB logs within 20min I reset the client logging again to spare our small system disks - if these logs are needed I think I will have to set up some remote logging facility. The observation I made from the scanning the logs: After the restart the mirror daemon spends some hours comparing the incomplete transfers (Lots of limes with do_synchronize: 18 entries in stack do_synchronize: top of stack path=./...(FILENAME)... do_synchronize: entry=./(FILENAME), data_sync=0, attr_sync=1 then there is the point where the number of items in the stack goes down: cephfs::mirror::PeerReplayer () do_synchronize: 8 entries in stack The top of stack moves up in the diretory levesl: cephfs::mirror::PeerReplayer () do_synchronize: top of stack path=./...FILENAME... but then it just stops, without any error message visible in the logfile and switches to the repeating sequence I posted already. Should I try to upload the logs - even gzipped they are quite huge: 388M syslog.restart-mirror-client_debug_20.gz 98M syslog.restart-mirror-no_client_debug.gz 54M syslog.stalled-no_client_debug.gz (as our servers are in an isolated net I will have to see if the 'ceph-post-file' method works from another system. Regards, Holger Naundorf Regards, Holger -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naund...@rz.uni-kiel.de ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naund...@rz.uni-kiel.de -- Dr. Holger Naundorf Christian-Albr
[ceph-users] Re: cephfs snap-mirror stalled
On 15.12.22 14:06, Venky Shankar wrote: Hi Holger, (sorry for the late reply) On Fri, Dec 9, 2022 at 6:22 PM Holger Naundorf wrote: As an update: After the third restart now the mirror-daemon is running normal again - only change to the restarts before was that during the restart dbug_client was set to 20. (First restart was after ~48h of no data movemnent to the receiver side and no changes, 2nd was with debug_mirror set to 20 and the thisrd - maybe to quickly - after only ~12h of no data movement with debug_mirror + _client set to 20). Currently I reset the debug levels back down to reduce the load on the system disks. Do you see ESTALE related messages in mirror daemon logs (with debug_client: 20)? We have run into a couple of instances where the mirror daemon would be stuck on a directory entry. The workaround for that is to find the directory path (where the daemon is stuck) and from another client (mount), list the entries in that directory (or at times restarting the daemon works, as in your case). Sounds like our bug - I do have the ESTALE messages with debug 20: syslog.restart-mirror-client_debug_20.gz:Dec 7 10:05:30 ceph07 bash[1740788]: debug 2022-12-07T09:05:30.299+ 7f05907a7700 20 client.29690499 got ESTALE on tid 2258671 from mds.0 syslog.restart-mirror-client_debug_20.gz:Dec 7 10:05:30 ceph07 bash[1740788]: debug 2022-12-07T09:05:30.299+ 7f05907a7700 20 client.29690499 got ESTALE on tid 2258671 from mds.0 syslog.restart-mirror-client_debug_20.gz:Dec 7 10:05:30 ceph07 bash[1740788]: debug 2022-12-07T09:05:30.299+ 7f05907a7700 20 client.29690499 got ESTALE on tid 2258671 from mds.0 (...) And when the mirror got stuck again after some time it mysteriously (at that time) started to work again while I was poking around in the logfiles and the system - and doing an 'ls' in the stuck dir was definitly included in the poking. This will be fixed in the next pacific release (tracker: https://tracker.ceph.com/issues/55935). Next means the .11 release or the current .10 we have not yet upgraded to? Thanks for the update. Regards, Holger Naundorf Regards, Holger Naundorf On 07.12.22 15:53, Holger Naundorf wrote: On 06.12.22 14:17, Venky Shankar wrote: On Tue, Dec 6, 2022 at 6:34 PM Holger Naundorf wrote: On 06.12.22 09:54, Venky Shankar wrote: Hi Holger, On Tue, Dec 6, 2022 at 1:42 PM Holger Naundorf wrote: Hello, we have set up a snap-mirror for a directory on one of our clusters - running ceph version ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) to get mirrorred our other cluster - running ceph version ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable) The initial setup went ok, when the first snapshot was created data started to flow at a decent (for our HW) rate of 100-200MB/s. As the directory contains ~200TB this was expected to take some time - but now the process has stalled completely after ~100TB were mirrored and ~7d running. Up to now I do not have any hints why it has stopped - I do not see any error messages from the cephfs-mirror daemon. Can the small version mismatch be a problem? Any hints where to look to find out what has got stuck are welcome. I'd look at the mirror daemon logs for any errors to start with. You might want to crank up the log level for debugging (debug cephfs_mirror=20). Even on max debug I do not see anything which looks like an error - but as this is the first time I try to dig into any cephfs-mirror logs I might not notice (as long as it is not red and flashing). The Log basically this type of sequence, repeating forever: (...) cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::Mirror update_fs_mirrors cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror update (0x556fe3a7f130) after 2 seconds cephfs::mirror::Watcher handle_notify: notify_id=751516198184655, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) run: trying to pick from 1 directories cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) pick_directory cephfs::mirror::Watcher handle_notify: notify_id=751516198184656, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify cephfs::mirror::Mirror update_fs_mirrors cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror update (0x556fe3a7fc70) after 2 seconds cephfs::mirror::Watcher handle_notify: notify_id=751516198184657, handle=93939050205568, notifier_id=25504530 cephfs::mirror::MirrorWatcher handle_notify (...) Basically, the interesting bit is not captured since it probably happened sometime back. Could you please set the following: debug cephfs_mirror = 20 debug client = 20 and restart the mirror daemon? The daemon would start synchronizing again. When synchronizing stalls, please share the daem
