[ceph-users] Re: ceph orch upgrade stuck between 16.2.7 and 16.2.13

2023-08-15 Thread Eugen Block
Hi, literally minutes before your email popped up in my inbox I had announced that I would upgrade our cluster from 16.2.10 to 16.2.13 tomorrow. Now I'm hesitating. ;-) I guess I would start looking on the nodes where it failed to upgrade OSDs and check out the cephadm.log as well as syslog

[ceph-users] Re: ceph orch upgrade stuck between 16.2.7 and 16.2.13

2023-08-15 Thread Robert Sander
On 8/15/23 11:02, Eugen Block wrote: I guess I would start looking on the nodes where it failed to upgrade OSDs and check out the cephadm.log as well as syslog. Did you see progress messages in the mgr log for the successfully updated OSDs (or MON/MGR)? The issue is that there is no informatio

[ceph-users] Re: ceph orch upgrade stuck between 16.2.7 and 16.2.13

2023-08-15 Thread Robert Sander
On 8/15/23 11:16, Curt wrote: Probably not the issue, but do all your osd servers have internet access?  I've had a similar experience when one of our osd servers default gateway got changed, so it was just waiting to download and took a bit to timeout. Yes, all nodes can manually pull the im

[ceph-users] Re: ceph orch upgrade stuck between 16.2.7 and 16.2.13

2023-08-15 Thread Adam King
with the log to cluster level already on debug, if you do a "ceph mgr fail" what does cephadm log to the cluster before it reports sleeping? It should at least be doing something if it's responsive at all. Also, in "ceph orch ps" and "ceph orch device ls" are the REFRESHED columns reporting that t