Hi Paul, To do better to answer you question, I'm following: http://docs.ceph.com/docs/nautilus/releases/nautilus/
At step 6, upgrade OSDs, I jumped on an OSD host and did a full 'yum update' for patching the host and rebooted to pick up the current centos kernel. I didn't do anything to specific commands for just updating the ceph RPMs in this process. peter Peter Eisch Senior Site Reliability Engineer T1.612.659.3228 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.59 From: Paul Emmerich <paul.emmer...@croit.io> Date: Wednesday, July 24, 2019 at 1:39 PM To: Peter Eisch <peter.ei...@virginpulse.com> Cc: Xavier Trilla <xavier.tri...@clouding.io>, "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Upgrading and lost OSDs On Wed, Jul 24, 2019 at 8:36 PM Peter Eisch <mailto:peter.ei...@virginpulse.com> wrote: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.7T 0 disk ├─sda1 8:1 0 100M 0 part ├─sda2 8:2 0 1.7T 0 part └─sda5 8:5 0 10M 0 part sdb 8:16 0 1.7T 0 disk ├─sdb1 8:17 0 100M 0 part ├─sdb2 8:18 0 1.7T 0 part └─sdb5 8:21 0 10M 0 part sdc 8:32 0 1.7T 0 disk ├─sdc1 8:33 0 100M 0 part That's ceph-disk which was removed, run "ceph-volume simple scan" -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://nam02.safelinks.protection.outlook.com/?url=https://croit.io&data=02|01|peter.ei...@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843215231&sdata=YEQI+UvikVPVeOFNSB2ikqVRiul8ElD3JEZDVOQI+NY=&reserved=0 croit GmbH Freseniusstr. 31h 81247 München https://nam02.safelinks.protection.outlook.com/?url=http://www.croit.io&data=02|01|peter.ei...@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55a6f8c7606a035|0|0|636995903843225224&sdata=83sD9wJHxE5W0renuDE7RGR/cPznR6jl9rEfl1AO+oA=&reserved=0 Tel: +49 89 1896585 90 ... I'm thinking the OSD would start (I can recreate the .service definitions in systemctl) if the above were mounted in a way like they are on another of my hosts: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.7T 0 disk ├─sda1 8:1 0 100M 0 part │ └─97712be4-1234-4acc-8102-2265769053a5 253:17 0 98M 0 crypt /var/lib/ceph/osd/ceph-16 ├─sda2 8:2 0 1.7T 0 part │ └─049b7160-1234-4edd-a5dc-fe00faca8d89 253:16 0 1.7T 0 crypt └─sda5 8:5 0 10M 0 part /var/lib/ceph/osd-lockbox/97712be4-9674-4acc-1234-2265769053a5 sdb 8:16 0 1.7T 0 disk ├─sdb1 8:17 0 100M 0 part │ └─f03f0298-1234-42e9-8b28-f3016e44d1e2 253:26 0 98M 0 crypt /var/lib/ceph/osd/ceph-17 ├─sdb2 8:18 0 1.7T 0 part │ └─51177019-1234-4963-82d1-5006233f5ab2 253:30 0 1.7T 0 crypt └─sdb5 8:21 0 10M 0 part /var/lib/ceph/osd-lockbox/f03f0298-1234-42e9-8b28-f3016e44d1e2 sdc 8:32 0 1.7T 0 disk ├─sdc1 8:33 0 100M 0 part │ └─0184df0c-1234-404d-92de-cb71b1047abf 253:8 0 98M 0 crypt /var/lib/ceph/osd/ceph-18 ├─sdc2 8:34 0 1.7T 0 part │ └─fdad7618-1234-4021-a63e-40d973712e7b 253:13 0 1.7T 0 crypt ... Thank you for your time on this, peter From: Xavier Trilla <mailto:xavier.tri...@clouding.io> Date: Wednesday, July 24, 2019 at 1:25 PM To: Peter Eisch <mailto:peter.ei...@virginpulse.com> Cc: "mailto:ceph-users@lists.ceph.com" <mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Upgrading and lost OSDs Hi Peter, Im not sure but maybe after some changes the OSDs are not being recongnized by ceph scripts. Ceph used to use udev to detect the OSDs and then moved to lvm, which kind of OSDs are you running? Blustore or filestore? Which version did you use to create them? Cheers! El 24 jul 2019, a les 20:04, Peter Eisch <mailto:mailto:peter.ei...@virginpulse.com> va escriure: Hi, I’m working through updating from 12.2.12/luminious to 14.2.2/nautilus on centos 7.6. The managers are updated alright: # ceph -s cluster: id: 2fdb5976-1234-4b29-ad9c-1ca74a9466ec health: HEALTH_WARN Degraded data redundancy: 24177/9555955 objects degraded (0.253%), 7 pgs degraded, 1285 pgs undersized 3 monitors have not enabled msgr2 ... I updated ceph on a OSD host with 'yum update' and then rebooted to grab the current kernel. Along the way, the contents of all the directories in /var/lib/ceph/osd/ceph-*/ were deleted. Thus I have 16 OSDs down from this. I can manage the undersized but I'd like to get these drives working again without deleting each OSD and recreating them. So far I've pulled the respective cephx key into the 'keyring' file and populated 'bluestore' into the 'type' files but I'm unsure how to get the lockboxes mounted to where I can get the OSDs running. The osd-lockbox directory is otherwise untouched from when the OSDs were deployed. Is there a way to run ceph-deploy or some other tool to rebuild the mounts for the drives? peter
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com