[ceph-users] Re: Howto upgrade AND change distro
I think it will depend on how you have your OSDs deployed currently. If they are bluestore deployed via ceph-volume using LVM, then it should mostly be pretty painless to migrate them to a new host, assuming everything is on the OSDs. The corner case would be if the WAL/DB is on a separate block device or something like that, but in that case, as long as the ownership on the block devices is correct. Copy the ceph keys, the ceph.conf, and the keyring file for the OSDs, and the requisite ceph-osd packages, you should be good to go. ceph-volume lvm activate --all Should scan lvm devices for OSDs, then create systemd targets for them, and then start them up. As was mentioned, set OSD flags for noout, norebalance, norecover before taking the node down, etc. Then once you activate the OSDs on the new host, make sure the crush topology looks as you expect, in case a host name changes/conflicts and it puts the "new" host in a weird crush location. Then unset the flags, and it shouldn't move too much data, if any. May want to try labbing this process, just to be sure, but I seem to recall this working when I lost the OS disk in an OSD node previously. Reed > On Aug 27, 2021, at 10:16 AM, Francois Legrand wrote: > > Hello, > > We are running a ceph nautilus cluster under centos 7. To upgrade to pacific > we need to change to a more recent distro (probably debian or ubuntu because > of the recent announcement about centos 8, but the distro doesn't matter very > much). > > However, I could'nt find a clear procedure to upgrade ceph AND the distro ! > As we have more than 100 osds and ~600TB of data, we would like to avoid as > far as possible to wipe the disks and rebuild/rebalance. It seems to be > possible to reinstall a server and reuse the osds, but the exact procedure > remains quite unclear to me. > > What is the best way to proceed ? Does someone have done that and have a > rather detailed doc on how to proceed ? > > Thanks for your help ! > > F. > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Howto upgrade AND change distro
Hi Fleg. We have reinstalled bunch of ceph nodes. You have to basically do following steps: 1. Back up /etc and /var/lib directories with configurations 2. Reinstall server with new OS (for example CentOS 8 Stream) and use the same disks 3. Install ceph packages and restore /etc/ceph and /var/lib/ceph, where you have to persist proper rights ceph:ceph 4. Start systemd units Michal Strnad On 8/30/21 5:49 PM, Francois Legrand wrote: Thanks, My point is how to reattach safely an osd from the previous server to the new installed distro ! Is there a detailed howto réinstall completely a server (or a cluster) ? F. Le 27/08/2021 à 19:47, Message: 1 Date: Fri, 27 Aug 2021 16:43:12 +0100 From: Matthew Vernon Subject: [ceph-users] Re: Howto upgrade AND change distro To: ceph-users@ceph.io Message-ID: <654262bf-b621-d534-7067-62a3a2abb...@wikimedia.org> Content-Type: text/plain; charset=utf-8; format=flowed Hi, On 27/08/2021 16:16, Francois Legrand wrote: We are running a ceph nautilus cluster under centos 7. To upgrade to pacific we need to change to a more recent distro (probably debian or ubuntu because of the recent announcement about centos 8, but the distro doesn't matter very much). However, I could'nt find a clear procedure to upgrade ceph AND the distro ! As we have more than 100 osds and ~600TB of data, we would like to avoid as far as possible to wipe the disks and rebuild/rebalance. It seems to be possible to reinstall a server and reuse the osds, but the exact procedure remains quite unclear to me. It's going to be least pain to do the operations separately, which means you may need to build a set of packages for one or other "end" of the operation, if you see what I mean? The Debian and Ubuntu installers both have an "expert mode" which gives you quite a lot of control which should enable you to upgrade the OS without touching the OSD disks - but make sure you have backups of all your Ceph config! If you're confident (and have enough redundancy), you can set noout while you upgrade a machine, which will reduce the amount of rebalancing you have to do when it rejoins the cluster post upgrade. Regards, Matthew [one good thing about Ubuntu's cloud archive is that e.g. you can get the same version that's default in 20.04 available as packages for 18.04 via UCA meaning you can upgrade Ceph first, and then do the distro upgrade, and it's pretty painless] ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Howto upgrade AND change distro
Thanks, My point is how to reattach safely an osd from the previous server to the new installed distro ! Is there a detailed howto réinstall completely a server (or a cluster) ? F. Le 27/08/2021 à 19:47, Message: 1 Date: Fri, 27 Aug 2021 16:43:12 +0100 From: Matthew Vernon Subject: [ceph-users] Re: Howto upgrade AND change distro To: ceph-users@ceph.io Message-ID: <654262bf-b621-d534-7067-62a3a2abb...@wikimedia.org> Content-Type: text/plain; charset=utf-8; format=flowed Hi, On 27/08/2021 16:16, Francois Legrand wrote: We are running a ceph nautilus cluster under centos 7. To upgrade to pacific we need to change to a more recent distro (probably debian or ubuntu because of the recent announcement about centos 8, but the distro doesn't matter very much). However, I could'nt find a clear procedure to upgrade ceph AND the distro ! As we have more than 100 osds and ~600TB of data, we would like to avoid as far as possible to wipe the disks and rebuild/rebalance. It seems to be possible to reinstall a server and reuse the osds, but the exact procedure remains quite unclear to me. It's going to be least pain to do the operations separately, which means you may need to build a set of packages for one or other "end" of the operation, if you see what I mean? The Debian and Ubuntu installers both have an "expert mode" which gives you quite a lot of control which should enable you to upgrade the OS without touching the OSD disks - but make sure you have backups of all your Ceph config! If you're confident (and have enough redundancy), you can set noout while you upgrade a machine, which will reduce the amount of rebalancing you have to do when it rejoins the cluster post upgrade. Regards, Matthew [one good thing about Ubuntu's cloud archive is that e.g. you can get the same version that's default in 20.04 available as packages for 18.04 via UCA meaning you can upgrade Ceph first, and then do the distro upgrade, and it's pretty painless] ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Howto upgrade AND change distro
Hi, On 27/08/2021 16:16, Francois Legrand wrote: We are running a ceph nautilus cluster under centos 7. To upgrade to pacific we need to change to a more recent distro (probably debian or ubuntu because of the recent announcement about centos 8, but the distro doesn't matter very much). However, I could'nt find a clear procedure to upgrade ceph AND the distro ! As we have more than 100 osds and ~600TB of data, we would like to avoid as far as possible to wipe the disks and rebuild/rebalance. It seems to be possible to reinstall a server and reuse the osds, but the exact procedure remains quite unclear to me. It's going to be least pain to do the operations separately, which means you may need to build a set of packages for one or other "end" of the operation, if you see what I mean? The Debian and Ubuntu installers both have an "expert mode" which gives you quite a lot of control which should enable you to upgrade the OS without touching the OSD disks - but make sure you have backups of all your Ceph config! If you're confident (and have enough redundancy), you can set noout while you upgrade a machine, which will reduce the amount of rebalancing you have to do when it rejoins the cluster post upgrade. Regards, Matthew [one good thing about Ubuntu's cloud archive is that e.g. you can get the same version that's default in 20.04 available as packages for 18.04 via UCA meaning you can upgrade Ceph first, and then do the distro upgrade, and it's pretty painless] ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
