Re: [ceph-users] Migrating from block to lvm
Losing a node is not a big deal for us (dual bonded 10G connection to each node). I’m thinking: 1. Drain node 2. Redeploy with Ceph Ansible It would require much less hands-on time for our group. I know the churn on the cluster would be high, which was my only concern. Mike Senior Systems Administrator Research Computing Services Team University of Victoria From: Martin Verges Date: Friday, November 15, 2019 at 11:52 AM To: Janne Johansson Cc: Cave Mike , ceph-users Subject: Re: [ceph-users] Migrating from block to lvm I would consider doing it host-by-host wise, as you should always be able to handle the complete loss of a node. This would be much faster in the end as you save a lot of time not migrating data back and forth. However this can lead to problems if your cluster is not configured according to the hardware performance given. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.ver...@croit.io<mailto:martin.ver...@croit.io> Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Fr., 15. Nov. 2019 um 20:46 Uhr schrieb Janne Johansson mailto:icepic...@gmail.com>>: Den fre 15 nov. 2019 kl 19:40 skrev Mike Cave mailto:mc...@uvic.ca>>: So would you recommend doing an entire node at the same time or per-osd? You should be able to do it per-OSD (or per-disk in case you run more than one OSD per disk), to minimize data movement over the network, letting other OSDs on the same host take a bit of the load while re-making the disks one by one. You can use "ceph osd reweight 0.0" to make the particular OSD release its data but still claim it supplies $crush-weight to the host, meaning the other disks will have to take its data more or less. Moving data between disks in the same host usually goes lots faster than over the network to other hosts. -- May the most significant bit of your life be positive. ___ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Migrating from block to lvm
Good points, thank you for the insight. Given that I’m hosting the journals (wal/block.dbs) on ssds, would I need to do all the OSDs hosts on each journal ssd at the same time? I’m fairly sure this would be the case. Senior Systems Administrator Research Computing Services Team University of Victoria O: 250.472.4997 From: Janne Johansson Date: Friday, November 15, 2019 at 11:46 AM To: Cave Mike Cc: Paul Emmerich , ceph-users Subject: Re: [ceph-users] Migrating from block to lvm Den fre 15 nov. 2019 kl 19:40 skrev Mike Cave mailto:mc...@uvic.ca>>: So would you recommend doing an entire node at the same time or per-osd? You should be able to do it per-OSD (or per-disk in case you run more than one OSD per disk), to minimize data movement over the network, letting other OSDs on the same host take a bit of the load while re-making the disks one by one. You can use "ceph osd reweight 0.0" to make the particular OSD release its data but still claim it supplies $crush-weight to the host, meaning the other disks will have to take its data more or less. Moving data between disks in the same host usually goes lots faster than over the network to other hosts. -- May the most significant bit of your life be positive. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Migrating from block to lvm
I would consider doing it host-by-host wise, as you should always be able to handle the complete loss of a node. This would be much faster in the end as you save a lot of time not migrating data back and forth. However this can lead to problems if your cluster is not configured according to the hardware performance given. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.ver...@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Fr., 15. Nov. 2019 um 20:46 Uhr schrieb Janne Johansson < icepic...@gmail.com>: > Den fre 15 nov. 2019 kl 19:40 skrev Mike Cave : > >> So would you recommend doing an entire node at the same time or per-osd? >> > > You should be able to do it per-OSD (or per-disk in case you run more than > one OSD per disk), to minimize data movement over the network, letting > other OSDs on the same host take a bit of the load while re-making the > disks one by one. You can use "ceph osd reweight 0.0" to make the > particular OSD release its data but still claim it supplies $crush-weight > to the host, meaning the other disks will have to take its data more or > less. > Moving data between disks in the same host usually goes lots faster than > over the network to other hosts. > > -- > May the most significant bit of your life be positive. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Migrating from block to lvm
Den fre 15 nov. 2019 kl 19:40 skrev Mike Cave : > So would you recommend doing an entire node at the same time or per-osd? > You should be able to do it per-OSD (or per-disk in case you run more than one OSD per disk), to minimize data movement over the network, letting other OSDs on the same host take a bit of the load while re-making the disks one by one. You can use "ceph osd reweight 0.0" to make the particular OSD release its data but still claim it supplies $crush-weight to the host, meaning the other disks will have to take its data more or less. Moving data between disks in the same host usually goes lots faster than over the network to other hosts. -- May the most significant bit of your life be positive. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Migrating from block to lvm
So would you recommend doing an entire node at the same time or per-osd? Senior Systems Administrator Research Computing Services Team University of Victoria O: 250.472.4997 On 2019-11-15, 10:28 AM, "Paul Emmerich" wrote: You'll have to tell LVM about multi-path, otherwise LVM gets confused. But that should be the only thing Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Nov 15, 2019 at 6:04 PM Mike Cave wrote: > > Greetings all! > > > > I am looking at upgrading to Nautilus in the near future (currently on Mimic). We have a cluster built on 480 OSDs all using multipath and simple block devices. I see that the ceph-disk tool is now deprecated and the ceph-volume tool doesn’t do everything that ceph-disk did for simple devices (e.g. I’m unable to activate a new osd and set the location of wal/block.db, so far as I have been able to figure out). So for disk replacements going forward it could get ugly. > > > > We deploy/manage using Ceph Ansible. > > > > I’m okay with updating the OSDs to LVM and understand that it will require a full rebuild of each OSD. > > > > I was thinking of going OSD by OSD through the cluster until they are all completed. However, someone suggested doing an entire node at a time (that would be 20 OSDs at a time in this case). Is one method going to be better than the other? > > > > Also a question about setting-up LVM: given I’m using multipath devices, do I have to preconfigure the LVM devices before running the ansible plays or will ansible take care of the LVM setup (even though they are on multipath)? > > > > I would then do the upgrade to Nautilus from Mimic after all the OSDs were converted. > > > > I’m looking for opinions on best practices to complete this as I’d like to minimize impact to our clients. > > > > Cheers, > > Mike Cave > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Migrating from block to lvm
You'll have to tell LVM about multi-path, otherwise LVM gets confused. But that should be the only thing Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Nov 15, 2019 at 6:04 PM Mike Cave wrote: > > Greetings all! > > > > I am looking at upgrading to Nautilus in the near future (currently on > Mimic). We have a cluster built on 480 OSDs all using multipath and simple > block devices. I see that the ceph-disk tool is now deprecated and the > ceph-volume tool doesn’t do everything that ceph-disk did for simple devices > (e.g. I’m unable to activate a new osd and set the location of wal/block.db, > so far as I have been able to figure out). So for disk replacements going > forward it could get ugly. > > > > We deploy/manage using Ceph Ansible. > > > > I’m okay with updating the OSDs to LVM and understand that it will require a > full rebuild of each OSD. > > > > I was thinking of going OSD by OSD through the cluster until they are all > completed. However, someone suggested doing an entire node at a time (that > would be 20 OSDs at a time in this case). Is one method going to be better > than the other? > > > > Also a question about setting-up LVM: given I’m using multipath devices, do I > have to preconfigure the LVM devices before running the ansible plays or will > ansible take care of the LVM setup (even though they are on multipath)? > > > > I would then do the upgrade to Nautilus from Mimic after all the OSDs were > converted. > > > > I’m looking for opinions on best practices to complete this as I’d like to > minimize impact to our clients. > > > > Cheers, > > Mike Cave > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
