[ceph-users] Migration of a Ceph cluster to a new datacenter and new IPs

[Marcus Müller]

Hi all,

we’re running a ceph hammer cluster with 3 mons and 24 osds (3 same nodes) and 
need to migrate all servers to a new datacenter and change the IPs of the 
nodes. 

I found this tutorial: 
http://docs.ceph.com/docs/hammer/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address-the-messy-way
 

 regarding the mons (should be easy) but nothing about the osds and which steps 
to do if you need to shutdown and migrate the cluster to a new datacenter.

Has anyone some ideas, how to and which steps I need? 

Regards,
Marcus___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migration of a Ceph cluster to a new datacenter and new IPs

[Burkhard Linke]

Hi,

On 12/19/18 8:55 PM, Marcus Müller wrote:

Hi all,

we’re running a ceph hammer cluster with 3 mons and 24 osds (3 same 
nodes) and need to migrate all servers to a new datacenter and change 
the IPs of the nodes.


I found this tutorial: 
http://docs.ceph.com/docs/hammer/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address-the-messy-way regarding 
the mons (should be easy) but nothing about the osds and which steps 
to do if you need to shutdown and migrate the cluster to a new datacenter.


Has anyone some ideas, how to and which steps I need?



AFAIK the OSDs use ceph.conf to retrieve the mons' ip addresses. So 
moving the mons to new address and updating ceph.conf prior to starting 
the OSDs should do the trick.



You might run into other problems afterwards, especially if you are 
using openstack. Cinder store the mon ip address in its database, and 
you need to update them manually via SQL statements...



Regards,

Burkhard


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migration of a Ceph cluster to a new datacenter and new IPs

[Paul Emmerich]

I'd do it like this:

* create 2 new mons with the new IPs
* update all clients to the 3 new mon IPs
* delete two old mons
* create 1 new mon
* delete the last old mon

I think it's easier to create/delete mons than to change the IP of an
existing mon. This doesn't even incur a downtime for the clients
because they get notified about the new mons.

For the OSDs: stop OSDs, change IP, start OSDs

Don't change the IP of a running OSD, they don't like that

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Wed, Dec 19, 2018 at 8:55 PM Marcus Müller  wrote:
>
> Hi all,
>
> we’re running a ceph hammer cluster with 3 mons and 24 osds (3 same nodes) 
> and need to migrate all servers to a new datacenter and change the IPs of the 
> nodes.
>
> I found this tutorial: 
> http://docs.ceph.com/docs/hammer/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address-the-messy-way
>  regarding the mons (should be easy) but nothing about the osds and which 
> steps to do if you need to shutdown and migrate the cluster to a new 
> datacenter.
>
> Has anyone some ideas, how to and which steps I need?
>
> Regards,
> Marcus
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migration of a Ceph cluster to a new datacenter and new IPs

[Marcus Müller]

Hi all,

Just wanted to explain my experience on how to stop the whole cluster and 
change the IPs.

First, we shut down the cluster with this procedure:

1.Stop the clients from using the RBD images/Rados Gateway on this
cluster or any other clients.
2.The cluster must be in healthy state before proceeding.
3.Set the noout, norecover, norebalance, nobackfill, nodown and pause flags
#ceph osd set noout
#ceph osd set norecover
#ceph osd set norebalance
#ceph osd set nobackfill
#ceph osd set nodown
#ceph osd set pause
4. Stop all ceph services 
4.1.First osd nodes one by one
4.2.Lastly monitor nodes one by one

Now we extracted the monmap with 'ceph-mon -i {mon-id} --extract-monmap 
/tmp/monmap‘
Followed this manual: 
http://docs.ceph.com/docs/hammer/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address-the-messy-way
 

And imported the new monmap to each monitor (while they all were stopped), 
changed ceph.conf on all nodes with the new IPs (don’t forget the clients).
The last step was to change the IP Config and the hosts files (in our case, 
again don’t forget the clients) and shutdown the nodes. 

In the new datacenter we started the nodes and everything came up as usual.

(1.Power on the admin node)
2.Power on the monitor nodes
3.Power on the osd nodes
4.Wait for all the nodes to come up , Verify all the services are
up and the connectivity is fine between the nodes.
5.Unset all the noout,norecover,noreblance, nobackfill, nodown and
pause flags.
#ceph osd unset noout
#ceph osd unset norecover
#ceph osd unset norebalance
#ceph osd unset nobackfill
#ceph osd unset nodown
#ceph osd unset pause
6.Check and verify the cluster is in healthy state, Verify all the
clients are able to access the cluster.

I hope this helps someone for the future!


> Am 20.12.2018 um 18:18 schrieb Paul Emmerich :
> 
> I'd do it like this:
> 
> * create 2 new mons with the new IPs
> * update all clients to the 3 new mon IPs
> * delete two old mons
> * create 1 new mon
> * delete the last old mon
> 
> I think it's easier to create/delete mons than to change the IP of an
> existing mon. This doesn't even incur a downtime for the clients
> because they get notified about the new mons.
> 
> For the OSDs: stop OSDs, change IP, start OSDs
> 
> Don't change the IP of a running OSD, they don't like that
> 
> Paul
> 
> -- 
> Paul Emmerich
> 
> Looking for help with your Ceph cluster? Contact us at https://croit.io
> 
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
> 
> On Wed, Dec 19, 2018 at 8:55 PM Marcus Müller  
> wrote:
>> 
>> Hi all,
>> 
>> we’re running a ceph hammer cluster with 3 mons and 24 osds (3 same nodes) 
>> and need to migrate all servers to a new datacenter and change the IPs of 
>> the nodes.
>> 
>> I found this tutorial: 
>> http://docs.ceph.com/docs/hammer/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address-the-messy-way
>>  regarding the mons (should be easy) but nothing about the osds and which 
>> steps to do if you need to shutdown and migrate the cluster to a new 
>> datacenter.
>> 
>> Has anyone some ideas, how to and which steps I need?
>> 
>> Regards,
>> Marcus
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com