Re: [ceph-users] Adding cluster network to running cluster

[mj]

On 06/07/2018 01:45 PM, Wido den Hollander wrote:


Removing cluster network is enough. After the restart the OSDs will not
publish a cluster network in the OSDMap anymore.

You can keep the public network in ceph.conf and can even remove that
after you removed the 10.10.x.x addresses from the system(s).

Wido


Thanks for the info, Wido. :-)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding cluster network to running cluster

[Wido den Hollander]

On 06/07/2018 01:39 PM, mj wrote:
> Hi,
> 
> Please allow me to ask one more question:
> 
> We currently have a seperated network: cluster on 10.10.x.x and public
> on 192.168.x.x.
> 
> I would like to migrate all network to 192.168.x.x setup, which would
> give us 2*10G.
> 
> Is simply changing the cluster network in ceph.conf and a restart enough?
> 

Removing cluster network is enough. After the restart the OSDs will not
publish a cluster network in the OSDMap anymore.

You can keep the public network in ceph.conf and can even remove that
after you removed the 10.10.x.x addresses from the system(s).

Wido

> (of course provided that the 2*10G bond is configured correctly, etc)
> 
> Or is there more to it?
> 
> MJ
> 
> On 06/07/2018 01:13 PM, Paul Emmerich wrote:
>> We also build almost all of our clusters with a single Ceph network.
>> 2x10 Gbit/s is almost never the bottleneck.
>>
>>
>>
>> Paul
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding cluster network to running cluster

[Paul Emmerich]

We also build almost all of our clusters with a single Ceph network.
2x10 Gbit/s is almost never the bottleneck.



Paul

2018-06-07 11:05 GMT+02:00 Wido den Hollander :

>
>
> On 06/07/2018 10:56 AM, Kevin Olbrich wrote:
> > Realy?
> >
> > I always thought that splitting the replication network is best practice.
> > Keeping everything in the same IPv6 network is much easier.
> >
>
> No, there is no big benefit unless your usecase (which 99% isn't) asks
> for it.
>
> Keep it simple, one network to run the cluster on. Less components which
> can fail or complicate things.
>
> Wido
>
> > Thank you.
> >
> > Kevin
> >
> > 2018-06-07 10:44 GMT+02:00 Wido den Hollander  > >:
> >
> >
> >
> > On 06/07/2018 09:46 AM, Kevin Olbrich wrote:
> > > Hi!
> > >
> > > When we installed our new luminous cluster, we had issues with the
> > > cluster network (setup of mon's failed).
> > > We moved on with a single network setup.
> > >
> > > Now I would like to set the cluster network again but the cluster
> is in
> > > use (4 nodes, 2 pools, VMs).
> >
> > Why? What is the benefit from having the cluster network? Back in the
> > old days when 10Gb was expensive you would run public on 1G and
> cluster
> > on 10G.
> >
> > Now with 2x10Gb going into each machine, why still bother with
> managing
> > two networks?
> >
> > I really do not see the benefit.
> >
> > I manage multiple 1000 ~ 2500 OSD clusters all running with all their
> > nodes on IPv6 and 2x10Gb in a single network. That works just fine.
> >
> > Try to keep the network simple and do not overcomplicate it.
> >
> > Wido
> >
> > > What happens if I set the cluster network on one of the nodes and
> reboot
> > > (maintenance, updates, etc.)?
> > > Will the node use both networks as the other three nodes are not
> > > reachable there?
> > >
> > > Both the MONs and OSDs have IPs in both networks, routing is not
> needed.
> > > This cluster is dualstack but we set ms_bind_ipv6 = true.
> > >
> > > Thank you.
> > >
> > > Kind regards
> > > Kevin
> > >
> > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com 
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > 
> > >
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding cluster network to running cluster

[Wido den Hollander]

On 06/07/2018 10:56 AM, Kevin Olbrich wrote:
> Realy?
> 
> I always thought that splitting the replication network is best practice.
> Keeping everything in the same IPv6 network is much easier.
> 

No, there is no big benefit unless your usecase (which 99% isn't) asks
for it.

Keep it simple, one network to run the cluster on. Less components which
can fail or complicate things.

Wido

> Thank you.
> 
> Kevin
> 
> 2018-06-07 10:44 GMT+02:00 Wido den Hollander  >:
> 
> 
> 
> On 06/07/2018 09:46 AM, Kevin Olbrich wrote:
> > Hi!
> > 
> > When we installed our new luminous cluster, we had issues with the
> > cluster network (setup of mon's failed).
> > We moved on with a single network setup.
> > 
> > Now I would like to set the cluster network again but the cluster is in
> > use (4 nodes, 2 pools, VMs).
> 
> Why? What is the benefit from having the cluster network? Back in the
> old days when 10Gb was expensive you would run public on 1G and cluster
> on 10G.
> 
> Now with 2x10Gb going into each machine, why still bother with managing
> two networks?
> 
> I really do not see the benefit.
> 
> I manage multiple 1000 ~ 2500 OSD clusters all running with all their
> nodes on IPv6 and 2x10Gb in a single network. That works just fine.
> 
> Try to keep the network simple and do not overcomplicate it.
> 
> Wido
> 
> > What happens if I set the cluster network on one of the nodes and reboot
> > (maintenance, updates, etc.)?
> > Will the node use both networks as the other three nodes are not
> > reachable there?
> > 
> > Both the MONs and OSDs have IPs in both networks, routing is not needed.
> > This cluster is dualstack but we set ms_bind_ipv6 = true.
> > 
> > Thank you.
> > 
> > Kind regards
> > Kevin
> > 
> > 
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> >
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding cluster network to running cluster

[Kevin Olbrich]

Realy?

I always thought that splitting the replication network is best practice.
Keeping everything in the same IPv6 network is much easier.

Thank you.

Kevin

2018-06-07 10:44 GMT+02:00 Wido den Hollander :

>
>
> On 06/07/2018 09:46 AM, Kevin Olbrich wrote:
> > Hi!
> >
> > When we installed our new luminous cluster, we had issues with the
> > cluster network (setup of mon's failed).
> > We moved on with a single network setup.
> >
> > Now I would like to set the cluster network again but the cluster is in
> > use (4 nodes, 2 pools, VMs).
>
> Why? What is the benefit from having the cluster network? Back in the
> old days when 10Gb was expensive you would run public on 1G and cluster
> on 10G.
>
> Now with 2x10Gb going into each machine, why still bother with managing
> two networks?
>
> I really do not see the benefit.
>
> I manage multiple 1000 ~ 2500 OSD clusters all running with all their
> nodes on IPv6 and 2x10Gb in a single network. That works just fine.
>
> Try to keep the network simple and do not overcomplicate it.
>
> Wido
>
> > What happens if I set the cluster network on one of the nodes and reboot
> > (maintenance, updates, etc.)?
> > Will the node use both networks as the other three nodes are not
> > reachable there?
> >
> > Both the MONs and OSDs have IPs in both networks, routing is not needed.
> > This cluster is dualstack but we set ms_bind_ipv6 = true.
> >
> > Thank you.
> >
> > Kind regards
> > Kevin
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding cluster network to running cluster

[Wido den Hollander]

On 06/07/2018 09:46 AM, Kevin Olbrich wrote:
> Hi!
> 
> When we installed our new luminous cluster, we had issues with the
> cluster network (setup of mon's failed).
> We moved on with a single network setup.
> 
> Now I would like to set the cluster network again but the cluster is in
> use (4 nodes, 2 pools, VMs).

Why? What is the benefit from having the cluster network? Back in the
old days when 10Gb was expensive you would run public on 1G and cluster
on 10G.

Now with 2x10Gb going into each machine, why still bother with managing
two networks?

I really do not see the benefit.

I manage multiple 1000 ~ 2500 OSD clusters all running with all their
nodes on IPv6 and 2x10Gb in a single network. That works just fine.

Try to keep the network simple and do not overcomplicate it.

Wido

> What happens if I set the cluster network on one of the nodes and reboot
> (maintenance, updates, etc.)?
> Will the node use both networks as the other three nodes are not
> reachable there?
> 
> Both the MONs and OSDs have IPs in both networks, routing is not needed.
> This cluster is dualstack but we set ms_bind_ipv6 = true.
> 
> Thank you.
> 
> Kind regards
> Kevin
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding cluster network to running cluster

[Burkhard Linke]

Hi,


I may be wrong, but AFAIK the cluster network is only used to bind the 
corresponding functionality to the correct network interface. There's no 
check for a common CIDR range or something similar in CEPH.



As long as the traffic is routeable from the current public network and 
the new cluster network, you should be able to transform the network 
setup host by host without any downtime. We used the same strategy to 
get rid of our IPoIB cluster network by defining one host as gateway to 
the cluster network and adding static routes on the other hosts during 
the transition. After all hosts were processed, we simply deleted the 
routes.


In case of the original poster it should simply be a matter of defining 
the cluster network in ceph.conf and restarting the OSDs, since the 
cluster network interfaces are already present on all hosts.


Regards,
Burkhard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding cluster network to running cluster

[David Turner]

2 Things about this.  First, Mons do not communicate over the cluster
network.  Only OSD daemons send any traffic over that network.  Mons, MDS,
RGW, MGR, etc all communicate over the public network.  OSDs communicate
with all clients, Mons, etc on the public network and with each other on
the cluster network.  That is for sharing extra replicas to other OSDs,
backfilling, recovering, pinging each other to see if things are up and
running. etc.

Second, What will happen if you change the cluster network on 1 node and
not the others is that the other nodes will report to the Mons that every
OSD on that node is down because they won't respond to pings and the 1 node
will report to the Mons that every other OSD in the cluster is down. Making
changes to the cluster network is easiest by stopping all OSDs and starting
them back up with the new settings.

On Thu, Jun 7, 2018 at 3:46 AM Kevin Olbrich  wrote:

> Hi!
>
> When we installed our new luminous cluster, we had issues with the cluster
> network (setup of mon's failed).
> We moved on with a single network setup.
>
> Now I would like to set the cluster network again but the cluster is in
> use (4 nodes, 2 pools, VMs).
> What happens if I set the cluster network on one of the nodes and reboot
> (maintenance, updates, etc.)?
> Will the node use both networks as the other three nodes are not reachable
> there?
>
> Both the MONs and OSDs have IPs in both networks, routing is not needed.
> This cluster is dualstack but we set ms_bind_ipv6 = true.
>
> Thank you.
>
> Kind regards
> Kevin
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com