On 15/11/17 12:58, Micha Krause wrote: > Hi, > > I've build a few clusters with separated public/cluster network, but I'm > wondering if this is really > the way to go. > > http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref > > states 2 reasons: > > 1. There is more traffic in the backend, which could cause latencies in the > public network. > > Is a low latency public network really an advantage, if my cluster network > has high latency? > > 2. Security: evil users could cause damage in the cluster net. > > Couldn't you cause the same kind, or even more damage in the public network? > > > On the other hand, if one host looses it's cluster network, it will report > random OSDs down over the > remaining public net. (yes I know about the "mon osd min down reporters" > workaround) > > > Advantages of a single, shared network: > > 1. Hosts with network problems, that can't reach other OSDs, all so can't > reach the mon. So our mon server doesn't get conflicting informations. > > 2. Given the same network bandwidth overall, OSDs can use a bigger part of > the bandwidth for backend traffic. > > 3. KISS principle. > > So if my server has 4 x 10GB/s network should I really split them in 2 x > 20GB/s (cluster/public) or am I > better off using 1 x 40GB/s (shared)? > > Micha Krause
I have two clusters, one running all-public-network and one with separated public/cluster networks. The latter is a bit of a pain because it's much more fiddly if I have to change anything, and also there is basically no point to it being set up this way (it all goes into the same switch so there's no real redundancy). To quote Wido (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017527.html): > I rarely use public/cluster networks as they don't add anything for most > systems. 20Gbit of bandwidth per node is more then enough in most cases and > my opinion is that multiple IPs per machine only add complexity. Unless you actually have to make your cluster available on a public network which you don't control/trust I really don't think there's much point in splitting things up; just bond your links together. Even if you still want to logically split cluster/public network so they're in different subnets, you can just assign multiple IPs to the link or potentially set up VLAN tagging on the switch/interfaces if you want your traffic a bit more securely segregated. Rich
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
