> Op 15 november 2017 om 15:03 schreef Richard Hesketh > <richard.hesk...@rd.bbc.co.uk>: > > > On 15/11/17 12:58, Micha Krause wrote: > > Hi, > > > > I've build a few clusters with separated public/cluster network, but I'm > > wondering if this is really > > the way to go. > > > > http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref > > > > states 2 reasons: > > > > 1. There is more traffic in the backend, which could cause latencies in the > > public network. > > > > Is a low latency public network really an advantage, if my cluster network > > has high latency? > > > > 2. Security: evil users could cause damage in the cluster net. > > > > Couldn't you cause the same kind, or even more damage in the public > > network? > > > > > > On the other hand, if one host looses it's cluster network, it will report > > random OSDs down over the > > remaining public net. (yes I know about the "mon osd min down reporters" > > workaround) > > > > > > Advantages of a single, shared network: > > > > 1. Hosts with network problems, that can't reach other OSDs, all so can't > > reach the mon. So our mon server doesn't get conflicting informations. > > > > 2. Given the same network bandwidth overall, OSDs can use a bigger part of > > the bandwidth for backend traffic. > > > > 3. KISS principle. > > > > So if my server has 4 x 10GB/s network should I really split them in 2 x > > 20GB/s (cluster/public) or am I > > better off using 1 x 40GB/s (shared)? > > > > Micha Krause > > I have two clusters, one running all-public-network and one with separated > public/cluster networks. The latter is a bit of a pain because it's much more > fiddly if I have to change anything, and also there is basically no point to > it being set up this way (it all goes into the same switch so there's no real > redundancy). > > To quote Wido > (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017527.html): > > I rarely use public/cluster networks as they don't add anything for most > > systems. 20Gbit of bandwidth per node is more then enough in most cases and > > my opinion is that multiple IPs per machine only add complexity. > > Unless you actually have to make your cluster available on a public network > which you don't control/trust I really don't think there's much point in > splitting things up; just bond your links together. Even if you still want to > logically split cluster/public network so they're in different subnets, you > can just assign multiple IPs to the link or potentially set up VLAN tagging > on the switch/interfaces if you want your traffic a bit more securely > segregated. >
Thanks for the quote! I still think about it that way. Public and Cluster networks might make sense in some cases, but imho it shouldn't be the default. One IP per machine is just KISS. It's up or down, not half-up, half-down. In the 7 years I've been running and building Ceph systems now I never ran into a case where I thought: I would really like/need a cluster network here. A 20Gbit LACP is sufficient for a bunch of disks in a system. And when you go full NVMe you might need more bandwidth, add a 40Gbit NIC in a system or so. To finish with, bandwidth usually isn't a really issue. Latency is. You'll run into latency problems (Ceph code, slow CPU's, disk latency) before you run out of network latency or bandwidth. Wido > Rich > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
