[ceph-users] Re: Networking Idea/Question

2021-03-17 Thread Stefan Kooman
On 3/17/21 7:44 AM, Janne Johansson wrote: Den ons 17 mars 2021 kl 02:04 skrev Tony Liu : What's the purpose of "cluster" network, simply increasing total bandwidth or for some isolations? Not having client traffic (that only occurs on the public network) fight over bandwidth with OSD<->OSD

[ceph-users] Re: Networking Idea/Question

2021-03-17 Thread Janne Johansson
Den ons 17 mars 2021 kl 02:04 skrev Tony Liu : > What's the purpose of "cluster" network, simply increasing total > bandwidth or for some isolations? Not having client traffic (that only occurs on the public network) fight over bandwidth with OSD<->OSD traffic (replication and recovery). Now,

[ceph-users] Re: Networking Idea/Question

2021-03-16 Thread Tony Liu
Tony > -Original Message- > From: Andrew Walker-Brown > Sent: Tuesday, March 16, 2021 9:18 AM > To: Tony Liu ; Stefan Kooman ; > Dave Hall ; ceph-users > Subject: RE: [ceph-users] Re: Networking Idea/Question > > https://docs.ceph.com/en/latest/rados/co

[ceph-users] Re: Networking Idea/Question

2021-03-16 Thread Andrew Walker-Brown
e Hall<mailto:kdh...@binghamton.edu>; ceph-users<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Networking Idea/Question > -Original Message- > From: Stefan Kooman > Sent: Tuesday, March 16, 2021 4:10 AM > To: Dave Hall ; ceph-users > Subject: [ceph-users] Re: Ne

[ceph-users] Re: Networking Idea/Question

2021-03-16 Thread Tony Liu
> -Original Message- > From: Stefan Kooman > Sent: Tuesday, March 16, 2021 4:10 AM > To: Dave Hall ; ceph-users > Subject: [ceph-users] Re: Networking Idea/Question > > On 3/15/21 5:34 PM, Dave Hall wrote: > > Hello, > > > > If anybody out there

[ceph-users] Re: Networking Idea/Question

2021-03-16 Thread Stefan Kooman
On 3/15/21 5:34 PM, Dave Hall wrote: Hello, If anybody out there has tried this or thought about it, I'd like to know... I've been thinking about ways to squeeze as much performance as possible from the NICs  on a Ceph OSD node.  The nodes in our cluster (6 x OSD, 3 x MGR/MON/MDS/RGW)

[ceph-users] Re: Networking Idea/Question

2021-03-16 Thread Dave Hall
Burkhard, I woke up with the same conclusion - LACP load balancing can break down when the traffic traverses a router since the IP headers have the router as the destination address and thus the Ethernet header has the same to MAC addresses. (I think that in a pure layer 2 fabric the MAC

[ceph-users] Re: Networking Idea/Question

2021-03-16 Thread Burkhard Linke
Hi, On 16.03.21 03:40, Dave Hall wrote: Andrew, I agree that the choice of hash function is important for LACP. My thinking has always been to stay down in layers 2 and 3.  With enough hosts it seems likely that traffic would be split close to evenly.  Heads or tails - 50% of the time

[ceph-users] Re: Networking Idea/Question

2021-03-16 Thread Dave Hall
Andrew, I agree that the choice of hash function is important for LACP. My thinking has always been to stay down in layers 2 and 3.  With enough hosts it seems likely that traffic would be split close to evenly.  Heads or tails - 50% of the time you're right.  TCP ports should also be nearly

[ceph-users] Re: Networking Idea/Question

2021-03-15 Thread Andrew Walker-Brown
Dave That’s the way our cluster is setup. It’s relatively small, 5 hosts, 12 osd’s. Each host has 2x10G with LACP to the switches. We’ve vlan’d public/private networks. Making best use of the LACP lag will to a greater extent be down to choosing the best hashing policy. At the moment