I am currently running our Ceph POC environment using dual Nexus 9372TX 10G-T switches, each OSD host has two connections to each switch and they are formed into a single 4 link VPC (MC-LAG), which is bonded under LACP on the host side.
What I have noticed is that the various hashing policies for LACP do not guarantee you will make full use of all the links. I tried various policies and from what I could see the normal L3+L4 IP and port hashing generally worked as good as anything else, but if you have lots of similar connections it doesn't seem to hash across all the links and say 2 will be heavily used while not much is hashed onto the other links. This might have just been because it was a fairly small pool of IPs and fairly similar port numbers that just happened to keep hashing to the same links (I ended up going to the point of tcpdumping traffic and scripting a calculation of what link it should use, it just happened to be so consistent). For two links it should be quite good - it seemed to balance across that quite well, but with 4 links it seemed to really prefer 2 in my case. > -----Original Message----- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > David Riedl > Sent: Thursday, 2 June 2016 2:12 AM > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Best Network Switches for Redundancy > > > > 4. As Ceph has lots of connections on lots of IP's and port's, LACP or > > the Linux ALB mode should work really well to balance connections. > Linux ALB Mode looks promising. Does that work with two switches? Each > server has 4 ports which are 'splitted' and connected to each switch. > _ > / _[switch] > / / || > [server] || > \ \_ || > \__[switch] > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake. _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
