On Tue, Nov 16, 2010 at 8:52 PM, Edward Ned Harvey <[email protected]> wrote: >> From: Sean Lutner [mailto:[email protected]] >... > > I just can't bear the idea of optimizing all my 6Gbit disks for performance, > only to funnel them all across a 1Gbit bottleneck.
Your message yesterday about point-to-point bandwidth limitations got me started thinking (and googling) about this. It seems like most of the methods for utilizing multiple network links for more then just redundancy use some kind of hash function to determine which link a particular packet is sent over. This works fairly well if you have many-to-many network flows. Unfortunately that's not what you have. With a single destination address, you are going to end up using a single link being used due to most hashing schemes just looking at the addresses of the packets. What you want is a simple round robin scheme to spray outgoing packets across both interfaces. BTW, the reason must systems don't do round robin is that network protocols prefer to receive packets in order and with consistent latency. With multiple active links per network flow, you get out of order and variable latency in the general case. This can screwup video streaming or VOIP pretty badly. You are in a single datacenter and aren't using those protocols, so we can hope it won't matter to you. (Be sure to test this. :-) If this was pure Linux-to-Linux, you could look into equal cost multi-path routing. Here's a random link that I found http://etutorials.org/networking/integrated%2Bcisco%2Band%2Bunix%2Bnetwork%2Barchitectures/chapter%2B8.%2Bstatic%2Brouting%2Bconcepts/equal-cost%2Bmulti-path%2Becmp%2Brouting/) If both servers are setup with machine local subnets, you should be able to use multipath to force use of the redundant paths equally for routing packets between the two machine local subnets. Note: This could still fail due to issues with Linux ARPing on a per machine rather then per interface for its IP addresses. This can screw you up by having all of your per-link IP addresses associated with a single Ethernet address and Ethernet spanning tree is only going to use one incoming link on the destination as a result. Manually setting up the ARP tables on the machines is probably the easiest way to deal with this. Dedicated wiring/separate VLANs/distinct subnets per link could all be useful/needed as well. Feel free to burn lots of 10.*.*.* non-routable subnets to get this to work. I also tried looking into iSCSI multipath and haven't figured out if that would be useful or not. Still it bears more investigation. If no single VM on the compute server needs more then 1Gbit you might be able to associate each one with a separate IP address and give your file server multiple IPs as well. In general, I would suggest starting by thinking about what goes across the wire(s) and working your way up through the hardware/protocol stack and see at what point you can introduce a mapping between a single higher level endpoint to multiple lower level endpoints. Then check to make sure nothing merges the lower level streams until you want it. (i.e. Linux's weird ARPing.) Finally, all of the above is based on "reading knowledge". I haven't tried it myself. (I also don't know about Solaris/ESXi capabilities.) Good Luck, Bill Bogstad _______________________________________________ bblisa mailing list [email protected] http://www.bblisa.org/mailman/listinfo/bblisa
