On May 09, 2009 09:18 -0700, Arden Wiebe wrote: > This might help answer some questions. > http://ioio.ca/Lustre-tcp-bonding/OST2.png which shows my mostly not > tuned OSS and OST's pulling 400+MiB/s over TCP Bonding provided by the > kernel complete with a cat of the modeprobe.conf file. You have the other > links I've sent you but the picture above is relevant to your questions.
Arden, thanks for sharing this info. Any chance you could post it to wiki.lustre.org? It would seem there is one bit of info missing somewhere - how does bond0 know which interfaces to use? Also, another oddity - the network monitor is showing 450MiB/s Received, yet the disk is showing only about 170MiB/s going to the disk. Either something is wacky with the monitoring (e.g. it is counting Received for both the eth* networks AND bond0), or Lustre is doing something very wierd and retransmitting the bulk data like crazy (seems unlikely). > --- On Thu, 5/7/09, Michael Ruepp <[email protected]> wrote: > > > From: Michael Ruepp <[email protected]> > > Subject: [Lustre-discuss] tcp network load balancing understanding lustre > > 1.8 > > To: [email protected] > > Date: Thursday, May 7, 2009, 5:50 AM > > Hi there, > > > > I am configured a simple tcp lustre 1.8 with one mdc (one > > nic) and two > > oss (four nic per oss) > > As well as in the 1.6 documentation, the multihomed > > sections is a > > little bit unclear to me. > > > > I give every NID a IP in the same subnet, eg: > > 10.111.20.35-38 - oss0 > > and 10.111.20.39-42 oss1 > > > > Do I have to make modprobe.conf.local look like this to > > force lustre > > to use all four interfaces parallel: > > > > options lnet networks=tcp0(eth0,eth1,eth2,eth3) > > Because on Page 138 the 1.8 Manual says: > > "Note – In the case of TCP-only clients, the first > > available non- > > loopback IP interface > > is used for tcp0 since the interfaces are not specified. " > > > > or do I have to specify it like this: > > options lnet networks=tcp > > Because on Page 112 the lustre 1.6 Manual says: > > "Note – In the case of TCP-only clients, all available IP > > interfaces > > are used for tcp0 > > since the interfaces are not specified. If there is more > > than one, the > > IP of the first one > > found is used to construct the tcp0 ID." > > > > Which is the opposite of the 1.8 Manual > > > > My goal ist to let lustre utilize all four Gb Links > > parallel. And my > > Lustre Clients are equipped with two Gb links which should > > be utilized > > by the lustre clients as well (eth0, eth1) > > > > Or is bonding the better solution in terms of performance? > > > > Thanks very much for input, > > > > Michael Ruepp > > Schwarzfilm AG > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > [email protected] > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
