[EMAIL PROTECTED] wrote on Tue, 03 Jul 2007 16:51 +1000:
> I'd appreciate the advice or insight anyone could provide on a pvfs2
> configuration change I'm contemplating.
>
> Specifically, I'd like to switch from using my current (congested)
> Ethernet connections to using some Infiniband cards I have.
> I have 'googled' the pvfs2 email list and one question about
> multihoming didn't seem to have any result/thread. I didn't see
> anything in the pvfs/doc/ folder, unless I missed it.
>
> I'd appreciate any comments or suggestions about the options I should
> set in the configuration set out below.
>
> The setup:
> --------------
> I have three IO servers each with a 2-port Infiniband card. All ports
> are connected to each other (no Infiniband switch). Using IPoIB I can
> ping all IB ports from all machines, and running: "osmtest -f a",
> indicates pass.
> Each PVFS IO server server is also a PVFS client. One machine is the
> metadata server.
We tend not to like to have clients and servers on the same machines
if one cares about performance. Independent of multi-homing.
> The performance questions:
> ---------------------------------------
> 1) Is multi-homing in this setup likely to give benefits, or just headaches?
> (a.k.a Is this a sane thing to contemplate?)
It is more for functionality: a given server can talk to multiple
networks at the same time. For performance, you would configure
pvfs only to use your second network and keep the traffic off your
congested ethernet.
> The configuration questions:
> ----------------------------------------
> 2) In pvfs2-conf.fs: should BMIModules refer to both tcp and ib to use
> IPoIB?
IPoIB is TCP, it is not IB. There is a native IB driver, or you can
use the TCP driver but send the packets through the IPoIB device by
choosing the right IP addresses. I'll say "no" here and assume you
are just trying to use two TCP addresses.
> 3) How should each IO server be "told" to listen on the two IP
> addresses assigned to the 2 IB ports? Specifically:
> a) Does HostID in the server config file need to be changed? If so,
> b) Does the Alias in the pvfs2-conf.fs get changed?
> c) Should TrustedPorts and TrustedNetwork values be set?
> d) Should the port set on each IPoIB/IP address be different or the
> same (I recall seeing an email about this but can't seem to find it
> again...)
The pvfs server listens to *:3334 by default. You do not have to
choose a particular listening _address_ (just port). On the
clients, you need to tell them to connect to the IPoIB address, not
the ethernet address. This can be done by choosing hostnames for IO
and meta servers that correspond to IP addresses on those NICs. So
yes, HostID and Alias are different. Also /etc/pvfs2tab.
Trusted* is always optional. I wouldn't bother unless you have
security concerns.
Same port.
> 4) Given I will be using IPoIB I assume I still set tcp as the PVFS
> protocol nominated in the HostID and Alias?
That helps my assumption in 2.
You might want to read "pvfs2-genconfig --spec-usage" if you want to
do the multi-homing thing.
-- Pete
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
