[EMAIL PROTECTED] wrote on Wed, 05 Sep 2007 12:53 -0500:
> Installed pvfs-2.6.3 on a SLES 10 SP1 x86_64 server running ofed-1.2 and
> on an IA64 client running SLES 10 with ofed-1.2.
>
> Start the server on x86_64 pvfs server (both metadata and I/O server).
> -----------------------------------------------------------------------
> hpcxe007:/
> # /usr/local/opt/pvfs-2.6.3/sbin/pvfs2-server /etc/pvfs2-fs.conf
> /etc/pvfs2-server.conf-hpcxe007 -d
> [D 12:39:31.785717] PVFS2 Server version 2.6.3 starting.
>
> start the IA64 client:
> ---------------------
> hpca4000:# /usr/local/opt/pvfs-2.6.3/sbin/pvfs2-client -f
> -p /usr/local/opt/pvfs-2.6.3/sbin/pvfs2-client-core
> pvfs2-client starting
> Spawning new child process
> Waiting on child with pid 23453
> About to exec /usr/local/opt/pvfs-2.6.3/sbin/pvfs2-client-core
>
> mount on the client:
> --------------------
> hpca4000:/test # mount -t pvfs2 ib://hpcxe007:3335/pvfs2-fs /test/pvfs2
Looks good. By the way, once this starts working, you may want to
grab CVS head rather than 2.6.3 as it has the important configure
option "--disable-tcp" that is required to get good IB performance.
> run a simple pvfs2-cp on the client:
> ------------------------------------
> hpca4000(salmr0)69:/usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp
> -t /tmp/testfile1 /tes
> t/pvfs2/
> [E 12:39:45.341786] Warning: get_nic_handle: found 3 HCAs, choosing the
> first.
> [E 12:39:45.343789] Error: openib_new_connection: register_mr eager
> recv.
You have three IB cards in the client? Wow. You might take a look
at the output of "ibv_devices" and "ibv_devinfo" to see if the first
NIC is the one that is connected to your server. Please send me the
output too. If not, we will have to add code to let you specify an
interface name to PVFS.
The error comes from this line (simplified):
mr = ibv_reg_mr(pd, buf, len, IBV_ACCESS_LOCAL_WRITE |
IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ);
where len is 20 * 8 kB == 160 kB. That's a pretty small amount
of memory. This is the first memory registration call on the
client.
Maybe the output from ibv_devinfo will tell us that you have some
interesting NIC that perhaps doesn't support the remote access flags,
or some other clue.
> this is the message I see on the server as it segfaults:
> ----------------------------------------------------------
> [E 09/05 12:39] Warning: exchange_data: partial read, 1/12 bytes.
> Segmentation fault
It's just complaining that the client died. But it shouldn't SEGV.
I'll take a look at that.
-- Pete
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
