On Mar 5, 2009, at 8:46 AM, Robert Latham wrote:
On Wed, Mar 04, 2009 at 07:15:24PM -0500, Bradley Settlemyer wrote:
Hello
I am trying to use PAV to run pvfs with the MX protocol. I've
updated pav so that servers start and ping correctly. But when I try
and run an mpi code, I'm getting client timeouts like the client
cannot contact the servers:
Lots of this stuff:
[E 19:11:02.573509] job_time_mgr_expire: job time out: cancelling bmi
operation, job_id: 3.
[E 19:11:02.583659] msgpair failed, will retry: Operation cancelled
(possibly due to timeout)
Brad, which version of MX and PVFS2?
OK, so pvfs utilities are all hunky-dory? not just pvfs2-ping but
pvfs2-cp and pvfs2-ls?
On Jazz, I usually configure MPICH2 to communicate over TCP and have
the PVFS system interface communicate over MX. This keeps the
situation fairly simple, but of course you get awful MPI performance.
Does MX still have the "ports" restriction that GM has? I wonder if
MPI communication is getting in the way of PVFS communication...
In short, I don't exactly know what's wrong myself. just tossing out
some theories.
==rob
Rob, MX is limited to 8 endpoints per NIC. One can use mx_info to get
the number:
8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
This can be increased to 16 with a module parameter.
Generally, you want no more than one endpoint per process and one
process per core for MPI. When you want to use MPI-IO over PVFS2, each
process will need two endpoints (one for MPI and one for PVFS2). If
you have eight cores, you should increase the max endpoints to 16 (if
you have eight cores).
Generally, I would not want to limit my MPI to TCP and IO to MX
especially if my TCP is over gigabit Ethernet. Unless your IO can
exceed the link rate, there will be plenty of bandwidth left over for
MPI and your latency will stay much lower than TCP.
What is PAV?
Scott
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
