I am running PVFS2 on OpenIB, with IBM's ehca.
When we start writing/reading large files, either with the NetPIPE
PVFS module we have or a modified GAMESS executable that uses
libpvfs2 directly, the 'ibv_reg_mr' function fails, and we get an error.
This is also correlated with kernel log messages like this:
Oct 16 11:14:45 p5l8 kernel: PU0003 000e0091:ehca_hcall_7arg_7ret
HCAD_ERROR opco
de=160 ret=fffffffffffffff7 arg1=1000000003000004 arg2=5
arg3=14f0ebc8 arg4=10000
arg5=e0000000000000 arg6=e3e9f200 arg7=0 out1=0 out2=0 out3=0 out4=0
out5=0 out6=0
out7=0
Oct 16 11:14:45 p5l8 kernel: PU0003 00090454:ehca_reg_mr HCAD_ERROR
hipz_alloc_mr
failed, h_ret=fffffffffffffff7 hca_hndl=1000000003000004
Oct 16 11:14:45 p5l8 kernel: PU0003 00090478:ehca_reg_mr <<<
ret=ffffffea shca=c00
00000e796b000 e_mr=c0000000d22c7d80 iova_start=0000000014f0ebc8
size=10000 acl=7 e
_pd=c0000000e3e9f200 pginfo=c0000001ad37fa70 num_pages=11 num_4k=11
Oct 16 11:14:45 p5l8 kernel: PU0003 00090176:ehca_reg_user_mr <<<
rc=fffffffffffff
fea pd=c0000000e3e9f200 region=c0000000cb73a9d0 mr_access_flags=7
udata=c0000001ad
37fba0
We are able to run on a 4x PCI-X Mellanox HCA, but obviously I'd like
to be using the 12x ehca.
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
