Re: [OMPI devel] Sending large messages over RDMA fails

2010-12-07 Thread Jeff Squyres
Doron -- I chatted with George about this today (we're both at the Forum together this week). We're in this situation because of some complicated history. 1. At one time, both PUT and GET protocols worked fine in OB1. 2. PUT was the default. 3. Over time, GET got broken (because it was rarely u

Re: [OMPI devel] Sending large messages over RDMA fails

2010-12-05 Thread Doron Shoham
Jeff Squyres wrote: On Nov 29, 2010, at 3:51 AM, Doron Shoham wrote: If only the PUT flag is set and/or the btl supports only PUT method then the sender will allocate a rendezvous header and will not eager send any data. The receiver will schedule rdma PUT(s) of the entire message. It is d

Re: [OMPI devel] Sending large messages over RDMA fails

2010-11-30 Thread Jeff Squyres
On Nov 29, 2010, at 3:51 AM, Doron Shoham wrote: > If only the PUT flag is set and/or the btl supports only PUT method then the > sender will allocate a rendezvous header and will not eager send any data. > The receiver will schedule rdma PUT(s) of the entire message. > It is done in mca_pml_ob1

[OMPI devel] Sending large messages over RDMA fails

2010-11-29 Thread Doron Shoham
Hi, The maximum message size of ConnectX HCAs is 1GB (older cards have a maximum of 2GB). Trying to send larger messages over RDMA direct protocol will fail. A reminder - RDMA direct will be used if RDMA writes or reads are allowed by |btl_openib_flags| and the sender's message is already