I've been looking into OpenMPI's support for RoCE (Mellanox's recent Infiniband-over-Ethernet) lately. While it's promising, I've hit a snag: RoCE requires lossless ethernet, and on my switches the only way to guarantee this is with CoS. RoCE adapters cannot emit CoS priority tags unless the client program selects an IB service level and uses a non-default GID.
There's a command-line option in OpenMPI to pick an IB SL, but I can't find one for picking a different GID. Does this exist for the openib btl? Or am I going about this the wrong way? -- Mike Shuey
