------
As most of you know, Jon and I are working on the new openib "CPC" (connect pseudo-component) stuff in /tmp-public/openib-cpc2. There are two main reasons for it:
1. Add support for RDMA CM (they need it for iWarp support)2. Add support for IB CM (which will hopefully be a more scalable connect system as compared to the current RML/OOB-based method of making IB QPs)
When complete, there will be 4 CPCs: RDMA CM, IB CM, OOB, and XOOB (same as OOB but with ConnectX XRC extensions).
RDMA CM has some known scaling issues, and at least some known workarounds -- I won't discuss the merits/drawbacks of RDMA CM here. IB CM has unknown scaling characteristics, but seems to look good on paper (e.g., it uses UD for a 3-way handshake to make an IB QP).
On the trunk, it's a per-MPI process decision as to which CPC you'll use. Per ticket #1191, one of the goals of the /tmp-public branch is to make CPC decision be a per-openib-BTL-module decision. So you can mix iWarp and IB hardware in a single host, for example. This fits in quite well with the "mpirun should work out of the box" philosophy of Open MPI.
In the openib BTL, each BTL module is paired with a specific HCA/NIC (verbs) port. And depending on the interface hardware and software, one or more CPCs may be available for each. Hence, for each BTL module in each MPI process, we may send one or more CPC connect information blobs in the modex (note that the oob and xoob CPCs don't need to send anything additional in the modex).
Jon and I are actually getting closer to completion on the branch, and it seems to be working.
In conjunction with several other scalability discussions that are ongoing right now, several of us have toyed with two basic ideas to improve scalability of job launch / startup:
1. the possibility of eliminating the modex altogether (e.g., have ORTE dump enough information to each MPI process to figure out/ calculate/locally lookup [in local files?] BTL addressing information for all peers in MPI_COMM_WORLD, etc.), a la Portals.
2. reducing the amount of data in the modex.One obvious idea for #2 is to have only one process on each host send all/the majority of openib BTL modex information for that host. The rationale here is that all MPI processes on a single host will share much of the same BTL addressing information, so why send it N times? Local rank 0 can modex send all/the majority of the modex for the openib BTL modules; local ranks 1-N can either send nothing or a [very] small piece of differentiating information (e.g., IBCM service ID).
This effectively makes the modex info for the openib BTL scale with the number of nodes, not the number of processes. This can be a big win in terms of overall modex size that needs to be both gathered and bcast.
I worked up a spreadsheet showing the current size of the modex in the openib-cpc2 branch right now (using some "somewhat" contrived machine size/ppn/port combinations), and then compared it to the size after implementing the #2 idea shown above (see attached PDF).
I also included a 3rd comparison for if Jon/I are able to reduce the CPC modex blob sizes -- we don't know yet if that'll work or not. But the numbers show that reducing the blobs by a few bytes clearly has [much] less of an impact than the "eliminating redundant modex information" idea, so we'll work on that one first.
Additionally, reducing the modex size, paired with other ongoing ORTE scalability efforts, may obviate the need to eliminate the modex (at least for now...). Or, more specifically, efforts for eliminating the modex can be pushed to beyond v1.3.
Of course, the same ideas can apply to other BTLs. We're only working on the openib BTL for now.
-- Jeff Squyres Cisco Systems
modex-sizes.pdf
Description: Adobe PDF document
