Hi Matias, Relaxing the restriction on the number of ranks is definitively a good thing. The cost will be reflected on the number of communicators and tags, and we must be careful how we balance this.
Assuming context_id is the communicator cid, with 10 bits you can only support 1024. A little low, even lower than MVAPICH. The way we allocate cid is very sparse, and with a limited number of possible cid, we might run in troubles very quickly for the few applications that are using a large number of communicators, and for the resilience support. Yet another reason to revisit the cid allocation in the short term. Regarding the tag with your proposal the OFI MTL will support a wider range of tags than the OB1 PML, where we are limited to 16 bits. Just make sure you correctly expose your tag limit via the MPI_TAG_UB. I personally would prefer a solution where we can alter the distribution of bits between bits in the cid and tag at compile time. We can also envision this selection to be driven by an MCA parameter, but this might be too costly. George. On Sat, Mar 3, 2018 at 2:56 AM, Cabral, Matias A <matias.a.cab...@intel.com> wrote: > Hi all, > > > > I’m working on extending the OFI MTL to support FI_REMOTE_CQ_DATA (1) to > extend the number of ranks currently supported by the MTL. Currently > limited to only 16 bits included in the OFI tag (2). After the feature is > implemented there will be no limitation for providers that support > FI_REMOTE_CQ_DATA and FI_DIRECTED_RECEIVE (3). However, there will be a > fallback mode for providers that do not support these features and I would > like to get consensus on the default tag distribution. This is my proposal: > > > > * Default: No FI_REMOTE_CQ_DATA > > * 01234567 01| 234567 01234567 0123| 4567 |01234567 01234567 01234567 > 01234567 > > * context_id | source rank |proto| message > tag > > > > #define MTL_OFI_CONTEXT_MASK (0xFFC0000000000000ULL) > > #define MTL_OFI_SOURCE_MASK (0x003FFFF0000000000ULL) > > #define MTL_OFI_SOURCE_BITS_COUNT (18) /* 262,143 ranks */ > > #define MTL_OFI_CONTEXT_BITS_COUNT (10) /* 1,023 communicators */ > > #define MTL_OFI_TAG_BITS_COUNT (32) /* no restrictions */ > > #define MTL_OFI_PROTO_BITS_COUNT (4) > > > > Notes: > > - More ranks and fewer context ids than the current > implementation. > > - Moved the protocol bits from the most significant bits because > some providers may reserve starting from there (see mem_tag_format (4)) and > sync send will not work. > > > > Thoughts? > > > > Today we had a call with Howard (LANL), John and Hamuri (HPE) and briefly > talked about this, and also thought about sending this email as a query to > find other developers keeping an eye on OFI support in OMPI. > > > > Thanks, > > _MAC > > > > > > (1) https://ofiwg.github.io/libfabric/master/man/fi_cq.3.html > > (2) https://github.com/open-mpi/ompi/blob/master/ompi/mca/mtl/ > ofi/mtl_ofi_types.h#L70 > > (3) https://ofiwg.github.io/libfabric/master/man/fi_getinfo.3.html > > (4) https://ofiwg.github.io/libfabric/master/man/fi_endpoint.3.html > > > > > > > > > > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/devel >
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/devel
