Final version of the case -- including a summary of the last parts of the discussion, about motivation, versioning and scope.
-ted Diffs: 11c11,14 < such as memory DR. --- > such as memory DR. In fact, one of the primary motivations of the work > is to enable fixing various IB bugs on large systems, where we need to > be careful about being in/out of the cage and the possible impact to > memory DR. 37c40,44 < the TI and CI as noted below. --- > the TI and CI as noted below. Each ABI version is incompatible with > each other and so this change will require a recompile of kernel ULPs, > framework and HCA drivers. All IB kernel components are delivered by > Sun engineering and are delivered together in our releases, so this > type of ABI change is not noticed by our customers. 38a46,49 > The scope of the project includes changes at the framework and HCA > driver level, which will be delivered together. The ULPs using these > new features, however, will be phased according to customer based > priorities. ==== A. Background The DDI distinguishes between different "types" of memory. Memory from ddi_dma_mem_alloc(9F) is usable for DMA and takes into account various factors such alignment and other device attributes. Memory from kmem_(z)alloc is not guaranteed to be usable for DMA, though most of the time it does work for that purpose, because of the capability of modern platforms. Nevertheless, there is value in maintaining these DDI distinctions, especially when considering certain platform issues such as memory DR. In fact, one of the primary motivations of the work is to enable fixing various IB bugs on large systems, where we need to be careful about being in/out of the cage and the possible impact to memory DR. In the context of InfiniBand, registered memory is the target of DMA operations. This case introduces new InfiniBand related interfaces analogous to the ddi_dma_mem_alloc family of functions to IBTF (InfiniBand Transport Framework, PSARC/2002/132 and follow-on cases). This addition will help the InfiniBand stack maintain the proper DDI memory distinctions important for certain types of platforms. B. Proposal The proposal is to make additions to the IBTF Channel and Transport interfaces. The functionality added to the Transport Interface (TI) is used by the ULPs to allocate memory suitable for DMA and IB memory registration. In turn, the framework uses new entry points in the Channel Interface (CI) to request memory allocation from the underlying HCA driver. These interfaces are basically a wrapper for DDI functions which on the one hand abstract away HCA device specific details at the ULP level, but at the same time allow for the HCA driver to adjust the memory attributes (alignment, etc.) as necessary for efficiency. These additions include an IBTF ABI change, so this case also marks an internal flag day, incrementing our interface version numbers for both the TI and CI as noted below. Each ABI version is incompatible with each other and so this change will require a recompile of kernel ULPs, framework and HCA drivers. All IB kernel components are delivered by Sun engineering and are delivered together in our releases, so this type of ABI change is not noticed by our customers. The scope of the project includes changes at the framework and HCA driver level, which will be delivered together. The ULPs using these new features, however, will be phased according to customer based priorities. All interface additions and changes in this proposal have a micro/patch binding. Transport Interface (ON Consolidation Private): ibt_alloc_io_mem() - Allocates DMA memory (at the transport level) ibt_free_io_mem() - Deallocates DMA memory IBTI_V3 - TI version change Channel Interface (ON Consolidation Private): ibc_alloc_io_mem() - Allocates DMA memory (at the HCA driver level) ibc_free_io_mem() - Deallocates DMA memory IBCI_V3 - CI version change C. Summary of Changes by man page See materials directory for copies of man pages. Modified man pages have change bars in the left margin. ibci.9 - modfied (new CI entry points added) ibc_alloc_io_mem.9e - new (alloc & free CI entry points) ibt_alloc_io_mem.9f - new (alloc & free TI functions) ibt_clnt_modinfo_t.9s - modified (IBTI_V3 version) ibc_hca_info_t.9s - modified (IBCI_V3 version) ibc_operations_t.9s - modified (new CI entry points added) -- Ted H. Kim Sun Microsystems, Inc. ted.kim at sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX
