Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI
This information is Copyright (c) 2010, Oracle and/or its affiliates. All
rights reserved.
1. Introduction
1.1. Project/Component Working Name:
IBTF 2010.Q2 Enhancements
1.2. Name of Document Author/Supplier:
Author: William Taylor
1.3 Date of This Document:
23 June, 2010
4. Technical Description
IBTF 2010.Q2 Enhancements
-------------------------
Table of Contents:
4.1 Background
4.2 Proposal
A. CQ Interrupt Enhancements
B. "DMA" Memory Regions
C. Zone Addressing
D. Corrected RDMA Limits
E. Kernel Map of User Buf Flag
F. Corrected Memory Management Error
G. "clone" Flag Restrictions
H. Version 4 ABI Change
4.3 Man Page Summary
4.1 Background
A number of upcoming InfiniBand projects require new capabilities from
our Solaris InfiniBand (IB) support. This case introduces new and
modified interfaces for our Solaris InfiniBand Transport Framework
(IBTF, PSARC/2002/132 and follow-on cases). The main subjects of this
case include: CQ interrupt enhancements, "DMA" memory regions and zone
addressing. In addition, these minor items are also included:
corrected RDMA limits, kernel map of user bufs flag, corrected memory
management error, "clone" flag restrictions, and the version 4 ABI
change for IBTF.
The first consumer for CQ interrupt enhancements is expected to be
RDSv3 (PSARC/2010/043). The consumer of "DMA" memory regions is
Lustre. Zones addressing will initially be consumed by OFUV Kernel
Components (PSARC/2009/421).
4.2 Proposal
The proposal is to make additions to the IBTF Transport Interface (TI)
for IB Upper Level Protocol (ULP) clients and the Channel Interface
(CI) for HCA drivers.
All interface changes and additions in the proposal have a micro/patch
binding.
Transport Interface (ON Consolidation Private)
ibt_free_cq_sched(): revise arguments
ibt_free_srcip_info(): new function to free ibt_srcip_info_t results
ibt_get_src_ip(): revise arguments
ibt_query_cq_handler_id(): new function to query CQ handler ID attributes
ibt_register_dma_mr(): new function for defining DMA MR
ibt_alt_ip_path_attr_t: add new zoneid_t field
ibt_cq_attr_t: add cq_hid field
ibt_cq_flags_t: add IBT_CQ_HID flag
ibt_cq_handler_attr_t: output for ibt_query_cq_handler_id()
ibt_cq_sched_attr_t: new input struct to ibt_alloc_cq_sched()
ibt_dmr_attr_t: new struct to define DMA MR attributes
ibt_hca_attr_t: add hca_dip, hca_conn_rdma_sgl_sz,
hca_conn_rdma_write_sgl_sz fields
ibt_hca_flags2_t: add IBT_HCA2_DMA_MR
ibt_iov_attr_t: add iov_alt_lkey field
ibt_iov_flags_t: add IBT_IOV_ALT_LKEY, IBT_IOV_USER_BUF flags
ibt_ip_path_attr_t: add new zoneid_t field
ibt_mr_flags_t: add IBT_MR_USER_BUF
ibt_va_flags_t: add IBT_VA_USER_BUF
ibt_scrip_attr_t: input to ibt_get_src_ip()
ibt_srcip_info_t: output of ibt_get_src_ip()
ibt_status_t: add IBT_CQ_SCHED_INVALID, IBT_CQ_NO_SCHED_GROUP,
IBT_CQ_HID_INVALID, IBT_WC_MEM_MGT_OP_ERR enum values
ibt_version_t: add IBTI_V4
ibt_wc_status_t: add IBT_WC_MEM_MGT_OP_ERR
Channel Interface (ON Consolidation Private)
ibc_operations_t: add ibc_query_cq_handler_id and ibc_register_dma_mr,
revise ibc_alloc_cq_sched and ibc_free_cq_sched
ibc_sched_hdl_t: new type for CQ sched object
ibt_cq_attr_t: add cq_hid field
ibt_cq_flags_t: add IBT_CQ_HID flag
ibt_cq_handler_attr_t: output for ibc_query_cq_handler_id
ibt_cq_sched_attr_t: new input struct to ibc_alloc_cq_sched
ibt_dmr_attr_t: new struct to define DMA MR attributes
ibt_hca_attr_t: add hca_dip, hca_conn_rdma_sgl_sz,
hca_conn_rdma_write_sgl_sz fields
ibt_hca_flags2_t: add IBT_HCA2_DMA_MR
ibt_iov_attr_t: add iov_alt_lkey field
ibt_iov_flags_t: add IBT_IOV_ALT_LKEY, IBT_IOV_USER_BUF flags
ibt_mr_flags_t: add IBT_MR_USER_BUF
ibt_va_flags_t: add IBT_VA_USER_BUF
ibt_status_t: add IBT_CQ_SCHED_INVALID, IBT_CQ_NO_SCHED_GROUP,
IBT_CQ_HID_INVALID, IBT_WC_MEM_MGT_OP_ERR enum values
ibc_version_t: add IBCI_V4
ibt_wc_status_t: add IBT_WC_MEM_MGT_OP_ERR
Because these changes cause binary incompatibility, we are also
incrementing our IBTF binary interface version number to four.
Copies of all new and modified man pages are in the materials
directory (see section 4.3 below).
A. CQ Interrupt Enhancements
PSARC/2009/060 introduced the IB version of MSI-X ("Multiple
Completion Handlers") to support RSS. The new modifications add the
ability to specify a particular interrupt or "group" and exposing DDI
interrupt information.
Previously, the assignment of MSI-X interrupts was not explicit and
implementation dependent (the implementation was using round robin
assignment of available HCA interrupts). But this can lead to
unintended (and random) sharing of interrupts between unrelated CQs
and applications. We propose two ways to improve the situation.
First, we add a way to specify the particular vector to use when
creating a CQ. Using this mechanism allows certain related functions
(e.g. send and receive of same QP) to be handled on the same interrupt
without requiring the CQs (and the handling code) to be combined.
Add flag and field to ibt_cq_attr_t:
IBT_CQ_HID = 1 << 3 /* new flag in ibt_cq_flags_t */
ibt_cq_handler_id_t cq_hid; /* new field in ibt_cq_attr_t */
Add error code to ibt_status_t:
IBT_CQ_HID_INVALID = 552, /* CQ Handler ID invalid */
The second mechanism allows one to specify assigning an interrupt
from a pre-reserved set ("group"). This feature provides a way to
separate interrupts from different ULPs. The interface for this
feature is a modification of the previously defined (but unused),
"CQ sched" functions.
Add new flags to ibt_cq_sched_flags_t:
/* if named group not found: */
IBT_CQS_EXACT_SCHED_GROUP = 1 << 1, /* return error */
IBT_CQS_SCHED_GROUP = 1 << 2, /* return default group instead */
Revise ibt_cq_sched_attr_t:
ibt_cq_sched_flags_t cqs_flags;
char *cqs_group_name;
Revise function signatures:
ibt_status_t ibt_free_cq_sched(ibt_hca_hdl_t hca_hdl, /* TI */
ibt_sched_hdl_t sched_hdl);
ibt_status_t (*ibc_alloc_cq_sched)(ibc_hca_hdl_t hca, /* CI */
ibt_cq_sched_attr_t *attr, ibc_sched_hdl_t *sched_hdl_p);
ibt_status_t (*ibc_free_cq_sched)(ibc_hca_hdl_t hca,
ibc_sched_hdl_t sched_hdl);
Add error codes to ibt_status_t:
IBT_CQ_SCHED_INVALID = 550, /* Invalid CQ Sched Handle */
IBT_CQ_NO_SCHED_GROUP = 551, /* Schedule group not found */
We also provide a function to provide the DDI information for
particular interrupts used by CQ handling:
Add the dev_info_t in ibt_hca_attr_t:
dev_info_t *hca_dip;
Add new ibt_cq_handler_attr_t:
dev_info_t *cha_dip;
ibt_intr_handle_t cha_ih;
Add new functions:
ibt_status_t ibt_query_cq_handler_id(ibt_hca_hdl_t hca_hdl, /* TI */
ibt_cq_handler_id_t hid, ibt_cq_handler_attr_t *attrs);
ibt_status_t (*ibc_query_cq_handler_id)(ibc_hca_hdl_t hca, /* CI */
ibt_cq_handler_id_t hid, ibt_cq_handler_attr_t *attrs);
Changed man pages: ibci.9, ibti.9, ibc_alloc_cq.9e,
ibc_alloc_cq_sched.9e, ibc_modify_cq.9e, ibc_query_cq.9e,
ibc_query_cq_handler_id.9e, ibt_alloc_cq.9f, ibt_alloc_cq_sched.9f,
ibt_modify_cq.9f, ibt_query_cq.9f, ibt_query_cq_handler_id.9f,
ibc_operations_t.9s, ibt_cq_sched_attr_t.9s, ibt_hca_attr_t.9s
B. "DMA" Memory Regions
Linux (OFED) has long provided a way to define a memory region which
can use physical addresses (ib_get_dma_mr). We now provide a similar
API to aid in the porting of Lustre from Linux.
Add capability flag to ibt_hca_flags2_t to indicate support:
IBT_HCA2_DMA_MR = 1 << 11 /* DMA MR */
Add operation to define memory region by physical address and length:
ibt_status_t ibt_register_dma_mr(ibt_hca_hdl_t hca_hdl, /* TI */
ibt_pd_hdl_t pd, ibt_dmr_attr_t *mem_attr,
ibt_mr_hdl_t *mr_hdl_p, ibt_mr_desc_t *mem_desc);
ibt_status_t (*ibc_register_dma_mr)(ibc_hca_hdl_t hca, /* CI */
ibc_pd_hdl_t pd, ibt_dmr_attr_t *attr_p, void *ibtl_reserved,
ibc_mr_hdl_t *mr_p, ibt_mr_desc_t *mem_desc);
Add ability to use other L_Keys (i.e. from register_dma_mr) with
map_mem_iov mapping function:
IBT_IOV_ALT_LKEY = (1 << 3) /* new flag in ibt_iov_flags_t */
ibt_lkey_t iov_alt_lkey; /* new field in ibt_iov_attr_t */
Changed man pages: ibci.9, ibti.9, ibc_map_mem_iov.9e,
ibc_register_dma_mr.9e, ibt_map_mem_iov.9f, ibt_register_dma_mr.9f,
ibc_operations_t.9s, ibt_dmr_attr_t.9s, ibt_hca_attr_t.9s
C. Zone Addressing
This first increment of zones capability for IB concentrates on IP
addressing issues when multiple IP stacks exist. When querying for an
IB path by IP address, we add information about the zone ID (so we can
use the correct IP stack).
New fields for zoneid_t in path queries:
zoneid_t ipa_zoneid; /* added to ibt_ip_path_attr_t */
zoneid_t apa_zoneid; /* added to ibt_alt_ip_path_attr_s */
When dispatching connection requests, we also need to distinguish
among zones which have IPoIB instances using the same port/P_Key
combination. To do this, we revise ibt_get_src_ip() to also allow zone
information on input and output:
typedef struct ibt_srcip_attr_s {
ib_gid_t sip_gid; /* REQUIRED: Local Port GID */
zoneid_t sip_zoneid; /* Zero means Global Zone */
ib_pkey_t sip_pkey; /* Optional */
sa_family_t sip_family; /* Optional : IPv4 or IPv6 */
} ibt_srcip_attr_t;
/* ip_flag : Flag to indicate whether list has any duplicate records. */
#define IBT_IPADDR_NO_FLAGS 0
#define IBT_IPADDR_DUPLICATE 1
typedef struct ibt_srcip_info_s {
ibt_ip_addr_t ip_addr;
zoneid_t ip_zoneid; /* ZoneId of this ip-addr */
uint_t ip_flag; /* Flag to indicate any gotchas */
} ibt_srcip_info_t;
ibt_status_t ibt_get_src_ip(ibt_srcip_attr_t *srcip_attr,
ibt_srcip_info_t **src_info_p, uint_t *entries_p);
void ibt_free_srcip_info(ibt_srcip_info_t *src_info, uint_t entries);
Changes man pages: ibti.9, ibt_get_src_ip.9f,
ibt_alt_ip_path_attr_t.9s, ibt_ip_path_attr_t.9s
D. Corrected RDMA Limits
Previously defined "detailed" WQE sizes in PSARC/2008/726 did not take
into account adapter RDMA-Read limits. We remove the misleading RDMA
"sgl overhead" quantity (hca_conn_rdma_sgl_overhead) and just give
explicit RDMA-Read and RDMA-Write SGL limits. Added to ibt_hca_attr_t:
uint_t hca_conn_rdma_read_sgl_sz; /* max RDMA-R SGL len */
uint_t hca_conn_rdma_write_sgl_sz; /* max RDMA-W SGL len */
Changed man pages: ibt_hca_attr_t.9s
E. Kernel Map of User Buf Flag
When a "buf" is registered, it might refer to an area of kernel or
user address space. We add an explicit flag for the case of user
address space so platform optimizations can be done during
registration (e.g. turn on relaxed ordering). Added to mr_flags in
ibt_smr_attr_t:
IBT_IOV_USER_BUF = (1 << 3) /* added to ibt_iov_flags_t */
IBT_MR_USER_BUF = (1 << 15) /* added to ibt_mr_flags_t */
IBT_VA_USER_BUF = (1 << 6) /* added to ibt_va_flags_t */
Changed man pages: ibc_map_mem_area.9e, ibc_map_mem_iov.9e,
ibt_map_mem_area.9f, ibt_map_mem_iov.9f, ibt_smr_attr_t.9s
F. Corrected Memory Management Error
In the IB spec Release 1.2, the meaning of the "bind" memory window
completion error was generalized as the "memory management"
error. This increased scope allowed the error to include cases such as
problems with memory registration by WR (PSARC/2009/060).
Rename error as "Memory management operation" error:
#define IBT_WC_MEM_MGT_OP_ERR 15 /* bind plus 1.2 memory extensions */
But also provide an alternate name for backward compatibility:
#define IBT_WC_MEM_WIN_BIND_ERR IBT_WC_MEM_MGT_OP_ERR
Changed man pages: ibc_wc_status_t.9s, ibt_wc_status_t.9s
G. "clone" Flag Restrictions
When allocation of QP ranges was defined in PSARC/2009/060 most of the
attributes were inherited from the definition of the single QP
allocation. However, we have decided not allow the IBT_ACHAN_CLONE
flag for QP range allocation (which copied the attributes from another
QP). We haven't come up with a real need for this flag.
Changed man pages: ibt_alloc_ud_channel_range.9f
H. Version 4 ABI Change
Because of the binary changes in this case, our interface version is
incremented to version 4.
Change the TI version number in enum ibt_version_e (ibt_version_t):
IBTI_V4 = 4 /* TI interface version */
Change the CI version number in enum ibc_version_e (ibc_version_t):
IBCI_V4 = 4 /* CI interface version */
Changed man pages: ibc_hca_info_t.9s, ibt_clnt_modinfo_t.9s
4.3 Man Page Summary
Man Page Disposition Reasons for change
(sorted by section) (section of 4.2)
------------------------------------------------------------------
ibci.9 changed A, B
ibti.9 changed A, B, C
ibc_alloc_cq.9e changed A
ibc_alloc_cq_sched.9e changed A
ibc_map_mem_area.9e changed E
ibc_map_mem_iov.9e changed B, E
ibc_modify_cq.9e changed A
ibc_query_cq.9e changed A
ibc_query_cq_handler_id.9e new A
ibc_register_dma_mr.9e new B
ibt_alloc_cq.9f changed A
ibt_alloc_cq_sched.9f changed A
ibt_alloc_ud_channel_range.9f changed G
ibt_get_src_ip.9f changed C
ibt_map_mem_area.9f changed E
ibt_map_mem_iov.9f changed B, E
ibt_modify_cq.9f changed A
ibt_query_cq.9f changed A
ibt_query_cq_handler_id.9f new A
ibt_register_dma_mr.9f new B
ibc_hca_info_t.9s changed H
ibc_operations_t.9s changed A, B
ibc_wc_status_t.9s changed F
ibt_alt_ip_path_attr_t.9s changed C
ibt_clnt_modinfo_t.9s changed H
ibt_cq_sched_attr_t.9s new A
ibt_dmr_attr_t.9s new B
ibt_hca_attr_t.9s changed A, B, D
ibt_ip_path_attr_t.9s changed C
ibt_smr_attr_t.9s changed E
ibt_wc_status_t.9s changed F
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
ON
6.5. ARC review type: FastTrack
6.6. ARC Exposure: open
_______________________________________________
opensolaris-arc mailing list
opensolaris-arc@opensolaris.org
