This patch adds a new file to the kernel infiniband documentation directory to briefly describe how to use memory regions.
Note: I will be on vacation from Nov. 11 through Nov. 26. Signed-off-by: Ralph Campbell <[EMAIL PROTECTED]> diff -r b9d92097f918 Documentation/infiniband/memory_regions.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/Documentation/infiniband/memory_regions.txt Wed Nov 08 18:35:46 2006 -0800 @@ -0,0 +1,110 @@ +INFINIBAND MEMORY REGIONS + + This is an overview of memory region usage for the user and kernel + verbs interface. The verbs API to send and receive data does not + specify memory addresses directly. Instead, a memory region + is constructed and a Lkey or Rkey is used to refer to the region. + +User Space Memory Regions + + User space memory regions are created by calling ibv_reg_mr(). + It returns a pointer to a struct ibv_mr which contains the + 'lkey' field and 'rkey' field. The lkey should be copied + into the 'lkey' field of struct ibv_sge when posting buffers + with ibv_post_send(), ibv_post_recv(), and ibv_post_srq_recv(). + The 'addr' field of the ibv_sge should be a user address between + the address and address + length passed to ibv_reg_mr(). + + The 'rkey' can be sent to another process and used by the + remote process in RDMA write, read, and atomic operations + to access the local process' memory region. + The 'remote_addr' field in the ibv_send_wr should be the local + process' address within the memory region. At some point in + the future, the interface may be extended to allow zero based + remote addresses which would mean the remote_addr would be + an offset within the local process' memory region. + + A memory region is destroyed by calling ibv_dereg_mr(). + + Note that creating and destroying memory regions results + in kernel system calls which lock the user's virtual memory + to physical memory. This means the system administrator must set + the RLIMIT memory lock limit high enough for processes to + be able to create memory regions of the desired size. + It is therefore best to limit the size of memory regions created. + +Kernel Space Memory Regions + + ib_get_dma_mr() This function returns a pointer to struct ib_mr + which contains the 'lkey' and 'rkey' fields similar to user + memory regions. The memory region represents all of physical + memory so no base address or length is needed when creating it. + The addresses used for the 'addr' field of struct ib_sge need + to be hardware device addresses suitable for DMA. + Since this mapping may be device specific, there are a set + of kernel verbs functions corresponding to the DMA mapping + functions described in DMA-API.txt. Another useful reference + is the "Linux Device Drivers" book, 3rd edition, by Rubini and Corbet. + + ib_dma_mapping_error() + ib_dma_map_single() + ib_dma_unmap_single() + ib_dma_map_page() + ib_dma_unmap_page() + ib_dma_map_sg() + ib_dma_unmap_sg() + ib_sg_dma_address() + ib_sg_dma_len() + ib_dma_sync_single_for_cpu() + ib_dma_sync_single_for_device() + + Remote processes should use the same address for 'remote_addr' + as the local kernel's address as returned by the mapping functions + listed above. The only difference is the local kernel uses the + 'lkey' and the remote kernel uses the 'rkey'. + + Note that the mapped addresses need to be unmapped after they + are no longer needed. This may require the local and remote + kernels to pass messages at the middle or upper layers to + sychronize. + + ib_reg_phys_mr() This function returns a pointer to struct ib_mr. + It takes an array of device DMA addresses and lengths which are used + to describe the memory region. These addresses are created by + calling the mapping functions listed for ib_get_dma_mr(). + The 'iova' argument is the starting address of the memory region + which should be used with the 'lkey' or 'rkey' returned in the + struct ib_mr. + + ib_dereg_mr() is used to destroy memory regions created by + either ib_get_dma_mr() or ib_reg_phys_mr(). + + ib_alloc_fmr() This returns a pointer to a struct ib_fmr. + The struct ib_fmr_attr argument specifies the size of each + FMR "page" as a power of two in 'page_shift'. This size + is assumed by ib_map_phys_fmr() described below. + A FMR cannot be used until ib_map_phys_fmr() is called. + The 'lkey' and 'rkey' fields are defined in struct ib_fmr + and used the same way as the other memory regions. + + ib_map_phys_fmr() The function takes an array of u64 and a length + for the number of entries in the array. Each u64 value should be + a DMA address created with the mapping functions listed for + ib_get_dma_mr(). The length of each u64 address region is the + FMR page size set when ib_alloc_fmr() was called. + Note that this now defines the memory region to start at address + 'iova' and is the base address used for 'addr' and 'remote_addr'. + The size of the memory region is the array length times the + FMR page size. + + FMR memory regions should be unmapped by calling ib_unmap_fmr() + and then the ib_fmr destroyed by calling ib_dealloc_fmr(). + + See also ib_create_fmr_pool(), ib_fmr_pool_map_phys(), and + ib_fmr_pool_unmap() which are defined in the ib_core module + to assist in caching FMRs. This can help performance when + the same memory is mapped/unmapped frequently. + + Despite the name FMR, the memory region allocation and deallocation + functions perform very differently depending on device, processor, + and platform differences. _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
