You are right to assume that all connections on the server side are put in a single protection domain, so they share memory registration done in that protection domain, consequently accesses as well. Multiple protection domains within a single server-process on a NIC would have been another design, but with the current design it is meant to be deployed/developed as what Patrick described.
Cheers, -- Animesh On Mon, Mar 25, 2019 at 9:56 PM Patrick Stuedi <[email protected]> wrote: > Hi William, > > Not sure I completely understand the question but here is an to what I > think you're asking. A storage server is always bound to one NIC, so one > protection domain. You can start multiple storage servers per node, for > instance one per NIC (or any number of storage servers you like) which > would then involve different protection domains. Running multiple storage > servers per node is a perfectly valid configuration, Crail storage servers > are designed to be "micro-servers" and are meant to be deployed like this. > > From the moment the storage server registers a region with the metadata > server, the region itself looses any meaning. Regions are broken into > blocks, and blocks are assigned to files and different clients my read or > write different files concurrently. > > -Patrick > > > > On Mon, Mar 25, 2019 at 7:53 PM West Lafayette L <[email protected]> > wrote: > > > Hello Patrick, > > > > Thank you for your detailed explanation. My question actually focuses on > > the storage server side. > > > > I am also aware that multiple memory registrations will take significant > > memory space for the state on the RDMA NIC and it will also increase the > > message exchange between the storage server and the metadata server. > > > > More specifically, my question is does Crail register a Crail region > > (assume crail.regionsize = 1GB) with different RDMA memory regions > (IbvMr) > > for different clients? In Crail, a storage server will host connections > to > > multiple clients, and clients use these connections to issue RDMA > requests > > to get data from Crail storage server. Does Crail > > 1. use different RDMA memory regions (IbvMr) for single Crail region? > (put > > each connection under different protection domains, register the single > > Crail region by N times to get N different RDMA memory regions, and give > > different connections different memory regions) > > 2. or it's a kind of shared memory region idea? (put all connections > under > > the same protection domain, register the Crail region once, and share the > > single RDMA memory region between different connections which can keep > > state on the NIC small)? > > Based on code study and discussion, I suppose it's close to 2. > > > > Again, many thanks. > > > > William > > > > On Mon, Mar 25, 2019 at 2:42 AM Patrick Stuedi <[email protected]> > wrote: > > > > > HI William, > > > > > > You have to differentiate the server side registration from the client > > side > > > registration. The links above are server side. There we allocate memory > > in > > > larger segments (defined by the config variable crail.allocationSize). > > > AllocationSize must be a multiples of crail.bufferSize which is the > basic > > > allocatoin unit for files. Reason we choose to allocate larger segments > > is > > > to keep the state on the NIC small (each registration consumes state), > > and > > > to amortize the regMr calls, but also to minimize the number of > messages > > > between the storage server and the metadata server (a storage server > > needs > > > to inform the metadata server about each allocated segment). > > > > > > On the client side we allocate memory that is used for buffered > streams. > > > This memory is also allocated in larger segments for the same reason to > > > keep the registration state on the NIC small and to amortize the > overhead > > > of regMr calls. We further try to use huge pages for both client and > > server > > > side which further reduces the state on the NIC. > > > > > > Let me know if you need further information. > > > > > > -Patrick > > > > > > > > > On Mon, Mar 25, 2019 at 12:18 AM [email protected] < > [email protected]> > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > I am learning Crail and RDMA implementation. I have a question > > regarding > > > > RDMA + Crail and I was hoping you could help me figure it out. In > > RDMA, a > > > > memory space is required to be registered with a protection domain to > > > get a > > > > memory region (IbvMr) and respected rkey. However, each registration > > will > > > > take extra overhead. > > > > > > > > After studying Crail code, I found it seems Crail only registers > memory > > > > space once with a single protection domain to get a single memory > > region > > > > and save registration overhead, and all incoming > > > connections/storageClients > > > > share the same memory space, memory region, and protection domain > with > > > the > > > > same RDMA privilege. Did I understand this correctly? > > > > > > > > Reference code in Crail: > > > > 1 [create endpoint]. > > > > > > > > > > https://github.com/apache/incubator-crail/blob/v1.1/storage-rdma/src/main/java/org/apache/crail/storage/rdma/RdmaStorageServer.java#L68 > > > > 2 [register memory]. > > > > > > > > > > https://github.com/apache/incubator-crail/blob/v1.1/storage-rdma/src/main/java/org/apache/crail/storage/rdma/RdmaStorageServer.java#L112 > > > > > > > > William > > > > > > > > > >
