I'm not sure I follow what a "completion channel" is. My understanding is that work completions are stored in user-accessible memory (typically a ring buffer). This enables fast-path reaping of work completions. The OS has no involvement unless notifications are enabled.
The "completion vector" is used to report completion notifications. So is the completion vector a *single* resource used by the driver/verbs to report completions, where said notifications are then split into user context dependent "completion channels"? The RDMAC verbs did not define callbacks to userspace at all. Instead it is assumed that the proxy for user mode services will receive the callbacks, and how it relays those notifications to userspace is outside the scope of the verbs. Both uDAPL and ITAPI define relays of notifications to AEVDS/CNOs and/or file descriptors. Forwarding a completion notification to userspace in order to make a callback in userspace so that it can kick an fd to wake up another thread doesn't make much sense. The uDAPL/ITAPI/whatever proxy can perform all of these functions without any device dependencies and in a way that is fully optimal for the usermode API that is being used. For kernel clients, I don't see any need for anything beyond the already defined callbacks direct from the device-dependent code. Even in the typical case where the usermode application does an evd_wait() on the DAT or ITAPI endpoint, the DAT/ITAPI proxy will be able to determine which thread should be woken and could even do so optimally. It also allows the proxy to implemenet Access Layer features such as EVD thresholding without device-specific support. > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Roland Dreier > Sent: Wednesday, September 21, 2005 12:22 PM > To: openib-general@openib.org > Subject: [openib-general] [RFC] libibverbs completion event handling > > While thinking about how to handle some of the issues raised > by Al Viro in <http://lkml.org/lkml/2005/9/16/146>, I > realized that our verbs interface could be improved to make > delivery of completion events more flexible. For example, > Arlin's request for using one FD for each CQ can be > accomodated quite nicely. > > The basic idea is to create new objects that I call > "completion vectors" and "completion channels." Completion > vectors refer to the interrupt generated when a completion > event occurs. With the current drivers, there will always be > a single completion vector, but once we have full MSI-X > support, multiple completion vectors will be possible. > Orthogonal to this is the notion of a completion channel. > This is a FD used for delivering completion events to userspace. > > Completion vectors are handled by the kernel, and userspace > cannot change the number of vectors that available. On the > other hand, completion channels are created at the request of > a userspace process, and userspace can create as many > channels as it wants. > > Every userspace CQ has a completion vector and a completion channel. > Multiple CQs can share the same completion vector and/or the > same completion channel. CQs with different completion > vectors can still share a completion channel, and vice versa. > > The exact API would be something like the below. Thoughts? > > Thanks, > Roland > > struct ibv_comp_channel { > int fd; > }; > > /** > * ibv_create_comp_channel - Create a completion event > channel */ extern struct ibv_comp_channel > *ibv_create_comp_channel(struct ibv_context *context); > > /** > * ibv_destroy_comp_channel - Destroy a completion event > channel */ extern int ibv_destroy_comp_channel(struct > ibv_comp_channel *channel); > > /** > * ibv_create_cq - Create a completion queue > * @context - Context CQ will be attached to > * @cqe - Minimum number of entries required for CQ > * @cq_context - Consumer-supplied context returned for > completion events > * @channel - Completion channel where completion events will > be queued. > * May be NULL if completion events will not be used. > * @comp_vector - Completion vector used to signal completion events. > * Must be >= 0 and < context->num_comp_vectors. > */ > extern struct ibv_cq *ibv_create_cq(struct ibv_context > *context, int cqe, > void *cq_context, > struct ibv_comp_channel *channel, > int comp_vector); > > /** > * ibv_get_cq_event - Read next CQ event > * @channel: Channel to get next event from. > * @cq: Used to return pointer to CQ. > * @cq_context: Used to return consumer-supplied CQ context. > * > * All completion events returned by ibv_get_cq_event() must > * eventually be acknowledged with ibv_ack_cq_events(). > */ > extern int ibv_get_cq_event(struct ibv_comp_channel *channel, > struct ibv_cq **cq, void > **cq_context); _______________________________________________ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general