[openib-general] Re: uCM create connection ID
On Thu, Jun 30, 2005 at 07:01:22PM -0700, Arlin Davis wrote: Libor Michalek wrote: The listen id is in the req rcvd event. (event-param.req_rcvd.listen_id) Do you mean that it is not being set correctly? Ok, I didn't look deep enough. It is set correctly and the polling seems to be working. The uDAPL code is now connecting properly but I am having difficulty setting the QP states properly without the ib_cm_init_qp_attr() call. Any chance of providing this call in uCM? To recreate that call, I was going to expand the ib_cm_attr_id() call that's currently there, to retreive the available connection information, and then provide a ib_cm_init_qp_attr() like wrapper around it. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: uCM create connection ID
Libor Michalek wrote: Assume that the userspace 'struct ib_cm_event' contains the cm_id as well as a new 'u64 context' which is inherited from the cm_id, and is set at the time of the cm_id creation. This is what I'm assuming that Arlin would like to see. In the case of two threads accessing the CM at once there's a race condition if you are going to use the 'context' variable as a pointer to memory: Thread 1 Thread 2 - --- cm_object = malloc(sizeof(*cm_object) ib_cm_create_id(cm_object-cm_id, (u64)cm_object) ib_cm_event_get(event) ib_cm_destroy_id(cm_object-cm_id) free(cm_object); process_event((void *)event-context); I see. This appears to come from a difference between the event reporting model used by the kernel CM versus the usermode CM (callback versus calldown). Maybe there's a way to assist the user here. Can we report a destruction event, or require a second call to indicate that an event has been processed? In the latter case, destruction could block while the event is being processed. Not sure if either of these would help if the user processed events using multiple threads, but I think with additional serialization in the CM it might be able to work. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: uCM create connection ID
On Thu, Jun 30, 2005 at 09:13:28AM -0700, Sean Hefty wrote: Libor Michalek wrote: Assume that the userspace 'struct ib_cm_event' contains the cm_id as well as a new 'u64 context' which is inherited from the cm_id, and is set at the time of the cm_id creation. This is what I'm assuming that Arlin would like to see. In the case of two threads accessing the CM at once there's a race condition if you are going to use the 'context' variable as a pointer to memory: Thread 1 Thread 2 - --- cm_object = malloc(sizeof(*cm_object) ib_cm_create_id(cm_object-cm_id, (u64)cm_object) ib_cm_event_get(event) ib_cm_destroy_id(cm_object-cm_id) free(cm_object); process_event((void *)event-context); I see. This appears to come from a difference between the event reporting model used by the kernel CM versus the usermode CM (callback versus calldown). Do you block the destroy on a lock while a callback for that cm_id is active? I wouldn't say that the difference is attributed to callback vs. calldown, in both cases it's a matter of serializing the destroy with the event. Maybe there's a way to assist the user here. Can we report a destruction event, or require a second call to indicate that an event has been processed? A destruction event could work, but with some limits which might make it impracticle. The user would have to be really carefull not to do _anything_ with the object after calling destroy, and only cleanup in the same thread that is used to get the destroy completion event. The destroy completion event could be retreived and processed before the original destroy call returns. Also, the user would need to make sure that they are getting events in a _single_ thread, since multiple event get threads could pose the same problem as before. Blocking on the destroy seems like it could be error prone, that you could easily deadlock the user, who probably has a lock around the object which contains the cm_id... We could build the serialization table for the API consumer, have all cm_id calls and events go through a level of indirection in a table locked against multiple threads. This was the way we ended up doing it in our old code for the userCM that we used for uDAPL. I had left this out since it seems reasonable that not all apps would want/need this guarantee from the API, and that they could implement it themselves if they did want it... I could be wrong. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: uCM create connection ID
Libor Michalek wrote: The listen id is in the req rcvd event. (event-param.req_rcvd.listen_id) Do you mean that it is not being set correctly? Ok, I didn't look deep enough. It is set correctly and the polling seems to be working. The uDAPL code is now connecting properly but I am having difficulty setting the QP states properly without the ib_cm_init_qp_attr() call. Any chance of providing this call in uCM? Thanks, -arlin ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: uCM create connection ID
On Tue, Jun 28, 2005 at 01:59:01PM -0700, Arlin Davis wrote: Hi Libor, I have a couple of uCM questions regarding create_id and events... Is it possible for a consumer of uCM to provide a context with the create_id that could be returned with the event? I will have some scale up issues if I have to walk a list looking for a uCM provided connection ID instead of a context that could point directly to the appropriate uDAPL CM object. It would be easy to add in a context variable. I had left it out on purpose, since it's easy to get into a situation where using the context as a pointer you can end up referencing deallocated memory. However, I suppose it should be there for flexability. It would also be very helpful if the CM event could be woke from user space. You mean break CM event out of it's wait? I would instead recommend that you call poll on the file descriptor for read readiness, and only call for the CM event when there's an event available. Also, if you insist on using threads, tou can break out of poll by sending a signal to the thread that is waiting on the poll, which will return -1 with an errno of EINTR. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: uCM create connection ID
Libor Michalek wrote: Is it possible for a consumer of uCM to provide a context with the create_id that could be returned with the event? I will have some scale up issues if I have to walk a list looking for a uCM provided connection ID instead of a context that could point directly to the appropriate uDAPL CM object. It would be easy to add in a context variable. I had left it out on purpose, since it's easy to get into a situation where using the context as a pointer you can end up referencing deallocated memory. However, I suppose it should be there for flexability. Can you explain the situation where the application could reference deallocated memory? I would think that the uCM could take steps that would make it impossible for a well written app from doing this. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: uCM create connection ID
Libor Michalek wrote: On Tue, Jun 28, 2005 at 01:59:01PM -0700, Arlin Davis wrote: Hi Libor, I have a couple of uCM questions regarding create_id and events... Is it possible for a consumer of uCM to provide a context with the create_id that could be returned with the event? I will have some scale up issues if I have to walk a list looking for a uCM provided connection ID instead of a context that could point directly to the appropriate uDAPL CM object. It would be easy to add in a context variable. I had left it out on purpose, since it's easy to get into a situation where using the context as a pointer you can end up referencing deallocated memory. However, I suppose it should be there for flexability. Thanks. Also, in the case with listen's (conn_req), the event returns the new conn_id but not the original conn_id associated with the listen. I will need the listen conn_id to associate back to the proper listen service point. It would also be very helpful if the CM event could be woke from user space. You mean break CM event out of it's wait? I would instead recommend that you call poll on the file descriptor for read readiness, and only call for the CM event when there's an event available. Also, if you insist on using threads, tou can break out of poll by sending a signal to the thread that is waiting on the poll, which will return -1 with an errno of EINTR. Fair enough. Have you tried the polling method? I am having problems waking up on the event even though I see the following kernel message from ucm. kernel: UCM: Event. CM ID 10 event 1 -arlin -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: uCM create connection ID
On Wed, Jun 29, 2005 at 12:42:09PM -0700, Arlin Davis wrote: Libor Michalek wrote: On Tue, Jun 28, 2005 at 01:59:01PM -0700, Arlin Davis wrote: I have a couple of uCM questions regarding create_id and events... Is it possible for a consumer of uCM to provide a context with the create_id that could be returned with the event? I will have some scale up issues if I have to walk a list looking for a uCM provided connection ID instead of a context that could point directly to the appropriate uDAPL CM object. It would be easy to add in a context variable. I had left it out on purpose, since it's easy to get into a situation where using the context as a pointer you can end up referencing deallocated memory. However, I suppose it should be there for flexability. Thanks. Also, in the case with listen's (conn_req), the event returns the new conn_id but not the original conn_id associated with the listen. I will need the listen conn_id to associate back to the proper listen service point. The listen id is in the req rcvd event. (event-param.req_rcvd.listen_id) Do you mean that it is not being set correctly? It would also be very helpful if the CM event could be woke from user space. You mean break CM event out of it's wait? I would instead recommend that you call poll on the file descriptor for read readiness, and only call for the CM event when there's an event available. Also, if you insist on using threads, tou can break out of poll by sending a signal to the thread that is waiting on the poll, which will return -1 with an errno of EINTR. Fair enough. Have you tried the polling method? I am having problems waking up on the event even though I see the following kernel message from ucm. kernel: UCM: Event. CM ID 10 event 1 Yes, it should be working, for example I just tried this: struct pollfd ufds; int result; ufds.fd = ib_cm_get_fd(); ufds.events = POLLIN; ufds.revents = 0; *event = NULL; result = poll(ufds, 1, 10); if (!result) return -ETIMEDOUT; if (result 0) return result; return ib_cm_event_get(event); -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
