Re: [openib-general] Re: Re: RFC: e2e credits
Hal Rosenstock wrote: I'd be curious what the CM authors would think of this as to whether it is a clarification and their original intent or not. I doubt it. It sounds like this is intended to be an HCA wide attribute. C17-15.2.1 says: "HCA receive queues that are not associated with an SRQ shall generate E-to-E flow control credits." - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
On Tue, 2006-03-14 at 16:37, Michael S. Tsirkin wrote: > I thought about this, and I noted that with current CM > the flow control flag is already set by ULP, per connection. > I currently don't see this as spec violation: there's no compliance statement > that requires this bit to have the same value for all connections. The spec > says: > "Signifies whether the local CA actually implements End-to-End Flow Control > (1), or instead always advertises .invalid credits.(0)." > and we just take this meaning, with a clarification: "for this connection". I'd be curious what the CM authors would think of this as to whether it is a clarification and their original intent or not. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: [openib-general] Re: Re: RFC: e2e credits > > Michael S. Tsirkin wrote: > >"Signifies whether the local CA actually implements End-to-End Flow Control > >(1), or instead always advertises .invalid credits.(0)." > >and we just take this meaning, with a clarification: "for this connection". > > This sounds reasonable. > > >Anyway, we have two options: removing this field from CM/CMA, > >and adding it as a low-level driver module option, or adding > >it to ib_qp_init_attr? > > > >In the later case, I think we can avoid ABI breakage by packing this > >flag in unused space near other flags. > > > >Personally, I think the second option is the less disruptive: it moves the > >policy of using the hardware flow control out to the ULPs, even if I can't > >come > >up with an example where this is useful - there's close to zero overhead > >to have > >it either. > > My personal preference would be to have this policy controlled by the ULPs. > > - Sean OK. I'll write a patch. -- Michael S. Tsirkin Staff Engineer, Mellanox Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Michael S. Tsirkin wrote: "Signifies whether the local CA actually implements End-to-End Flow Control (1), or instead always advertises .invalid credits.(0)." and we just take this meaning, with a clarification: "for this connection". This sounds reasonable. Anyway, we have two options: removing this field from CM/CMA, and adding it as a low-level driver module option, or adding it to ib_qp_init_attr? In the later case, I think we can avoid ABI breakage by packing this flag in unused space near other flags. Personally, I think the second option is the less disruptive: it moves the policy of using the hardware flow control out to the ULPs, even if I can't come up with an example where this is useful - there's close to zero overhead to have it either. My personal preference would be to have this policy controlled by the ULPs. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: RFC: e2e credits
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: Re: RFC: e2e credits > > >> As far as I can tell, it doesn't do anything. Flow control is a parameter > >> of > >> the IB CM REQ/REP messages, so are exposed to the user. > >> > >> I don't see how a user can make use of it in any practical way, however. > >> It > >> isn't used to configure anything on the QP from what I can tell. > > > >So we can either add an ability to enable flow control per RQ, > >or remove this option altogether. > > I'm fine with doing this, but it requires changes to the spec wrt to the > definition of the flow control field in the CM messages. I thought about this, and I noted that with current CM the flow control flag is already set by ULP, per connection. I currently don't see this as spec violation: there's no compliance statement that requires this bit to have the same value for all connections. The spec says: "Signifies whether the local CA actually implements End-to-End Flow Control (1), or instead always advertises .invalid credits.(0)." and we just take this meaning, with a clarification: "for this connection". Anyway, we have two options: removing this field from CM/CMA, and adding it as a low-level driver module option, or adding it to ib_qp_init_attr? In the later case, I think we can avoid ABI breakage by packing this flag in unused space near other flags. Personally, I think the second option is the less disruptive: it moves the policy of using the hardware flow control out to the ULPs, even if I can't come up with an example where this is useful - there's close to zero overhead to have it either. And I think me and Roland share an aversion to module options - they tend to get out of hand pretty quickly. -- Michael S. Tsirkin Staff Engineer, Mellanox Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: RFC: e2e credits
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: Re: RFC: e2e credits > > >What does the flow_control parameter in ib_cm and rdma_cma conn_param do? > >If that's what it is, I think its easier to just add API to ib verbs > >making it possible to implement it than invent a device-specific way > >to do it per driver. > > As far as I can tell, it doesn't do anything. Flow control is a parameter of > the IB CM REQ/REP messages, so are exposed to the user. > > I don't see how a user can make use of it in any practical way, however. It > isn't used to configure anything on the QP from what I can tell. So we can either add an ability to enable flow control per RQ, or remove this option altogether. -- Michael S. Tsirkin Staff Engineer, Mellanox Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: Re: RFC: e2e credits
Sean Hefty wrote: >> What does the flow_control parameter in ib_cm and rdma_cma >> conn_param do? If that's what it is, I think its easier to just add >> API to ib verbs making it possible to implement it than invent a >> device-specific way to do it per driver. > > As far as I can tell, it doesn't do anything. Flow control > is a parameter of the IB CM REQ/REP messages, so are exposed to the > user. > > I don't see how a user can make use of it in any practical > way, however. It isn't used to configure anything on the QP > from what I can tell. This is why I mentioned that the only > use I can think of is for an app to change its algorithms > based on whether this value is set or not, but I have a hard > time imagining an app doing so. > > - Sean The approach taken in both DAT and IT-API is to provide the flow control information to both the Connection Manager *and* in the private data. IT-API even defines a standard pre-header for the Private Data over iWARP to carry this information. When the transport specific Connection Manager can use this data it does so directly. Otherwise the peer uses the information in the Private Data to configure/select its QP to match. With IT-API this is handled by middleware, while with DAT the application itself is expected to do this. An application can maintain transport neutrality by simply complying with the flow control limit at all times (in selection/configuration of the Endpoint and in how many unacknowledged Sends it generates) whether or not there is any hardware enforcement. The type of application that can use hardware flow control is indeed relatively rare. It is one that has no real need for explicit positive acknowledgement, and instead infers acceptance from a transport layer ack and a lack of a complaint/nack. Those applications *could* make use of such an indicator, by explicitly generating periodic acks in the absence of hardware flow control. Whether checking the flag is worthwhile, as opposed to always using explicit acks, would depend on the application. I would only rely on hardware flow control when my application had very specific needs that were not addressed by the more transport neutral logic. But as a very layer interface (as opposed to an application layer interface such as DAPL) making presumptions about what an application will find useful is something that should be kept to a minimum. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: Re: RFC: e2e credits
>What does the flow_control parameter in ib_cm and rdma_cma conn_param do? >If that's what it is, I think its easier to just add API to ib verbs >making it possible to implement it than invent a device-specific way >to do it per driver. As far as I can tell, it doesn't do anything. Flow control is a parameter of the IB CM REQ/REP messages, so are exposed to the user. I don't see how a user can make use of it in any practical way, however. It isn't used to configure anything on the QP from what I can tell. This is why I mentioned that the only use I can think of is for an app to change its algorithms based on whether this value is set or not, but I have a hard time imagining an app doing so. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: RFC: e2e credits
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: Re: RFC: e2e credits > > >> So, how about we add a flag to ib_qp_init_attr? > >> Something like > >>responder_invalid_credits > >> > >> Which will, for RC QPs, cause the responder to always > >> generate invalid credits, effectively disabling hardware E2E > >> flow control for this receive queue. ULPs will turn this on > >> if they do implement flow control in software and make sure that RNR > >> is never generated. > > At first glance, this sounds reasonable. This could end up breaking the > ABI somewhere (verbs, CM, CMA?), but I'm not sure. How does this map to > the IB CM flow control parameter? (I'm at a traveler's PC at the > moment, and don't have access to the specs.) Is the IB CM flow control > parameter actually used anywhere, or just exchanged by the remote sides? What does the flow_control parameter in ib_cm and rdma_cma conn_param do? If that's what it is, I think its easier to just add API to ib verbs making it possible to implement it than invent a device-specific way to do it per driver. -- Michael S. Tsirkin Staff Engineer, Mellanox Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Hal Rosenstock wrote: From an architectural view, this may require changing the meaning of the value carried in the IB CM messages from 'signifying whether the local CA actually implements flow control', to 'signifying whether the local QP will implement flow control'. Should this be an IBA spec comment as well ? It would need to be if this is the route that we wanted to go. At this point, I don't think that we're leaning that direction though. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
On Tue, 2006-03-07 at 16:20, Sean Hefty wrote: > From an architectural view, this may require > changing the meaning of the value carried in the IB CM messages from > 'signifying > whether the local CA actually implements flow control', to 'signifying > whether > the local QP will implement flow control'. Should this be an IBA spec comment as well ? -- Hal > - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
At 04:39 PM 3/7/2006, Michael S. Tsirkin wrote: >Anyway, sine ULPs don't seem to need it, another approach would be an option to >disable these flow controls globally, and probably add a module option >to enable >them back just in case. That's much simpler, isn't it? Thumbs up! If nothing uses them, why hang them around, enabled, just to cause problems? As an upper layer implementor, I sure don't want them in the way, nor do I want to add special code to turn something off I wasn't even aware was in the provider. Tom. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Michael S. Tsirkin wrote: Okay - I'm confused now. Why have this value at all? How does the statement above match up with C9-150.2.1: For QPs that are not associated with an SRQ, each HCA receive queue shall generate end-to-end flow control credits? Donnu. All this is a bit theoretical since all ULPs today always post receive WQEs before remote side posts sends. So they all can just safely disable this bit and avoid the chance of getting into limited state. Right now, I'm leaning towards the idea of disabling flow control by default, and enabling it by request. Are we sure that none of the ULPs (SDP, SRP, iSCSI, etc.) make use of this? Making this a QP tunable parameter seems more difficult. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: [openib-general] Re: Re: RFC: e2e credits > > Michael S. Tsirkin wrote: > >>The only way I can find for consumers to make use of this value is for > >>them to change their algorithms based on whether flow control is > >>supported or not. But it's not clear to me how the consumer determines > >>what value to set flow control to. > > > >Oh, if you always post receive WQE before remote side posts send WQEs, then > >you can disable flow control. SDP does this. > > But as you're trying to address, there's no way to disable flow control. > I.e. this value doesn't get associated with the QP. So, currently, the > only use that this value has is for the ULP to change its algorithm based > on whether or not it's set. Right. Even if its associated with the QP, note how this is a parameter of the remote peer's RQ, so you can't change it. > My interpretation of the spec is that the value carried in the CM message > applies to the HCA as a whole, such that all REQs originating from the same > HCA should have the same value. But I don't see how to determine what that > value should be. Right, but I don't see that it matters much. If you think per-QP is nicer we can change the meaning a bit. > >>>>9.7.7.2.4 > >>>> > >>>>Even a responder which does generate end-to-end credits may choose > >>>>to send the 'invalid' code in the AETH. > > > > > >So an application can't really depend on the e2e credits for correctness - > >its just an optimization hint. > > Okay - I'm confused now. Why have this value at all? How does the > statement above match up with C9-150.2.1: For QPs that are not associated > with an SRQ, each HCA receive queue shall generate end-to-end flow control > credits? Donnu. All this is a bit theoretical since all ULPs today always post receive WQEs before remote side posts sends. So they all can just safely disable this bit and avoid the chance of getting into limited state. > >Anyway, sine ULPs don't seem to need it, another approach would be an > >option to > >disable these flow controls globally, and probably add a module option to > >enable > >them back just in case. That's much simpler, isn't it? > > > >What do you think? > > Examining the spec, it looks like flow control is defined as a per HCA > feature. Do we need an ib_device_cap_flag to indicate if flow control is > enabled or not on the device? That could be fine, too. -- Michael S. Tsirkin Staff Engineer, Mellanox Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Michael S. Tsirkin wrote: The only way I can find for consumers to make use of this value is for them to change their algorithms based on whether flow control is supported or not. But it's not clear to me how the consumer determines what value to set flow control to. Oh, if you always post receive WQE before remote side posts send WQEs, then you can disable flow control. SDP does this. But as you're trying to address, there's no way to disable flow control. I.e. this value doesn't get associated with the QP. So, currently, the only use that this value has is for the ULP to change its algorithm based on whether or not it's set. My interpretation of the spec is that the value carried in the CM message applies to the HCA as a whole, such that all REQs originating from the same HCA should have the same value. But I don't see how to determine what that value should be. 9.7.7.2.4 Even a responder which does generate end-to-end credits may choose to send the 'invalid' code in the AETH. So an application can't really depend on the e2e credits for correctness - its just an optimization hint. Okay - I'm confused now. Why have this value at all? How does the statement above match up with C9-150.2.1: For QPs that are not associated with an SRQ, each HCA receive queue shall generate end-to-end flow control credits? Anyway, sine ULPs don't seem to need it, another approach would be an option to disable these flow controls globally, and probably add a module option to enable them back just in case. That's much simpler, isn't it? What do you think? Examining the spec, it looks like flow control is defined as a per HCA feature. Do we need an ib_device_cap_flag to indicate if flow control is enabled or not on the device? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: [openib-general] Re: Re: RFC: e2e credits > > Michael S. Tsirkin wrote: > >I think that the CM flow control parameter affects the requestor. > >This option is for responder. > > The CM flow control parameter indicates if the local HCA implements flow > control or always advertises invalid credits. It's carried in both the REQ > and REP messages, so we end up knowing the value used by both sides. (See > 12.7.26) > > The only way I can find for consumers to make use of this value is for them > to change their algorithms based on whether flow control is supported or > not. But it's not clear to me how the consumer determines what value to > set flow control to. Oh, if you always post receive WQE before remote side posts send WQEs, then you can disable flow control. SDP does this. But: >>> 9.7.7.2.4 >>> >>> Even a responder which does generate end-to-end credits may choose >>> to send the 'invalid' code in the AETH. So an application can't really depend on the e2e credits for correctness - its just an optimization hint. > I think that we need to relate anything that we do with flow control in > with the value exchanged as part of the IB CM protocol. (This value is > also exposed through the RDMA CM as well.) From an architectural view, > this may require changing the meaning of the value carried in the IB CM > messages from 'signifying whether the local CA actually implements flow > control', to 'signifying whether the local QP will implement flow control'. I guess its fine to do it this way - its just an optimization hint to the application. Anyway, sine ULPs don't seem to need it, another approach would be an option to disable these flow controls globally, and probably add a module option to enable them back just in case. That's much simpler, isn't it? What do you think? -- Michael S. Tsirkin Staff Engineer, Mellanox Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: Re: RFC: e2e credits
Michael S. Tsirkin wrote: I think that the CM flow control parameter affects the requestor. This option is for responder. The CM flow control parameter indicates if the local HCA implements flow control or always advertises invalid credits. It's carried in both the REQ and REP messages, so we end up knowing the value used by both sides. (See 12.7.26) The only way I can find for consumers to make use of this value is for them to change their algorithms based on whether flow control is supported or not. But it's not clear to me how the consumer determines what value to set flow control to. I think that we need to relate anything that we do with flow control in with the value exchanged as part of the IB CM protocol. (This value is also exposed through the RDMA CM as well.) From an architectural view, this may require changing the meaning of the value carried in the IB CM messages from 'signifying whether the local CA actually implements flow control', to 'signifying whether the local QP will implement flow control'. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: RFC: e2e credits
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: Re: RFC: e2e credits > > >> So, how about we add a flag to ib_qp_init_attr? > >> Something like > >>responder_invalid_credits > >> > >> Which will, for RC QPs, cause the responder to always > >> generate invalid credits, effectively disabling hardware E2E > >> flow control for this receive queue. ULPs will turn this on > >> if they do implement flow control in software and make sure that RNR > >> is never generated. > > At first glance, this sounds reasonable. This could end up breaking the > ABI somewhere (verbs, CM, CMA?), but I'm not sure. How does this map to > the IB CM flow control parameter? (I'm at a traveler's PC at the > moment, and don't have access to the specs.) Is the IB CM flow control > parameter actually used anywhere, or just exchanged by the remote sides? > > - Sean > (Sorry if this breaks threading.) I think that the CM flow control parameter affects the requestor. This option is for responder. -- Michael S. Tsirkin Staff Engineer, Mellanox Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
