RE: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
"Caitlin Bestler" <[EMAIL PROTECTED]> wrote on 13.12.2007 22:08:34: > To clarify, an FMR Work Request is simply posted to the SendQ like > any other Work Request (of course the QP has to be privileged, or > it will complete in error). An SQ Post should never block. This would require hardware support, wouldn't it? eHCA2 doesn't have this kind of support, so FMR WRs are not an option here. J. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
RE: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
> -Original Message- > From: Joachim Fenkes [mailto:[EMAIL PROTECTED] > Sent: Thursday, December 13, 2007 1:00 PM > To: Caitlin Bestler > Cc: Arnd Bergmann; [EMAIL PROTECTED]; OF-General; LKML; > linuxppc-dev@ozlabs.org; Or Gerlitz; Roland Dreier; Stefan Roscher > Subject: Re: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize > HCA-related hCalls on POWER5 > > [EMAIL PROTECTED] wrote on 13.12.2007 20:22:49: > > > On Dec 13, 2007 12:30 AM, Or Gerlitz <[EMAIL PROTECTED]> wrote: > > > The current implementation of the open iscsi initiator makes sure > to > > > issue commands in thread (sleepable) context, see iscsi_xmitworker > and > > > references to it in drivers/scsi/libiscsi.c , so this keeps ehca > users > > > safe for the time being. > > > I agree, *some* form of FMR support is important for iSER (and > probably > > for NFS over RDMA as well). Rather than adding a crippled NO FMR > > mode it would make more sense to add support for FMR Work Requests. > > I'm not certain what, if any, impact that would have on the Power5 > problem, > > but that's certainly a cleaner path for iWARP. > > Well, FMR WRs wouldn't change the eHCA issue -- the driver would have > to > make an hCall in any case, and the architecture says that the hCalls > used > in this scenario might return H_LONG_BUSY, causing the driver to sleep. > No > way around that. Because of this, eHCA's FMRs are actually standard MRs > with a different API. > > If, as Or said, the iSCSI initiator issues commands in sleepable > context > anyway, nothing would be lost by using standard MRs as a fallback > solution > if FMRs aren't available, would it? > To clarify, an FMR Work Request is simply posted to the SendQ like any other Work Request (of course the QP has to be privileged, or it will complete in error). An SQ Post should never block. But yes, if the current iSCSI initiator always does all call-based FMRs in a sleepable context then I would agree then any changes can wait for the first vendor that wants to support FMR Work Requests. FMR Work Requests can be pipelined, so anyone with hardware that supported them would have strong motivation to enable the open iSCSI initiator to take advantage of this. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
[EMAIL PROTECTED] wrote on 13.12.2007 20:22:49: > On Dec 13, 2007 12:30 AM, Or Gerlitz <[EMAIL PROTECTED]> wrote: > > The current implementation of the open iscsi initiator makes sure to > > issue commands in thread (sleepable) context, see iscsi_xmitworker and > > references to it in drivers/scsi/libiscsi.c , so this keeps ehca users > > safe for the time being. > I agree, *some* form of FMR support is important for iSER (and probably > for NFS over RDMA as well). Rather than adding a crippled NO FMR > mode it would make more sense to add support for FMR Work Requests. > I'm not certain what, if any, impact that would have on the Power5 problem, > but that's certainly a cleaner path for iWARP. Well, FMR WRs wouldn't change the eHCA issue -- the driver would have to make an hCall in any case, and the architecture says that the hCalls used in this scenario might return H_LONG_BUSY, causing the driver to sleep. No way around that. Because of this, eHCA's FMRs are actually standard MRs with a different API. If, as Or said, the iSCSI initiator issues commands in sleepable context anyway, nothing would be lost by using standard MRs as a fallback solution if FMRs aren't available, would it? J. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
On Dec 13, 2007 12:30 AM, Or Gerlitz <[EMAIL PROTECTED]> wrote: > Roland Dreier wrote: > > I think the right fix for iSER would be to make iSER work even for > > devices that don't support FMRs. For example cxgb3 doesn't implement > > FMRs so if anyone ever updates iSER to work on iWARP and not just IB, > > then this is something that has to be tackled anyway. Then ehca could > > just get rid of the FMR support it has. > > OK, The iSER design took into account the case of many initiators > running on strong/modern machines talking to possibly lightweight > embedded target for which the processing cost per I/O at the target side > should be minimized, that is at most --one-- RDMA operation should be > issued by the target to serve an I/O request. > > For that end, iSER works with one descriptor (called stag in iWARP and > rkey in IB) per I/O direction sent from the initiator to the target and > hence can't work without some sort of FMR implementation. > > The current implementation of the open iscsi initiator makes sure to > issue commands in thread (sleepable) context, see iscsi_xmitworker and > references to it in drivers/scsi/libiscsi.c , so this keeps ehca users > safe for the time being. > > Or. > I agree, *some* form of FMR support is important for iSER (and probably for NFS over RDMA as well). Rather than adding a crippled NO FMR mode it would make more sense to add support for FMR Work Requests. I'm not certain what, if any, impact that would have on the Power5 problem, but that's certainly a cleaner path for iWARP. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
Roland Dreier wrote: > I think the right fix for iSER would be to make iSER work even for > devices that don't support FMRs. For example cxgb3 doesn't implement > FMRs so if anyone ever updates iSER to work on iWARP and not just IB, > then this is something that has to be tackled anyway. Then ehca could > just get rid of the FMR support it has. OK, The iSER design took into account the case of many initiators running on strong/modern machines talking to possibly lightweight embedded target for which the processing cost per I/O at the target side should be minimized, that is at most --one-- RDMA operation should be issued by the target to serve an I/O request. For that end, iSER works with one descriptor (called stag in iWARP and rkey in IB) per I/O direction sent from the initiator to the target and hence can't work without some sort of FMR implementation. The current implementation of the open iscsi initiator makes sure to issue commands in thread (sleepable) context, see iscsi_xmitworker and references to it in drivers/scsi/libiscsi.c , so this keeps ehca users safe for the time being. Or. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
> What is the fix you suggest, to add a device query that tells you for > which verbs the documentation does not apply? or enhance the code of the > map_phys_fmr verb within the ehca driver to return error if called > from non-sleepable context? I think the right fix for iSER would be to make iSER work even for devices that don't support FMRs. For example cxgb3 doesn't implement FMRs so if anyone ever updates iSER to work on iWARP and not just IB, then this is something that has to be tackled anyway. Then ehca could just get rid of the FMR support it has. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
Joachim Fenkes wrote: > Roland Dreier <[EMAIL PROTECTED]> wrote on 10.12.2007 22:47:37: >> It's an optional device feature, so this should be OK >> (although the iSER driver currently seems to depend on a device >> supporting FMRs, which is probably going to be a problem with iWARP >> support in the future anyway). > I don't feel very well with removing code from the driver that iSER seems > to depend on. Are there plans to fix this in iSER? What is the fix you suggest, to add a device query that tells you for which verbs the documentation does not apply? or enhance the code of the map_phys_fmr verb within the ehca driver to return error if called from non-sleepable context? Or. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5
Or Gerlitz <[EMAIL PROTECTED]> wrote on 12.12.2007 13:14:25: > Joachim Fenkes wrote: > > Roland Dreier <[EMAIL PROTECTED]> wrote on 10.12.2007 22:47:37: > > >> It's an optional device feature, so this should be OK > >> (although the iSER driver currently seems to depend on a device > >> supporting FMRs, which is probably going to be a problem with iWARP > >> support in the future anyway). > > > I don't feel very well with removing code from the driver that iSER seems > > to depend on. Are there plans to fix this in iSER? > > What is the fix you suggest, to add a device query that tells you for > which verbs the documentation does not apply? or enhance the code of the > map_phys_fmr verb within the ehca driver to return error if called > from non-sleepable context? Roland, what is your suggestion here? We could implement both versions Or is proposing, but having both at the same time sound like overkill. Christoph R. ___ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
