On Sat, 2016-10-08 at 19:35 +0200, Hannes Reinecke wrote: > You might actually be hitting a limitation in the exchange manager code. > The libfc exchange manager tries to be really clever and will assign a > per-cpu exchange manager (probably to increase locality). However, we > only have a limited number of exchanges, so on large systems we might > actually run into a exchange starvation problem, where we have in theory > enough free exchanges, but none for the submitting cpu. > > (Personally, the exchange manager code is in urgent need of reworking. > It should be replaced by the sbitmap code from Omar). > > Do check how many free exchanges are actually present for the stalling > CPU; it might be that you run into a starvation issue.
We are still looking into this but one thing that looks bad is that the exchange manager code rounds up the number of CPUs to the next power of 2 before dividing up the exchange id space (and uses the lsbs of the xid to extract the CPU when looking up an xid). We have a machine with 288 CPUs, this code is just begging for a rewrite as it looks to be wasting most of the limited xid space on ixgbe FCoE. Looks like we get 512 offloaded xids on this adapter and 4096-512 non-offloaded xids. This would give 1 + 7 xids per CPU. However, I'm not sure that even 4096 / 288 = 14 would be enough to prevent stalling. And, of course, potentially most of the CPUs aren't submitting I/O, so the whole idea of per-CPU xid space is questionable. -Ewan -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
