akepner> At least with the workload that we used to reproduce this akepner> bug, yes. (The workload was simply 2 ttcp processes, each akepner> placed on a different node of an Altix.) Without the akepner> mmiowb()s things would hang very reliably and very akepner> quickly (within a second). With the additional mmiowb() akepner> calls I never observed a problem after 10's of minutes.
OK, cool. Sounds convincing to me. BTW -- are there Altix systems with PCIe? Have you tested the mthca_arbel_xxx (mem-free PCIe HCA) changes, or just the mthca_tavor_xxx (PCI-X HCA) parts? akepner> I wanted to put it in the "if (likely(nreq))" block so akepner> that we don't do the mmiowb() unless it's really akepner> necessary. A very minor optimization (but a co-worker akepner> reports that it does produce a measurable, but small akepner> performance improvement.) I see -- the other mmiowb()s are next to the spin_unlock()s elsewhere because the other routines might ring doorbells during the loop if someone passes in a ton of work requests, right? (All the mmiowb()s look necessary to me but I'm just curious about the level of testing) I'm still a little puzzled by the fact that it affects performance, because that "if (likely(nreq))" is super-super-likely: under any normal workload, I would expect it always to be true. It's really strange that there's any difference between mmiowb(); qp->sq.next_ind = ind; qp->sq.head += nreq; and qp->sq.next_ind = ind; qp->sq.head += nreq; mmiowb(); Anyway, it all looks good. I'll apply your patch and submit it to -stable for 2.6.18.x. Thanks, Roland _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general