> In the applications I'm familiar with InfiniBand is being used not > only because of its low latency but also because of its high > throughput.
Yes, I seem to recall hearing that people care about throughput as well. > In order to handle such loads efficiently, interrupts have to be > spread over multiple CPUs. Let's look at what you say you want here: - strict in-order processing of completions - work spread across multiple CPUs Do you see that the two goals are contradictory? If you are running work on multiple CPUs in parallel, then there can't be an order assumed between CPUs -- otherwise you serialize the processing and lose all the benefit of parallelism. > Switching from a single receive queue to multiple receive queues is an > interesting alternative, but is not possible without changing the > communication protocol between client and server. Changing the > communication protocol is not always possible, especially when the > communication protocol has been defined by a standards organization. If you only have a single client talking to a single server over a single connection, then yes the opportunities for parallelism are limited. By the way, looking at VipCQNotify further, I'm not sure I follow exactly the race you're worried about. If you're willing to do your processing from the completion notification callback (which seems to be the approach that VipCQNotify forces), then doesn't the following (from Documentation/infiniband/core_locking.txt): The low-level driver is responsible for ensuring that multiple completion event handlers for the same CQ are not called simultaneously. The driver must guarantee that only one CQ event handler for a given CQ is running at a time. In other words, the following situation is not allowed: CPU1 CPU2 low-level driver -> consumer CQ event callback: /* ... */ ib_req_notify_cq(cq, ...); low-level driver -> /* ... */ consumer CQ event callback: /* ... */ return from CQ event handler mean that the problem you are complaining about doesn't actually exist? - R. -- Roland Dreier <rola...@cisco.com> || For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html