On 06/27/13 23:24, David Dillow wrote:
On Wed, 2013-06-12 at 15:35 +0200, Bart Van Assche wrote:
Several InfiniBand HCA's allow to configure the completion vector
per queue pair. This allows to spread the workload created by IB
completion interrupts over multiple MSI-X vectors and hence over
multiple CPU cores. In other words, configuring the completion
vector properly not only allows to reduce latency on an initiator
connected to multiple SRP targets but also allows to improve
throughput.
Shouldn't there be some documentation about how to set this? What
happens when you try to set it on an HCA that doesn't support this how
does the user know what's wrong?
Hello Dave,
HCA drivers that support only one completion vector ignore the
comp_vector parameter. See e.g. c2_create_cq() in
drivers/infiniband/hw/amso1100/c2_provider.c for an example. The third
argument of that function, called "vector", is ignored.
What will happen if an out-of-range value is chosen for this parameter
depends on the HCA driver. This is how the mlx4 driver handles the
comp_vector parameter:
vector = dev->eq_table[vector % ibdev->num_comp_vectors];
So an out-of-range value for comp_vector won't cause a failure. And if
ever a HCA driver would be added to the kernel tree that causes
create_cq() to fail if the comp_vector parameter is out of range then
the ib_srp driver will propagate the error code returned by
ib_create_cq() to the process that wrote into the add_target sysfs
attribute.
The procedure for choosing a value for this parameter is as follows:
* Look up the interrupt numbers that have been assigned to each HCA
completion vector, e.g. via grep mlx4-ib- /proc/interrupts.
* Decide which CPU to bind each interrupt to and update
/proc/irq/$irq/smp_affinity.
* Decide which CPU and completion vector should be used for processing
SRP completions and specify that completion vector number in the SRP
login string.
To me this looks like a standard procedure that applies to any HCA that
supports multiple MSI-X vectors ?
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
