Re: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
On Thu, Jun 17, 2010 at 05:53:58PM +0300, Yevgeny Petrilin wrote: > I think that this patch would do the job, (Is that an ack?) > Anyway we are thinking of ways to change our interrupt allocation scheme. > Would be interested to know what you've got in mind. -- Arthur -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
> The mlx4_core driver allocates 'nreq' msix vectors (and irqs), > where: > > nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs, > num_possible_cpus() + 1); > > ConnectX HCAs support 512 event queues (4 reserved). On a system with enough > processors, we get: > > mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors > available, trying again > > Further attempts (by other drivers) to allocate interrupts fail, because > mlx4_core got 'em all. > > How about this? Hi, I think that this patch would do the job, Anyway we are thinking of ways to change our interrupt allocation scheme. --Yevgeny-- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
On Sun, Jun 13, 2010 at 09:53:24AM +0300, Eli Cohen wrote: > > how many CPU cores are in your system? What kernel version did you > use? I'm almost certain that it was a 2048 core system (it's not available right now for me to verify). We used 2.6.32.12 (sles11 sp1). -- Arthur -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
On Thu, Jun 10, 2010 at 09:59:21AM -0700, Arthur Kepner wrote: > > The mlx4_core driver allocates 'nreq' msix vectors (and irqs), > where: > > nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs, >num_possible_cpus() + 1); > > ConnectX HCAs support 512 event queues (4 reserved). On a system > with enough processors, we get: > > mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors > available, trying again > > Further attempts (by other drivers) to allocate interrupts fail, > because mlx4_core got 'em all. > > How about this? > Hi Arthur, how many CPU cores are in your system? What kernel version did you use? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH/RFC] mlx4_core: module param to limit msix vec allocation
The mlx4_core driver allocates 'nreq' msix vectors (and irqs), where: nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs, num_possible_cpus() + 1); ConnectX HCAs support 512 event queues (4 reserved). On a system with enough processors, we get: mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again Further attempts (by other drivers) to allocate interrupts fail, because mlx4_core got 'em all. How about this? Signed-off-by: Arthur Kepner --- main.c |8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index e3e0d54..0a316d0 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -68,6 +68,10 @@ static int msi_x = 1; module_param(msi_x, int, 0444); MODULE_PARM_DESC(msi_x, "attempt to use MSI-X if nonzero"); +static int max_msi_x_vec = 64; +module_param(max_msi_x_vec, int, 0444); +MODULE_PARM_DESC(max_msi_x_vec, "max MSI-X vectors we'll attempt to allocate"); + #else /* CONFIG_PCI_MSI */ #define msi_x (0) @@ -968,8 +972,10 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev) int i; if (msi_x) { + nreq = min_t(int, num_possible_cpus() + 1, max_msi_x_vec); nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs, -num_possible_cpus() + 1); +nreq); + entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL); if (!entries) goto no_msi; -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html