Re: rmmod mlx4_core panic 3.16-rc1
On Fri, Jun 20, 2014 at 6:34 AM, Or Gerlitz or.gerl...@gmail.com wrote:
On Thu, Jun 19, 2014 at 6:33 AM, Shirley Ma shirley...@oracle.com wrote:
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it
failed with mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs with
IB port - single port VFs syntax is only supported when all ports are
configured as ethernet
What do you mean by port1 and port2 -- can you give the exact
command line you used?
Single ported VFs are currently supported for Ethernet only
configuration, that is not for only IB nor for VPI, that is only if
you use port_type_arrary=2,2
Note that you can still use dual-ported VFs, for both IB, Ethernet and
VPI, that is
num_vfs=N will create N dual-ported VFs, are you on IB?
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
On Fri, Jun 20, 2014 at 07:02:41AM +0300, Or Gerlitz wrote: On Fri, Jun 20, 2014 at 6:51 AM, Wei Yang weiy...@linux.vnet.ibm.com wrote: mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet BUG: unable to handle kernel NULL pointer dereference at 038c IP: [a0350450] __mlx4_remove_one+0x20/0x380 [mlx4_core] From this log, it happens during probe? If not, any action after probe? yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it I tried this modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 It looks good to me. BTW, I didn't test 3.16-rc1, sine the SRIOV patch on power platform is not rebased to the latest kernel yet. PGD 45d3ba067 PUD 45ace8067 PMD 0 Oops: [#1] SMP DEBUG_PAGEALLOC Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas scsi_transport_sas raid_class [last unloaded: mlx4_core] CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1 Hardware name: Oracle Corporation SUN FIRE X4170 M3 /ASSY,MOTHERBOARD,1U , BIOS 17050100 08/29/2013 task: 880461540110 ti: 88046500 task.ti: 88046500 RIP: 0010:[a0350450] [a0350450] __mlx4_remove_one+0x20/0x380 [mlx4_core] RSP: 0018:880465003d88 EFLAGS: 00010296 RAX: 0001 RBX: RCX: RDX: 0026 RSI: 0292 RDI: 880468b8f000 RBP: 880465003db8 R08: R09: R10: 09f911029d74e35b R11: 09f911029d74e35b R12: R13: 880468b8f000 R14: a036de40 R15: 0001 FS: 7ff287fc2700() GS:88046fce() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 038c CR3: 00045cfae000 CR4: 000407e0 Stack: 880465003da8 880468b8f000 880468b8f000 a036de40 0001 880465003dd8 a0350805 880468b8f098 a036dd60 880465003e08 812ebaa6 Call Trace: [a0350805] mlx4_remove_one+0x25/0x50 [mlx4_core] [812ebaa6] pci_device_remove+0x46/0xc0 [813ce08f] __device_release_driver+0x7f/0xf0 [813ce1c8] driver_detach+0xc8/0xd0 [813cced9] bus_remove_driver+0x59/0xd0 [813cef80] driver_unregister+0x30/0x70 [812ebc13] pci_unregister_driver+0x23/0x80 [a03650e4] mlx4_cleanup+0x10/0x1e [mlx4_core] [810ceff9] SyS_delete_module+0x189/0x210 [815d2f12] system_call_fastpath+0x16/0x1b Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 9f 58 01 00 00 49 89 fd 44 8b b3 8c 03 00 00 45 85 f6 0f 85 41 02 00 00 f6 43 08 04 44 RIP [a0350450] __mlx4_remove_one+0x20/0x380 [mlx4_core] RSP 880465003d88 CR2: 038c -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Richard Yang Help you, Help me -- Richard Yang Help you, Help me -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
On Fri, Jun 20, 2014 at 9:21 AM, Wei Yang weiy...@linux.vnet.ibm.com wrote: From this log, it happens during probe? If not, any action after probe? yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it I tried this modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 It looks good to me. NO. I wanted you to hit the error flow where this bug seems to remain... so you need to try and use single ported VFs with IB e.g $ modprobe mlx4_core num_vfs=1,1,1 port_type_array=1,1 BTW, I didn't test 3.16-rc1, sine the SRIOV patch on power platform is not rebased to the latest kernel yet. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
On Fri, Jun 20, 2014 at 9:33 AM, Or Gerlitz or.gerl...@gmail.com wrote: On Fri, Jun 20, 2014 at 9:21 AM, Wei Yang weiy...@linux.vnet.ibm.com wrote: From this log, it happens during probe? If not, any action after probe? yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it I tried this modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 It looks good to me. NO. I wanted you to hit the error flow where this bug seems to remain... so you need to try and use single ported VFs with IB e.g $ modprobe mlx4_core num_vfs=1,1,1 port_type_array=1,1 or $ modprobe mlx4_core num_vfs=1,1,1 probe_vf=1,1,1 rt_type_array=1,1 -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] ibacm/prov: export a function to return config file name
From: Kaike Wan kaike@intel.com
Instead of exposing the ibacm configuration file name directly, this patch
exports a function to return the configuration file name. This provides better
encapsulation.
Signed-off-by: Kaike Wan kaike@intel.com
---
include/infiniband/acm_prov.h |3 ++-
prov/acmp/src/acmp.c |1 +
src/acm.c |7 ++-
3 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/include/infiniband/acm_prov.h b/include/infiniband/acm_prov.h
index dcfdf5f..9e299b9 100644
--- a/include/infiniband/acm_prov.h
+++ b/include/infiniband/acm_prov.h
@@ -82,7 +82,6 @@ struct acm_provider {
/* Variables exported from core */
extern atomic_t counter[ACM_MAX_COUNTER];
-extern char *opts_file;
int provider_query(struct acm_provider **info, uint32_t *version);
@@ -117,4 +116,6 @@ acm_alloc_sa_mad(const struct acm_endpoint *endpoint, void
*context,
extern void acm_free_sa_mad(struct acm_sa_mad *mad);
extern int acm_send_sa_mad(struct acm_sa_mad *mad);
+extern char * acm_get_opts_file(void);
+
#endif /* ACM_PROV_H */
diff --git a/prov/acmp/src/acmp.c b/prov/acmp/src/acmp.c
index 45ad03b..5db62e2 100644
--- a/prov/acmp/src/acmp.c
+++ b/prov/acmp/src/acmp.c
@@ -2776,6 +2776,7 @@ static void acmp_set_options(void)
FILE *f;
char s[120];
char opt[32], value[256];
+ char *opts_file = acm_get_opts_file();
if (!(f = fopen(opts_file, r)))
return;
diff --git a/src/acm.c b/src/acm.c
index 6d18ac5..97b773e 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -186,7 +186,7 @@ static struct sa_data {
* Service options - may be set through ibacm_opts.cfg file.
*/
static char *acme = IBACM_BIN_PATH /ib_acme -A;
-char *opts_file = ACM_CONF_DIR / ACM_OPTS_FILE;
+static char *opts_file = ACM_CONF_DIR / ACM_OPTS_FILE;
static char *addr_file = ACM_CONF_DIR / ACM_ADDR_FILE;
static char log_file[128] = /var/log/ibacm.log;
static int log_level = 0;
@@ -256,6 +256,11 @@ int ib_any_gid(union ibv_gid *gid)
return ((gid-global.subnet_prefix | gid-global.interface_id) == 0);
}
+char * acm_get_opts_file(void)
+{
+ return opts_file;
+}
+
static struct acmc_prov_context *
acm_alloc_prov_context(struct acm_provider *prov)
{
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] ibacm/prov: export a function to increment performance counters
From: Kaike Wan kaike@intel.com
Instead of exposing the combined counters directly, this patch exports a
function to providers to increment the combined counters. This offers better
encapsulation and avoids exposing the private type atomic_t.
Signed-off-by: Kaike Wan kaike@intel.com
---
include/infiniband/acm_prov.h |4 +---
prov/acmp/src/acmp.c | 14 +++---
src/acm.c |8 +++-
3 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/include/infiniband/acm_prov.h b/include/infiniband/acm_prov.h
index 9e299b9..890e6ba 100644
--- a/include/infiniband/acm_prov.h
+++ b/include/infiniband/acm_prov.h
@@ -80,9 +80,6 @@ struct acm_provider {
void(*query_perf)(void *ep_context, uint64_t *values, uint8_t *cnt);
};
-/* Variables exported from core */
-extern atomic_t counter[ACM_MAX_COUNTER];
-
int provider_query(struct acm_provider **info, uint32_t *version);
/* Functions exported from core */
@@ -117,5 +114,6 @@ extern void acm_free_sa_mad(struct acm_sa_mad *mad);
extern int acm_send_sa_mad(struct acm_sa_mad *mad);
extern char * acm_get_opts_file(void);
+extern void acm_increment_counter(int type);
#endif /* ACM_PROV_H */
diff --git a/prov/acmp/src/acmp.c b/prov/acmp/src/acmp.c
index 5db62e2..2dd356d 100644
--- a/prov/acmp/src/acmp.c
+++ b/prov/acmp/src/acmp.c
@@ -798,7 +798,7 @@ static uint8_t acmp_resolve_path_sa(struct acmp_ep *ep,
struct acmp_dest *dest,
memcpy(mad-data, dest-path, sizeof(dest-path));
mad-comp_mask = acm_path_comp_mask(dest-path);
- atomic_inc(counter[ACM_CNTR_ROUTE_QUERY]);
+ acm_increment_counter(ACM_CNTR_ROUTE_QUERY);
atomic_inc(ep-counters[ACM_CNTR_ROUTE_QUERY]);
dest-state = ACMP_QUERY_ROUTE;
if (acm_send_sa_mad(sa_mad)) {
@@ -1630,7 +1630,7 @@ acmp_query(void *addr_context, struct acm_msg *msg,
uint64_t id)
sizeof(struct ibv_path_record));
mad-comp_mask = acm_path_comp_mask(msg-resolve_data[0].info.path);
- atomic_inc(counter[ACM_CNTR_ROUTE_QUERY]);
+ acm_increment_counter(ACM_CNTR_ROUTE_QUERY);
atomic_inc(ep-counters[ACM_CNTR_ROUTE_QUERY]);
if (acm_send_sa_mad(sa_mad)) {
acm_log(0, Error - Failed to send sa mad\n);
@@ -1691,8 +1691,8 @@ acmp_send_resolve(struct acmp_ep *ep, struct acmp_dest
*dest,
rec-gid_cnt = (uint8_t) ep-mc_cnt;
for (i = 0; i ep-mc_cnt; i++)
memcpy(rec-gid[i], ep-mc_dest[i].address, 16);
-
- atomic_inc(counter[ACM_CNTR_ADDR_QUERY]);
+
+ acm_increment_counter(ACM_CNTR_ADDR_QUERY);
atomic_inc(ep-counters[ACM_CNTR_ADDR_QUERY]);
acmp_post_send(ep-resolve_queue, msg);
return 0;
@@ -1758,13 +1758,13 @@ test:
if (acmp_dest_timeout(dest))
goto test;
acm_log(2, request satisfied from local cache\n);
- atomic_inc(counter[ACM_CNTR_ROUTE_CACHE]);
+ acm_increment_counter(ACM_CNTR_ROUTE_CACHE);
atomic_inc(ep-counters[ACM_CNTR_ROUTE_CACHE]);
status = ACM_STATUS_SUCCESS;
break;
case ACMP_ADDR_RESOLVED:
acm_log(2, have address, resolving route\n);
- atomic_inc(counter[ACM_CNTR_ADDR_CACHE]);
+ acm_increment_counter(ACM_CNTR_ADDR_CACHE);
atomic_inc(ep-counters[ACM_CNTR_ADDR_CACHE]);
status = acmp_resolve_path_sa(ep, dest, acmp_dest_sa_resp);
if (status) {
@@ -1833,7 +1833,7 @@ test:
if (acmp_dest_timeout(dest))
goto test;
acm_log(2, request satisfied from local cache\n);
- atomic_inc(counter[ACM_CNTR_ROUTE_CACHE]);
+ acm_increment_counter(ACM_CNTR_ROUTE_CACHE);
atomic_inc(ep-counters[ACM_CNTR_ROUTE_CACHE]);
status = ACM_STATUS_SUCCESS;
break;
diff --git a/src/acm.c b/src/acm.c
index 97b773e..8f147ef 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -163,7 +163,7 @@ static struct acmc_client client_array[FD_SETSIZE - 1];
static FILE *flog;
static lock_t log_lock;
PER_THREAD char log_data[ACM_MAX_ADDRESS];
-atomic_t counter[ACM_MAX_COUNTER];
+static atomic_t counter[ACM_MAX_COUNTER];
static struct acmc_device *
acm_get_device_from_gid(union ibv_gid *sgid, uint8_t *port);
@@ -261,6 +261,12 @@ char * acm_get_opts_file(void)
return opts_file;
}
+void acm_increment_counter(int type)
+{
+ if (type = 0 type ACM_MAX_COUNTER)
+ atomic_inc(counter[type]);
+}
+
static struct acmc_prov_context *
acm_alloc_prov_context(struct acm_provider *prov)
{
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 2/2] ibacm/prov: export a function to increment performance counters
thanks! - applied both -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
On 06/19/2014 08:34 PM, Or Gerlitz wrote:
On Thu, Jun 19, 2014 at 6:33 AM, Shirley Ma shirley...@oracle.com wrote:
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it
failed with mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs with
IB port - single port VFs syntax is only supported when all ports are
configured as ethernet
What do you mean by port1 and port2 -- can you give the exact
command line you used?
Single ported VFs are currently supported for Ethernet only
configuration, that is not for only IB nor for VPI, that is only if
you use port_type_arrary=2,2
I tried command line with num_vfs without port_type_array=2,2.
num_vfs=2
num_vfs={1,1,2}
both failed.
2. After mlx4_core module is being loaded with with num_vfs={} parameters,
when removing mlx4_core, it consistently hits below panic. Whether this
problem is being tracked?
what do you mean by num_vfs={}, is it num_vfs=N or {N}, also here,
please send the exact setting you used. The crash you indicated below
is supposed to be fixed by the upstream commit
da1de8dfff09d33d4a5345762c21b487028e25f5 net/mlx4_core: Keep only one
driver entry release - are you sure to have this commit in the tree
you are working with?
Or.
Yes, I tried net-next tree with this commit
a1de8dfff09d33d4a5345762c21b487028e25f5.
mlx4_ib mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1
(Feb 2014)
mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
mlx4_core: Initializing :40:00.0
mlx4_core :40:00.0: Enabling SR-IOV with 2 VFs
pci :40:00.1: [15b3:1002] type 00 class 0x0c0600
mlx4_core: Initializing :40:00.1
mlx4_core :40:00.1: enabling device ( - 0002)
mlx4_core :40:00.1: Skipping virtual function:1
pci :40:00.2: [15b3:1002] type 00 class 0x0c0600
mlx4_core: Initializing :40:00.2
mlx4_core :40:00.2: enabling device ( - 0002)
mlx4_core :40:00.2: Skipping virtual function:2
mlx4_core :40:00.0: Running in master mode
mlx4_core :40:00.0: PCIe BW is different than device's capability
mlx4_core :40:00.0: PCIe link speed is 5.0GT/s, device supports 8.0GT/s
mlx4_core :40:00.0: PCIe link width is x8, device supports x8
mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port -
single port VFs syntax is only supported when all ports are configured as
ethernet
BUG: unable to handle kernel NULL pointer dereference at 038c
IP: [a0350450] __mlx4_remove_one+0x20/0x380 [mlx4_core]
PGD 45d3ba067 PUD 45ace8067 PMD 0
Oops: [#1] SMP DEBUG_PAGEALLOC
Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE
iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash
dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt
iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr
i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core
ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod
crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas
scsi_transport_sas raid_class [last unloaded: mlx4_core]
CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
Hardware name: Oracle Corporation SUN FIRE X4170 M3 /ASSY,MOTHERBOARD,1U
, BIOS 17050100 08/29/2013
task: 880461540110 ti: 88046500 task.ti: 88046500
RIP: 0010:[a0350450] [a0350450]
__mlx4_remove_one+0x20/0x380 [mlx4_core]
RSP: 0018:880465003d88 EFLAGS: 00010296
RAX: 0001 RBX: RCX:
RDX: 0026 RSI: 0292 RDI: 880468b8f000
RBP: 880465003db8 R08: R09:
R10: 09f911029d74e35b R11: 09f911029d74e35b R12:
R13: 880468b8f000 R14: a036de40 R15: 0001
FS: 7ff287fc2700() GS:88046fce() knlGS:
CS: 0010 DS: ES: CR0: 80050033
CR2: 038c CR3: 00045cfae000 CR4: 000407e0
Stack:
880465003da8 880468b8f000 880468b8f000
a036de40 0001 880465003dd8 a0350805
880468b8f098 a036dd60 880465003e08 812ebaa6
Call Trace:
[a0350805] mlx4_remove_one+0x25/0x50 [mlx4_core]
[812ebaa6] pci_device_remove+0x46/0xc0
[813ce08f] __device_release_driver+0x7f/0xf0
[813ce1c8] driver_detach+0xc8/0xd0
[813cced9] bus_remove_driver+0x59/0xd0
[813cef80] driver_unregister+0x30/0x70
[812ebc13] pci_unregister_driver+0x23/0x80
Re: rmmod mlx4_core panic 3.16-rc1
On 06/19/2014 11:17 PM, Or Gerlitz wrote:
On Fri, Jun 20, 2014 at 6:34 AM, Or Gerlitz or.gerl...@gmail.com wrote:
On Thu, Jun 19, 2014 at 6:33 AM, Shirley Ma shirley...@oracle.com wrote:
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it
failed with mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs
with IB port - single port VFs syntax is only supported when all ports are
configured as ethernet
What do you mean by port1 and port2 -- can you give the exact
command line you used?
Single ported VFs are currently supported for Ethernet only
configuration, that is not for only IB nor for VPI, that is only if
you use port_type_arrary=2,2
Note that you can still use dual-ported VFs, for both IB, Ethernet and
VPI, that is
num_vfs=N will create N dual-ported VFs, are you on IB?
Yes, I tried num_vfs=N. It failed. I didn't combine with port_type_array=2,2.
If it's required, then the code needs to check the arguments.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
When loading the modules, the proper arguments need to be checked if one depends on another. You can easily reproduce this problem by only using num_vfs = N. Shirley On 06/19/2014 11:34 PM, Or Gerlitz wrote: On Fri, Jun 20, 2014 at 9:33 AM, Or Gerlitz or.gerl...@gmail.com wrote: On Fri, Jun 20, 2014 at 9:21 AM, Wei Yang weiy...@linux.vnet.ibm.com wrote: From this log, it happens during probe? If not, any action after probe? yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it I tried this modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 It looks good to me. NO. I wanted you to hit the error flow where this bug seems to remain... so you need to try and use single ported VFs with IB e.g $ modprobe mlx4_core num_vfs=1,1,1 port_type_array=1,1 or $ modprobe mlx4_core num_vfs=1,1,1 probe_vf=1,1,1 rt_type_array=1,1 -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH librdmacm] [TRIVIAL] cma: In ucma_convert_path, fix selector values
Thanks, Hal. All of your recent patches should be applied. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
re: mlx5: Add driver for Mellanox Connect-IB adapters
Hello Eli Cohen,
This is a semi-automatic email about new static checker warnings.
The patch e126ba97dba9: mlx5: Add driver for Mellanox Connect-IB
adapters from Jul 7, 2013, leads to the following Smatch complaint:
drivers/infiniband/hw/mlx5/qp.c:979 create_qp_common()
error: we previously assumed 'pd-uobject' could be null (see line 847)
drivers/infiniband/hw/mlx5/qp.c
846 if (pd) {
847 if (pd-uobject) {
^^^
There are a bunch of checks for pd-uobject.
848 mlx5_ib_dbg(dev, requested sq_wqe_count
(%d)\n, ucmd.sq_wqe_count);
849 if (ucmd.rq_wqe_shift != qp-rq.wqe_shift ||
850 ucmd.rq_wqe_count != qp-rq.wqe_cnt) {
851 mlx5_ib_dbg(dev, invalid rq params\n);
852 return -EINVAL;
853 }
854 if (ucmd.sq_wqe_count
dev-mdev.caps.max_wqes) {
855 mlx5_ib_dbg(dev, requested
sq_wqe_count (%d) max allowed (%d)\n,
[ snip ]
971 qp-doorbell_qpn = swab32(qp-mqp.qpn 8);
972
973 qp-mqp.event = mlx5_ib_qp_event;
974
975 return 0;
976
977 err_create:
978 if (qp-create_type == MLX5_QP_USER)
979 destroy_qp_user(pd, qp);
^^^
But we dereference it inside destroy_qp_user() unconditionally.
980 else if (qp-create_type == MLX5_QP_KERNEL)
981 destroy_qp_kernel(dev, qp);
regards,
dan carpenter
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
how to re-use a QP for a new connection
Hi- I’m considering a change to xprtrdma that would re-use the QP and rdma_cm_id after a transport disconnect. I use rdma_disconnect() and then wait for the TIMEWAIT_EXIT upcall. But after that, rdma_resolve_addr() always fails (-EINVAL). What does xprtrdma need to do to get the rdma_cm_id back to the RDMA_CM_IDLE state so I can reset the QP? Feel free to tell me this doesn’t make sense. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] iw_cxgb4: call iwpm_init() only once
We need to only register with the iwpm core once. Currently it is being
done for every adapter, which causes a failure for each adapter but the
first, making multiple adapters unusable.
This was introduced by
9eccfe1 RDMA/cxgb4: Add support for iWARP Port Mapper user space service
which hit 3.16-rc1. So we need the fix included in 3.16.
Signed-off-by: Steve Wise sw...@opengridcomputing.com
---
drivers/infiniband/hw/cxgb4/cm.c |2 +-
drivers/infiniband/hw/cxgb4/device.c | 17 ++---
drivers/infiniband/hw/cxgb4/iw_cxgb4.h |2 +-
3 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c
index 5e153f6..dc5bdd9 100644
--- a/drivers/infiniband/hw/cxgb4/cm.c
+++ b/drivers/infiniband/hw/cxgb4/cm.c
@@ -3917,7 +3917,7 @@ int __init c4iw_cm_init(void)
return 0;
}
-void __exit c4iw_cm_term(void)
+void c4iw_cm_term(void)
{
WARN_ON(!list_empty(timeout_list));
flush_workqueue(workq);
diff --git a/drivers/infiniband/hw/cxgb4/device.c
b/drivers/infiniband/hw/cxgb4/device.c
index dd93aad..cbcb72d 100644
--- a/drivers/infiniband/hw/cxgb4/device.c
+++ b/drivers/infiniband/hw/cxgb4/device.c
@@ -729,7 +729,6 @@ static void c4iw_dealloc(struct uld_ctx *ctx)
if (ctx-dev-rdev.oc_mw_kva)
iounmap(ctx-dev-rdev.oc_mw_kva);
ib_dealloc_device(ctx-dev-ibdev);
- iwpm_exit(RDMA_NL_C4IW);
ctx-dev = NULL;
}
@@ -826,12 +825,6 @@ static struct c4iw_dev *c4iw_alloc(const struct
cxgb4_lld_info *infop)
setup_debugfs(devp);
}
- ret = iwpm_init(RDMA_NL_C4IW);
- if (ret) {
- pr_err(port mapper initialization failed with %d\n, ret);
- ib_dealloc_device(devp-ibdev);
- return ERR_PTR(ret);
- }
return devp;
}
@@ -1332,6 +1325,15 @@ static int __init c4iw_init_module(void)
pr_err(%s[%u]: Failed to add netlink callback\n
, __func__, __LINE__);
+ err = iwpm_init(RDMA_NL_C4IW);
+ if (err) {
+ pr_err(port mapper initialization failed with %d\n, err);
+ ibnl_remove_client(RDMA_NL_C4IW);
+ c4iw_cm_term();
+ debugfs_remove_recursive(c4iw_debugfs_root);
+ return err;
+ }
+
cxgb4_register_uld(CXGB4_ULD_RDMA, c4iw_uld_info);
return 0;
@@ -1349,6 +1351,7 @@ static void __exit c4iw_exit_module(void)
}
mutex_unlock(dev_mutex);
cxgb4_unregister_uld(CXGB4_ULD_RDMA);
+ iwpm_exit(RDMA_NL_C4IW);
ibnl_remove_client(RDMA_NL_C4IW);
c4iw_cm_term();
debugfs_remove_recursive(c4iw_debugfs_root);
diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
index 125bc5d..361fff7 100644
--- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
+++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
@@ -908,7 +908,7 @@ int c4iw_destroy_ctrl_qp(struct c4iw_rdev *rdev);
int c4iw_register_device(struct c4iw_dev *dev);
void c4iw_unregister_device(struct c4iw_dev *dev);
int __init c4iw_cm_init(void);
-void __exit c4iw_cm_term(void);
+void c4iw_cm_term(void);
void c4iw_release_dev_ucontext(struct c4iw_rdev *rdev,
struct c4iw_dev_ucontext *uctx);
void c4iw_init_dev_ucontext(struct c4iw_rdev *rdev,
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: how to re-use a QP for a new connection
I'm considering a change to xprtrdma that would re-use the QP and rdma_cm_id after a transport disconnect. I use rdma_disconnect() and then wait for the TIMEWAIT_EXIT upcall. But after that, rdma_resolve_addr() always fails (-EINVAL). What does xprtrdma need to do to get the rdma_cm_id back to the RDMA_CM_IDLE state so I can reset the QP? I don't know that the kernel rdma cm code supports this. It's likely that the id would need to have its state reset and data structures cleaned up. That said, I doubt re-use of the rdma_cm_id would provide any real advantage. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: how to re-use a QP for a new connection
Hi Sean- On Jun 20, 2014, at 3:41 PM, Hefty, Sean sean.he...@intel.com wrote: I'm considering a change to xprtrdma that would re-use the QP and rdma_cm_id after a transport disconnect. I use rdma_disconnect() and then wait for the TIMEWAIT_EXIT upcall. But after that, rdma_resolve_addr() always fails (-EINVAL). What does xprtrdma need to do to get the rdma_cm_id back to the RDMA_CM_IDLE state so I can reset the QP? I don't know that the kernel rdma cm code supports this. It's likely that the id would need to have its state reset and data structures cleaned up. That said, I doubt re-use of the rdma_cm_id would provide any real advantage. During a remote transport disconnect, the QP leaves RTS. xprtrdma deals with this in a separate transport connect worker process, where it creates a new id and qp, and replaces the existing id and qp. Unfortunately there are parts of xprtrdma (namely FRMR deregistration) that are not easy to serialize with this reconnect logic. Re-using the QP would mean no serialization would be needed between transport reconnect and FRMR deregistration. If QP re-use is not supported, though, it’s not worth considering any further. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: how to re-use a QP for a new connection
During a remote transport disconnect, the QP leaves RTS. xprtrdma deals with this in a separate transport connect worker process, where it creates a new id and qp, and replaces the existing id and qp. Unfortunately there are parts of xprtrdma (namely FRMR deregistration) that are not easy to serialize with this reconnect logic. Re-using the QP would mean no serialization would be needed between transport reconnect and FRMR deregistration. If QP re-use is not supported, though, it's not worth considering any further. It may be possible to reuse the QP, just not the rdma_cm_id without additional code changes. Reuse of the rdma_cm_id may also require changes in the underlying IB/iWarp CMs. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: how to re-use a QP for a new connection
The QP can be reused. The rdma_id_private has a field reuseaddr. What additional change is needed besides rdma_set_reuseaddr? Shirley On 06/20/2014 02:17 PM, Hefty, Sean wrote: During a remote transport disconnect, the QP leaves RTS. xprtrdma deals with this in a separate transport connect worker process, where it creates a new id and qp, and replaces the existing id and qp. Unfortunately there are parts of xprtrdma (namely FRMR deregistration) that are not easy to serialize with this reconnect logic. Re-using the QP would mean no serialization would be needed between transport reconnect and FRMR deregistration. If QP re-use is not supported, though, it's not worth considering any further. It may be possible to reuse the QP, just not the rdma_cm_id without additional code changes. Reuse of the rdma_cm_id may also require changes in the underlying IB/iWarp CMs. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: how to re-use a QP for a new connection
Hi Shirley- I’ve found that to move the QP back to the IB_QPS_INIT state, I need to call ib_modify_qp() with a specific set of attributes, including the pkey_index and port_num. rdma_init_qp_attr() extracts those attributes. But, when I try to call it after rdma_disconnect(), the rdma_cm_id is not in the RDMA_CM_IDLE state, and the call fails. So I can’t get the QP back to the INIT state unless the rdma_cm_id has somehow been reset. I suppose I could call rdma_init_qp_attr() while the transport is still connected, and save the returned attributes. On Jun 20, 2014, at 6:24 PM, Shirley Ma shirley...@oracle.com wrote: The QP can be reused. The rdma_id_private has a field reuseaddr. What additional change is needed besides rdma_set_reuseaddr? Shirley On 06/20/2014 02:17 PM, Hefty, Sean wrote: During a remote transport disconnect, the QP leaves RTS. xprtrdma deals with this in a separate transport connect worker process, where it creates a new id and qp, and replaces the existing id and qp. Unfortunately there are parts of xprtrdma (namely FRMR deregistration) that are not easy to serialize with this reconnect logic. Re-using the QP would mean no serialization would be needed between transport reconnect and FRMR deregistration. If QP re-use is not supported, though, it's not worth considering any further. It may be possible to reuse the QP, just not the rdma_cm_id without additional code changes. Reuse of the rdma_cm_id may also require changes in the underlying IB/iWarp CMs. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: how to re-use a QP for a new connection
On 06/20/2014 03:30 PM, Chuck Lever wrote: Hi Shirley- I’ve found that to move the QP back to the IB_QPS_INIT state, I need to call ib_modify_qp() with a specific set of attributes, including the pkey_index and port_num. rdma_init_qp_attr() extracts those attributes. But, when I try to call it after rdma_disconnect(), the rdma_cm_id is not in the RDMA_CM_IDLE state, and the call fails. So I can’t get the QP back to the INIT state unless the rdma_cm_id has somehow been reset. I see, we need to have rdma_reset_id() to change the cm_id state to RDMA_CM_IDLE. I suppose I could call rdma_init_qp_attr() while the transport is still connected, and save the returned attributes. Maybe we can save ib_qp_attr in xprtrdma rpcrdma_ia? On Jun 20, 2014, at 6:24 PM, Shirley Ma shirley...@oracle.com wrote: The QP can be reused. The rdma_id_private has a field reuseaddr. What additional change is needed besides rdma_set_reuseaddr? Shirley On 06/20/2014 02:17 PM, Hefty, Sean wrote: During a remote transport disconnect, the QP leaves RTS. xprtrdma deals with this in a separate transport connect worker process, where it creates a new id and qp, and replaces the existing id and qp. Unfortunately there are parts of xprtrdma (namely FRMR deregistration) that are not easy to serialize with this reconnect logic. Re-using the QP would mean no serialization would be needed between transport reconnect and FRMR deregistration. If QP re-use is not supported, though, it's not worth considering any further. It may be possible to reuse the QP, just not the rdma_cm_id without additional code changes. Reuse of the rdma_cm_id may also require changes in the underlying IB/iWarp CMs. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
