Re: how to re-use a QP for a new connection
On 06/20/2014 03:30 PM, Chuck Lever wrote: > Hi Shirley- > > I’ve found that to move the QP back to the IB_QPS_INIT state, I need to > call ib_modify_qp() with a specific set of attributes, including the > pkey_index and port_num. > > rdma_init_qp_attr() extracts those attributes. But, when I try to call it > after rdma_disconnect(), the rdma_cm_id is not in the RDMA_CM_IDLE state, > and the call fails. > > So I can’t get the QP back to the INIT state unless the rdma_cm_id has > somehow been reset. I see, we need to have rdma_reset_id() to change the cm_id state to RDMA_CM_IDLE. > I suppose I could call rdma_init_qp_attr() while the transport is still > connected, and save the returned attributes. Maybe we can save ib_qp_attr in xprtrdma rpcrdma_ia? > On Jun 20, 2014, at 6:24 PM, Shirley Ma wrote: > >> The QP can be reused. The rdma_id_private has a field reuseaddr. What >> additional change is needed besides rdma_set_reuseaddr? >> >> Shirley >> >> On 06/20/2014 02:17 PM, Hefty, Sean wrote: During a remote transport disconnect, the QP leaves RTS. xprtrdma deals with this in a separate transport connect worker process, where it creates a new id and qp, and replaces the existing id and qp. Unfortunately there are parts of xprtrdma (namely FRMR deregistration) that are not easy to serialize with this reconnect logic. Re-using the QP would mean no serialization would be needed between transport reconnect and FRMR deregistration. If QP re-use is not supported, though, it's not worth considering any further. >>> >>> It may be possible to reuse the QP, just not the rdma_cm_id without >>> additional code changes. Reuse of the rdma_cm_id may also require changes >>> in the underlying IB/iWarp CMs. >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >>> the body of a message to majord...@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> > > -- > Chuck Lever > chuck[dot]lever[at]oracle[dot]com > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: how to re-use a QP for a new connection
Hi Shirley- I’ve found that to move the QP back to the IB_QPS_INIT state, I need to call ib_modify_qp() with a specific set of attributes, including the pkey_index and port_num. rdma_init_qp_attr() extracts those attributes. But, when I try to call it after rdma_disconnect(), the rdma_cm_id is not in the RDMA_CM_IDLE state, and the call fails. So I can’t get the QP back to the INIT state unless the rdma_cm_id has somehow been reset. I suppose I could call rdma_init_qp_attr() while the transport is still connected, and save the returned attributes. On Jun 20, 2014, at 6:24 PM, Shirley Ma wrote: > The QP can be reused. The rdma_id_private has a field reuseaddr. What > additional change is needed besides rdma_set_reuseaddr? > > Shirley > > On 06/20/2014 02:17 PM, Hefty, Sean wrote: >>> During a remote transport disconnect, the QP leaves RTS. >>> >>> xprtrdma deals with this in a separate transport connect worker process, >>> where it creates a new id and qp, and replaces the existing id and qp. >>> >>> Unfortunately there are parts of xprtrdma (namely FRMR deregistration) >>> that are not easy to serialize with this reconnect logic. >>> >>> Re-using the QP would mean no serialization would be needed between >>> transport reconnect and FRMR deregistration. >>> >>> If QP re-use is not supported, though, it's not worth considering any >>> further. >> >> It may be possible to reuse the QP, just not the rdma_cm_id without >> additional code changes. Reuse of the rdma_cm_id may also require changes >> in the underlying IB/iWarp CMs. >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: how to re-use a QP for a new connection
The QP can be reused. The rdma_id_private has a field reuseaddr. What additional change is needed besides rdma_set_reuseaddr? Shirley On 06/20/2014 02:17 PM, Hefty, Sean wrote: >> During a remote transport disconnect, the QP leaves RTS. >> >> xprtrdma deals with this in a separate transport connect worker process, >> where it creates a new id and qp, and replaces the existing id and qp. >> >> Unfortunately there are parts of xprtrdma (namely FRMR deregistration) >> that are not easy to serialize with this reconnect logic. >> >> Re-using the QP would mean no serialization would be needed between >> transport reconnect and FRMR deregistration. >> >> If QP re-use is not supported, though, it's not worth considering any >> further. > > It may be possible to reuse the QP, just not the rdma_cm_id without > additional code changes. Reuse of the rdma_cm_id may also require changes in > the underlying IB/iWarp CMs. > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: how to re-use a QP for a new connection
> During a remote transport disconnect, the QP leaves RTS. > > xprtrdma deals with this in a separate transport connect worker process, > where it creates a new id and qp, and replaces the existing id and qp. > > Unfortunately there are parts of xprtrdma (namely FRMR deregistration) > that are not easy to serialize with this reconnect logic. > > Re-using the QP would mean no serialization would be needed between > transport reconnect and FRMR deregistration. > > If QP re-use is not supported, though, it's not worth considering any > further. It may be possible to reuse the QP, just not the rdma_cm_id without additional code changes. Reuse of the rdma_cm_id may also require changes in the underlying IB/iWarp CMs. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: how to re-use a QP for a new connection
Hi Sean- On Jun 20, 2014, at 3:41 PM, Hefty, Sean wrote: >> I'm considering a change to xprtrdma that would re-use the QP and >> rdma_cm_id after a transport disconnect. >> >> I use rdma_disconnect() and then wait for the TIMEWAIT_EXIT upcall. >> But after that, rdma_resolve_addr() always fails (-EINVAL). >> >> What does xprtrdma need to do to get the rdma_cm_id back to the >> RDMA_CM_IDLE state so I can reset the QP? > > I don't know that the kernel rdma cm code supports this. It's likely that > the id would need to have its state reset and data structures cleaned up. > That said, I doubt re-use of the rdma_cm_id would provide any real advantage. During a remote transport disconnect, the QP leaves RTS. xprtrdma deals with this in a separate transport connect worker process, where it creates a new id and qp, and replaces the existing id and qp. Unfortunately there are parts of xprtrdma (namely FRMR deregistration) that are not easy to serialize with this reconnect logic. Re-using the QP would mean no serialization would be needed between transport reconnect and FRMR deregistration. If QP re-use is not supported, though, it’s not worth considering any further. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: how to re-use a QP for a new connection
> I'm considering a change to xprtrdma that would re-use the QP and > rdma_cm_id after a transport disconnect. > > I use rdma_disconnect() and then wait for the TIMEWAIT_EXIT upcall. > But after that, rdma_resolve_addr() always fails (-EINVAL). > > What does xprtrdma need to do to get the rdma_cm_id back to the > RDMA_CM_IDLE state so I can reset the QP? I don't know that the kernel rdma cm code supports this. It's likely that the id would need to have its state reset and data structures cleaned up. That said, I doubt re-use of the rdma_cm_id would provide any real advantage. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] iw_cxgb4: call iwpm_init() only once
We need to only register with the iwpm core once. Currently it is being
done for every adapter, which causes a failure for each adapter but the
first, making multiple adapters unusable.
This was introduced by
9eccfe1 RDMA/cxgb4: Add support for iWARP Port Mapper user space service
which hit 3.16-rc1. So we need the fix included in 3.16.
Signed-off-by: Steve Wise
---
drivers/infiniband/hw/cxgb4/cm.c |2 +-
drivers/infiniband/hw/cxgb4/device.c | 17 ++---
drivers/infiniband/hw/cxgb4/iw_cxgb4.h |2 +-
3 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c
index 5e153f6..dc5bdd9 100644
--- a/drivers/infiniband/hw/cxgb4/cm.c
+++ b/drivers/infiniband/hw/cxgb4/cm.c
@@ -3917,7 +3917,7 @@ int __init c4iw_cm_init(void)
return 0;
}
-void __exit c4iw_cm_term(void)
+void c4iw_cm_term(void)
{
WARN_ON(!list_empty(&timeout_list));
flush_workqueue(workq);
diff --git a/drivers/infiniband/hw/cxgb4/device.c
b/drivers/infiniband/hw/cxgb4/device.c
index dd93aad..cbcb72d 100644
--- a/drivers/infiniband/hw/cxgb4/device.c
+++ b/drivers/infiniband/hw/cxgb4/device.c
@@ -729,7 +729,6 @@ static void c4iw_dealloc(struct uld_ctx *ctx)
if (ctx->dev->rdev.oc_mw_kva)
iounmap(ctx->dev->rdev.oc_mw_kva);
ib_dealloc_device(&ctx->dev->ibdev);
- iwpm_exit(RDMA_NL_C4IW);
ctx->dev = NULL;
}
@@ -826,12 +825,6 @@ static struct c4iw_dev *c4iw_alloc(const struct
cxgb4_lld_info *infop)
setup_debugfs(devp);
}
- ret = iwpm_init(RDMA_NL_C4IW);
- if (ret) {
- pr_err("port mapper initialization failed with %d\n", ret);
- ib_dealloc_device(&devp->ibdev);
- return ERR_PTR(ret);
- }
return devp;
}
@@ -1332,6 +1325,15 @@ static int __init c4iw_init_module(void)
pr_err("%s[%u]: Failed to add netlink callback\n"
, __func__, __LINE__);
+ err = iwpm_init(RDMA_NL_C4IW);
+ if (err) {
+ pr_err("port mapper initialization failed with %d\n", err);
+ ibnl_remove_client(RDMA_NL_C4IW);
+ c4iw_cm_term();
+ debugfs_remove_recursive(c4iw_debugfs_root);
+ return err;
+ }
+
cxgb4_register_uld(CXGB4_ULD_RDMA, &c4iw_uld_info);
return 0;
@@ -1349,6 +1351,7 @@ static void __exit c4iw_exit_module(void)
}
mutex_unlock(&dev_mutex);
cxgb4_unregister_uld(CXGB4_ULD_RDMA);
+ iwpm_exit(RDMA_NL_C4IW);
ibnl_remove_client(RDMA_NL_C4IW);
c4iw_cm_term();
debugfs_remove_recursive(c4iw_debugfs_root);
diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
index 125bc5d..361fff7 100644
--- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
+++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
@@ -908,7 +908,7 @@ int c4iw_destroy_ctrl_qp(struct c4iw_rdev *rdev);
int c4iw_register_device(struct c4iw_dev *dev);
void c4iw_unregister_device(struct c4iw_dev *dev);
int __init c4iw_cm_init(void);
-void __exit c4iw_cm_term(void);
+void c4iw_cm_term(void);
void c4iw_release_dev_ucontext(struct c4iw_rdev *rdev,
struct c4iw_dev_ucontext *uctx);
void c4iw_init_dev_ucontext(struct c4iw_rdev *rdev,
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
how to re-use a QP for a new connection
Hi- I’m considering a change to xprtrdma that would re-use the QP and rdma_cm_id after a transport disconnect. I use rdma_disconnect() and then wait for the TIMEWAIT_EXIT upcall. But after that, rdma_resolve_addr() always fails (-EINVAL). What does xprtrdma need to do to get the rdma_cm_id back to the RDMA_CM_IDLE state so I can reset the QP? Feel free to tell me this doesn’t make sense. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
re: mlx5: Add driver for Mellanox Connect-IB adapters
Hello Eli Cohen,
This is a semi-automatic email about new static checker warnings.
The patch e126ba97dba9: "mlx5: Add driver for Mellanox Connect-IB
adapters" from Jul 7, 2013, leads to the following Smatch complaint:
drivers/infiniband/hw/mlx5/qp.c:979 create_qp_common()
error: we previously assumed 'pd->uobject' could be null (see line 847)
drivers/infiniband/hw/mlx5/qp.c
846 if (pd) {
847 if (pd->uobject) {
^^^
There are a bunch of checks for pd->uobject.
848 mlx5_ib_dbg(dev, "requested sq_wqe_count
(%d)\n", ucmd.sq_wqe_count);
849 if (ucmd.rq_wqe_shift != qp->rq.wqe_shift ||
850 ucmd.rq_wqe_count != qp->rq.wqe_cnt) {
851 mlx5_ib_dbg(dev, "invalid rq params\n");
852 return -EINVAL;
853 }
854 if (ucmd.sq_wqe_count >
dev->mdev.caps.max_wqes) {
855 mlx5_ib_dbg(dev, "requested
sq_wqe_count (%d) > max allowed (%d)\n",
[ snip ]
971 qp->doorbell_qpn = swab32(qp->mqp.qpn << 8);
972
973 qp->mqp.event = mlx5_ib_qp_event;
974
975 return 0;
976
977 err_create:
978 if (qp->create_type == MLX5_QP_USER)
979 destroy_qp_user(pd, qp);
^^^
But we dereference it inside destroy_qp_user() unconditionally.
980 else if (qp->create_type == MLX5_QP_KERNEL)
981 destroy_qp_kernel(dev, qp);
regards,
dan carpenter
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH librdmacm] [TRIVIAL] cma: In ucma_convert_path, fix selector values
Thanks, Hal. All of your recent patches should be applied. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
When loading the modules, the proper arguments need to be checked if one depends on another. You can easily reproduce this problem by only using num_vfs = N. Shirley On 06/19/2014 11:34 PM, Or Gerlitz wrote: > On Fri, Jun 20, 2014 at 9:33 AM, Or Gerlitz wrote: >> On Fri, Jun 20, 2014 at 9:21 AM, Wei Yang wrote: >> > From this log, it happens during probe? > If not, any action after probe? >> yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it >> >>> I tried this >>> modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 >>> It looks good to me. >> >> NO. I wanted you to hit the error flow where this bug seems to >> remain... so you need to try and use single ported VFs with IB e.g >> $ modprobe mlx4_core num_vfs=1,1,1 port_type_array=1,1 > > or > > $ modprobe mlx4_core num_vfs=1,1,1 probe_vf=1,1,1 rt_type_array=1,1 > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
On 06/19/2014 11:17 PM, Or Gerlitz wrote:
> On Fri, Jun 20, 2014 at 6:34 AM, Or Gerlitz wrote:
>> On Thu, Jun 19, 2014 at 6:33 AM, Shirley Ma wrote:
>>> 1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
>>> I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it
>>> failed with mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs
>>> with IB port - single port VFs syntax is only supported when all ports are
>>> configured as ethernet
>
>> What do you mean by "port1" and "port2" -- can you give the exact
>> command line you used?
>
>> Single ported VFs are currently supported for Ethernet only
>> configuration, that is not for only IB nor for VPI, that is only if
>> you use port_type_arrary=2,2
>
> Note that you can still use dual-ported VFs, for both IB, Ethernet and
> VPI, that is
> num_vfs=N will create N dual-ported VFs, are you on IB?
Yes, I tried num_vfs=N. It failed. I didn't combine with port_type_array=2,2.
If it's required, then the code needs to check the arguments.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rmmod mlx4_core panic 3.16-rc1
On 06/19/2014 08:34 PM, Or Gerlitz wrote:
> On Thu, Jun 19, 2014 at 6:33 AM, Shirley Ma wrote:
>>
>> 1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
>>
>> I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it
>> failed with mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs with
>> IB port - single port VFs syntax is only supported when all ports are
>> configured as ethernet
>
>
> What do you mean by "port1" and "port2" -- can you give the exact
> command line you used?
>
> Single ported VFs are currently supported for Ethernet only
> configuration, that is not for only IB nor for VPI, that is only if
> you use port_type_arrary=2,2
>
I tried command line with num_vfs without port_type_array=2,2.
num_vfs=2
num_vfs={1,1,2}
both failed.
>
>>
>>
>> 2. After mlx4_core module is being loaded with with num_vfs={} parameters,
>> when removing mlx4_core, it consistently hits below panic. Whether this
>> problem is being tracked?
>
>
> what do you mean by "num_vfs={}", is it num_vfs=N or {N}, also here,
> please send the exact setting you used. The crash you indicated below
> is supposed to be fixed by the upstream commit
> da1de8dfff09d33d4a5345762c21b487028e25f5 "net/mlx4_core: Keep only one
> driver entry release" - are you sure to have this commit in the tree
> you are working with?
>
> Or.
Yes, I tried net-next tree with this commit
a1de8dfff09d33d4a5345762c21b487028e25f5.
>>
>> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1
>> (Feb 2014)
>> mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
>> mlx4_core: Initializing :40:00.0
>> mlx4_core :40:00.0: Enabling SR-IOV with 2 VFs
>> pci :40:00.1: [15b3:1002] type 00 class 0x0c0600
>> mlx4_core: Initializing :40:00.1
>> mlx4_core :40:00.1: enabling device ( -> 0002)
>> mlx4_core :40:00.1: Skipping virtual function:1
>> pci :40:00.2: [15b3:1002] type 00 class 0x0c0600
>> mlx4_core: Initializing :40:00.2
>> mlx4_core :40:00.2: enabling device ( -> 0002)
>> mlx4_core :40:00.2: Skipping virtual function:2
>> mlx4_core :40:00.0: Running in master mode
>> mlx4_core :40:00.0: PCIe BW is different than device's capability
>> mlx4_core :40:00.0: PCIe link speed is 5.0GT/s, device supports 8.0GT/s
>> mlx4_core :40:00.0: PCIe link width is x8, device supports x8
>> mlx4_core :40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port -
>> single port VFs syntax is only supported when all ports are configured as
>> ethernet
>> BUG: unable to handle kernel NULL pointer dereference at 038c
>> IP: [] __mlx4_remove_one+0x20/0x380 [mlx4_core]
>> PGD 45d3ba067 PUD 45ace8067 PMD 0
>> Oops: [#1] SMP DEBUG_PAGEALLOC
>> Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE
>> iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
>> autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
>> iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
>> xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash
>> dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt
>> iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr
>> i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core
>> ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod
>> crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas
>> scsi_transport_sas raid_class [last unloaded: mlx4_core]
>> CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
>> Hardware name: Oracle Corporation SUN FIRE X4170 M3 /ASSY,MOTHERBOARD,1U
>> , BIOS 17050100 08/29/2013
>> task: 880461540110 ti: 88046500 task.ti: 88046500
>> RIP: 0010:[] []
>> __mlx4_remove_one+0x20/0x380 [mlx4_core]
>> RSP: 0018:880465003d88 EFLAGS: 00010296
>> RAX: 0001 RBX: RCX:
>> RDX: 0026 RSI: 0292 RDI: 880468b8f000
>> RBP: 880465003db8 R08: R09:
>> R10: 09f911029d74e35b R11: 09f911029d74e35b R12:
>> R13: 880468b8f000 R14: a036de40 R15: 0001
>> FS: 7ff287fc2700() GS:88046fce() knlGS:
>> CS: 0010 DS: ES: CR0: 80050033
>> CR2: 038c CR3: 00045cfae000 CR4: 000407e0
>> Stack:
>> 880465003da8 880468b8f000 880468b8f000
>> a036de40 0001 880465003dd8 a0350805
>> 880468b8f098 a036dd60 880465003e08 812ebaa6
>> Call Trace:
>> [] mlx4_remove_one+0x25/0x50 [mlx4_core]
>> [] pci_device_remove+0x46/0xc0
>> [] __device_release_driver+0x7f/0xf0
>> [] driver_detach+0xc8/0xd0
>> [] bus_remove_driver+0x59/0xd0
>> [] driver_unregister+0x30/0x70
>> [] pci_unregister_driver+0x23/0x80
>> [] mlx4_
RE: [PATCH 2/2] ibacm/prov: export a function to increment performance counters
thanks! - applied both -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] ibacm/prov: export a function to increment performance counters
From: Kaike Wan
Instead of exposing the combined counters directly, this patch exports a
function to providers to increment the combined counters. This offers better
encapsulation and avoids exposing the private type atomic_t.
Signed-off-by: Kaike Wan
---
include/infiniband/acm_prov.h |4 +---
prov/acmp/src/acmp.c | 14 +++---
src/acm.c |8 +++-
3 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/include/infiniband/acm_prov.h b/include/infiniband/acm_prov.h
index 9e299b9..890e6ba 100644
--- a/include/infiniband/acm_prov.h
+++ b/include/infiniband/acm_prov.h
@@ -80,9 +80,6 @@ struct acm_provider {
void(*query_perf)(void *ep_context, uint64_t *values, uint8_t *cnt);
};
-/* Variables exported from core */
-extern atomic_t counter[ACM_MAX_COUNTER];
-
int provider_query(struct acm_provider **info, uint32_t *version);
/* Functions exported from core */
@@ -117,5 +114,6 @@ extern void acm_free_sa_mad(struct acm_sa_mad *mad);
extern int acm_send_sa_mad(struct acm_sa_mad *mad);
extern char * acm_get_opts_file(void);
+extern void acm_increment_counter(int type);
#endif /* ACM_PROV_H */
diff --git a/prov/acmp/src/acmp.c b/prov/acmp/src/acmp.c
index 5db62e2..2dd356d 100644
--- a/prov/acmp/src/acmp.c
+++ b/prov/acmp/src/acmp.c
@@ -798,7 +798,7 @@ static uint8_t acmp_resolve_path_sa(struct acmp_ep *ep,
struct acmp_dest *dest,
memcpy(mad->data, &dest->path, sizeof(dest->path));
mad->comp_mask = acm_path_comp_mask(&dest->path);
- atomic_inc(&counter[ACM_CNTR_ROUTE_QUERY]);
+ acm_increment_counter(ACM_CNTR_ROUTE_QUERY);
atomic_inc(&ep->counters[ACM_CNTR_ROUTE_QUERY]);
dest->state = ACMP_QUERY_ROUTE;
if (acm_send_sa_mad(sa_mad)) {
@@ -1630,7 +1630,7 @@ acmp_query(void *addr_context, struct acm_msg *msg,
uint64_t id)
sizeof(struct ibv_path_record));
mad->comp_mask = acm_path_comp_mask(&msg->resolve_data[0].info.path);
- atomic_inc(&counter[ACM_CNTR_ROUTE_QUERY]);
+ acm_increment_counter(ACM_CNTR_ROUTE_QUERY);
atomic_inc(&ep->counters[ACM_CNTR_ROUTE_QUERY]);
if (acm_send_sa_mad(sa_mad)) {
acm_log(0, "Error - Failed to send sa mad\n");
@@ -1691,8 +1691,8 @@ acmp_send_resolve(struct acmp_ep *ep, struct acmp_dest
*dest,
rec->gid_cnt = (uint8_t) ep->mc_cnt;
for (i = 0; i < ep->mc_cnt; i++)
memcpy(&rec->gid[i], ep->mc_dest[i].address, 16);
-
- atomic_inc(&counter[ACM_CNTR_ADDR_QUERY]);
+
+ acm_increment_counter(ACM_CNTR_ADDR_QUERY);
atomic_inc(&ep->counters[ACM_CNTR_ADDR_QUERY]);
acmp_post_send(&ep->resolve_queue, msg);
return 0;
@@ -1758,13 +1758,13 @@ test:
if (acmp_dest_timeout(dest))
goto test;
acm_log(2, "request satisfied from local cache\n");
- atomic_inc(&counter[ACM_CNTR_ROUTE_CACHE]);
+ acm_increment_counter(ACM_CNTR_ROUTE_CACHE);
atomic_inc(&ep->counters[ACM_CNTR_ROUTE_CACHE]);
status = ACM_STATUS_SUCCESS;
break;
case ACMP_ADDR_RESOLVED:
acm_log(2, "have address, resolving route\n");
- atomic_inc(&counter[ACM_CNTR_ADDR_CACHE]);
+ acm_increment_counter(ACM_CNTR_ADDR_CACHE);
atomic_inc(&ep->counters[ACM_CNTR_ADDR_CACHE]);
status = acmp_resolve_path_sa(ep, dest, acmp_dest_sa_resp);
if (status) {
@@ -1833,7 +1833,7 @@ test:
if (acmp_dest_timeout(dest))
goto test;
acm_log(2, "request satisfied from local cache\n");
- atomic_inc(&counter[ACM_CNTR_ROUTE_CACHE]);
+ acm_increment_counter(ACM_CNTR_ROUTE_CACHE);
atomic_inc(&ep->counters[ACM_CNTR_ROUTE_CACHE]);
status = ACM_STATUS_SUCCESS;
break;
diff --git a/src/acm.c b/src/acm.c
index 97b773e..8f147ef 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -163,7 +163,7 @@ static struct acmc_client client_array[FD_SETSIZE - 1];
static FILE *flog;
static lock_t log_lock;
PER_THREAD char log_data[ACM_MAX_ADDRESS];
-atomic_t counter[ACM_MAX_COUNTER];
+static atomic_t counter[ACM_MAX_COUNTER];
static struct acmc_device *
acm_get_device_from_gid(union ibv_gid *sgid, uint8_t *port);
@@ -261,6 +261,12 @@ char * acm_get_opts_file(void)
return opts_file;
}
+void acm_increment_counter(int type)
+{
+ if (type >= 0 && type < ACM_MAX_COUNTER)
+ atomic_inc(&counter[type]);
+}
+
static struct acmc_prov_context *
acm_alloc_prov_context(struct acm_provider *prov)
{
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] ibacm/prov: export a function to return config file name
From: Kaike Wan
Instead of exposing the ibacm configuration file name directly, this patch
exports a function to return the configuration file name. This provides better
encapsulation.
Signed-off-by: Kaike Wan
---
include/infiniband/acm_prov.h |3 ++-
prov/acmp/src/acmp.c |1 +
src/acm.c |7 ++-
3 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/include/infiniband/acm_prov.h b/include/infiniband/acm_prov.h
index dcfdf5f..9e299b9 100644
--- a/include/infiniband/acm_prov.h
+++ b/include/infiniband/acm_prov.h
@@ -82,7 +82,6 @@ struct acm_provider {
/* Variables exported from core */
extern atomic_t counter[ACM_MAX_COUNTER];
-extern char *opts_file;
int provider_query(struct acm_provider **info, uint32_t *version);
@@ -117,4 +116,6 @@ acm_alloc_sa_mad(const struct acm_endpoint *endpoint, void
*context,
extern void acm_free_sa_mad(struct acm_sa_mad *mad);
extern int acm_send_sa_mad(struct acm_sa_mad *mad);
+extern char * acm_get_opts_file(void);
+
#endif /* ACM_PROV_H */
diff --git a/prov/acmp/src/acmp.c b/prov/acmp/src/acmp.c
index 45ad03b..5db62e2 100644
--- a/prov/acmp/src/acmp.c
+++ b/prov/acmp/src/acmp.c
@@ -2776,6 +2776,7 @@ static void acmp_set_options(void)
FILE *f;
char s[120];
char opt[32], value[256];
+ char *opts_file = acm_get_opts_file();
if (!(f = fopen(opts_file, "r")))
return;
diff --git a/src/acm.c b/src/acm.c
index 6d18ac5..97b773e 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -186,7 +186,7 @@ static struct sa_data {
* Service options - may be set through ibacm_opts.cfg file.
*/
static char *acme = IBACM_BIN_PATH "/ib_acme -A";
-char *opts_file = ACM_CONF_DIR "/" ACM_OPTS_FILE;
+static char *opts_file = ACM_CONF_DIR "/" ACM_OPTS_FILE;
static char *addr_file = ACM_CONF_DIR "/" ACM_ADDR_FILE;
static char log_file[128] = "/var/log/ibacm.log";
static int log_level = 0;
@@ -256,6 +256,11 @@ int ib_any_gid(union ibv_gid *gid)
return ((gid->global.subnet_prefix | gid->global.interface_id) == 0);
}
+char * acm_get_opts_file(void)
+{
+ return opts_file;
+}
+
static struct acmc_prov_context *
acm_alloc_prov_context(struct acm_provider *prov)
{
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
