NFS/RDMA - client crash on mount
Hi, I'm currently trying to setup a NFS/RDMA server/client pair. base system is CentOS release 6.2 I tried both the provided kernel 2.6.32-220.el6.x86_64 and a custom-built 3.4.0 (x86_64), and the result is exactly the same. Both server and client use nfs-utils version 1.2.3. Hardware-wise, both servers are virtual machines using softiwarp ( http://gitorious.org/softiwarp ) to emulate the rdma layer. It might be the source of the problem, but both rping (obviously) and diod/9pnet_rdma work flawlessly, so I suppose there might be something else. Server setup: # cat /etc/exports /tmp 10.3.4.3/255.255.255.255(fsid=0,rw,async,insecure,no_root_squash) # cat /proc/fs/nfsd/portlist rdma 20049 udp 2049 tcp 2049 udp 2049 tcp 2049 # echo 32767 > /proc/sys/sunrpc/rpc_debug (rdma also on with full debug) Client setup: # modprobe xprtrdma # echo 32767 > /proc/sys/sunrpc/rpc_debug # mount -vvv -t nfs -o proto=rdma,port=20049 10.3.4.2:/tmp /mnt/t mount: fstab path: "/etc/fstab" mount: mtab path: "/etc/mtab" mount: lock path: "/etc/mtab~" mount: temp path: "/etc/mtab.tmp" mount: UID:0 mount: eUID: 0 mount: spec: "10.3.4.2:/tmp" mount: node: "/mnt/t" mount: types: "nfs" mount: opts: "proto=rdma,port=20049" mount: external mount: argv[0] = "/sbin/mount.nfs" mount: external mount: argv[1] = "10.3.4.2:/tmp" mount: external mount: argv[2] = "/mnt/t" mount: external mount: argv[3] = "-v" mount: external mount: argv[4] = "-o" mount: external mount: argv[5] = "rw,proto=rdma,port=20049" mount.nfs: timeout set for Wed May 30 08:55:22 2012 mount.nfs: trying text-based options 'proto=rdma,port=20049,vers=4,addr=10.3.4.2,clientaddr=10.3.4.3' Connection to 10.3.4.3 closed by remote host. After this, I've lost ssh access, the VM doesn't reply to ping won't reboot with a soft-reboot. The login prompt is apparently still live, but won't accept a user. After reboot, there is no log of what happened whatsoever on the client, but I could get this backtrace with a netconsole: RPC: looking up machine cred for service * RPC: xprt_setup_rdma: 10.3.4.2:20049 ( 7644/1) siw_add_obj(OBJ9975): IDR New Object ( 7644/1) siw_pd_add(PD9975): New Object RPC: rpcrdma_ia_open: FRMR registration specified but not supported by adapter, using riskier RPCRDMA_ALLPHYSICAL ( 7644/1) siw_mem_add(IDR10286): New Object ( 7644/1) siw_alloc_mr(MEM10286): New Object, UMEM (null) ( 7644/1) siw_pd_get(PD9975): New refcount: 2 RPC: rpcrdma_ia_open: memory registration strategy is 6 RPC: rpcrdma_ep_create: requested max: dtos: send 32 recv 32; iovs: send 2 recv 1 ( 7644/0) siw_add_obj(OBJ23494): IDR New Object ( 7644/0) siw_cq_add(CQ23494): New Object ( 7644/0) siw_req_notify_cq(CQ23494:) flags: 0x 2 RPC: rpcrdma_buffer_create: max_requests 32 RPC: created transport 88020532 with 32 slots RPC: creating nfs client for 10.3.4.2 (xprt 88020532) RPC: creating UNIX authenticator for client 8802047f6800 RPC: new task initialized, procpid 7644 RPC: allocated task 880216294080 RPC: 1 __rpc_execute flags=0x680 RPC: 1 call_start nfs4 proc NULL (sync) RPC: 1 call_reserve (status 0) RPC: xprt_rdma_reserve_xprt: cwndscale 256 RPC: 1 reserved req 88020463f200 xid def103c4 RPC: wake_up_first(880205320178 "xprt_sending") RPC: 1 call_reserveresult (status 0) RPC: 1 call_refresh (status 0) RPC: 1 holding NULL cred 8185b650 RPC: 1 refreshing NULL cred 8185b650 RPC: 1 call_refreshresult (status 0) RPC: 1 call_allocate (status 0) RPC: xprt_rdma_allocate: size 88, request 0x88020b45 RPC: 1 call_bind (status 0) RPC: 1 call_connect xprt 88020532 is not connected RPC: 1 xprt_connect xprt 88020532 is not connected RPC: 1 xprt_cwnd_limited cong = 0 cwnd = 256 RPC: 1 sleep_on(queue "xprt_pending" time 4308025979) RPC: 1 added to queue 880205320228 "xprt_pending" RPC: 1 setting alarm for 6 ms RPC: xprt_rdma_connect_worker: connect ( 41/3) siw_create_qp: new QP on device siw_eth0 ( 41/3) siw_add_obj(OBJ26807): IDR New Object ( 41/3) siw_qp_add(QP26807): New Object ( 41/3) siw_pd_get(PD9975): New refcount: 3 siw_ofed_modify_qp --- IB_QP_STATE IB_QP_ACCESS_FLAGS siw_ofed_modify_qp -(end)- ( 41/3) siw_ofed_modify_qp(QP26807): Desired IB QP state: INIT ( 41/3) siw_qp_modify(QP26807) ( 41/3) siw_qp_modify(QP26807): SIW QP state: IDLE => IDLE ( 41/3) siw_qp_modify QP state transition undefined: IDLE => IDLE ( 41/3) siw_ofed_modify_qp(QP26807): Exit with 0 BUG: unable to handle kernel NULL pointer dereference at (null) IP: [] rpcrdma_ep_connect+0x126/0x2e6 [xprtrdma] PGD 0 Oops: [#1] SMP CPU 3 Modules linked in: netconsole xprtrdma 9pnet_rdma 9p 9pnet fscache siw(O) rdma_ucm ib_uverbs rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 joydev microcode pcspkr virtio_net virtio
Re: NFS/RDMA - client crash on mount
Hi,
Well don't mind me, it was soft-iwarp related afterall.
It would look like siw doesn't like 2K MTU like a few other devices,
adding it in the list in net/sunrpc/xprtrdma/verbs.c around line 880
made the crash just disappear.
It's not something easy to patch though given siw is out of tree, why is
the MTU 2k by default? Isn't there a way to get capatibilities out of
the devices, other than an hard-coded list like this?
(this:
/* XXX Tavor device performs badly with 2K MTU! */
if (strnicmp(ia->ri_id->device->dma_device->bus->name, "pci", 3) == 0)
{
struct pci_dev *pcid = to_pci_dev(ia->ri_id->device->dma_device);
if (pcid->device == PCI_DEVICE_ID_MELLANOX_TAVOR &&
(pcid->vendor == PCI_VENDOR_ID_MELLANOX ||
pcid->vendor == PCI_VENDOR_ID_TOPSPIN)) {
struct ib_qp_attr attr = {
.path_mtu = IB_MTU_1024
};
rc = ib_modify_qp(ia->ri_id->qp, &attr, IB_QP_PATH_MTU);
}
})
Sorry for the noise then, thanks to everyone who bothered to read my
mail.
--
Dominique Martinet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: NFS/RDMA - client crash on mount
For siw, the dma_device->bus is currently not set, which will cause
the crash.
As we were testing NFS/RDMA we just disabled that fix for some Mellanox
device.
I agree I probably have to find a more consistent way to fake real dma
hardware. Any suggestions?
Thanks,
Bernard.
linux-rdma-ow...@vger.kernel.org wrote on 05/30/2012 02:28:58 PM:
> From: Dominique Martinet
> To: linux-...@vger.kernel.org, linux-rdma@vger.kernel.org,
> Date: 05/30/2012 02:29 PM
> Subject: Re: NFS/RDMA - client crash on mount
> Sent by: linux-rdma-ow...@vger.kernel.org
>
> Hi,
>
> Well don't mind me, it was soft-iwarp related afterall.
> It would look like siw doesn't like 2K MTU like a few other devices,
> adding it in the list in net/sunrpc/xprtrdma/verbs.c around line 880
> made the crash just disappear.
>
> It's not something easy to patch though given siw is out of tree, why is
> the MTU 2k by default? Isn't there a way to get capatibilities out of
> the devices, other than an hard-coded list like this?
>
> (this:
> /* XXX Tavor device performs badly with 2K MTU! */
> if (strnicmp(ia->ri_id->device->dma_device->bus->name, "pci", 3) == 0)
> {
>struct pci_dev *pcid = to_pci_dev(ia->ri_id->device->dma_device);
>if (pcid->device == PCI_DEVICE_ID_MELLANOX_TAVOR &&
>(pcid->vendor == PCI_VENDOR_ID_MELLANOX ||
>pcid->vendor == PCI_VENDOR_ID_TOPSPIN)) {
> struct ib_qp_attr attr = {
> .path_mtu = IB_MTU_1024
> };
> rc = ib_modify_qp(ia->ri_id->qp, &attr, IB_QP_PATH_MTU);
>}
> })
>
>
> Sorry for the noise then, thanks to everyone who bothered to read my
> mail.
>
> --
> Dominique Martinet
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] opensm/osm_ucast_ftree.c: Add a couple of asserts
Also, fix some commentary typos Signed-off-by: Hal Rosenstock --- diff --git a/opensm/osm_ucast_ftree.c b/opensm/osm_ucast_ftree.c index ee4c34b..1233310 100644 --- a/opensm/osm_ucast_ftree.c +++ b/opensm/osm_ucast_ftree.c @@ -519,6 +519,7 @@ static void port_group_add_port(IN ftree_port_group_t * p_group, } p_port = port_create(port_num, remote_port_num); + CL_ASSERT(p_port); cl_ptr_vector_insert(&p_group->ports, p_port, NULL); } @@ -765,7 +766,7 @@ static int set_hops_on_remote_sw(IN ftree_port_group_t * p_group, CL_ASSERT(p_group->remote_node_type == IB_NODE_TYPE_SWITCH); p_remote_sw->hops[target_lid] = hops; - /* If taget lid is a switch we set the min hop table values + /* If target lid is a switch we set the min hop table values * for each port on the associated osm_sw struct */ if (!is_target_sw) return 0; @@ -911,6 +912,7 @@ static void hca_add_port(IN ftree_hca_t * p_hca, IN uint8_t port_num, remote_port_guid, remote_node_guid, remote_node_type, p_remote_hca_or_sw, is_cn, is_io); + CL_ASSERT(p_group); p_hca->up_port_groups[p_hca->up_port_groups_num++] = p_group; } port_group_add_port(p_group, port_num, remote_port_num); @@ -2278,7 +2280,7 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, target_lid, /* LID that we're routing to */ is_real_lid, /* whether the target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ + is_target_a_sw, /* Whether target lid is a switch or not */ current_hops + 1); /* Number of hops done to this point */ created_route |= routed; /* Counters are promoted only if a route toward a node is created */ @@ -2345,7 +2347,7 @@ fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_lid, /* LID that we're routing to */ is_real_lid, /* whether this target LID is real or dummy */ is_main_path, /* whether this path to HCA should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ + is_target_a_sw, /* Whether target lid is a switch or not */ current_hops); /* Number of hops done up to this point */ /* recursion stop condition - if it's a root switch, */ @@ -2373,7 +2375,7 @@ fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_lid, /* LID that we're routing to */ is_real_lid,/* whether this target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ - is_target_a_sw, /* Wheter target lid is a switch or not */ + is_target_a_sw, /* Whether target lid is a switch or not */ reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ reverse_hops + 1, /* Number of reverse_hops done up to this point */ current_hops @@ -2502,7 +2504,7 @@ fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_lid, /* LID that we're routing to */ is_real_lid,/* whether this target LID is real or dummy */ is_main_path,
Re: [PATCH 1/1] ib_srp: Infiniband srp fast failover patch.
Hi Dave, As long as we get faster failover I am happy with Bart's patch. Currently when I run IO to several luns over multipath and the preferred path goes down, the system hangs until the IO fails over. Even ssh'ing into the systems take 20-30 seconds. I *suspect* that is because IO is being queued up somewhere which brings the whole system to its knees. Thank you for looking at the patch. Thanks Karan On 05/30/2012 01:06 AM, David Dillow wrote: On Tue, 2012-05-29 at 17:07 -0400, Karandeep Chahal wrote: Subject: [PATCH] Infiniband srp fast failover patch. This conflicts with Bart's patches to improve failover; it will be much better to use his approach to block the target rather than remove it wholesale -- we could have lost connectivity as a transient and may get it back quickly if someone grabbed the wrong cable, etc. Also, we should only kill the one target on DREQ, and we already have a pointer to it from the CM context -- no need to search. It is a good idea to hook into the event mechanism; this is something I've long wanted to incorporate (as Vu did in OFED). I'm looking at getting Bart's series to a point I can merge it, and I'll pull in your ideas -- with credit -- there. Thanks, -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH][TRIVIAL] opensm/include/opensm/osm_log.h: Fix commentary cut 'n paste error
Signed-off-by: Hal Rosenstock --- diff --git a/include/opensm/osm_log.h b/include/opensm/osm_log.h index b2f105a..4214cd4 100644 --- a/include/opensm/osm_log.h +++ b/include/opensm/osm_log.h @@ -102,7 +102,7 @@ typedef uint8_t osm_log_level_t; */ #define OSM_LOG_DEFAULT_LEVEL OSM_LOG_ERROR | OSM_LOG_INFO -/s* OpenSM: MAD Wrapper/osm_log_t +/s* OpenSM: Log/osm_log_t * NAME * osm_log_t * -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH][TRIVIAL] opensm/complib/cl_fleximap.h: Cosmetic changes
Fix typo and some formatting changes Signed-off-by: Hal Rosenstock --- diff --git a/include/complib/cl_fleximap.h b/include/complib/cl_fleximap.h index 7390a14..89bc7c0 100644 --- a/include/complib/cl_fleximap.h +++ b/include/complib/cl_fleximap.h @@ -462,8 +462,8 @@ static inline cl_fmap_item_t *cl_fmap_head(IN const cl_fmap_t * const p_map) /* * PARAMETERS * p_map -* [in] Pointer to a cl_fmap_t structure whose item with the lowest key -* is returned. +* [in] Pointer to a cl_fmap_t structure whose item with the lowest +* key is returned. * * RETURN VALUES * Pointer to the map item with the lowest key in the flexi map. @@ -601,7 +601,7 @@ cl_fmap_item_t *cl_fmap_insert(IN cl_fmap_t * const p_map, * [in] Pointer to the key value to assign to the item. Storage * for the key must be persistant, as only the pointer is stored. * Users are responsible for maintaining the validity of key -* pointers while they are in use. +* pointers while they are in use. * * p_item * [in] Pointer to a cl_fmap_item_t stucture to insert into the flexi map. @@ -693,8 +693,8 @@ cl_fmap_item_t *cl_fmap_get(IN const cl_fmap_t * const p_map, * cl_fmap_get_next * * DESCRIPTION -* The cl_fmap_get_next function returns the first map item associated with a -* key > the key specified. +* The cl_fmap_get_next function returns the first map item associated with +* a key > the key specified. * * SYNOPSIS */ @@ -744,7 +744,7 @@ cl_fmap_remove_item(IN cl_fmap_t * const p_map, * This function does not return a value. * * In a debug build, cl_fmap_remove_item asserts that the item being -* removed es in the specified map. +* removed is in the specified map. * * NOTES * Removes the map item pointed to by p_item from its flexi map. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH][RESEND] opensm/osm_ucast_dfsssp.c: Use osm_log_is_active
rather than osm_log_get_level to determine whether log level is active.
Log level is bit mask so >= doesn't always work properly.
Signed-off-by: Hal Rosenstock
---
diff --git a/opensm/osm_ucast_dfsssp.c b/opensm/osm_ucast_dfsssp.c
index 1e3ade1..3b7b83a 100644
--- a/opensm/osm_ucast_dfsssp.c
+++ b/opensm/osm_ucast_dfsssp.c
@@ -1185,7 +1185,7 @@ static int dfsssp_build_graph(void *context)
}
}
/* print the discovered graph */
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG))
dfsssp_print_graph(p_mgr, adj_list, adj_list_size);
dfsssp_ctx->adj_list = adj_list;
@@ -1814,7 +1814,7 @@ static int dfsssp_remove_deadlocks(dfsssp_context_t *
dfsssp_ctx)
OSM_LOG(p_mgr->p_log, OSM_LOG_INFO,
"Virtual Lanes needed: %" PRIu8 "\n", vl_needed);
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_INFO) {
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_INFO)) {
OSM_LOG(p_mgr->p_log, OSM_LOG_INFO,
"Paths per VL (before balancing):\n");
for (i = 0; i < vl_avail; i++)
@@ -1869,7 +1869,7 @@ static int dfsssp_remove_deadlocks(dfsssp_context_t *
dfsssp_ctx)
goto ERROR;
}
/* else { no balancing } */
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_INFO) {
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_INFO)) {
OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG,
"Virtual Lanes per src/dest combination after
balancing:\n");
vltable_print(p_mgr, srcdest2vl_table);
@@ -1958,8 +1958,8 @@ static int dfsssp_do_dijkstra_routing(void *context)
port, lid);
if (err)
return err;
- if (osm_log_get_level(p_mgr->p_log) >=
- OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log,
+ OSM_LOG_DEBUG))
print_routes(p_mgr, adj_list,
adj_list_size, port);
@@ -1974,8 +1974,8 @@ static int dfsssp_do_dijkstra_routing(void *context)
update_weights(p_mgr, adj_list, adj_list_size,
port);
- if (osm_log_get_level(p_mgr->p_log) >=
- OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log,
+ OSM_LOG_DEBUG))
dfsssp_print_graph(p_mgr, adj_list,
adj_list_size);
}
@@ -1998,7 +1998,7 @@ static int dfsssp_do_dijkstra_routing(void *context)
dijkstra(p_mgr, adj_list, adj_list_size, port, lid);
if (err)
return err;
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG))
print_routes(p_mgr, adj_list, adj_list_size,
port);
@@ -2010,7 +2010,7 @@ static int dfsssp_do_dijkstra_routing(void *context)
/* add weights for calculated routes to adjust the
weights for the next cycle */
update_weights(p_mgr, adj_list, adj_list_size, port);
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG))
dfsssp_print_graph(p_mgr, adj_list,
adj_list_size);
}
@@ -2032,7 +2032,7 @@ static int dfsssp_do_dijkstra_routing(void *context)
}
/* print the new_lft for each switch after routing is done */
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG) {
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) {
for (item = cl_qmap_head(sw_tbl); item != cl_qmap_end(sw_tbl);
item = cl_qmap_next(item)) {
sw = (osm_switch_t *) item;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCHv2] opensm/osm_sa_informinfo.c: Add trusted support for InformInfo/InformInfoRecord
per C15-0.2-1.16
Compile tested only
Also, fixed one error number (so not duplicated)
Signed-off-by: Hal Rosenstock
---
Change since v1:
Rebased against latest upstream master
diff --git a/opensm/osm_sa_informinfo.c b/opensm/osm_sa_informinfo.c
index e3f6ffa..772a293 100644
--- a/opensm/osm_sa_informinfo.c
+++ b/opensm/osm_sa_informinfo.c
@@ -74,6 +74,7 @@ typedef struct osm_iir_search_ctxt {
ib_net16_t subscriber_enum;
osm_sa_t *sa;
osm_physp_t *p_req_physp;
+ ib_net64_t sm_key;
} osm_iir_search_ctxt_t;
/**
@@ -291,6 +292,16 @@ static void sa_inform_info_rec_by_comp_mask(IN osm_sa_t *
sa,
memcpy(&p_rec_item->rec, &p_infr->inform_record,
sizeof(ib_inform_info_record_t));
+
+ /*
+* Per C15-0.2-1.16, InformInfoRecords shall always be
+* provided with the QPN set to 0, except for the case
+* of a trusted request, in which case the actual
+* subscriber QPN shall be returned.
+*/
+ if (p_ctxt->sm_key == 0)
+ ib_inform_info_set_qpn(&p_rec_item->rec.inform_info, 0);
+
cl_qlist_insert_tail(p_ctxt->p_list, &p_rec_item->list_item);
Exit:
@@ -349,6 +360,7 @@ static void infr_rcv_process_get_method(osm_sa_t * sa, IN
osm_madw_t * p_madw)
context.subscriber_enum = p_rcvd_rec->subscriber_enum;
context.sa = sa;
context.p_req_physp = p_req_physp;
+ context.sm_key = p_rcvd_mad->sm_key;
OSM_LOG(sa->p_log, OSM_LOG_DEBUG,
"Query Subscriber GID:%s(%02X) Enum:0x%X(%02X)\n",
@@ -440,7 +452,7 @@ static void infr_rcv_process_set_method(osm_sa_t * sa, IN
osm_madw_t * p_madw)
if (p_recvd_inform_info->subscribe > 1) {
cl_plock_release(sa->p_lock);
- OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4308 "
+ OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 430A "
"Invalid subscribe: %d\n",
p_recvd_inform_info->subscribe);
osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID);
@@ -448,6 +460,24 @@ static void infr_rcv_process_set_method(osm_sa_t * sa, IN
osm_madw_t * p_madw)
}
/*
+* Per C15-0.2-1.16, SubnAdmSet(InformInfo) subscriptions for
+* SM security traps shall be provided only if they come from a
+* trusted source.
+*/
+ if ((p_sa_mad->sm_key == 0) && p_recvd_inform_info->is_generic &&
+ ((cl_ntoh16(p_recvd_inform_info->g_or_v.generic.trap_num) >= 256) &&
+(cl_ntoh16(p_recvd_inform_info->g_or_v.generic.trap_num) <= 259)))
{
+ cl_plock_release(sa->p_lock);
+
+ OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 430B "
+ "Request for security trap from non-trusted requester: "
+ "Given SM_Key:0x%016" PRIx64 "\n",
+ cl_ntoh64(p_sa_mad->sm_key));
+ osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID);
+ goto Exit;
+ }
+
+ /*
* MODIFICATIONS DONE ON INCOMING REQUEST:
*
* QPN:
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] opensm/osm_ucast_dfsssp.c: Use osm_log_is_active
rather than osm_log_get_level to determine whether log level is active.
Log level is bit mask so >= doesn't always work properly.
Signed-off-by: Hal Rosenstock
---
diff --git a/opensm/osm_ucast_dfsssp.c b/opensm/osm_ucast_dfsssp.c
index 1e3ade1..3b7b83a 100644
--- a/opensm/osm_ucast_dfsssp.c
+++ b/opensm/osm_ucast_dfsssp.c
@@ -1185,7 +1185,7 @@ static int dfsssp_build_graph(void *context)
}
}
/* print the discovered graph */
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG))
dfsssp_print_graph(p_mgr, adj_list, adj_list_size);
dfsssp_ctx->adj_list = adj_list;
@@ -1814,7 +1814,7 @@ static int dfsssp_remove_deadlocks(dfsssp_context_t *
dfsssp_ctx)
OSM_LOG(p_mgr->p_log, OSM_LOG_INFO,
"Virtual Lanes needed: %" PRIu8 "\n", vl_needed);
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_INFO) {
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_INFO)) {
OSM_LOG(p_mgr->p_log, OSM_LOG_INFO,
"Paths per VL (before balancing):\n");
for (i = 0; i < vl_avail; i++)
@@ -1869,7 +1869,7 @@ static int dfsssp_remove_deadlocks(dfsssp_context_t *
dfsssp_ctx)
goto ERROR;
}
/* else { no balancing } */
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_INFO) {
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_INFO)) {
OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG,
"Virtual Lanes per src/dest combination after
balancing:\n");
vltable_print(p_mgr, srcdest2vl_table);
@@ -1958,8 +1958,8 @@ static int dfsssp_do_dijkstra_routing(void *context)
port, lid);
if (err)
return err;
- if (osm_log_get_level(p_mgr->p_log) >=
- OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log,
+ OSM_LOG_DEBUG))
print_routes(p_mgr, adj_list,
adj_list_size, port);
@@ -1974,8 +1974,8 @@ static int dfsssp_do_dijkstra_routing(void *context)
update_weights(p_mgr, adj_list, adj_list_size,
port);
- if (osm_log_get_level(p_mgr->p_log) >=
- OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log,
+ OSM_LOG_DEBUG))
dfsssp_print_graph(p_mgr, adj_list,
adj_list_size);
}
@@ -1998,7 +1998,7 @@ static int dfsssp_do_dijkstra_routing(void *context)
dijkstra(p_mgr, adj_list, adj_list_size, port, lid);
if (err)
return err;
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG))
print_routes(p_mgr, adj_list, adj_list_size,
port);
@@ -2010,7 +2010,7 @@ static int dfsssp_do_dijkstra_routing(void *context)
/* add weights for calculated routes to adjust the
weights for the next cycle */
update_weights(p_mgr, adj_list, adj_list_size, port);
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG)
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG))
dfsssp_print_graph(p_mgr, adj_list,
adj_list_size);
}
@@ -2032,7 +2032,7 @@ static int dfsssp_do_dijkstra_routing(void *context)
}
/* print the new_lft for each switch after routing is done */
- if (osm_log_get_level(p_mgr->p_log) >= OSM_LOG_DEBUG) {
+ if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) {
for (item = cl_qmap_head(sw_tbl); item != cl_qmap_end(sw_tbl);
item = cl_qmap_next(item)) {
sw = (osm_switch_t *) item;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] opensm/opensm_release_notes-3.3.txt: Update Unsupported IB Compliance Statements
GUIDInfo is now supported (alias GUID support) Signed-off-by: Hal Rosenstock --- diff --git a/doc/opensm_release_notes-3.3.txt b/doc/opensm_release_notes-3.3.txt index 888485e..19e6bc6 100644 --- a/doc/opensm_release_notes-3.3.txt +++ b/doc/opensm_release_notes-3.3.txt @@ -295,9 +295,6 @@ information regarding each compliance statement. If no permission to forward, the subscription should be removed and no further forwarding should occur. -* C14-24.1.1.5 and C14-62.1.1.22 (Initialization): - GUIDInfo - SM should enable assigning Port GUIDInfo. - * C14-44 (Initialization): If the SM discovers that it is missing an M_Key to update CA/RT/SW, it should notify the higher level. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH][TRIVIAL] infiniband-diags: clean up exclude arguments to ibdiag_process_opts()
Alphabetize for standardization and greater clarity Signed-off-by: Jim Foraker --- src/ibcacheedit.c |2 +- src/iblinkinfo.c|2 +- src/ibnetdiscover.c |2 +- src/ibqueryerrors.c |2 +- src/ibsendtrap.c|2 +- src/ibstat.c|2 +- src/saquery.c |2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/ibcacheedit.c b/src/ibcacheedit.c index 28b8b21..f9e7dc9 100644 --- a/src/ibcacheedit.c +++ b/src/ibcacheedit.c @@ -269,7 +269,7 @@ int main(int argc, char **argv) }; char *usage_args = " "; - ibdiag_process_opts(argc, argv, NULL, "edCPDLGtsv", + ibdiag_process_opts(argc, argv, NULL, "CDdeGLPstv", opts, process_opt, usage_args, NULL); diff --git a/src/iblinkinfo.c b/src/iblinkinfo.c index 36aa6d1..a817920 100644 --- a/src/iblinkinfo.c +++ b/src/iblinkinfo.c @@ -619,7 +619,7 @@ int main(int argc, char **argv) }; char usage_args[] = ""; - ibdiag_process_opts(argc, argv, &config, "SDandlpgRGL", opts, + ibdiag_process_opts(argc, argv, &config, "aDdGgLlnpRS", opts, process_opt, usage_args, NULL); argc -= optind; diff --git a/src/ibnetdiscover.c b/src/ibnetdiscover.c index 8ec9421..4748be9 100644 --- a/src/ibnetdiscover.c +++ b/src/ibnetdiscover.c @@ -1074,7 +1074,7 @@ int main(int argc, char **argv) }; char usage_args[] = "[topology-file]"; - ibdiag_process_opts(argc, argv, &config, "sGDL", opts, process_opt, + ibdiag_process_opts(argc, argv, &config, "DGLs", opts, process_opt, usage_args, NULL); f = stdout; diff --git a/src/ibqueryerrors.c b/src/ibqueryerrors.c index 4fec240..eb57418 100644 --- a/src/ibqueryerrors.c +++ b/src/ibqueryerrors.c @@ -901,7 +901,7 @@ int main(int argc, char **argv) char usage_args[] = ""; memset(suppressed_fields, 0, sizeof suppressed_fields); - ibdiag_process_opts(argc, argv, &config, "scnSrRDGL", opts, process_opt, + ibdiag_process_opts(argc, argv, &config, "cDGLnRrSs", opts, process_opt, usage_args, NULL); argc -= optind; diff --git a/src/ibsendtrap.c b/src/ibsendtrap.c index 2d63e35..f98d9cf 100644 --- a/src/ibsendtrap.c +++ b/src/ibsendtrap.c @@ -192,7 +192,7 @@ int main(int argc, char **argv) snprintf(usage_args + n, sizeof(usage_args) - n, "\n default behavior is to send \"%s\"", traps[0].trap_name); - ibdiag_process_opts(argc, argv, NULL, "DLG", NULL, NULL, + ibdiag_process_opts(argc, argv, NULL, "DGL", NULL, NULL, usage_args, NULL); argc -= optind; diff --git a/src/ibstat.c b/src/ibstat.c index f953cf4..3d971ed 100644 --- a/src/ibstat.c +++ b/src/ibstat.c @@ -292,7 +292,7 @@ int main(int argc, char *argv[]) NULL }; - ibdiag_process_opts(argc, argv, NULL, "sDGLCPte", opts, process_opt, + ibdiag_process_opts(argc, argv, NULL, "CDeGLPst", opts, process_opt, usage_args, usage_examples); argc -= optind; diff --git a/src/saquery.c b/src/saquery.c index e5fdb25..1e730a4 100644 --- a/src/saquery.c +++ b/src/saquery.c @@ -1638,7 +1638,7 @@ int main(int argc, char **argv) q = NULL; ibd_timeout = DEFAULT_SA_TIMEOUT_MS; - ibdiag_process_opts(argc, argv, ¶ms, "DLGs", opts, process_opt, + ibdiag_process_opts(argc, argv, ¶ms, "DGLs", opts, process_opt, usage_args, NULL); argc -= optind; -- 1.7.9.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3 0/8] Mkey support in infiniband-diags
I am about to send V3 of the patch set. Changes include: . There is now a common "-K" option which must be passed in order to view "sensitive" credentials. Data is shown as "" rather than as invalid/all zeros, as an indication to the user that they should pass -K in order to see it. . Documentation is migrated to restructured text and cleaned up significantly. . The config file now accepts a "sa_smkey" parameter, which sets the smkey used in SA requests. The default config file continues to set this to 0 to discourage bad security practice, but the user can now configure their own default as they wish. . The changes to ibportstate to display and modify mkey-related parameters have been broken out into two separate patches, and the patch series reordered into a more logical progression. Jim -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/16] librdmacm: Check that send and recv CQs are different before destroying
ucma_destroy_cqs() destroys both the send and recv CQs if they
are non-null. If the two CQs are actually the same one, this
results in a crash when trying to destroy the second CQ. Check
that the CQs are different before destroying the second CQ.
This fixes a crash when using rsockets, which sets the send and
recv CQs to the same CQ.
Signed-off-by: Sean Hefty
---
src/cma.c |4 ++--
src/rsocket.c |5 +
2 files changed, 3 insertions(+), 6 deletions(-)
diff --git a/src/cma.c b/src/cma.c
index c32803d..9cd34cf 100755
--- a/src/cma.c
+++ b/src/cma.c
@@ -1096,10 +1096,10 @@ static void ucma_destroy_cqs(struct rdma_cm_id *id)
if (id->recv_cq_channel)
ibv_destroy_comp_channel(id->recv_cq_channel);
- if (id->send_cq)
+ if (id->send_cq && (id->send_cq != id->recv_cq))
ibv_destroy_cq(id->send_cq);
- if (id->send_cq_channel)
+ if (id->send_cq_channel && (id->send_cq_channel != id->recv_cq_channel))
ibv_destroy_comp_channel(id->send_cq_channel);
}
diff --git a/src/rsocket.c b/src/rsocket.c
index de0ec3e..01b7248 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -366,11 +366,8 @@ static void rs_free(struct rsocket *rs)
rdma_dereg_mr(rs->target_mr);
if (rs->cm_id) {
- if (rs->cm_id->qp) {
- rs->cm_id->send_cq_channel = NULL;
- rs->cm_id->send_cq = NULL;
+ if (rs->cm_id->qp)
rdma_destroy_qp(rs->cm_id);
- }
rdma_destroy_id(rs->cm_id);
}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/16] librdmacm: rsocket improvements
The following patch set provides fixes, adds user configurability, and optimizes rsockets based on the needs and results of performance analysis and wider testing. Additional optimizations will follow, but I wanted to go ahead and push these changes out. Signed-off-by: Sean Hefty -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/16] librdmacm/rstream: Check for connection error on async connect
Signed-off-by: Sean Hefty
---
examples/rstream.c | 14 +-
1 files changed, 13 insertions(+), 1 deletions(-)
diff --git a/examples/rstream.c b/examples/rstream.c
index 104b318..c440f04 100644
--- a/examples/rstream.c
+++ b/examples/rstream.c
@@ -448,7 +448,8 @@ static int client_connect(void)
{
struct addrinfo *res;
struct pollfd fds;
- int ret, rs;
+ int ret, rs, err;
+ socklen_t len;
ret = getaddrinfo(dst_addr, port, NULL, &res);
if (ret) {
@@ -477,6 +478,17 @@ static int client_connect(void)
ret = do_poll(&fds);
if (ret)
goto err;
+
+ len = sizeof err;
+ ret = rs_getsockopt(rs, SOL_SOCKET, SO_ERROR, &err, &len);
+ if (ret)
+ goto err;
+ if (err) {
+ ret = -1;
+ errno = err;
+ perror("async rconnect");
+ goto err;
+ }
}
free:
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 16/16] librdmacm/rstream: Use separate connections for latency/bw tests
Optimize each connection for either latency or bandwidth
results. This improves small message latency under 384
bytes by .5 - 1 us, while increasing bandwidth by
1 - 1.5 Gbps.
Signed-off-by: Sean Hefty
---
examples/rstream.c | 146 ++--
1 files changed, 96 insertions(+), 50 deletions(-)
diff --git a/examples/rstream.c b/examples/rstream.c
index 054d11e..48e2771 100644
--- a/examples/rstream.c
+++ b/examples/rstream.c
@@ -71,12 +71,20 @@ static struct test_size_param test_size[] = {
};
#define TEST_CNT (sizeof test_size / sizeof test_size[0])
+enum rs_optimization {
+ opt_mixed,
+ opt_latency,
+ opt_bandwidth
+};
+
+static int rs, lrs;
static int use_rs = 1;
static int use_async;
static int verify;
static int flags = MSG_DONTWAIT;
static int poll_timeout = 0;
static int custom;
+static enum rs_optimization optimization;
static int size_option;
static int iterations = 1;
static int transfer_size = 1000;
@@ -242,7 +250,7 @@ static int do_poll(struct pollfd *fds)
return ret == 1 ? 0 : ret;
}
-static int send_xfer(int rs, int size)
+static int send_xfer(int size)
{
struct pollfd fds;
int offset, ret;
@@ -274,7 +282,7 @@ static int send_xfer(int rs, int size)
return 0;
}
-static int recv_xfer(int rs, int size)
+static int recv_xfer(int size)
{
struct pollfd fds;
int offset, ret;
@@ -309,37 +317,37 @@ static int recv_xfer(int rs, int size)
return 0;
}
-static int sync_test(int rs)
+static int sync_test(void)
{
int ret;
- ret = dst_addr ? send_xfer(rs, 4) : recv_xfer(rs, 4);
+ ret = dst_addr ? send_xfer(16) : recv_xfer(16);
if (ret)
return ret;
- return dst_addr ? recv_xfer(rs, 4) : send_xfer(rs, 4);
+ return dst_addr ? recv_xfer(16) : send_xfer(16);
}
-static int run_test(int rs)
+static int run_test(void)
{
int ret, i, t;
- ret = sync_test(rs);
+ ret = sync_test();
if (ret)
goto out;
gettimeofday(&start, NULL);
for (i = 0; i < iterations; i++) {
for (t = 0; t < transfer_count; t++) {
- ret = dst_addr ? send_xfer(rs, transfer_size) :
-recv_xfer(rs, transfer_size);
+ ret = dst_addr ? send_xfer(transfer_size) :
+recv_xfer(transfer_size);
if (ret)
goto out;
}
for (t = 0; t < transfer_count; t++) {
- ret = dst_addr ? recv_xfer(rs, transfer_size) :
-send_xfer(rs, transfer_size);
+ ret = dst_addr ? recv_xfer(transfer_size) :
+send_xfer(transfer_size);
if (ret)
goto out;
}
@@ -361,6 +369,10 @@ static void set_options(int rs)
sizeof buffer_size);
rs_setsockopt(rs, SOL_SOCKET, SO_RCVBUF, (void *) &buffer_size,
sizeof buffer_size);
+ } else {
+ val = 1 << 19;
+ rs_setsockopt(rs, SOL_SOCKET, SO_SNDBUF, (void *) &val, sizeof
val);
+ rs_setsockopt(rs, SOL_SOCKET, SO_RCVBUF, (void *) &val, sizeof
val);
}
val = 1;
@@ -368,13 +380,23 @@ static void set_options(int rs)
if (flags & MSG_DONTWAIT)
rs_fcntl(rs, F_SETFL, O_NONBLOCK);
+
+ if (use_rs) {
+ /* Inline size based on experimental data */
+ if (optimization == opt_latency) {
+ val = 384;
+ rs_setsockopt(rs, SOL_RDMA, RDMA_INLINE, &val, sizeof
val);
+ } else if (optimization == opt_bandwidth) {
+ val = 0;
+ rs_setsockopt(rs, SOL_RDMA, RDMA_INLINE, &val, sizeof
val);
+ }
+ }
}
-static int server_connect(void)
+static int server_listen(void)
{
- struct pollfd fds;
struct addrinfo hints, *res;
- int rs, lrs, ret;
+ int val, ret;
memset(&hints, 0, sizeof hints);
hints.ai_flags = RAI_PASSIVE;
@@ -387,30 +409,41 @@ static int server_connect(void)
lrs = rs_socket(res->ai_family, res->ai_socktype, res->ai_protocol);
if (lrs < 0) {
perror("rsocket");
- rs = lrs;
+ ret = lrs;
goto free;
}
- set_options(lrs);
- rs = 1;
- rs = rs_setsockopt(lrs, SOL_SOCKET, SO_REUSEADDR, &rs, sizeof rs);
- if (rs) {
+ val = 1;
+ ret = rs_setsockopt(lrs, SOL_SOCKET, SO_REUSEADDR, &val, sizeof val);
+ if (ret) {
perror("rsetsockopt SO_REUSEADDR");
goto close;
}
- rs = rs_b
[PATCH 3/16] librdmacm/rsocket: Fix hang in rrecv/rsend after disconnecting
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem and the cause was reported by Sridhar Samudrala
. It can be reproduced by running netserver -f -D
using the rs-preload library. A similar issue exists with rsend().
Fix this by not blocking on a CQ unless we're connected.
Signed-off-by: Sean Hefty
---
Sridhar, can you please let me know if this fixes the hang you were seeing?
I moved the connected check inside holding the cq lock from the patch that
you sent me.
src/rsocket.c | 26 +++---
1 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/src/rsocket.c b/src/rsocket.c
index 01b7248..8c96dc1 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -908,6 +908,11 @@ static int rs_can_send(struct rsocket *rs)
(rs->target_sgl[rs->target_sge].length != 0);
}
+static int rs_conn_can_send(struct rsocket *rs)
+{
+ return rs_can_send(rs) || (rs->state != rs_connected);
+}
+
static int rs_can_send_ctrl(struct rsocket *rs)
{
return rs->ctrl_avail;
@@ -918,6 +923,11 @@ static int rs_have_rdata(struct rsocket *rs)
return (rs->rmsg_head != rs->rmsg_tail);
}
+static int rs_conn_have_rdata(struct rsocket *rs)
+{
+ return rs_have_rdata(rs) || (rs->state != rs_connected);
+}
+
static int rs_all_sends_done(struct rsocket *rs)
{
return (rs->sqe_avail + rs->ctrl_avail) == RS_QP_SIZE;
@@ -980,7 +990,7 @@ ssize_t rrecv(int socket, void *buf, size_t len, int flags)
}
fastlock_acquire(&rs->rlock);
if (!rs_have_rdata(rs)) {
- ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_have_rdata);
+ ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_conn_have_rdata);
if (ret && errno != ECONNRESET)
goto out;
}
@@ -1084,9 +1094,14 @@ ssize_t rsend(int socket, const void *buf, size_t len,
int flags)
fastlock_acquire(&rs->slock);
for (left = len; left; left -= xfer_size, buf += xfer_size) {
if (!rs_can_send(rs)) {
- ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_can_send);
+ ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
+ rs_conn_can_send);
if (ret)
break;
+ if (rs->state != rs_connected) {
+ ret = ERR(ECONNRESET);
+ break;
+ }
}
if (olen < left) {
@@ -1193,9 +1208,14 @@ static ssize_t rsendv(int socket, const struct iovec
*iov, int iovcnt, int flags
fastlock_acquire(&rs->slock);
for (left = len; left; left -= xfer_size) {
if (!rs_can_send(rs)) {
- ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_can_send);
+ ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
+ rs_conn_can_send);
if (ret)
break;
+ if (rs->state != rs_connected) {
+ ret = ERR(ECONNRESET);
+ break;
+ }
}
if (olen < left) {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 5/16] librdmacm: Delay ACM connection until resolving an address
Avoid creating a connection to the ACM service when
it's not needed. For example, if the user of the librdmacm
is a server application, it will not use ACM services.
Signed-off-by: Sean Hefty
---
src/acm.c | 22 +-
src/cma.c |2 --
src/cma.h |2 --
3 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/src/acm.c b/src/acm.c
index 9c65919..5bdeb88 100755
--- a/src/acm.c
+++ b/src/acm.c
@@ -89,26 +89,30 @@ static void ucma_set_server_port(void)
void ucma_ib_init(void)
{
struct sockaddr_in addr;
+ static int init;
int ret;
+ if (init)
+ return;
+
+ pthread_mutex_lock(&acm_lock);
ucma_set_server_port();
sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (sock < 0)
- return;
+ goto out;
memset(&addr, 0, sizeof addr);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
addr.sin_port = htons(server_port);
ret = connect(sock, (struct sockaddr *) &addr, sizeof(addr));
- if (ret)
- goto err;
-
- return;
-
-err:
- close(sock);
- sock = -1;
+ if (ret) {
+ close(sock);
+ sock = -1;
+ }
+out:
+ init = 1;
+ pthread_mutex_unlock(&acm_lock);
}
void ucma_ib_cleanup(void)
diff --git a/src/cma.c b/src/cma.c
index 9cd34cf..2121d06 100755
--- a/src/cma.c
+++ b/src/cma.c
@@ -272,8 +272,6 @@ int ucma_init(void)
ib += (cma_dev->verbs->device->transport_type ==
IBV_TRANSPORT_IB);
}
- if (ib)
- ucma_ib_init();
cma_dev_cnt = dev_cnt;
ucma_set_af_ib_support();
pthread_mutex_unlock(&mut);
diff --git a/src/cma.h b/src/cma.h
index 7703fe8..0d94e8b 100644
--- a/src/cma.h
+++ b/src/cma.h
@@ -117,11 +117,9 @@ extern int af_ib_support;
#define RAI_ROUTEONLY 0x0100
#ifdef USE_IB_ACM
-void ucma_ib_init();
void ucma_ib_cleanup();
void ucma_ib_resolve(struct rdma_addrinfo **rai, struct rdma_addrinfo *hints);
#else
-#define ucma_ib_init()
#define ucma_ib_cleanup()
#define ucma_ib_resolve(x, y)
#endif
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/16] librdmacm/acm: Use -1 to indicate an invalid socket rather than 0
socket() can return 0 as a valid socket. This can happen
when using a daemon that closes stdin/out/err.
Signed-off-by: Sean Hefty
---
src/acm.c |8
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/acm.c b/src/acm.c
index bcf11da..9c65919 100755
--- a/src/acm.c
+++ b/src/acm.c
@@ -61,7 +61,7 @@ typedef struct acm_msg cma_acm_msg_t;
#endif
static pthread_mutex_t acm_lock = PTHREAD_MUTEX_INITIALIZER;
-static int sock;
+static int sock = -1;
static short server_port = 6125;
struct ib_connect_hdr {
@@ -108,12 +108,12 @@ void ucma_ib_init(void)
err:
close(sock);
- sock = 0;
+ sock = -1;
}
void ucma_ib_cleanup(void)
{
- if (sock > 0) {
+ if (sock >= 0) {
shutdown(sock, SHUT_RDWR);
close(sock);
}
@@ -322,7 +322,7 @@ void ucma_ib_resolve(struct rdma_addrinfo **rai, struct
rdma_addrinfo *hints)
struct acm_ep_addr_data *data;
int ret;
- if (sock <= 0)
+ if (sock < 0)
return;
memset(&msg, 0, sizeof msg);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 7/16] librdmacm/rsockets: Reduce QP size if larger than hardware maximums
When porting rsockets to iwarp, it was discovered that the default
QP size (512) was larger than that supported by the hardware.
Decrease the size of the QP if the default size is larger than
the maximum supported by the hardware.
Signed-off-by: Sean Hefty
---
src/cma.c | 10
src/cma.h |3 ++-
src/rsocket.c | 68 ++---
3 files changed, 62 insertions(+), 19 deletions(-)
diff --git a/src/cma.c b/src/cma.c
index 2121d06..f688c4c 100755
--- a/src/cma.c
+++ b/src/cma.c
@@ -76,6 +76,7 @@ struct cma_device {
struct ibv_pd *pd;
uint64_tguid;
int port_cnt;
+ int max_qpsize;
uint8_t max_initiator_depth;
uint8_t max_responder_resources;
};
@@ -267,6 +268,7 @@ int ucma_init(void)
}
cma_dev->port_cnt = attr.phys_port_cnt;
+ cma_dev->max_qpsize = attr.max_qp_wr;
cma_dev->max_initiator_depth = (uint8_t)
attr.max_qp_init_rd_atom;
cma_dev->max_responder_resources = (uint8_t)
attr.max_qp_rd_atom;
ib += (cma_dev->verbs->device->transport_type ==
IBV_TRANSPORT_IB);
@@ -2215,3 +2217,11 @@ void rdma_destroy_ep(struct rdma_cm_id *id)
rdma_destroy_id(id);
}
+
+int ucma_max_qpsize(struct rdma_cm_id *id)
+{
+ struct cma_id_private *id_priv;
+
+ id_priv = container_of(id, struct cma_id_private, id);
+ return id_priv->cma_dev->max_qpsize;
+}
diff --git a/src/cma.h b/src/cma.h
index 0d94e8b..ffbcbf2 100644
--- a/src/cma.h
+++ b/src/cma.h
@@ -67,7 +67,7 @@ static inline uint64_t htonll(uint64_t x) { return x; }
static inline uint64_t ntohll(uint64_t x) { return x; }
#endif
-#define min(a, b) (a < b ? a : b)
+#define min(a, b) ((a) < (b) ? a : b)
/*
* Fast synchronization for low contention locking.
@@ -104,6 +104,7 @@ static inline void fastlock_release(fastlock_t *lock)
}
#endif /* DEFINE_ATOMICS */
+int ucma_max_qpsize(struct rdma_cm_id *id);
int ucma_complete(struct rdma_cm_id *id);
static inline int ERR(int err)
{
diff --git a/src/rsocket.c b/src/rsocket.c
index 8c96dc1..59cc13c 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -55,6 +55,8 @@
#define RS_OLAP_START_SIZE 2048
#define RS_MAX_TRANSFER 65536
#define RS_QP_SIZE 512
+#define RS_QP_MAX_SIZE 0xFFFE
+#define RS_QP_MIN_SIZE 8
#define RS_QP_CTRL_SIZE 4
#define RS_CONN_RETRIES 6
#define RS_SGL_SIZE 2
@@ -160,7 +162,9 @@ struct rsocket {
int sbuf_bytes_avail;
uint16_t sseq_no;
uint16_t sseq_comp;
+ uint16_t sq_size;
+ uint16_t rq_size;
uint16_t rseq_no;
uint16_t rseq_comp;
int rbuf_bytes_avail;
@@ -168,7 +172,7 @@ struct rsocket {
int rbuf_offset;
int rmsg_head;
int rmsg_tail;
- struct rs_msg rmsg[RS_QP_SIZE + 1];
+ struct rs_msg *rmsg;
int remote_sge;
struct rs_sge remote_sgl;
@@ -222,8 +226,17 @@ static struct rsocket *rs_alloc(struct rsocket
*inherited_rs)
return NULL;
rs->index = -1;
- rs->sbuf_size = inherited_rs ? inherited_rs->sbuf_size : RS_BUF_SIZE;
- rs->rbuf_size = inherited_rs ? inherited_rs->rbuf_size : RS_BUF_SIZE;
+ if (inherited_rs) {
+ rs->sbuf_size = inherited_rs->sbuf_size;
+ rs->rbuf_size = inherited_rs->rbuf_size;
+ rs->sq_size = inherited_rs->sq_size;
+ rs->rq_size = inherited_rs->rq_size;
+ rs->ctrl_avail = inherited_rs->ctrl_avail;
+ } else {
+ rs->sbuf_size = rs->rbuf_size = RS_BUF_SIZE;
+ rs->sq_size = rs->rq_size = RS_QP_SIZE;
+ rs->ctrl_avail = RS_QP_CTRL_SIZE;
+ }
fastlock_init(&rs->slock);
fastlock_init(&rs->rlock);
fastlock_init(&rs->cq_lock);
@@ -244,8 +257,24 @@ static int rs_set_nonblocking(struct rsocket *rs, long arg)
return ret;
}
+static void rs_set_qp_size(struct rsocket *rs)
+{
+ uint16_t max_size;
+
+ max_size = min(ucma_max_qpsize(rs->cm_id), RS_QP_MAX_SIZE);
+
+ if (rs->sq_size > max_size)
+ rs->sq_size = max_size;
+ if (rs->rq_size > max_size)
+ rs->rq_size = max_size;
+}
+
static int rs_init_bufs(struct rsocket *rs)
{
+ rs->rmsg = calloc(rs->rq_size + 1, sizeof(*rs->rmsg));
+ if (!rs->rmsg)
+ return -1;
+
rs->sbuf = calloc(rs->sbuf_size, sizeof(*rs->sbuf));
if (!rs->sbuf)
return -1;
@@ -273,9 +302,8 @@ static int rs_init_bufs(struct rsocket *rs)
rs->rbuf_free_offset = rs->rbuf_size >> 1;
rs->rbuf_bytes_avail = rs->rbuf_size >> 1;
- rs->ctrl_avail = RS_QP_CTRL_SIZE;
- rs->sqe_avail = RS_QP_S
[PATCH 6/16] librdmacm/rs-preload: Handle recursive socket() calls
When ACM support is enabled in the librdmacm, it will attempt to
establish a socket connection to the ACM daemon. When the rsocket
preload library is in use, this can result in a recursive call
to socket() that results in the library hanging. The resulting
call stack is:
socket() -> rsocket() -> rdma_create_id() -> ucma_init() ->
socket() -> rsocket() -> rdma_create_id() -> ucma_init()
The second call to ucma_init() hangs because initialization is
still pending.
Fix this by checking for recursive calls to socket() in the preload
library. When detected, call the real socket() call instead of
directing the call back into rsockets(). Since rsockets is a part
of the librdmacm, it can call rsockets directly if it wants to use
rsockets instead of standard sockets.
This problem and the cause was reported by Chet Murthy
Signed-off-by: Sean Hefty
---
src/preload.c | 12 +---
1 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/src/preload.c b/src/preload.c
index 87778eb..99afd26 100644
--- a/src/preload.c
+++ b/src/preload.c
@@ -287,21 +287,27 @@ err:
int socket(int domain, int type, int protocol)
{
+ static __thread int recursive;
int index, ret;
+ if (recursive)
+ goto real;
+
init_preload();
index = fd_open();
if (index < 0)
return index;
+ recursive = 1;
ret = rsocket(domain, type, protocol);
+ recursive = 0;
if (ret >= 0) {
fd_store(index, ret, fd_rsocket);
return index;
- } else {
- fd_close(index, &ret);
- return real_socket(domain, type, protocol);
}
+ fd_close(index, &ret);
+real:
+ return real_socket(domain, type, protocol);
}
int bind(int socket, const struct sockaddr *addr, socklen_t addrlen)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 14/16] librdmacm/rstream: Add option to specify size of send/recv buffers
Signed-off-by: Sean Hefty
---
examples/rstream.c | 34 +-
man/rstream.1 |5 -
2 files changed, 17 insertions(+), 22 deletions(-)
diff --git a/examples/rstream.c b/examples/rstream.c
index c440f04..df36e34 100644
--- a/examples/rstream.c
+++ b/examples/rstream.c
@@ -81,6 +81,7 @@ static int size_option;
static int iterations = 1;
static int transfer_size = 1000;
static int transfer_count = 1000;
+static int buffer_size;
static char test_name[9] = "custom";
static char *port = "7471";
static char *dst_addr;
@@ -349,33 +350,20 @@ out:
static void set_options(int rs)
{
- int val, optname, ret;
- long long bytes;
- socklen_t size;
+ int val;
- bytes = transfer_size * transfer_count * iterations;
- for (optname = SO_SNDBUF; ; optname = SO_RCVBUF) {
- size = sizeof val;
- ret = rs_getsockopt(rs, SOL_SOCKET, optname, (void *) &val,
&size);
- if (ret)
- break;
-
- if (val < bytes) {
- size = sizeof val;
- val = ((val << 2) > bytes) ? bytes : (val << 2);
- rs_setsockopt(rs, SOL_SOCKET, optname, (void *) &val,
size);
- }
-
- if (optname == SO_RCVBUF)
- break;
+ if (buffer_size) {
+ rs_setsockopt(rs, SOL_SOCKET, SO_SNDBUF, (void *) &buffer_size,
+ sizeof buffer_size);
+ rs_setsockopt(rs, SOL_SOCKET, SO_RCVBUF, (void *) &buffer_size,
+ sizeof buffer_size);
}
val = 1;
rs_setsockopt(rs, IPPROTO_TCP, TCP_NODELAY, (void *) &val, sizeof(val));
- if (flags & MSG_DONTWAIT) {
+ if (flags & MSG_DONTWAIT)
rs_fcntl(rs, F_SETFL, O_NONBLOCK);
- }
}
static int server_connect(void)
@@ -586,7 +574,7 @@ int main(int argc, char **argv)
{
int op, ret;
- while ((op = getopt(argc, argv, "s:b:I:C:S:p:T:")) != -1) {
+ while ((op = getopt(argc, argv, "s:b:B:I:C:S:p:T:")) != -1) {
switch (op) {
case 's':
dst_addr = optarg;
@@ -594,6 +582,9 @@ int main(int argc, char **argv)
case 'b':
src_addr = optarg;
break;
+ case 'B':
+ buffer_size = atoi(optarg);
+ break;
case 'I':
custom = 1;
iterations = atoi(optarg);
@@ -621,6 +612,7 @@ int main(int argc, char **argv)
printf("usage: %s\n", argv[0]);
printf("\t[-s server_address]\n");
printf("\t[-b bind_address]\n");
+ printf("\t[-B buffer_size]\n");
printf("\t[-I iterations]\n");
printf("\t[-C transfer_count]\n");
printf("\t[-S transfer_size or all]\n");
diff --git a/man/rstream.1 b/man/rstream.1
index 0909932..ea87b95 100644
--- a/man/rstream.1
+++ b/man/rstream.1
@@ -4,7 +4,7 @@ rstream \- streaming over RDMA ping-pong test.
.SH SYNOPSIS
.sp
.nf
-\fIrstream\fR [-s server_address] [-b bind_address]
+\fIrstream\fR [-s server_address] [-b bind_address] [-B buffer_size]
[-I iterations] [-C transfer_count]
[-S transfer_size] [-p server_port] [-T test_option]
.fi
@@ -21,6 +21,9 @@ This option must be specified by the client.
\-b bind_address
The local network address to bind to.
.TP
+\-B buffer_size
+Indicates the size of the send and receive network buffers.
+.TP
\-I iterations
The number of times that the specified number of messages will be
exchanged between the client and server. (default 1000)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 13/16] librdmacm/rsockets: Change the default QP size from 512 to 384
Simple latency/bandwidth tests using rstream showed minimal difference in performance between using a QP sized to 384 entries versus 512. Reduce the overhead of a default rsocket by using 384 entries. A user can request a larger size by calling rsetsockopt. Signed-off-by: Sean Hefty --- src/rsocket.c |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/rsocket.c b/src/rsocket.c index b89ef42..c111797 100644 --- a/src/rsocket.c +++ b/src/rsocket.c @@ -54,7 +54,7 @@ #define RS_INLINE 64 #define RS_OLAP_START_SIZE 2048 #define RS_MAX_TRANSFER 65536 -#define RS_QP_SIZE 512 +#define RS_QP_SIZE 384 #define RS_QP_MAX_SIZE 0xFFFE #define RS_QP_MIN_SIZE 8 #define RS_QP_CTRL_SIZE 4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 15/16] librdmacm/rstream: Use snprintf in place of sprintf
Avoid possible buffer overrun.
Signed-off-by: Sean Hefty
---
examples/rstream.c | 38 +-
1 files changed, 21 insertions(+), 17 deletions(-)
diff --git a/examples/rstream.c b/examples/rstream.c
index df36e34..054d11e 100644
--- a/examples/rstream.c
+++ b/examples/rstream.c
@@ -82,7 +82,7 @@ static int iterations = 1;
static int transfer_size = 1000;
static int transfer_count = 1000;
static int buffer_size;
-static char test_name[9] = "custom";
+static char test_name[10] = "custom";
static char *port = "7471";
static char *dst_addr;
static char *src_addr;
@@ -105,7 +105,7 @@ static void *buf;
#define rs_getsockopt(s,l,n,v,ol) \
use_rs ? rgetsockopt(s,l,n,v,ol) : getsockopt(s,l,n,v,ol)
-static void size_str(char *str, long long size)
+static void size_str(char *str, size_t ssize, long long size)
{
long long base, fraction = 0;
char mag;
@@ -127,22 +127,22 @@ static void size_str(char *str, long long size)
if (size / base < 10)
fraction = (size % base) * 10 / base;
if (fraction) {
- sprintf(str, "%lld.%lld%c", size / base, fraction, mag);
+ snprintf(str, ssize, "%lld.%lld%c", size / base, fraction, mag);
} else {
- sprintf(str, "%lld%c", size / base, mag);
+ snprintf(str, ssize, "%lld%c", size / base, mag);
}
}
-static void cnt_str(char *str, long long cnt)
+static void cnt_str(char *str, size_t ssize, long long cnt)
{
if (cnt >= 10)
- sprintf(str, "%lldb", cnt / 10);
+ snprintf(str, ssize, "%lldb", cnt / 10);
else if (cnt >= 100)
- sprintf(str, "%lldm", cnt / 100);
+ snprintf(str, ssize, "%lldm", cnt / 100);
else if (cnt >= 1000)
- sprintf(str, "%lldk", cnt / 1000);
+ snprintf(str, ssize, "%lldk", cnt / 1000);
else
- sprintf(str, "%lld", cnt);
+ snprintf(str, ssize, "%lld", cnt);
}
static void show_perf(void)
@@ -156,13 +156,13 @@ static void show_perf(void)
/* name size transfers iterations bytes seconds Gb/sec usec/xfer */
printf("%-10s", test_name);
- size_str(str, transfer_size);
+ size_str(str, sizeof str, transfer_size);
printf("%-8s", str);
- cnt_str(str, transfer_count);
+ cnt_str(str, sizeof str, transfer_count);
printf("%-8s", str);
- cnt_str(str, iterations);
+ cnt_str(str, sizeof str, iterations);
printf("%-8s", str);
- size_str(str, bytes);
+ size_str(str, sizeof str, bytes);
printf("%-8s", str);
printf("%8.2fs%10.2f%11.2f\n",
usec / 100., (bytes * 8) / (1000. * usec),
@@ -185,8 +185,10 @@ static int size_to_count(int size)
static void init_latency_test(int size)
{
- size_str(test_name, size);
- sprintf(test_name, "%s_lat", test_name);
+ char sstr[5];
+
+ size_str(sstr, sizeof sstr, size);
+ snprintf(test_name, sizeof test_name, "%s_lat", sstr);
transfer_count = 1;
transfer_size = size;
iterations = size_to_count(transfer_size);
@@ -194,8 +196,10 @@ static void init_latency_test(int size)
static void init_bandwidth_test(int size)
{
- size_str(test_name, size);
- sprintf(test_name, "%s_bw", test_name);
+ char sstr[5];
+
+ size_str(sstr, sizeof sstr, size);
+ snprintf(test_name, sizeof test_name, "%s_bw", sstr);
iterations = 1;
transfer_size = size;
transfer_count = size_to_count(transfer_size);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 12/16] librdmacm/rsockets: Simplify state checks
Signed-off-by: Sean Hefty
---
src/rsocket.c | 26 +++---
1 files changed, 11 insertions(+), 15 deletions(-)
diff --git a/src/rsocket.c b/src/rsocket.c
index ef070a8..b89ef42 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -126,6 +126,9 @@ union rs_wr_id {
};
};
+/*
+ * rsocket states are ordered as passive, connecting, connected, disconnected.
+ */
enum rs_state {
rs_init,
rs_bound,
@@ -134,9 +137,9 @@ enum rs_state {
rs_resolving_route,
rs_connecting,
rs_accepting,
+ rs_connect_error,
rs_connected,
rs_disconnected,
- rs_connect_error,
rs_error
};
@@ -1021,9 +1024,7 @@ ssize_t rrecv(int socket, void *buf, size_t len, int
flags)
int ret;
rs = idm_at(&idm, socket);
- if (rs->state != rs_connected &&
- (rs->state == rs_resolving_addr || rs->state == rs_resolving_route
||
-rs->state == rs_connecting || rs->state == rs_accepting)) {
+ if (rs->state < rs_connected) {
ret = rs_do_connect(rs);
if (ret) {
if (errno == EINPROGRESS)
@@ -1125,7 +1126,7 @@ ssize_t rsend(int socket, const void *buf, size_t len,
int flags)
int ret = 0;
rs = idm_at(&idm, socket);
- if (rs->state != rs_connected) {
+ if (rs->state < rs_connected) {
ret = rs_do_connect(rs);
if (ret) {
if (errno == EINPROGRESS)
@@ -1234,7 +1235,7 @@ static ssize_t rsendv(int socket, const struct iovec
*iov, int iovcnt, int flags
int i, ret = 0;
rs = idm_at(&idm, socket);
- if (rs->state != rs_connected) {
+ if (rs->state < rs_connected) {
ret = rs_do_connect(rs);
if (ret) {
if (errno == EINPROGRESS)
@@ -1380,7 +1381,7 @@ static int rs_poll_rs(struct rsocket *rs, int events,
revents |= POLLOUT;
if (rs->state == rs_disconnected)
revents |= POLLHUP;
- if (rs->state == rs_error)
+ else if (rs->state == rs_error)
revents |= POLLERR;
return revents;
@@ -1421,16 +1422,11 @@ static int rs_poll_arm(struct pollfd *rfds, struct
pollfd *fds, nfds_t nfds)
if (fds[i].revents)
return 1;
- switch (rs->state) {
- case rs_connected:
- case rs_disconnected:
- case rs_error:
+ if (rs->state >= rs_connected)
rfds[i].fd = rs->cm_id->recv_cq_channel->fd;
- break;
- default:
+ else
rfds[i].fd = rs->cm_id->channel->fd;
- break;
- }
+
rfds[i].events = POLLIN;
} else {
rfds[i].fd = fds[i].fd;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 11/16] librdmacm/rs-preload: Use environment variable to set QP size
Allow the user to specify the size of the send/receive queues
and inline data size through environment variables:
RS_SQ_SIZE, RS_RQ_SIZE, and RS_INLINE.
Signed-off-by: Sean Hefty
---
src/preload.c | 39 +++
1 files changed, 39 insertions(+), 0 deletions(-)
diff --git a/src/preload.c b/src/preload.c
index 99afd26..d21389f 100644
--- a/src/preload.c
+++ b/src/preload.c
@@ -97,6 +97,10 @@ static int (*real_fcntl)(int socket, int cmd, ... /* arg */);
static struct index_map idm;
static pthread_mutex_t mut = PTHREAD_MUTEX_INITIALIZER;
+static int sq_size;
+static int rq_size;
+static int sq_inline;
+
enum fd_type {
fd_normal,
fd_rsocket
@@ -196,6 +200,23 @@ static enum fd_type fd_close(int index, int *fd)
return type;
}
+void getenv_options(void)
+{
+ char *var;
+
+ var = getenv("RS_SQ_SIZE");
+ if (var)
+ sq_size = atoi(var);
+
+ var = getenv("RS_RQ_SIZE");
+ if (var)
+ rq_size = atoi(var);
+
+ var = getenv("RS_INLINE");
+ if (var)
+ sq_inline = atoi(var);
+}
+
static void init_preload(void)
{
static int init;
@@ -231,6 +252,8 @@ static void init_preload(void)
real_setsockopt = dlsym(RTLD_NEXT, "setsockopt");
real_getsockopt = dlsym(RTLD_NEXT, "getsockopt");
real_fcntl = dlsym(RTLD_NEXT, "fcntl");
+
+ getenv_options();
init = 1;
out:
pthread_mutex_unlock(&mut);
@@ -285,6 +308,21 @@ err:
return ret;
}
+/*
+ * Use defaults on failure.
+ */
+void set_rsocket_options(int rsocket)
+{
+ if (sq_size)
+ rsetsockopt(rsocket, SOL_RDMA, RDMA_SQSIZE, &sq_size, sizeof
sq_size);
+
+ if (rq_size)
+ rsetsockopt(rsocket, SOL_RDMA, RDMA_RQSIZE, &rq_size, sizeof
rq_size);
+
+ if (sq_inline)
+ rsetsockopt(rsocket, SOL_RDMA, RDMA_INLINE, &sq_inline, sizeof
sq_inline);
+}
+
int socket(int domain, int type, int protocol)
{
static __thread int recursive;
@@ -303,6 +341,7 @@ int socket(int domain, int type, int protocol)
recursive = 0;
if (ret >= 0) {
fd_store(index, ret, fd_rsocket);
+ set_rsocket_options(ret);
return index;
}
fd_close(index, &ret);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 10/16] librdmacm/rsocket: Add option to specify size of inline data
Allow the user to override the default inline data size.
We still require a minimum size in order to transfer receive
buffer update message.
Signed-off-by: Sean Hefty
---
We can eliminate the need for inline entirely by reserving some of the
send buffer space for control messages, but that work is deferred for
a later patch.
include/rdma/rsocket.h |3 ++-
src/rsocket.c | 21 +
2 files changed, 19 insertions(+), 5 deletions(-)
diff --git a/include/rdma/rsocket.h b/include/rdma/rsocket.h
index 9fd9929..65feda9 100644
--- a/include/rdma/rsocket.h
+++ b/include/rdma/rsocket.h
@@ -75,7 +75,8 @@ int rgetsockname(int socket, struct sockaddr *addr, socklen_t
*addrlen);
#define SOL_RDMA 0x1
enum {
RDMA_SQSIZE,
- RDMA_RQSIZE
+ RDMA_RQSIZE,
+ RDMA_INLINE
};
int rsetsockopt(int socket, int level, int optname,
diff --git a/src/rsocket.c b/src/rsocket.c
index f94e48a..ef070a8 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -105,6 +105,7 @@ struct rs_sge {
uint32_t length;
};
+#define RS_MIN_INLINE(sizeof(struct rs_sge))
#define rs_host_is_net() (1 == htonl(1))
#define RS_CONN_FLAG_NET 1
@@ -163,6 +164,7 @@ struct rsocket {
uint16_t sseq_no;
uint16_t sseq_comp;
uint16_t sq_size;
+ uint16_t sq_inline;
uint16_t rq_size;
uint16_t rseq_no;
@@ -229,11 +231,13 @@ static struct rsocket *rs_alloc(struct rsocket
*inherited_rs)
if (inherited_rs) {
rs->sbuf_size = inherited_rs->sbuf_size;
rs->rbuf_size = inherited_rs->rbuf_size;
+ rs->sq_inline = inherited_rs->sq_inline;
rs->sq_size = inherited_rs->sq_size;
rs->rq_size = inherited_rs->rq_size;
rs->ctrl_avail = inherited_rs->ctrl_avail;
} else {
rs->sbuf_size = rs->rbuf_size = RS_BUF_SIZE;
+ rs->sq_inline = RS_INLINE;
rs->sq_size = rs->rq_size = RS_QP_SIZE;
rs->ctrl_avail = RS_QP_CTRL_SIZE;
}
@@ -367,7 +371,7 @@ static int rs_create_ep(struct rsocket *rs)
qp_attr.cap.max_recv_wr = rs->rq_size;
qp_attr.cap.max_send_sge = 2;
qp_attr.cap.max_recv_sge = 1;
- qp_attr.cap.max_inline_data = RS_INLINE;
+ qp_attr.cap.max_inline_data = rs->sq_inline;
ret = rdma_create_qp(rs->cm_id, NULL, &qp_attr);
if (ret)
@@ -1156,7 +1160,7 @@ ssize_t rsend(int socket, const void *buf, size_t len,
int flags)
if (xfer_size > rs->target_sgl[rs->target_sge].length)
xfer_size = rs->target_sgl[rs->target_sge].length;
- if (xfer_size <= RS_INLINE) {
+ if (xfer_size <= rs->sq_inline) {
sge.addr = (uintptr_t) buf;
sge.length = xfer_size;
sge.lkey = 0;
@@ -1277,7 +1281,7 @@ static ssize_t rsendv(int socket, const struct iovec
*iov, int iovcnt, int flags
ret = rs_write_data(rs, rs_wrid(1, xfer_size),
rs->ssgl, 1,
rs_msg_set(RS_OP_DATA, xfer_size),
- xfer_size <= RS_INLINE ?
IBV_SEND_INLINE : 0);
+ xfer_size <= rs->sq_inline ?
IBV_SEND_INLINE : 0);
if (xfer_size < rs_sbuf_left(rs))
rs->ssgl[0].addr += xfer_size;
else
@@ -1291,7 +1295,7 @@ static ssize_t rsendv(int socket, const struct iovec
*iov, int iovcnt, int flags
ret = rs_write_data(rs, rs_wrid(1, xfer_size),
rs->ssgl, 2,
rs_msg_set(RS_OP_DATA, xfer_size),
- xfer_size <= RS_INLINE ?
IBV_SEND_INLINE : 0);
+ xfer_size <= rs->sq_inline ?
IBV_SEND_INLINE : 0);
rs->ssgl[0].addr = (uintptr_t) rs->sbuf +
rs->ssgl[1].length;
}
if (ret)
@@ -1730,6 +1734,11 @@ int rsetsockopt(int socket, int level, int optname,
case RDMA_RQSIZE:
rs->rq_size = min((*(uint32_t *) optval),
RS_QP_MAX_SIZE);
break;
+ case RDMA_INLINE:
+ rs->sq_inline = min(*(uint32_t *) optval,
RS_QP_MAX_SIZE);
+ if (rs->sq_inline < RS_MIN_INLINE)
+ rs->sq_inline = RS_MIN_INLINE;
+ break;
default:
break;
}
@@ -1809,6 +1818,10 @@ int rgetsockopt(int socket, int level, int optname,
*((int *) optval) = rs->rq_size;
*optlen = sizeof(int);
[PATCH 8/16] librdmacm/rsockets: Define options specific to rsockets
Allow a user to control some of the RDMA related attributes of an rsocket through setsockopt/getsockopt. A user specifies that the rsocket should be modified through SOL_RDMA level. This patch provides the initial framework. Subsequent patches will add the configurable parameters. Signed-off-by: Sean Hefty --- This deviates from standard sockets, but it seems to be the best way to provide the user some control over the RDMA portion of an rsocket. include/rdma/rsocket.h |2 ++ src/rsocket.c |5 + 2 files changed, 7 insertions(+), 0 deletions(-) diff --git a/include/rdma/rsocket.h b/include/rdma/rsocket.h index 5310d33..87ee943 100644 --- a/include/rdma/rsocket.h +++ b/include/rdma/rsocket.h @@ -72,6 +72,8 @@ int rselect(int nfds, fd_set *readfds, fd_set *writefds, int rgetpeername(int socket, struct sockaddr *addr, socklen_t *addrlen); int rgetsockname(int socket, struct sockaddr *addr, socklen_t *addrlen); +#define SOL_RDMA 0x1 + int rsetsockopt(int socket, int level, int optname, const void *optval, socklen_t optlen); int rgetsockopt(int socket, int level, int optname, diff --git a/src/rsocket.c b/src/rsocket.c index 59cc13c..ee31e0c 100644 --- a/src/rsocket.c +++ b/src/rsocket.c @@ -1710,6 +1710,8 @@ int rsetsockopt(int socket, int level, int optname, break; } break; + case SOL_RDMA: + break; default: break; } @@ -1775,6 +1777,9 @@ int rgetsockopt(int socket, int level, int optname, break; } break; + case SOL_RDMA: + ret = ENOTSUP; + break; default: ret = ENOTSUP; break; -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 9/16] librdmacm/rsockets: Allow user to specify the QP sizes
Add setsockopt options that allow the user to specify the desired
size of the underlying QP. The provided sizes are used as the
maximum size when creating the QP. The actual sizes of the QP
are the smaller of the user provided maximum and the maximum
sizes supported by the underlying hardware.
A user may retrieve the actual sizes of the QP through the
getsockopt call.
The send and receive queue sizes are specified separately.
Signed-off-by: Sean Hefty
---
include/rdma/rsocket.h |4
src/cma.h |1 +
src/rsocket.c | 38 --
3 files changed, 41 insertions(+), 2 deletions(-)
diff --git a/include/rdma/rsocket.h b/include/rdma/rsocket.h
index 87ee943..9fd9929 100644
--- a/include/rdma/rsocket.h
+++ b/include/rdma/rsocket.h
@@ -73,6 +73,10 @@ int rgetpeername(int socket, struct sockaddr *addr,
socklen_t *addrlen);
int rgetsockname(int socket, struct sockaddr *addr, socklen_t *addrlen);
#define SOL_RDMA 0x1
+enum {
+ RDMA_SQSIZE,
+ RDMA_RQSIZE
+};
int rsetsockopt(int socket, int level, int optname,
const void *optval, socklen_t optlen);
diff --git a/src/cma.h b/src/cma.h
index ffbcbf2..2ee4767 100644
--- a/src/cma.h
+++ b/src/cma.h
@@ -67,6 +67,7 @@ static inline uint64_t htonll(uint64_t x) { return x; }
static inline uint64_t ntohll(uint64_t x) { return x; }
#endif
+#define max(a, b) ((a) > (b) ? a : b)
#define min(a, b) ((a) < (b) ? a : b)
/*
diff --git a/src/rsocket.c b/src/rsocket.c
index ee31e0c..f94e48a 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -265,8 +265,15 @@ static void rs_set_qp_size(struct rsocket *rs)
if (rs->sq_size > max_size)
rs->sq_size = max_size;
+ else if (rs->sq_size < 2)
+ rs->sq_size = 2;
+ if (rs->sq_size <= (RS_QP_CTRL_SIZE << 2))
+ rs->ctrl_avail = 1;
+
if (rs->rq_size > max_size)
rs->rq_size = max_size;
+ else if (rs->rq_size < 2)
+ rs->rq_size = 2;
}
static int rs_init_bufs(struct rsocket *rs)
@@ -1711,12 +1718,27 @@ int rsetsockopt(int socket, int level, int optname,
}
break;
case SOL_RDMA:
+ if (rs->state > rs_listening) {
+ ret = ERR(EINVAL);
+ break;
+ }
+
+ switch (optname) {
+ case RDMA_SQSIZE:
+ rs->sq_size = min((*(uint32_t *) optval),
RS_QP_MAX_SIZE);
+ break;
+ case RDMA_RQSIZE:
+ rs->rq_size = min((*(uint32_t *) optval),
RS_QP_MAX_SIZE);
+ break;
+ default:
+ break;
+ }
break;
default:
break;
}
- if (!ret) {
+ if (!ret && opts) {
if (opt_on)
*opts |= (1 << optname);
else
@@ -1778,7 +1800,19 @@ int rgetsockopt(int socket, int level, int optname,
}
break;
case SOL_RDMA:
- ret = ENOTSUP;
+ switch (optname) {
+ case RDMA_SQSIZE:
+ *((int *) optval) = rs->sq_size;
+ *optlen = sizeof(int);
+ break;
+ case RDMA_RQSIZE:
+ *((int *) optval) = rs->rq_size;
+ *optlen = sizeof(int);
+ break;
+ default:
+ ret = ENOTSUP;
+ break;
+ }
break;
default:
ret = ENOTSUP;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 1/8] infiniband-diags: Obfuscate sensitive output by default
Cause security credentials to not be displayed by default. Instead, the "-K" flag is supported, which must be passed to the command to display valid results. Signed-off-by: Jim Foraker --- doc/rst/common/opt_K.rst |4 +++ doc/rst/saquery.8.in.rst |2 ++ doc/rst/smpquery.8.in.rst |2 ++ include/ibdiag_common.h |9 ++ src/ibaddr.c |2 +- src/ibcacheedit.c |2 +- src/ibccconfig.c |2 +- src/ibccquery.c |2 +- src/ibdiag_common.c | 71 + src/iblinkinfo.c |2 +- src/ibnetdiscover.c |2 +- src/ibping.c |2 +- src/ibqueryerrors.c |2 +- src/ibroute.c |2 +- src/ibsendtrap.c |2 +- src/ibstat.c |2 +- src/ibsysstat.c |2 +- src/ibtracert.c |2 +- src/perfquery.c |2 +- src/saquery.c | 66 - src/sminfo.c |2 +- src/smpdump.c |2 +- src/smpquery.c|6 ++-- src/vendstat.c|2 +- 24 files changed, 139 insertions(+), 55 deletions(-) create mode 100644 doc/rst/common/opt_K.rst diff --git a/doc/rst/common/opt_K.rst b/doc/rst/common/opt_K.rst new file mode 100644 index 000..2b96bf2 --- /dev/null +++ b/doc/rst/common/opt_K.rst @@ -0,0 +1,4 @@ +.. Define the common option -K + +**-K, --show_keys** + show security keys (mkey, smkey, etc.) associated with the request. diff --git a/doc/rst/saquery.8.in.rst b/doc/rst/saquery.8.in.rst index 36696c7..79c8bd3 100644 --- a/doc/rst/saquery.8.in.rst +++ b/doc/rst/saquery.8.in.rst @@ -84,6 +84,8 @@ OPTIONS queries. If non-numeric value (like 'x') is specified then saquery will prompt for a value. +.. include:: common/opt_K.rst + **--slid ** Source LID (PathRecord) **--dlid ** Destination LID (PathRecord) diff --git a/doc/rst/smpquery.8.in.rst b/doc/rst/smpquery.8.in.rst index 3b249df..b422731 100644 --- a/doc/rst/smpquery.8.in.rst +++ b/doc/rst/smpquery.8.in.rst @@ -46,6 +46,8 @@ Current supported operations and their parameters: Set SMSupportsExtendedSpeeds bit 31 in AttributeModifier (only impacts PortInfo queries). +.. include:: common/opt_K.rst + Addressing Flags diff --git a/include/ibdiag_common.h b/include/ibdiag_common.h index 6645322..e989088 100644 --- a/include/ibdiag_common.h +++ b/include/ibdiag_common.h @@ -39,6 +39,7 @@ #ifndef _IBDIAG_COMMON_H_ #define _IBDIAG_COMMON_H_ +#include #include #include #include @@ -50,6 +51,7 @@ extern enum MAD_DEST ibd_dest_type; extern ib_portid_t *ibd_sm_id; extern int ibd_timeout; extern uint32_t ibd_ibnetdisc_flags; +extern int show_keys; /**/ /*External interface */ @@ -64,6 +66,8 @@ extern uint32_t ibd_ibnetdisc_flags; } while (0) #define IBERROR(fmt, ...) iberror(__FUNCTION__, fmt, ## __VA_ARGS__) +#define NOT_DISPLAYED_STR "" + /* not all versions of ib_types.h will have this define */ #ifndef IB_PM_PC_XMIT_WAIT_SUP #define IB_PM_PC_XMIT_WAIT_SUP (CL_HTON16(((uint16_t)1)<<12)) @@ -150,4 +154,9 @@ int resolve_self(char *ca_name, uint8_t ca_port, ib_portid_t *portid, int resolve_portid_str(char *ca_name, uint8_t ca_port, ib_portid_t * portid, char *addr_str, enum MAD_DEST dest_type, ib_portid_t *sm_id, const struct ibmad_port *srcport); +int vsnprint_field(char *buf, size_t n, enum MAD_FIELDS f, int spacing, + const char *format, va_list va_args); +int snprint_field(char *buf, size_t n, enum MAD_FIELDS f, int spacing, + const char *format, ...); +void dump_portinfo(void *pi, int pisize, int tabs); #endif /* _IBDIAG_COMMON_H_ */ diff --git a/src/ibaddr.c b/src/ibaddr.c index 455d941..0735837 100644 --- a/src/ibaddr.c +++ b/src/ibaddr.c @@ -133,7 +133,7 @@ int main(int argc, char **argv) NULL }; - ibdiag_process_opts(argc, argv, NULL, "L", opts, process_opt, + ibdiag_process_opts(argc, argv, NULL, "KL", opts, process_opt, usage_args, usage_examples); argc -= optind; diff --git a/src/ibcacheedit.c b/src/ibcacheedit.c index f9e7dc9..2f9a634 100644 --- a/src/ibcacheedit.c +++ b/src/ibcacheedit.c @@ -269,7 +269,7 @@ int main(int argc, char **argv) }; char *usage_args = " "; - ibdiag_process_opts(argc, argv, NULL, "CDdeGLPstv", + ibdiag_process_opts(argc, argv, NULL, "CDdeGKLPstv", opts, process_opt, usage_args, NULL); diff --git a/src/ibccconfig.c b/src/ibccconfig.c index f84c63c..d796ed8 100644 --- a/src/ibccconfig.c +++ b/src/ibccconfig.c @@ -615,7 +615,7 @@ int main(int argc, char **argv)
[PATCH V3 2/8] infiniband-diags: install config file mode 400
Necessary so that we may add credentials (ie, mkey) to it Signed-off-by: Jim Foraker --- Makefile.am |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile.am b/Makefile.am index f861000..33b5f90 100644 --- a/Makefile.am +++ b/Makefile.am @@ -148,4 +148,4 @@ install-data-hook: fi $(top_srcdir)/config/install-sh -c -m 444 $(top_srcdir)/scripts/IBswcountlimits.pm $(DESTDIR)/$(PERL_INSTALLDIR)/IBswcountlimits.pm $(top_srcdir)/config/install-sh -c -m 444 $(top_srcdir)/etc/error_thresholds $(DESTDIR)/$(sysconfdir)/infiniband-diags - $(top_srcdir)/config/install-sh -c -m 444 $(top_srcdir)/etc/ibdiag.conf $(DESTDIR)/$(sysconfdir)/infiniband-diags + $(top_srcdir)/config/install-sh -c -m 400 $(top_srcdir)/etc/ibdiag.conf $(DESTDIR)/$(sysconfdir)/infiniband-diags -- 1.7.9.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 3/8] infiniband-diags/ibportstate.c: Display MKey, lease, and protect bits
By default, display mkey lease and protection bits.
With a flag, also display mkey itself
Signed-off-by: Jim Foraker
---
doc/rst/ibportstate.8.in.rst |1 +
src/ibportstate.c| 30 --
2 files changed, 25 insertions(+), 6 deletions(-)
diff --git a/doc/rst/ibportstate.8.in.rst b/doc/rst/ibportstate.8.in.rst
index 3af2aaa..1fc9bb2 100644
--- a/doc/rst/ibportstate.8.in.rst
+++ b/doc/rst/ibportstate.8.in.rst
@@ -89,6 +89,7 @@ Debugging flags
.. include:: common/opt_h.rst
.. include:: common/opt_d.rst
.. include:: common/opt_e.rst
+.. include:: common/opt_K.rst
.. include:: common/opt_v.rst
.. include:: common/opt_V.rst
diff --git a/src/ibportstate.c b/src/ibportstate.c
index 5559d18..b2df212 100644
--- a/src/ibportstate.c
+++ b/src/ibportstate.c
@@ -142,7 +142,7 @@ static int get_port_info(ib_portid_t * dest, uint8_t *
data, int portnum,
}
static void show_port_info(ib_portid_t * dest, uint8_t * data, int portnum,
- int espeed_cap)
+ int espeed_cap, int is_switch)
{
char buf[2300];
char val[64];
@@ -201,18 +201,36 @@ static void show_port_info(ib_portid_t * dest, uint8_t *
data, int portnum,
val);
sprintf(buf + strlen(buf), "%s", "\n");
}
+ if (!is_switch || portnum == 0) {
+ if (show_keys) {
+ mad_decode_field(data, IB_PORT_MKEY_F, val);
+ mad_dump_field(IB_PORT_MKEY_F, buf + strlen(buf),
+ sizeof buf - strlen(buf), val);
+ } else
+ snprint_field(buf+strlen(buf), sizeof(buf)-strlen(buf),
+ IB_PORT_MKEY_F, 32, NOT_DISPLAYED_STR);
+ sprintf(buf+strlen(buf), "%s", "\n");
+ mad_decode_field(data, IB_PORT_MKEY_LEASE_F, val);
+ mad_dump_field(IB_PORT_MKEY_LEASE_F, buf + strlen(buf),
+ sizeof buf - strlen(buf), val);
+ sprintf(buf+strlen(buf), "%s", "\n");
+ mad_decode_field(data, IB_PORT_MKEY_PROT_BITS_F, val);
+ mad_dump_field(IB_PORT_MKEY_PROT_BITS_F, buf + strlen(buf),
+ sizeof buf - strlen(buf), val);
+ sprintf(buf+strlen(buf), "%s", "\n");
+ }
printf("# Port info: %s port %d\n%s", portid2str(dest), portnum, buf);
}
static void set_port_info(ib_portid_t * dest, uint8_t * data, int portnum,
- int espeed_cap)
+ int espeed_cap, int is_switch)
{
if (!smp_set_via(data, dest, IB_ATTR_PORT_INFO, portnum, 0, srcport))
IBERROR("smp set portinfo failed");
printf("\nAfter PortInfo set:\n");
- show_port_info(dest, data, portnum, espeed_cap);
+ show_port_info(dest, data, portnum, espeed_cap, is_switch);
}
static void get_ext_port_info(ib_portid_t * dest, uint8_t * data, int portnum)
@@ -460,7 +478,7 @@ int main(int argc, char **argv)
else
printf("%s PortInfo:\n", is_switch ? "Switch" : "CA");
espeed_cap = get_port_info(&portid, data, portnum, is_switch);
- show_port_info(&portid, data, portnum, espeed_cap);
+ show_port_info(&portid, data, portnum, espeed_cap, is_switch);
if (is_mlnx_ext_port_info_supported(devid)) {
get_ext_port_info(&portid, data2, portnum);
show_ext_port_info(&portid, data2, portnum);
@@ -527,7 +545,7 @@ int main(int argc, char **argv)
fdr10);
set_ext_port_info(&portid, data2, portnum);
}
- set_port_info(&portid, data, portnum, is_switch);
+ set_port_info(&portid, data, portnum, espeed_cap, is_switch);
} else if (is_switch && portnum) {
/* Now, make sure PortState is Active */
@@ -596,7 +614,7 @@ int main(int argc, char **argv)
get_ext_port_info(&peerportid, data2,
peerlocalportnum);
show_port_info(&peerportid, data, peerlocalportnum,
- peer_espeed_cap);
+ peer_espeed_cap, is_peer_switch);
if (is_mlnx_ext_port_info_supported(rem_devid))
show_ext_port_info(&peerportid, data2,
peerlocalportnum);
--
1.7.9.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 6/8] infiniband-diags: Add m_key option to config file
Signed-off-by: Jim Foraker
---
etc/ibdiag.conf |2 ++
src/ibdiag_common.c |2 ++
2 files changed, 4 insertions(+)
diff --git a/etc/ibdiag.conf b/etc/ibdiag.conf
index 77f3ce9..2a2334f 100644
--- a/etc/ibdiag.conf
+++ b/etc/ibdiag.conf
@@ -15,3 +15,5 @@
# Default = true
#MLX_EPI=false
+# define a default m_key
+#m_key=0x00
diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
index 069f6e5..d7d784f 100644
--- a/src/ibdiag_common.c
+++ b/src/ibdiag_common.c
@@ -157,6 +157,8 @@ void read_ibdiag_config(const char *file)
} else {
ibd_ibnetdisc_flags &= ~IBND_CONFIG_MLX_EPI;
}
+ } else if (strncmp(name, "m_key", strlen("m_key")) == 0) {
+ ibd_mkey = strtoull(val_str, 0, 0);
}
}
--
1.7.9.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 4/8] infiniband-diags: Allow specification of an mkey to use on the command line
Signed-off-by: Jim Foraker --- doc/rst/common/opt_m.rst|6 ++ doc/rst/ibaddr.8.in.rst |1 + doc/rst/ibccconfig.8.in.rst |1 + doc/rst/ibccquery.8.in.rst |1 + doc/rst/iblinkinfo.8.in.rst |1 + doc/rst/ibnetdiscover.8.in.rst |1 + doc/rst/ibportstate.8.in.rst|1 + doc/rst/ibqueryerrors.8.in.rst |1 + doc/rst/ibroute.8.in.rst|1 + doc/rst/ibtracert.8.in.rst |6 ++ doc/rst/perfquery.8.in.rst |1 + doc/rst/sminfo.8.in.rst |1 + doc/rst/smpquery.8.in.rst |1 + include/ibdiag_common.h |1 + libibnetdisc/include/infiniband/ibnetdisc.h |3 ++- libibnetdisc/src/ibnetdisc.c|1 + src/ibaddr.c|2 ++ src/ibcacheedit.c |2 +- src/ibccconfig.c|2 ++ src/ibccquery.c |2 ++ src/ibdiag_common.c | 13 + src/iblinkinfo.c|3 +++ src/ibnetdiscover.c |2 +- src/ibping.c|2 +- src/ibportstate.c |2 ++ src/ibqueryerrors.c |3 +++ src/ibroute.c |2 ++ src/ibsendtrap.c|2 ++ src/ibstat.c|2 +- src/ibsysstat.c |2 +- src/ibtracert.c |4 +++- src/perfquery.c |2 ++ src/saquery.c |2 +- src/sminfo.c|2 ++ src/smpquery.c |2 ++ src/vendstat.c |2 +- 36 files changed, 74 insertions(+), 9 deletions(-) create mode 100644 doc/rst/common/opt_m.rst diff --git a/doc/rst/common/opt_m.rst b/doc/rst/common/opt_m.rst new file mode 100644 index 000..13082e8 --- /dev/null +++ b/doc/rst/common/opt_m.rst @@ -0,0 +1,6 @@ +.. Define the common option -m + +**-m, --m_key ** +use the specified M_key for requests. If non-numeric value (like 'x') +is specified then a value will be prompted for. + diff --git a/doc/rst/ibaddr.8.in.rst b/doc/rst/ibaddr.8.in.rst index 93de897..a1fc42b 100644 --- a/doc/rst/ibaddr.8.in.rst +++ b/doc/rst/ibaddr.8.in.rst @@ -68,6 +68,7 @@ Port Selection flags Configuration flags --- +.. include:: common/opt_m.rst .. include:: common/opt_z-config.rst FILES diff --git a/doc/rst/ibccconfig.8.in.rst b/doc/rst/ibccconfig.8.in.rst index 3ab31b6..44d8c72 100644 --- a/doc/rst/ibccconfig.8.in.rst +++ b/doc/rst/ibccconfig.8.in.rst @@ -67,6 +67,7 @@ Port Selection flags Configuration flags --- +.. include:: common/opt_m.rst .. include:: common/opt_z-config.rst diff --git a/doc/rst/ibccquery.8.in.rst b/doc/rst/ibccquery.8.in.rst index 89b741c..183f2e3 100644 --- a/doc/rst/ibccquery.8.in.rst +++ b/doc/rst/ibccquery.8.in.rst @@ -66,6 +66,7 @@ Port Selection flags Configuration flags --- +.. include:: common/opt_m.rst .. include:: common/opt_z-config.rst FILES diff --git a/doc/rst/iblinkinfo.8.in.rst b/doc/rst/iblinkinfo.8.in.rst index 1496bfc..de23de2 100644 --- a/doc/rst/iblinkinfo.8.in.rst +++ b/doc/rst/iblinkinfo.8.in.rst @@ -99,6 +99,7 @@ Configuration flags .. include:: common/opt_o-outstanding_smps.rst .. include:: common/opt_node_name_map.rst .. include:: common/opt_t.rst +.. include:: common/opt_m.rst Debugging flags --- diff --git a/doc/rst/ibnetdiscover.8.in.rst b/doc/rst/ibnetdiscover.8.in.rst index fb7c4c9..c858200 100644 --- a/doc/rst/ibnetdiscover.8.in.rst +++ b/doc/rst/ibnetdiscover.8.in.rst @@ -83,6 +83,7 @@ Configuration flags .. include:: common/opt_o-outstanding_smps.rst .. include:: common/opt_node_name_map.rst .. include:: common/opt_t.rst +.. include:: common/opt_m.rst Debugging flags --- diff --git a/doc/rst/ibportstate.8.in.rst b/doc/rst/ibportstate.8.in.rst index 1fc9bb2..fca3f7a 100644 --- a/doc/rst/ibportstate.8.in.rst +++ b/doc/rst/ibportstate.8.in.rst @@ -82,6 +82,7 @@ Configuration flags .. include:: common/opt_z-config.rst .. include:: common/opt_t.rst +.. include:: common/opt_m.rst Debugging flags --- diff --git a/doc/rst/ibqueryerrors.8.in.rst b/doc/rst/ibqueryerrors.8.in.rst index 15d488d..d0e9bc9 100644 --- a/doc/rst/ibqueryerrors.8.in.rst +++ b/doc/rst/ibqueryerrors.8.in.rst @@ -102,6 +102,7 @@ Configuration flags .. include:: common/opt_o-outstanding_smps.rst .. include:: common/opt_node_name_map.rst .. include:: common/opt_t.rst +.. include:: common/opt_m.rst Debugging flags -
[PATCH V3 8/8] infiniband-diags: Add sa_smkey option to config file
Defines a default for the smkey used in SA requests
Signed-off-by: Jim Foraker
---
etc/ibdiag.conf |3 +++
src/ibdiag_common.c |3 +++
2 files changed, 6 insertions(+)
diff --git a/etc/ibdiag.conf b/etc/ibdiag.conf
index 2a2334f..1e6149d 100644
--- a/etc/ibdiag.conf
+++ b/etc/ibdiag.conf
@@ -17,3 +17,6 @@
# define a default m_key
#m_key=0x00
+
+# default smkey to be used for SA requests
+#sa_smkey=0x00
diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
index c40490a..1301810 100644
--- a/src/ibdiag_common.c
+++ b/src/ibdiag_common.c
@@ -160,6 +160,9 @@ void read_ibdiag_config(const char *file)
}
} else if (strncmp(name, "m_key", strlen("m_key")) == 0) {
ibd_mkey = strtoull(val_str, 0, 0);
+ } else if (strncmp(name, "sa_smkey",
+ strlen("sa_smkey")) == 0) {
+ ibd_sa_smkey = strtoull(val_str, 0, 0);
}
}
--
1.7.9.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 7/8] ib-diags/saquery: Fix smkey handling
smkey is already defined as a global inside saquery.c, so remove
broken support for passing it around as a function parameter
Signed-off-by: Jim Foraker
---
include/ibdiag_common.h |1 +
src/ibdiag_common.c |1 +
src/saquery.c | 60 +++
3 files changed, 31 insertions(+), 31 deletions(-)
diff --git a/include/ibdiag_common.h b/include/ibdiag_common.h
index 0120203..63b8ab0 100644
--- a/include/ibdiag_common.h
+++ b/include/ibdiag_common.h
@@ -52,6 +52,7 @@ extern ib_portid_t *ibd_sm_id;
extern int ibd_timeout;
extern uint32_t ibd_ibnetdisc_flags;
extern uint64_t ibd_mkey;
+extern uint64_t ibd_sa_smkey;
extern int show_keys;
/**/
diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
index d7d784f..c40490a 100644
--- a/src/ibdiag_common.c
+++ b/src/ibdiag_common.c
@@ -72,6 +72,7 @@ int ibd_ca_port = 0;
int ibd_timeout = 0;
uint32_t ibd_ibnetdisc_flags = IBND_CONFIG_MLX_EPI;
uint64_t ibd_mkey;
+uint64_t ibd_sa_smkey;
int show_keys = 0;
static const char *prog_name;
diff --git a/src/saquery.c b/src/saquery.c
index dec990a..9ca7375 100644
--- a/src/saquery.c
+++ b/src/saquery.c
@@ -85,7 +85,6 @@ struct query_cmd {
static char *node_name_map_file = NULL;
static nn_map_t *node_name_map = NULL;
-static uint64_t smkey = 1;
/**
* Declare some globals because I don't want this to be too complex.
@@ -724,11 +723,11 @@ static void dump_results(struct sa_query_result *r, void
(*dump_func) (void *))
*/
static int get_any_records(bind_handle_t h,
uint16_t attr_id, uint32_t attr_mod,
- ib_net64_t comp_mask, void *attr, uint64_t sm_key,
+ ib_net64_t comp_mask, void *attr,
struct sa_query_result *result)
{
int ret = sa_query(h, IB_MAD_METHOD_GET_TABLE, attr_id, attr_mod,
- cl_ntoh64(comp_mask), sm_key, attr, result);
+ cl_ntoh64(comp_mask), ibd_sa_smkey, attr, result);
if (ret) {
fprintf(stderr, "Query SA failed: %s\n", strerror(ret));
return ret;
@@ -744,12 +743,12 @@ static int get_any_records(bind_handle_t h,
static int get_and_dump_any_records(bind_handle_t h, uint16_t attr_id,
uint32_t attr_mod, ib_net64_t comp_mask,
- void *attr, uint64_t sm_key,
+ void *attr,
void (*dump_func) (void *))
{
struct sa_query_result result;
int ret = get_any_records(h, attr_id, attr_mod, comp_mask, attr,
- sm_key, &result);
+ &result);
if (ret)
return ret;
@@ -761,18 +760,17 @@ static int get_and_dump_any_records(bind_handle_t h,
uint16_t attr_id,
/**
* Get all the records available for requested query type.
*/
-static int get_all_records(bind_handle_t h, uint16_t attr_id, int trusted,
+static int get_all_records(bind_handle_t h, uint16_t attr_id,
struct sa_query_result *result)
{
- return get_any_records(h, attr_id, 0, 0, NULL, trusted ? smkey : 0,
- result);
+ return get_any_records(h, attr_id, 0, 0, NULL, result);
}
static int get_and_dump_all_records(bind_handle_t h, uint16_t attr_id,
- int trusted, void (*dump_func) (void *))
+ void (*dump_func) (void *))
{
struct sa_query_result result;
- int ret = get_all_records(h, attr_id, 0, &result);
+ int ret = get_all_records(h, attr_id, &result);
if (ret)
return ret;
@@ -792,7 +790,7 @@ static int get_lid_from_name(bind_handle_t h, const char
*name, uint16_t * lid)
int ret;
struct sa_query_result result;
- ret = get_all_records(h, IB_SA_ATTR_NODERECORD, 0, &result);
+ ret = get_all_records(h, IB_SA_ATTR_NODERECORD, &result);
if (ret)
return ret;
@@ -892,7 +890,7 @@ static int get_issm_records(bind_handle_t h, ib_net32_t
capability_mask,
attr.port_info.capability_mask = capability_mask;
return get_any_records(h, IB_SA_ATTR_PORTINFORECORD, 1 << 31,
- IB_PIR_COMPMASK_CAPMASK, &attr, 0, result);
+ IB_PIR_COMPMASK_CAPMASK, &attr, result);
}
static int print_node_records(bind_handle_t h)
@@ -901,7 +899,7 @@ static int print_node_records(bind_handle_t h)
int ret;
struct sa_query_result result;
- ret = get_all_records(h, IB_SA_ATTR_NODERECORD, 0, &result);
+ ret = get_all_records(h, IB_SA_ATTR_NODERECORD, &result);
if (ret)
return ret;
@@ -944,7 +942,7 @@ static int get_print_class_port_info(bind_handle_t h)
{
struc
[PATCH V3 5/8] infiniband-diags/ibportstate.c: Support changing MKey, lease, and protect bits
Signed-off-by: Jim Foraker
---
doc/rst/ibportstate.8.in.rst |6
src/ibportstate.c| 72 ++
2 files changed, 65 insertions(+), 13 deletions(-)
diff --git a/doc/rst/ibportstate.8.in.rst b/doc/rst/ibportstate.8.in.rst
index fca3f7a..052e86a 100644
--- a/doc/rst/ibportstate.8.in.rst
+++ b/doc/rst/ibportstate.8.in.rst
@@ -61,6 +61,12 @@ OPTIONS
done on a switch. This peer port validation feature of query op
requires LID routing to be functioning in the subnet.
+**mkey, mkeylease, and mkeyprot** are only allowed on CAs, routers, or
+switch port 0 (An error is generated if attempted on external switch
+ports). Hexadecimal and octal mkeys may be specified by prepending the
+key with '0x' or '0', respectively. If a non-numeric value (like 'x')
+is specified for the mkey, then ibportstate will prompt for a value.
+
Addressing Flags
diff --git a/src/ibportstate.c b/src/ibportstate.c
index 02cf333..743a276 100644
--- a/src/ibportstate.c
+++ b/src/ibportstate.c
@@ -41,6 +41,7 @@
#include
#include
#include
+#include
#include
#include
@@ -64,22 +65,28 @@ enum port_ops {
LID,
SMLID,
LMC,
+ MKEY,
+ MKEYLEASE,
+ MKEYPROT,
};
struct ibmad_port *srcport;
-int speed = 0; /* no state change */
-int espeed = 0; /* no state change */
-int fdr10 = 0; /* no state change */
-int width = 0; /* no state change */
-int lid;
-int smlid;
-int lmc;
-int mtu;
-int vls = 0; /* no state change */
+uint64_t speed = 0; /* no state change */
+uint64_t espeed = 0; /* no state change */
+uint64_t fdr10 = 0; /* no state change */
+uint64_t width = 0; /* no state change */
+uint64_t lid;
+uint64_t smlid;
+uint64_t lmc;
+uint64_t mtu;
+uint64_t vls = 0; /* no state change */
+uint64_t mkey;
+uint64_t mkeylease;
+uint64_t mkeyprot;
struct {
const char *name;
- int *val;
+ uint64_t *val;
int set;
} port_args[] = {
{"query", NULL, 0}, /* QUERY */
@@ -98,6 +105,9 @@ struct {
{"lid", &lid, 0}, /* LID */
{"smlid", &smlid, 0}, /* SMLID */
{"lmc", &lmc, 0}, /* LMC */
+ {"mkey", &mkey, 0}, /* MKEY */
+ {"mkeylease", &mkeylease, 0}, /* MKEY LEASE */
+ {"mkeyprot", &mkeyprot, 0}, /* MKEY PROTECT BITS */
};
#define NPORT_ARGS (sizeof(port_args) / sizeof(port_args[0]))
@@ -367,9 +377,10 @@ int main(int argc, char **argv)
int i;
uint16_t devid, rem_devid;
long val;
+ char *endp;
char usage_args[] = " []\n"
"\nSupported ops: enable, disable, reset, speed, width, query,\n"
- "\tdown, arm, active, vls, mtu, lid, smlid, lmc\n";
+ "\tdown, arm, active, vls, mtu, lid, smlid, lmc, mkey, mkeylease,
mkeyprot\n";
const char *usage_examples[] = {
"3 1 disable\t\t\t# by lid",
"-G 0x2C9000100D051 1 enable\t# by guid",
@@ -423,7 +434,7 @@ int main(int argc, char **argv)
if (++i >= argc)
IBERROR("%s requires an additional parameter",
port_args[j].name);
- val = strtol(argv[i], 0, 0);
+ val = strtoull(argv[i], 0, 0);
switch (j) {
case SPEED:
if (val < 0 || val > 15)
@@ -461,8 +472,29 @@ int main(int argc, char **argv)
case LMC:
if (val < 0 || val > 7)
IBERROR("invalid lmc value %ld", val);
+ break;
+ case MKEY:
+ errno = 0;
+ val = strtoull(argv[i], &endp, 0);
+ if (errno || *endp != '\0') {
+ errno = 0;
+ val = strtoull(getpass("New M_Key: "),
+ &endp, 0);
+ if (errno || *endp != '\0') {
+ IBERROR("Bad new M_Key\n");
+ }
+ }
+ /* All 64-bit values are legal */
+ break;
+ case MKEYLEASE:
+ if (val < 0 || val > 0x)
+ IBERROR("invalid mkey lease time %ld",
val);
+ break;
+ case MKEYPROT:
+ if (val < 0 || val > 3)
+ IBERROR("invalid mkey protection bit
setting %ld", val);
}
- *port_args[j].val = (int)val;
+
Re: [PATCH][TRIVIAL] infiniband-diags: clean up exclude arguments to ibdiag_process_opts()
On Wed, 30 May 2012 10:09:02 -0700 Jim Foraker wrote: > Alphabetize for standardization and greater clarity Good idea, thanks applied, Ira > > Signed-off-by: Jim Foraker > --- > src/ibcacheedit.c |2 +- > src/iblinkinfo.c|2 +- > src/ibnetdiscover.c |2 +- > src/ibqueryerrors.c |2 +- > src/ibsendtrap.c|2 +- > src/ibstat.c|2 +- > src/saquery.c |2 +- > 7 files changed, 7 insertions(+), 7 deletions(-) > > diff --git a/src/ibcacheedit.c b/src/ibcacheedit.c > index 28b8b21..f9e7dc9 100644 > --- a/src/ibcacheedit.c > +++ b/src/ibcacheedit.c > @@ -269,7 +269,7 @@ int main(int argc, char **argv) > }; > char *usage_args = " "; > > - ibdiag_process_opts(argc, argv, NULL, "edCPDLGtsv", > + ibdiag_process_opts(argc, argv, NULL, "CDdeGLPstv", > opts, process_opt, usage_args, > NULL); > > diff --git a/src/iblinkinfo.c b/src/iblinkinfo.c > index 36aa6d1..a817920 100644 > --- a/src/iblinkinfo.c > +++ b/src/iblinkinfo.c > @@ -619,7 +619,7 @@ int main(int argc, char **argv) > }; > char usage_args[] = ""; > > - ibdiag_process_opts(argc, argv, &config, "SDandlpgRGL", opts, > + ibdiag_process_opts(argc, argv, &config, "aDdGgLlnpRS", opts, > process_opt, usage_args, NULL); > > argc -= optind; > diff --git a/src/ibnetdiscover.c b/src/ibnetdiscover.c > index 8ec9421..4748be9 100644 > --- a/src/ibnetdiscover.c > +++ b/src/ibnetdiscover.c > @@ -1074,7 +1074,7 @@ int main(int argc, char **argv) > }; > char usage_args[] = "[topology-file]"; > > - ibdiag_process_opts(argc, argv, &config, "sGDL", opts, process_opt, > + ibdiag_process_opts(argc, argv, &config, "DGLs", opts, process_opt, > usage_args, NULL); > > f = stdout; > diff --git a/src/ibqueryerrors.c b/src/ibqueryerrors.c > index 4fec240..eb57418 100644 > --- a/src/ibqueryerrors.c > +++ b/src/ibqueryerrors.c > @@ -901,7 +901,7 @@ int main(int argc, char **argv) > char usage_args[] = ""; > > memset(suppressed_fields, 0, sizeof suppressed_fields); > - ibdiag_process_opts(argc, argv, &config, "scnSrRDGL", opts, process_opt, > + ibdiag_process_opts(argc, argv, &config, "cDGLnRrSs", opts, process_opt, > usage_args, NULL); > > argc -= optind; > diff --git a/src/ibsendtrap.c b/src/ibsendtrap.c > index 2d63e35..f98d9cf 100644 > --- a/src/ibsendtrap.c > +++ b/src/ibsendtrap.c > @@ -192,7 +192,7 @@ int main(int argc, char **argv) > snprintf(usage_args + n, sizeof(usage_args) - n, >"\n default behavior is to send \"%s\"", traps[0].trap_name); > > - ibdiag_process_opts(argc, argv, NULL, "DLG", NULL, NULL, > + ibdiag_process_opts(argc, argv, NULL, "DGL", NULL, NULL, > usage_args, NULL); > > argc -= optind; > diff --git a/src/ibstat.c b/src/ibstat.c > index f953cf4..3d971ed 100644 > --- a/src/ibstat.c > +++ b/src/ibstat.c > @@ -292,7 +292,7 @@ int main(int argc, char *argv[]) > NULL > }; > > - ibdiag_process_opts(argc, argv, NULL, "sDGLCPte", opts, process_opt, > + ibdiag_process_opts(argc, argv, NULL, "CDeGLPst", opts, process_opt, > usage_args, usage_examples); > > argc -= optind; > diff --git a/src/saquery.c b/src/saquery.c > index e5fdb25..1e730a4 100644 > --- a/src/saquery.c > +++ b/src/saquery.c > @@ -1638,7 +1638,7 @@ int main(int argc, char **argv) > q = NULL; > ibd_timeout = DEFAULT_SA_TIMEOUT_MS; > > - ibdiag_process_opts(argc, argv, ¶ms, "DLGs", opts, process_opt, > + ibdiag_process_opts(argc, argv, ¶ms, "DGLs", opts, process_opt, > usage_args, NULL); > > argc -= optind; > -- > 1.7.9.2 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Ira Weiny Member of Technical Staff Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 8/8] infiniband-diags: Add sa_smkey option to config file
On 5/30/2012 1:21 PM, Jim Foraker wrote:
> Defines a default for the smkey used in SA requests
>
> Signed-off-by: Jim Foraker
> ---
> etc/ibdiag.conf |3 +++
> src/ibdiag_common.c |3 +++
> 2 files changed, 6 insertions(+)
>
> diff --git a/etc/ibdiag.conf b/etc/ibdiag.conf
> index 2a2334f..1e6149d 100644
> --- a/etc/ibdiag.conf
> +++ b/etc/ibdiag.conf
> @@ -17,3 +17,6 @@
>
> # define a default m_key
> #m_key=0x00
> +
> +# default smkey to be used for SA requests
> +#sa_smkey=0x00
> diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
> index c40490a..1301810 100644
> --- a/src/ibdiag_common.c
> +++ b/src/ibdiag_common.c
> @@ -160,6 +160,9 @@ void read_ibdiag_config(const char *file)
> }
> } else if (strncmp(name, "m_key", strlen("m_key")) == 0) {
> ibd_mkey = strtoull(val_str, 0, 0);
> + } else if (strncmp(name, "sa_smkey",
> +strlen("sa_smkey")) == 0) {
For consistency with OpenSM naming, how about sa_key for this ?
-- Hal
> + ibd_sa_smkey = strtoull(val_str, 0, 0);
> }
> }
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 8/8] infiniband-diags: Add sa_smkey option to config file
On Wed, 2012-05-30 at 11:06 -0700, Hal Rosenstock wrote:
> On 5/30/2012 1:21 PM, Jim Foraker wrote:
> > Defines a default for the smkey used in SA requests
> >
> > Signed-off-by: Jim Foraker
> > ---
> > etc/ibdiag.conf |3 +++
> > src/ibdiag_common.c |3 +++
> > 2 files changed, 6 insertions(+)
> >
> > diff --git a/etc/ibdiag.conf b/etc/ibdiag.conf
> > index 2a2334f..1e6149d 100644
> > --- a/etc/ibdiag.conf
> > +++ b/etc/ibdiag.conf
> > @@ -17,3 +17,6 @@
> >
> > # define a default m_key
> > #m_key=0x00
> > +
> > +# default smkey to be used for SA requests
> > +#sa_smkey=0x00
> > diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
> > index c40490a..1301810 100644
> > --- a/src/ibdiag_common.c
> > +++ b/src/ibdiag_common.c
> > @@ -160,6 +160,9 @@ void read_ibdiag_config(const char *file)
> > }
> > } else if (strncmp(name, "m_key", strlen("m_key")) == 0) {
> > ibd_mkey = strtoull(val_str, 0, 0);
> > + } else if (strncmp(name, "sa_smkey",
> > + strlen("sa_smkey")) == 0) {
>
> For consistency with OpenSM naming, how about sa_key for this ?
That's fine by me. IIRC, that was actually my first choice, but I
thought there might be some objections, since it's called smkey in the
spec.
Jim
>
> -- Hal
>
> > + ibd_sa_smkey = strtoull(val_str, 0, 0);
> > }
> > }
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/16] librdmacm/rsocket: Fix hang in rrecv/rsend after disconnecting
On 05/30/2012 10:21 AM, Hefty, Sean wrote:
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem and the cause was reported by Sridhar Samudrala
. It can be reproduced by running netserver -f -D
using the rs-preload library. A similar issue exists with rsend().
Fix this by not blocking on a CQ unless we're connected.
Signed-off-by: Sean Hefty
---
Sridhar, can you please let me know if this fixes the hang you were seeing?
I moved the connected check inside holding the cq lock from the patch that
you sent me.
src/rsocket.c | 26 +++---
1 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/src/rsocket.c b/src/rsocket.c
index 01b7248..8c96dc1 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -908,6 +908,11 @@ static int rs_can_send(struct rsocket *rs)
(rs->target_sgl[rs->target_sge].length != 0);
}
+static int rs_conn_can_send(struct rsocket *rs)
+{
+ return rs_can_send(rs) || (rs->state != rs_connected);
+}
+
static int rs_can_send_ctrl(struct rsocket *rs)
{
return rs->ctrl_avail;
@@ -918,6 +923,11 @@ static int rs_have_rdata(struct rsocket *rs)
return (rs->rmsg_head != rs->rmsg_tail);
}
+static int rs_conn_have_rdata(struct rsocket *rs)
+{
+ return rs_have_rdata(rs) || (rs->state != rs_connected);
+}
+
static int rs_all_sends_done(struct rsocket *rs)
{
return (rs->sqe_avail + rs->ctrl_avail) == RS_QP_SIZE;
@@ -980,7 +990,7 @@ ssize_t rrecv(int socket, void *buf, size_t len, int flags)
}
fastlock_acquire(&rs->rlock);
if (!rs_have_rdata(rs)) {
- ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_have_rdata);
+ ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_conn_have_rdata);
if (ret&& errno != ECONNRESET)
goto out;
}
@@ -1084,9 +1094,14 @@ ssize_t rsend(int socket, const void *buf, size_t len,
int flags)
fastlock_acquire(&rs->slock);
for (left = len; left; left -= xfer_size, buf += xfer_size) {
if (!rs_can_send(rs)) {
- ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_can_send);
+ ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
+ rs_conn_can_send);
if (ret)
break;
+ if (rs->state != rs_connected) {
+ ret = ERR(ECONNRESET);
+ break;
+ }
}
if (olen< left) {
@@ -1193,9 +1208,14 @@ static ssize_t rsendv(int socket, const struct iovec
*iov, int iovcnt, int flags
fastlock_acquire(&rs->slock);
for (left = len; left; left -= xfer_size) {
if (!rs_can_send(rs)) {
- ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_can_send);
+ ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
+ rs_conn_can_send);
if (ret)
break;
+ if (rs->state != rs_connected) {
+ ret = ERR(ECONNRESET);
+ break;
+ }
}
if (olen< left) {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sean, Have tested by applying only this patch in the entire series.
netperf now seems to be working.
Thanks
Pradeep
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] opensm: Add per module logging support
Add _v2 APIs for osm_log, osm_log_is_active, and osm_log_msg_box
Also, add these to libopensm.map
Add _V2 forms of the various OSM_LOG macros
All of above pass additional parameter to identify module.
Each module defines a unique FILE_ID from 0-254.
255 is reserved to indicate "name not found".
FILE_ID must match index in osm_subnet.c:module_name_str
table. Note also currently that subnet's
per_mod_log_tbl is 128 entries (which is enough
more than the number of modules).
Use version 2 API (osm_log_v2) and macros (OSM_LOG_V2_)
in OpenSM client code
Added options to enable/disable per module logging and
it's configuration file
Config file format
Set of lines with module name and logging level as follows:
where:
module name is file name including .c
separator is either = , space, or tab
logging level is the same levels as used in the coarse/overall logging
as follows:
BITLOG LEVEL ENABLED
-
0x01 - ERROR (error messages)
0x02 - INFO (basic messages, low volume)
0x04 - VERBOSE (interesting stuff, moderate volume)
0x08 - DEBUG (diagnostic, high volume)
0x10 - FUNCS (function entry/exit, very high volume)
0x20 - FRAMES (dumps all SMP and GMP frames)
0x40 - ROUTING (dump FDB routing information)
0x80 - currently unused
Add parsing of the configuration file into table of log levels
indexed by FILE_ID
Added osm_get_log_per_module routine to obtain the configured info
for a supplied module name. This is a new library function.
Signed-off-by: Hal Rosenstock
---
As this patch is too large for email, please find this @
https://www.openfabrics.org/~halr/0001-opensm-Add-per-module-logging-support.patch
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] ibacm/acm.c: Make sure shift for subnet timeout is not negative
in acm_port_up
Signed-off-by: Hal Rosenstock
---
diff --git a/src/acm.c b/src/acm.c
index b47ee8a..c40f079 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -2794,7 +2794,8 @@ static void acm_port_up(struct acm_port *port)
port->mtu = attr.active_mtu;
port->rate = acm_get_rate(attr.active_width, attr.active_speed);
- port->subnet_timeout = 1 << (attr.subnet_timeout - 8);
+ if (attr.subnet_timeout >= 8)
+ port->subnet_timeout = 1 << (attr.subnet_timeout - 8);
for (port->gid_cnt = 0;; port->gid_cnt++) {
ret = ibv_query_gid(port->dev->verbs, port->port_num,
port->gid_cnt, &gid);
if (ret || !gid.global.interface_id)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] ibacm/acme.c: Eliminate seg fault when source not supplied
Signed-off-by: Hal Rosenstock
---
diff --git a/src/acme.c b/src/acme.c
index d3f8174..533588c 100644
--- a/src/acme.c
+++ b/src/acme.c
@@ -618,12 +618,18 @@ static void resolve(char *svc)
ret = resolve_name(&path);
break;
case 'l':
- memset(&path, 0, sizeof path);
- ret = resolve_lid(&path);
+ if (src_addr) {
+ memset(&path, 0, sizeof path);
+ ret = resolve_lid(&path);
+ } else
+ ret = -1;
break;
case 'g':
- memset(&path, 0, sizeof path);
- ret = resolve_gid(&path);
+ if (src_addr) {
+ memset(&path, 0, sizeof path);
+ ret = resolve_gid(&path);
+ } else
+ ret = -1;
break;
default:
break;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3.1 7/8] ib-diags/saquery: Fix smkey handling
smkey is already defined as a global inside saquery.c, so remove
broken support for passing it around as a function parameter
Signed-off-by: Jim Foraker
---
include/ibdiag_common.h |1 +
src/ibdiag_common.c |1 +
src/saquery.c | 60 +++
3 files changed, 31 insertions(+), 31 deletions(-)
diff --git a/include/ibdiag_common.h b/include/ibdiag_common.h
index 0120203..6dade10 100644
--- a/include/ibdiag_common.h
+++ b/include/ibdiag_common.h
@@ -52,6 +52,7 @@ extern ib_portid_t *ibd_sm_id;
extern int ibd_timeout;
extern uint32_t ibd_ibnetdisc_flags;
extern uint64_t ibd_mkey;
+extern uint64_t ibd_sakey;
extern int show_keys;
/**/
diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
index d7d784f..eaf504f 100644
--- a/src/ibdiag_common.c
+++ b/src/ibdiag_common.c
@@ -72,6 +72,7 @@ int ibd_ca_port = 0;
int ibd_timeout = 0;
uint32_t ibd_ibnetdisc_flags = IBND_CONFIG_MLX_EPI;
uint64_t ibd_mkey;
+uint64_t ibd_sakey;
int show_keys = 0;
static const char *prog_name;
diff --git a/src/saquery.c b/src/saquery.c
index dec990a..7a7878b 100644
--- a/src/saquery.c
+++ b/src/saquery.c
@@ -85,7 +85,6 @@ struct query_cmd {
static char *node_name_map_file = NULL;
static nn_map_t *node_name_map = NULL;
-static uint64_t smkey = 1;
/**
* Declare some globals because I don't want this to be too complex.
@@ -724,11 +723,11 @@ static void dump_results(struct sa_query_result *r, void
(*dump_func) (void *))
*/
static int get_any_records(bind_handle_t h,
uint16_t attr_id, uint32_t attr_mod,
- ib_net64_t comp_mask, void *attr, uint64_t sm_key,
+ ib_net64_t comp_mask, void *attr,
struct sa_query_result *result)
{
int ret = sa_query(h, IB_MAD_METHOD_GET_TABLE, attr_id, attr_mod,
- cl_ntoh64(comp_mask), sm_key, attr, result);
+ cl_ntoh64(comp_mask), ibd_sakey, attr, result);
if (ret) {
fprintf(stderr, "Query SA failed: %s\n", strerror(ret));
return ret;
@@ -744,12 +743,12 @@ static int get_any_records(bind_handle_t h,
static int get_and_dump_any_records(bind_handle_t h, uint16_t attr_id,
uint32_t attr_mod, ib_net64_t comp_mask,
- void *attr, uint64_t sm_key,
+ void *attr,
void (*dump_func) (void *))
{
struct sa_query_result result;
int ret = get_any_records(h, attr_id, attr_mod, comp_mask, attr,
- sm_key, &result);
+ &result);
if (ret)
return ret;
@@ -761,18 +760,17 @@ static int get_and_dump_any_records(bind_handle_t h,
uint16_t attr_id,
/**
* Get all the records available for requested query type.
*/
-static int get_all_records(bind_handle_t h, uint16_t attr_id, int trusted,
+static int get_all_records(bind_handle_t h, uint16_t attr_id,
struct sa_query_result *result)
{
- return get_any_records(h, attr_id, 0, 0, NULL, trusted ? smkey : 0,
- result);
+ return get_any_records(h, attr_id, 0, 0, NULL, result);
}
static int get_and_dump_all_records(bind_handle_t h, uint16_t attr_id,
- int trusted, void (*dump_func) (void *))
+ void (*dump_func) (void *))
{
struct sa_query_result result;
- int ret = get_all_records(h, attr_id, 0, &result);
+ int ret = get_all_records(h, attr_id, &result);
if (ret)
return ret;
@@ -792,7 +790,7 @@ static int get_lid_from_name(bind_handle_t h, const char
*name, uint16_t * lid)
int ret;
struct sa_query_result result;
- ret = get_all_records(h, IB_SA_ATTR_NODERECORD, 0, &result);
+ ret = get_all_records(h, IB_SA_ATTR_NODERECORD, &result);
if (ret)
return ret;
@@ -892,7 +890,7 @@ static int get_issm_records(bind_handle_t h, ib_net32_t
capability_mask,
attr.port_info.capability_mask = capability_mask;
return get_any_records(h, IB_SA_ATTR_PORTINFORECORD, 1 << 31,
- IB_PIR_COMPMASK_CAPMASK, &attr, 0, result);
+ IB_PIR_COMPMASK_CAPMASK, &attr, result);
}
static int print_node_records(bind_handle_t h)
@@ -901,7 +899,7 @@ static int print_node_records(bind_handle_t h)
int ret;
struct sa_query_result result;
- ret = get_all_records(h, IB_SA_ATTR_NODERECORD, 0, &result);
+ ret = get_all_records(h, IB_SA_ATTR_NODERECORD, &result);
if (ret)
return ret;
@@ -944,7 +942,7 @@ static int get_print_class_port_info(bind_handle_t h)
{
struct sa_quer
[PATCH V3.1 8/8] infiniband-diags: Add sa_smkey option to config file
Defines a default for the smkey used in SA requests
Signed-off-by: Jim Foraker
---
etc/ibdiag.conf |3 +++
src/ibdiag_common.c |3 +++
2 files changed, 6 insertions(+)
diff --git a/etc/ibdiag.conf b/etc/ibdiag.conf
index 2a2334f..9686d14 100644
--- a/etc/ibdiag.conf
+++ b/etc/ibdiag.conf
@@ -17,3 +17,6 @@
# define a default m_key
#m_key=0x00
+
+# default smkey to be used for SA requests
+#sa_key=0x00
diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
index eaf504f..ba0ce0b 100644
--- a/src/ibdiag_common.c
+++ b/src/ibdiag_common.c
@@ -160,6 +160,9 @@ void read_ibdiag_config(const char *file)
}
} else if (strncmp(name, "m_key", strlen("m_key")) == 0) {
ibd_mkey = strtoull(val_str, 0, 0);
+ } else if (strncmp(name, "sa_key",
+ strlen("sa_key")) == 0) {
+ ibd_sakey = strtoull(val_str, 0, 0);
}
}
--
1.7.9.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V3 8/8] infiniband-diags: Add sa_smkey option to config file
I just posted a "V3.1" for patches 7 and 8, which converts variable
ibd_sa_smkey to ibd_sakey and config parameter sa_smkey to sa_key.
Jim
On Wed, 2012-05-30 at 11:43 -0700, Jim Foraker wrote:
> On Wed, 2012-05-30 at 11:06 -0700, Hal Rosenstock wrote:
> > On 5/30/2012 1:21 PM, Jim Foraker wrote:
> > > Defines a default for the smkey used in SA requests
> > >
> > > Signed-off-by: Jim Foraker
> > > ---
> > > etc/ibdiag.conf |3 +++
> > > src/ibdiag_common.c |3 +++
> > > 2 files changed, 6 insertions(+)
> > >
> > > diff --git a/etc/ibdiag.conf b/etc/ibdiag.conf
> > > index 2a2334f..1e6149d 100644
> > > --- a/etc/ibdiag.conf
> > > +++ b/etc/ibdiag.conf
> > > @@ -17,3 +17,6 @@
> > >
> > > # define a default m_key
> > > #m_key=0x00
> > > +
> > > +# default smkey to be used for SA requests
> > > +#sa_smkey=0x00
> > > diff --git a/src/ibdiag_common.c b/src/ibdiag_common.c
> > > index c40490a..1301810 100644
> > > --- a/src/ibdiag_common.c
> > > +++ b/src/ibdiag_common.c
> > > @@ -160,6 +160,9 @@ void read_ibdiag_config(const char *file)
> > > }
> > > } else if (strncmp(name, "m_key", strlen("m_key")) == 0) {
> > > ibd_mkey = strtoull(val_str, 0, 0);
> > > + } else if (strncmp(name, "sa_smkey",
> > > +strlen("sa_smkey")) == 0) {
> >
> > For consistency with OpenSM naming, how about sa_key for this ?
> That's fine by me. IIRC, that was actually my first choice, but I
> thought there might be some objections, since it's called smkey in the
> spec.
>
> Jim
>
> >
> > -- Hal
> >
> > > + ibd_sa_smkey = strtoull(val_str, 0, 0);
> > > }
> > > }
> > >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] opensm: Add enum for FILE_ID for per module logging
An osm_file_ids_enum was added to the SM header and
a FILE_ID const definition to each source file.
This patch follows patch to add per module logging.
Signed-off-by: Ilya Nelkenbaum
Signed-off-by: Hal Rosenstock
---
diff --git a/include/opensm/osm_sm.h b/include/opensm/osm_sm.h
index 4c7b120..a7b2556 100644
--- a/include/opensm/osm_sm.h
+++ b/include/opensm/osm_sm.h
@@ -90,6 +90,109 @@ BEGIN_C_DECLS
* Steve King, Intel
*
*/
+
+/d* OpenSM: Subnet/osm_file_ids_enum
+* NAME
+* osm_file_ids_enum
+*
+* DESCRIPTION
+* Enumerates all FILE_IDs used for logging support.
+*
+* SYNOPSIS
+*/
+typedef enum _osm_file_ids_enum {
+ OSM_FILE_MAIN_C = 0,
+ OSM_FILE_CONSOLE_C,
+ OSM_FILE_CONSOLE_IO_C,
+ OSM_FILE_DB_FILES_C,
+ OSM_FILE_DB_PACK_C,
+ OSM_FILE_DROP_MGR_C,
+ OSM_FILE_DUMP_C,
+ OSM_FILE_EVENT_PLUGIN_C,
+ OSM_FILE_GUID_INFO_RCV_C,
+ OSM_FILE_GUID_MGR_C,
+ OSM_FILE_HELPER_C,
+ OSM_FILE_INFORM_C,
+ OSM_FILE_LID_MGR_C,
+ OSM_FILE_LIN_FWD_RCV_C,
+ OSM_FILE_LINK_MGR_C,
+ OSM_FILE_LOG_C,
+ OSM_FILE_MAD_POOL_C,
+ OSM_FILE_MCAST_FWD_RCV_C,
+ OSM_FILE_MCAST_MGR_C,
+ OSM_FILE_MCAST_TBL_C,
+ OSM_FILE_MCM_PORT_C,
+ OSM_FILE_MESH_C,
+ OSM_FILE_MLNX_EXT_PORT_INFO_RCV_C,
+ OSM_FILE_MTREE_C,
+ OSM_FILE_MULTICAST_C,
+ OSM_FILE_NODE_C,
+ OSM_FILE_NODE_DESC_RCV_C,
+ OSM_FILE_NODE_INFO_RCV_C,
+ OSM_FILE_OPENSM_C,
+ OSM_FILE_PERFMGR_C,
+ OSM_FILE_PERFMGR_DB_C,
+ OSM_FILE_PKEY_C,
+ OSM_FILE_PKEY_MGR_C,
+ OSM_FILE_PKEY_RCV_C,
+ OSM_FILE_PORT_C,
+ OSM_FILE_PORT_INFO_RCV_C,
+ OSM_FILE_PRTN_C,
+ OSM_FILE_PRTN_CONFIG_C,
+ OSM_FILE_QOS_C,
+ OSM_FILE_QOS_PARSER_L_L,
+ OSM_FILE_QOS_PARSER_Y_Y,
+ OSM_FILE_QOS_POLICY_C,
+ OSM_FILE_REMOTE_SM_C,
+ OSM_FILE_REQ_C,
+ OSM_FILE_RESP_C,
+ OSM_FILE_ROUTER_C,
+ OSM_FILE_SA_C,
+ OSM_FILE_SA_CLASS_PORT_INFO_C,
+ OSM_FILE_SA_GUIDINFO_RECORD_C,
+ OSM_FILE_SA_INFORMINFO_C,
+ OSM_FILE_SA_LFT_RECORD_C,
+ OSM_FILE_SA_LINK_RECORD_C,
+ OSM_FILE_SA_MAD_CTRL_C,
+ OSM_FILE_SA_MCMEMBER_RECORD_C,
+ OSM_FILE_SA_MFT_RECORD_C,
+ OSM_FILE_SA_MULTIPATH_RECORD_C,
+ OSM_FILE_SA_NODE_RECORD_C,
+ OSM_FILE_SA_PATH_RECORD_C,
+ OSM_FILE_SA_PKEY_RECORD_C,
+ OSM_FILE_SA_PORTINFO_RECORD_C,
+ OSM_FILE_SA_SERVICE_RECORD_C,
+ OSM_FILE_SA_SLVL_RECORD_C,
+ OSM_FILE_SA_SMINFO_RECORD_C,
+ OSM_FILE_SA_SW_INFO_RECORD_C,
+ OSM_FILE_SA_VLARB_RECORD_C,
+ OSM_FILE_SERVICE_C,
+ OSM_FILE_SLVL_MAP_RCV_C,
+ OSM_FILE_SM_C,
+ OSM_FILE_SMINFO_RCV_C,
+ OSM_FILE_SM_MAD_CTRL_C,
+ OSM_FILE_SM_STATE_MGR_C,
+ OSM_FILE_STATE_MGR_C,
+ OSM_FILE_SUBNET_C,
+ OSM_FILE_SW_INFO_RCV_C,
+ OSM_FILE_SWITCH_C,
+ OSM_FILE_TORUS_C,
+ OSM_FILE_TRAP_RCV_C,
+ OSM_FILE_UCAST_CACHE_C,
+ OSM_FILE_UCAST_DNUP_C,
+ OSM_FILE_UCAST_FILE_C,
+ OSM_FILE_UCAST_FTREE_C,
+ OSM_FILE_UCAST_LASH_C,
+ OSM_FILE_UCAST_MGR_C,
+ OSM_FILE_UCAST_UPDN_C,
+ OSM_FILE_VENDOR_IBUMAD_C,
+ OSM_FILE_VL15INTF_C,
+ OSM_FILE_VL_ARB_RCV_C,
+ OSM_FILE_ST_C,
+ OSM_FILE_UCAST_DFSSSP_C,
+} osm_file_ids_enum;
+/***/
+
/s* OpenSM: SM/osm_sm_t
* NAME
* osm_sm_t
diff --git a/libvendor/osm_vendor_ibumad.c b/libvendor/osm_vendor_ibumad.c
index b0be770..e35dcce 100644
--- a/libvendor/osm_vendor_ibumad.c
+++ b/libvendor/osm_vendor_ibumad.c
@@ -65,9 +65,10 @@
#include
#include
#include
+#include
#include
-#define FILE_ID 84
+#define FILE_ID OSM_FILE_VENDOR_IBUMAD_C
/s* OpenSM: Vendor UMAD/osm_umad_bind_info_t
* NAME
diff --git a/opensm/main.c b/opensm/main.c
index 102237e..9957fc3 100644
--- a/opensm/main.c
+++ b/opensm/main.c
@@ -62,7 +62,7 @@
#include
#include
-#define FILE_ID 0
+#define FILE_ID OSM_FILE_MAIN_C
volatile unsigned int osm_exit_flag = 0;
diff --git a/opensm/osm_console.c b/opensm/osm_console.c
index 1a2e845..d894116 100644
--- a/opensm/osm_console.c
+++ b/opensm/osm_console.c
@@ -57,7 +57,7 @@
#include
#include
-#define FILE_ID 1
+#define FILE_ID OSM_FILE_CONSOLE_c
extern void osm_update_node_desc(IN osm_opensm_t *osm);
diff --git a/opensm/osm_console_io.c b/opensm/osm_console_io.c
index c02d7ba..be8c02c 100644
--- a/opensm/osm_console_io.c
+++ b/opensm/osm_console_io.c
@@ -58,7 +58,7 @@
#include
#include
-#define FILE_ID 2
+#define FILE_ID OSM_FILE_CONSOLE_IO_C
static int is_local(char *str)
{
diff --git a/opensm/osm_db_files.c b/opensm/osm_db_files.c
index ba37b4e..4a2d71c 100644
--- a/opensm/osm_db_files.c
+++ b/opensm/osm_db_files.c
@@ -48,8 +48,9 @@
#include
#include
#include
+#include
-#define FILE_ID 3
+#define FILE_ID OSM_FILE_DB_FILES_C
/d* Database/OSM_DB_MAX
Re: [PATCH] opensm: Add per module logging support
On Wed, 30 May 2012 17:30:50 -0400 Hal Rosenstock wrote: > > Add _v2 APIs for osm_log, osm_log_is_active, and osm_log_msg_box > Also, add these to libopensm.map > Add _V2 forms of the various OSM_LOG macros > > All of above pass additional parameter to identify module. > Each module defines a unique FILE_ID from 0-254. > 255 is reserved to indicate "name not found". > FILE_ID must match index in osm_subnet.c:module_name_str > table. Note also currently that subnet's > per_mod_log_tbl is 128 entries (which is enough > more than the number of modules). This is a great idea for debugging but do you really need to define the "FILE_ID's"? Could you use the __FILE__ macro to do a lookup? Or do you think that would affect performance? Also do you really think that 255 files is enough id's to reserve? Ira > > Use version 2 API (osm_log_v2) and macros (OSM_LOG_V2_) > in OpenSM client code > > Added options to enable/disable per module logging and > it's configuration file > > Config file format > Set of lines with module name and logging level as follows: > > where: > module name is file name including .c > separator is either = , space, or tab > logging level is the same levels as used in the coarse/overall logging > as follows: > > BITLOG LEVEL ENABLED > - > 0x01 - ERROR (error messages) > 0x02 - INFO (basic messages, low volume) > 0x04 - VERBOSE (interesting stuff, moderate volume) > 0x08 - DEBUG (diagnostic, high volume) > 0x10 - FUNCS (function entry/exit, very high volume) > 0x20 - FRAMES (dumps all SMP and GMP frames) > 0x40 - ROUTING (dump FDB routing information) > 0x80 - currently unused > > Add parsing of the configuration file into table of log levels > indexed by FILE_ID > > Added osm_get_log_per_module routine to obtain the configured info > for a supplied module name. This is a new library function. > > Signed-off-by: Hal Rosenstock > --- > As this patch is too large for email, please find this @ > https://www.openfabrics.org/~halr/0001-opensm-Add-per-module-logging-support.patch > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Ira Weiny Member of Technical Staff Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] opensm: Add per module logging support
On 5/30/2012 7:03 PM, Ira Weiny wrote: > On Wed, 30 May 2012 17:30:50 -0400 > Hal Rosenstock wrote: > >> >> Add _v2 APIs for osm_log, osm_log_is_active, and osm_log_msg_box >> Also, add these to libopensm.map >> Add _V2 forms of the various OSM_LOG macros >> >> All of above pass additional parameter to identify module. >> Each module defines a unique FILE_ID from 0-254. >> 255 is reserved to indicate "name not found". >> FILE_ID must match index in osm_subnet.c:module_name_str >> table. Note also currently that subnet's >> per_mod_log_tbl is 128 entries (which is enough >> more than the number of modules). > > This is a great idea for debugging but do you really need to define the > "FILE_ID's"? Could you use the __FILE__ macro to do a lookup? Or do you > think that would affect performance? That was the first implementation and the performance impact was too much. > Also do you really think that 255 files is enough id's to reserve? Yes but it's hard to predict the future. We're only at 88 now. We've only added a few per year and this leaves >160 more. -- Hal > Ira > >> >> Use version 2 API (osm_log_v2) and macros (OSM_LOG_V2_) >> in OpenSM client code >> >> Added options to enable/disable per module logging and >> it's configuration file >> >> Config file format >> Set of lines with module name and logging level as follows: >> >> where: >> module name is file name including .c >> separator is either = , space, or tab >> logging level is the same levels as used in the coarse/overall logging >> as follows: >> >> BITLOG LEVEL ENABLED >> - >> 0x01 - ERROR (error messages) >> 0x02 - INFO (basic messages, low volume) >> 0x04 - VERBOSE (interesting stuff, moderate volume) >> 0x08 - DEBUG (diagnostic, high volume) >> 0x10 - FUNCS (function entry/exit, very high volume) >> 0x20 - FRAMES (dumps all SMP and GMP frames) >> 0x40 - ROUTING (dump FDB routing information) >> 0x80 - currently unused >> >> Add parsing of the configuration file into table of log levels >> indexed by FILE_ID >> >> Added osm_get_log_per_module routine to obtain the configured info >> for a supplied module name. This is a new library function. >> >> Signed-off-by: Hal Rosenstock >> --- >> As this patch is too large for email, please find this @ >> https://www.openfabrics.org/~halr/0001-opensm-Add-per-module-logging-support.patch >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] opensm/osm_subnet.c: Fixed ftree/updn configuration failure when root_guid_file points to non-existing file
Signed-off-by: Ilya Nelkenbaum
Reviewed-by: Hal Rosenstock
---
diff --git a/opensm/osm_subnet.c b/opensm/osm_subnet.c
index 7dcde87..04c7b18 100644
--- a/opensm/osm_subnet.c
+++ b/opensm/osm_subnet.c
@@ -1315,6 +1315,17 @@ int osm_subn_verify_config(IN osm_subn_opt_t * p_opts)
}
#endif
+ if (p_opts->root_guid_file != NULL) {
+ FILE *root_file = fopen(p_opts->root_guid_file, "r");
+ if (!root_file) {
+ log_report("Root guid file provided: %s doesn't
exist.\n"
+ "Using default roots discovery algorithm\n",
+ p_opts->root_guid_file);
+ p_opts->root_guid_file = NULL;
+ } else
+ fclose(root_file);
+ }
+
return 0;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/16] librdmacm/rsocket: Fix hang in rrecv/rsend after disconnecting
On 5/30/2012 12:10 PM, Pradeep Satyanarayana wrote:
On 05/30/2012 10:21 AM, Hefty, Sean wrote:
If a user calls rrecv() after a blocking rsocket has been disconnected,
it will hang. This problem and the cause was reported by Sridhar
Samudrala
. It can be reproduced by running netserver -f -D
using the rs-preload library. A similar issue exists with rsend().
Fix this by not blocking on a CQ unless we're connected.
Signed-off-by: Sean Hefty
---
Sridhar, can you please let me know if this fixes the hang you were
seeing?
I moved the connected check inside holding the cq lock from the patch
that
you sent me.
src/rsocket.c | 26 +++---
1 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/src/rsocket.c b/src/rsocket.c
index 01b7248..8c96dc1 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -908,6 +908,11 @@ static int rs_can_send(struct rsocket *rs)
(rs->target_sgl[rs->target_sge].length != 0);
}
+static int rs_conn_can_send(struct rsocket *rs)
+{
+return rs_can_send(rs) || (rs->state != rs_connected);
+}
+
static int rs_can_send_ctrl(struct rsocket *rs)
{
return rs->ctrl_avail;
@@ -918,6 +923,11 @@ static int rs_have_rdata(struct rsocket *rs)
return (rs->rmsg_head != rs->rmsg_tail);
}
+static int rs_conn_have_rdata(struct rsocket *rs)
+{
+return rs_have_rdata(rs) || (rs->state != rs_connected);
+}
+
static int rs_all_sends_done(struct rsocket *rs)
{
return (rs->sqe_avail + rs->ctrl_avail) == RS_QP_SIZE;
@@ -980,7 +990,7 @@ ssize_t rrecv(int socket, void *buf, size_t len,
int flags)
}
fastlock_acquire(&rs->rlock);
if (!rs_have_rdata(rs)) {
-ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_have_rdata);
+ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_conn_have_rdata);
if (ret&& errno != ECONNRESET)
goto out;
}
@@ -1084,9 +1094,14 @@ ssize_t rsend(int socket, const void *buf,
size_t len, int flags)
fastlock_acquire(&rs->slock);
for (left = len; left; left -= xfer_size, buf += xfer_size) {
if (!rs_can_send(rs)) {
-ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_can_send);
+ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
+rs_conn_can_send);
if (ret)
break;
+if (rs->state != rs_connected) {
+ret = ERR(ECONNRESET);
+break;
+}
}
if (olen< left) {
@@ -1193,9 +1208,14 @@ static ssize_t rsendv(int socket, const struct
iovec *iov, int iovcnt, int flags
fastlock_acquire(&rs->slock);
for (left = len; left; left -= xfer_size) {
if (!rs_can_send(rs)) {
-ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
rs_can_send);
+ret = rs_process_cq(rs, rs_nonblocking(rs, flags),
+rs_conn_can_send);
if (ret)
break;
+if (rs->state != rs_connected) {
+ret = ERR(ECONNRESET);
+break;
+}
}
if (olen< left) {
Sean, Have tested by applying only this patch in the entire series.
netperf now seems to be working.
Yes. The patch fixes the hang in recv().
However, i still see a few other issues related to socket semantics that
need to be addressed.
# ldp netperf -H 192.168.0.198 -l 3 -t TCP_STREAM
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.0.198 (192.168.0.198) port 0 AF_INET
netperf: get_transport_info: getsockopt: errno 95
Recv SendSend
Socket Socket Message Elapsed
Size SizeSize Time Throughput
bytes bytes bytessecs.10^6bits/sec
131072 131072 1310723.00 6176.91
shutdown_control: no response received errno 95
1. netperf: get_transport_info: getsockopt: errno 95
This failure is due to the missing TCP_MAXSEG socket option support. May
be this is OK as this option
doesn't make much sense when using RDMA. Or we could return a reasonable
value.
2. shutdown_control: no response received errno 95
Here select() on control socket is failing with EOPNOTSUPP after doing a
shutdown(SHUT_WR) of the control socket
3. Once in a while netserver timesout in recv() after the client closes
the connection.
Thanks
Sridhar
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 3/16] librdmacm/rsocket: Fix hang in rrecv/rsend after disconnecting
Thanks for the feedback. > 1. netperf: get_transport_info: getsockopt: errno 95 > This failure is due to the missing TCP_MAXSEG socket option support. May > be this is OK as this option > doesn't make much sense when using RDMA. Or we could return a reasonable > value. Missing socket options are usually easy enough to add, so that setsockopt can return success. :) Maybe this option makes sense as the MTU? One issue I hit into with rsockets is that several options must be set before connecting. And in this case, if MAXSEG mapped to MTU, it would need to match on both sides. > 2. shutdown_control: no response received errno 95 > Here select() on control socket is failing with EOPNOTSUPP after doing a > shutdown(SHUT_WR) of the control socket rshutdown() pretty much assumes SHUT_RDWR. I need to think whether anything should be done to handle SHUT_RD or SHUT_WR, or if shutdown should just ignore those. The issue may be the result of rshutdown() switching the rsocket from nonblocking to blocking in order to process the shutdown properly. The code is waiting for all pending sends to complete to prevent data loss... > 3. Once in a while netserver timesout in recv() after the client closes > the connection. ...or maybe we still lose the disconnect message. Did this occur after you applied patch 3? - Sean -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/3] IB/mlx4: Fix max_wqe capacity report for query device
On Wed, May 30, 2012 at 2:25 AM, Hefty, Sean wrote: >> Did you try out the patches? was it helpful to address the problem >> you're facing? > I have not had time to test it yet so whenever you do have that time... -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
