Re: [RFC PATCH] Set HopLimit based on off subnet Re: SRP issues with OpenSM 3.3.3
On Thu, Dec 17, 2009 at 9:18 PM, Ira Weiny wei...@llnl.gov wrote:
On Wed, 16 Dec 2009 08:29:43 -0500
Hal Rosenstock hal.rosenst...@gmail.com wrote:
On Tue, Dec 15, 2009 at 9:55 PM, Ira Weiny wei...@llnl.gov wrote:
[snip]
diff --git a/opensm/opensm/osm_sa_path_record.c
b/opensm/opensm/osm_sa_path_record.c
index be0cd71..1fa83a1 100644
--- a/opensm/opensm/osm_sa_path_record.c
+++ b/opensm/opensm/osm_sa_path_record.c
@@ -757,6 +757,14 @@ Exit:
return (status);
}
+static int gid_is_off_subnet(IN osm_sa_t * sa, IN const ib_gid_t * p_dgid)
+{
+ return (!ib_gid_is_link_local(p_dgid)
+ !ib_gid_is_multicast(p_dgid)
+ ib_gid_get_subnet_prefix(p_dgid) !=
+ sa-p_subn-opt.subnet_prefix);
+}
+
/**
**/
static void pr_rcv_build_pr(IN osm_sa_t * sa, IN const osm_port_t *
p_src_port,
@@ -770,16 +778,12 @@ static void pr_rcv_build_pr(IN osm_sa_t * sa, IN
const osm_port_t * p_src_port,
{
const osm_physp_t *p_src_physp;
const osm_physp_t *p_dest_physp;
- boolean_t is_nonzero_gid = 0;
OSM_LOG_ENTER(sa-p_log);
p_src_physp = p_src_port-p_physp;
if (p_dgid)
- is_nonzero_gid = ib_gid_is_notzero(p_dgid);
-
- if (is_nonzero_gid)
p_pr-dgid = *p_dgid;
else {
p_dest_physp = p_dest_port-p_physp;
@@ -799,7 +803,7 @@ static void pr_rcv_build_pr(IN osm_sa_t * sa, IN const
osm_port_t * p_src_port,
p_pr-hop_flow_raw = cl_hton32(1 31);
/* Only set HopLimit if going through a router */
- if (is_nonzero_gid)
+ if (gid_is_off_subnet(sa, p_pr-dgid))
p_pr-hop_flow_raw |= cl_hton32(IB_HOPLIMIT_MAX);
p_pr-pkey = p_parms-pkey;
@@ -1248,10 +1252,7 @@ static ib_net16_t pr_rcv_get_end_points(IN osm_sa_t
* sa,
memset(p_dgid, 0, sizeof(*p_dgid));
if (comp_mask IB_PR_COMPMASK_DGID) {
- if (!ib_gid_is_link_local(p_pr-dgid)
- !ib_gid_is_multicast(p_pr-dgid)
- ib_gid_get_subnet_prefix(p_pr-dgid) !=
- sa-p_subn-opt.subnet_prefix) {
+ if (gid_is_off_subnet(sa, p_pr-dgid)) {
dest_guid = find_router(sa,
p_pr-dgid.unicast.prefix);
if (!dest_guid) {
char gid_str[INET6_ADDRSTRLEN];
This maintains the p_dgid found in pr_rcv_get_end_points but appropriately
set's the HopLimit when the DGID is off subnet.
Yes, this looks right to me. Would it be better to only determine
gid_is_off_subnet once by saving this in a variable and passing it as
a parameter where needed ?
It might be faster but it would mean a bigger change.
Yes.
It looks like having
the dgid or p_pr structures in some internal opensm structure to carry that
information along would be best but that is a lot of work.
It could also be done by passing this as an additional parameter
through the necessary routines.
Do you think this will adversely affect performance on a big system?
I don't know as I haven't measured the difference in the two
approaches. Just trying to minimize additions to computing PRs.
-- Hal
I plan
to roll this out internally and we will have a chance to test on bigger
systems here. But there are larger out there now.
Ira
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
wei...@llnl.gov
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 01/12] opensm: Make error message for torus-2QoS dateline specification match code check.
Signed-off-by: Jim Schutt jasc...@sandia.gov --- opensm/opensm/osm_ucast_torus.c |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/opensm/osm_ucast_torus.c b/opensm/opensm/osm_ucast_torus.c index 8eb2880..7108394 100644 --- a/opensm/opensm/osm_ucast_torus.c +++ b/opensm/opensm/osm_ucast_torus.c @@ -954,7 +954,7 @@ bool parse_dir_dateline(int c_dir, struct torus *t, const char *parse_sep) if ((*dl 0 *dl = -max_dl) || *dl = max_dl) OSM_LOG(t-osm-log, OSM_LOG_ERROR, Error: dateline value for coordinate direction %d - must be %d = dl = %d\n, + must be %d dl %d\n, c_dir, -max_dl, max_dl); else success = true; -- 1.5.6.GIT -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 10/12] opensm: Implement master spanning tree for torus-2QoS multicast support.
In order to route a 2D/3D torus without credit loops while providing
support for two QoS levels, torus-2QoS needs to use 3 VL bits and
all 4 available SL bits. This means that multicast traffic must
share SL values with unicast traffic, which in turn means that
multicast routing must be compatible with unicast routing to prevent
credit loops.
Torus-2QoS unicast routing is based on DOR, and it turns out to
be possible to construct spanning trees so that when multicast
and unicast traffic are overlaid, credit loops are not possible.
Here is a 2D example of such a spanning tree, where x is the
root switch, and each + is a non-root switch:
+ + + + +
| | | | |
+ + + + +
| | | | |
+--+--x--+--+
| | | | |
+ + + + +
For multicast traffic routed from root to tip, every turn in the
above spanning tree is a legal DOR turn.
For traffic routed from tip to root, and traffic routed through
the root, turns are not legal DOR turns. However, to construct
a credit loop, the union of multicast routing on this spanning
tree with DOR unicast routing can only provide 3 of the 4 turns
needed for the loop.
In addition, if none of the above spanning tree branches crosses
a dateline used for unicast credit loop avoidance on a torus,
and multicast traffic is confined to SL 0 or SL 8 (recall that
torus-2QoS uses SL bit 3 to differentiate QoS level), then
multicast traffic also cannot contribute to the ring credit
loops that are otherwise possible in a torus.
Torus-2QoS uses these ideas to create a master spanning tree.
Every multicast group spanning tree will be constructed as a
subset of the master tree, with the same root as the master
tree.
Such multicast group spanning trees will in general not be
optimal for groups which are a subset of the full fabric.
However, this compromise must be made to enable support for
two QoS levels on a torus while preventing credit loops.
Signed-off-by: Jim Schutt jasc...@sandia.gov
---
opensm/opensm/osm_ucast_torus.c | 267 +++
1 files changed, 267 insertions(+), 0 deletions(-)
diff --git a/opensm/opensm/osm_ucast_torus.c b/opensm/opensm/osm_ucast_torus.c
index 61e0bf3..082fcf5 100644
--- a/opensm/opensm/osm_ucast_torus.c
+++ b/opensm/opensm/osm_ucast_torus.c
@@ -154,6 +154,19 @@ struct link {
* type. Furthermore, if that type is PASSTHRU, then the connected links:
* 1) are parallel to a given coordinate direction
* 2) share the same two switches as endpoints.
+ *
+ * Torus-2QoS uses one master spanning tree for multicast, of which every
+ * multicast group spanning tree is a subtree. to_stree_root is a pointer
+ * to the next port_grp on the path to the master spanning tree root.
+ * to_stree_tip is a pointer to the next port_grp on the path to a master
+ * spanning tree branch tip.
+ *
+ * Each t_switch can have at most one port_grp with a non-NULL to_stree_root.
+ * Exactly one t_switch in the fabric will have all port_grp objects with
+ * to_stree_root NULL; it is the master spanning tree root.
+ *
+ * A t_switch with all port_grp objects where to_stree_tip is NULL is at a
+ * master spanning tree branch tip.
*/
struct port_grp {
enum endpt_type type;
@@ -163,6 +176,8 @@ struct port_grp {
unsigned sw_dlid_cnt; /* switch dlids routed through this group */
unsigned ca_dlid_cnt; /* CA dlids routed through this group */
struct t_switch *sw;/* what switch we're attached to */
+ struct port_grp *to_stree_root;
+ struct port_grp *to_stree_tip;
struct endpoint **port;
};
@@ -8499,6 +8514,256 @@ bool torus_lft(struct torus *t, struct t_switch *sw)
return success;
}
+static
+bool good_xy_ring(struct torus *t, int x, int y, int z)
+{
+ struct t_switch sw = t-sw;
+ bool good_ring = true;
+
+ for (x = 0; x t-x_sz good_ring; x++)
+ good_ring = sw[x][y][z];
+
+ for (y = 0; y t-y_sz good_ring; y++)
+ good_ring = sw[x][y][z];
+
+ return good_ring;
+}
+
+static
+struct t_switch *find_plane_mid(struct torus *t, int z)
+{
+ int x, dx, xm = t-x_sz / 2;
+ int y, dy, ym = t-y_sz / 2;
+ struct t_switch sw = t-sw;
+
+ if (good_xy_ring(t, xm, ym, z))
+ return sw[xm][ym][z];
+
+ for (dx = 1, dy = 1; dx = xm dy = ym; dx++, dy++) {
+
+ x = canonicalize(xm - dx, t-x_sz);
+ y = canonicalize(ym - dy, t-y_sz);
+ if (good_xy_ring(t, x, y, z))
+ return sw[x][y][z];
+
+ x = canonicalize(xm + dx, t-x_sz);
+ y = canonicalize(ym + dy, t-y_sz);
+ if (good_xy_ring(t, x, y, z))
+ return sw[x][y][z];
+ }
+ return NULL;
+}
+
+static
+struct t_switch *find_stree_root(struct torus *t)
+{
+ int x, y, z, dz, zm = t-z_sz / 2;
+ struct t_switch sw = t-sw;
+ struct t_switch *root;
+
[PATCH 11/12] opensm: Implement multicast support for torus-2QoS.
Every multicast spanning tree used by torus-2QoS is a subset
of the master spanning tree built when unicast routing is
computed. This is required because when QoS is enabled,
torus-2QoS needs to use the same SLs for unicast and multicast.
Thus, the multicast spanning trees must have special properties
to avoid credit loops between unicast and multicast traffic.
To build a spanning tree for a particular MLID, torus-2QoS just
needs to mark all the ports that participate in that multicast
group, then walk the master spanning tree and add switches
hosting the marked ports to the multicast group spanning tree.
Use a depth-first search of the master spanning tree for this.
Signed-off-by: Jim Schutt jasc...@sandia.gov
---
opensm/opensm/osm_ucast_torus.c | 250 +--
1 files changed, 239 insertions(+), 11 deletions(-)
diff --git a/opensm/opensm/osm_ucast_torus.c b/opensm/opensm/osm_ucast_torus.c
index 082fcf5..e2eb324 100644
--- a/opensm/opensm/osm_ucast_torus.c
+++ b/opensm/opensm/osm_ucast_torus.c
@@ -300,6 +300,7 @@ struct torus {
struct coord_dirs *origin;
struct t_switch sw;
+ struct t_switch *master_stree_root;
unsigned flags;
int debug;
@@ -8515,6 +8516,241 @@ bool torus_lft(struct torus *t, struct t_switch *sw)
}
static
+osm_mtree_node_t *mcast_stree_branch(struct t_switch *sw, osm_switch_t *osm_sw,
+osm_mgrp_box_t *mgb, unsigned depth,
+unsigned *port_cnt, unsigned *max_depth)
+{
+ osm_mtree_node_t *mtn = NULL;
+ osm_mcast_tbl_t *mcast_tbl, *ds_mcast_tbl;
+ osm_node_t *ds_node;
+ struct t_switch *ds_sw;
+ struct port_grp *ptgrp;
+ struct link *link;
+ struct endpoint *port;
+ unsigned g, p;
+ unsigned mcast_fwd_ports = 0, mcast_end_ports = 0;
+
+ depth++;
+
+ if (osm_sw-priv != sw) {
+ OSM_LOG(sw-torus-osm-log, OSM_LOG_INFO,
+ Error: osm_sw (GUID 0x%04llx)
+ not in our fabric description\n,
+ ntohllu(osm_node_get_node_guid(osm_sw-p_node)));
+ goto out;
+ }
+ if (!osm_switch_supports_mcast(osm_sw)) {
+ OSM_LOG(sw-torus-osm-log, OSM_LOG_ERROR,
+ Error: osm_sw (GUID 0x%04llx)
+ does not support multicast\n,
+ ntohllu(osm_node_get_node_guid(osm_sw-p_node)));
+ goto out;
+ }
+ mtn = osm_mtree_node_new(osm_sw);
+ if (!mtn) {
+ OSM_LOG(sw-torus-osm-log, OSM_LOG_ERROR,
+ Insufficient memory to build multicast tree\n);
+ goto out;
+ }
+ mcast_tbl = osm_switch_get_mcast_tbl_ptr(osm_sw);
+ /*
+* Recurse to downstream switches, i.e. those closer to master
+* spanning tree branch tips.
+*
+* Note that if there are multiple ports in this port group, i.e.,
+* multiple parallel links, we can pick any one of them to use for
+* any individual MLID without causing loops. Pick one based on MLID
+* for now, until someone turns up evidence we need to be smarter.
+*
+* Also, it might be we got called in a window between a switch getting
+* removed from the fabric, and torus-2QoS getting to rebuild its
+* fabric representation. If that were to happen, our next hop
+* osm_switch pointer might be stale. Look it up via opensm's fabric
+* description to be sure it's not.
+*/
+ for (g = 0; g 2 * TORUS_MAX_DIM; g++) {
+ ptgrp = sw-ptgrp[g];
+ if (!ptgrp-to_stree_tip)
+ continue;
+
+ p = mgb-mlid % ptgrp-port_cnt;/* port # in port group */
+ p = ptgrp-port[p]-port; /* now port # in switch */
+
+ ds_node = osm_node_get_remote_node(osm_sw-p_node, p, NULL);
+ ds_sw = ptgrp-to_stree_tip-sw;
+
+ if (!(ds_node ds_node-sw
+ ds_sw-osm_switch == ds_node-sw)) {
+ OSM_LOG(sw-torus-osm-log, OSM_LOG_ERROR,
+ Error: stale pointer to osm_sw
+ (GUID 0x%04llx)\n, ntohllu(ds_sw-n_id));
+ continue;
+ }
+ mtn-child_array[p] =
+ mcast_stree_branch(ds_sw, ds_node-sw, mgb,
+ depth, port_cnt, max_depth);
+ if (!mtn-child_array[p])
+ continue;
+
+ osm_mcast_tbl_set(mcast_tbl, mgb-mlid, p);
+ mcast_fwd_ports++;
+ /*
+* Since we forward traffic for this multicast group on this
+* port, cause the switch on the other end of the link
+* to forward traffic back to us. Do it now
[PATCH 09/12] opensm: Make mcast_mgr_purge_tree() available outside osm_mcast_mgr.c.
A routing engine that needs to compute multicast spanning trees with
special properties will need to delete old trees. There's already
a function that does this: mcast_mgr_purge_tree().
Make it available outside osm_mcast_mgr.c, and change the name
to follow the naming convention (osm_ prefix) for global functions.
Signed-off-by: Jim Schutt jasc...@sandia.gov
---
opensm/include/opensm/osm_multicast.h | 33 +
opensm/opensm/osm_mcast_mgr.c |4 ++--
2 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/opensm/include/opensm/osm_multicast.h
b/opensm/include/opensm/osm_multicast.h
index 1da575d..df6ac6c 100644
--- a/opensm/include/opensm/osm_multicast.h
+++ b/opensm/include/opensm/osm_multicast.h
@@ -53,6 +53,7 @@
#include opensm/osm_mcm_port.h
#include opensm/osm_subnet.h
#include opensm/osm_log.h
+#include opensm/osm_sm.h
#ifdef __cplusplus
# define BEGIN_C_DECLS extern C {
@@ -193,6 +194,38 @@ osm_mgrp_t *osm_mgrp_new(IN osm_subn_t * subn, IN
ib_net16_t mlid,
* Multicast Group, osm_mgrp_delete
*/
+/*
+ * Need a forward declaration to work around include loop:
+ * osm_sm.h - osm_multicast.h
+ */
+struct osm_sm;
+
+/f* OpenSM: Multicast Tree/osm_purge_mtree
+* NAME
+* osm_purge_mtree
+*
+* DESCRIPTION
+* Frees all the nodes in a multicast spanning tree
+*
+* SYNOPSIS
+*/
+void osm_purge_mtree(IN struct osm_sm * sm, IN osm_mgrp_box_t * mgb);
+/*
+* PARAMETERS
+* sm
+* [in] Pointer to osm_sm_t object.
+* mgb
+* [in] Pointer to an osm_mgrp_box_t object.
+*
+* RETURN VALUES
+* None.
+*
+*
+* NOTES
+*
+* SEE ALSO
+*/
+
/f* OpenSM: Multicast Group/osm_mgrp_is_guid
* NAME
* osm_mgrp_is_guid
diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c
index e65e459..11a10ce 100644
--- a/opensm/opensm/osm_mcast_mgr.c
+++ b/opensm/opensm/osm_mcast_mgr.c
@@ -146,7 +146,7 @@ static void mcast_mgr_purge_tree_node(IN osm_mtree_node_t *
p_mtn)
free(p_mtn);
}
-static void mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_box_t * mbox)
+void osm_purge_mtree(osm_sm_t * sm, IN osm_mgrp_box_t * mbox)
{
OSM_LOG_ENTER(sm-p_log);
@@ -695,7 +695,7 @@ static ib_api_status_t
mcast_mgr_build_spanning_tree(osm_sm_t * sm,
on multicast forwarding table information if the user wants to
preserve existing multicast routes.
*/
- mcast_mgr_purge_tree(sm, mbox);
+ osm_purge_mtree(sm, mbox);
/* build the first subset containing all member ports */
if (make_port_list(port_list, mbox)) {
--
1.5.6.GIT
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 12/12] opensm: Update documentation to describe torus-2QoS multicast support.
Signed-off-by: Jim Schutt jasc...@sandia.gov --- opensm/doc/current-routing.txt | 121 +++- 1 files changed, 118 insertions(+), 3 deletions(-) diff --git a/opensm/doc/current-routing.txt b/opensm/doc/current-routing.txt index 141d793..78a2e01 100644 --- a/opensm/doc/current-routing.txt +++ b/opensm/doc/current-routing.txt @@ -400,8 +400,18 @@ Torus-2QoS Routing Algorithm Torus-2QoS is routing algorithm designed for large-scale 2D/3D torus fabrics. - -It is a DOR-based algorithm that avoids deadlocks that would otherwise +The torus-2QoS routing engine can provide the following functionality on +a 2D/3D torus: +- routing that is free of credit loops +- two levels of QoS, assuming switches support 8 data VLs +- ability to route around a single failed switch, and/or multiple failed +links, without +- introducing credit loops +- changing path SL values +- very short run times, with good scaling properties as fabric size +increases + +Torus-2QoS is a DOR-based algorithm that avoids deadlocks that would otherwise occur in a torus using the concept of a dateline for each torus dimension. It encodes into a path SL which datelines the path crosses as follows: @@ -424,7 +434,7 @@ ports as follows: sl2vl(iport,oport,sl) = 0x1 (sl cdir(oport)); Thus torus-2QoS consumes 8 SL values (SL bits 0-2) and 2 VL values (VL bit 0) - per QoS level to provide deadlock-free routing on a 3D torus. +per QoS level to provide deadlock-free routing on a 3D torus. Torus-2QoS routes around link failure by taking the long way around any 1D ring interrupted by a link failure. For example, consider the 2D 6x5 @@ -538,3 +548,108 @@ path S-n-I-q-r-D, with illegal turn at switch I, and with hop I-q using a VL with bit 1 set. In contrast to the earlier examples, the second hop after the illegal turn, q-r, can be used to construct a credit loop encircling the failed switches. + +Since torus-2QoS uses all four available SL bits, and the three data VL +bits that are typically available in current switches, there is no way +to use SL/VL values to separate multicast traffic from unicast traffic. +Thus, torus-2QoS must generate multicast routing such that credit loops +cannot arise from a combination of multicast and unicast path segments. + +It turns out that it is possible to construct spanning trees for multicast +routing that have that property. For the 2D 6x5 torus example above, here +is the full-fabric spanning tree that torus-2QoS will construct, where x +is the root switch and each + is a non-root switch: + + 4++++++ +|||||| + 3++++++ +|||||| + 2+++x++ +|||||| + 1++++++ +|||||| + y=0++++++ + + x=012345 + +For multicast traffic routed from root to tip, every turn in the above +spanning tree is a legal DOR turn. + +For traffic routed from tip to root, and some traffic routed through the +root, turns are not legal DOR turns. However, to construct a credit loop, +the union of multicast routing on this spanning tree with DOR unicast +routing can only provide 3 of the 4 turns needed for the loop. + +In addition, if none of the above spanning tree branches crosses a dateline +used for unicast credit loop avoidance on a torus, and if multicast traffic +is confined to SL 0 or SL 8 (recall that torus-2QoS uses SL bit 3 to +differentiate QoS level), then multicast traffic also cannot contribute to +the ring credit loops that are otherwise possible in a torus. + +Torus-2QoS uses these ideas to create a master spanning tree. Every +multicast group spanning tree will be constructed as a subset of the master +tree, with the same root as the master tree. + +Such multicast group spanning trees will in general not be optimal for +groups which are a subset of the full fabric. However, this compromise must +be made to enable support for two QoS levels on a torus while preventing +credit loops. + +In the presence of link or switch failures that result in a fabric for +which torus-2QoS can generate credit-loop-free unicast routes, it is also +possible to generate a master spanning tree for multicast that retains the +required properties. For example, consider that same 2D 6x5 torus, with +the link from (2,2) to (3,2) failed. Torus-2QoS will generate the following +master spanning tree: + + 4++++++ +|||||| + 3++++++ +|||||| + 2 --+++x++-- +|||||| + 1++++++ +|||||| + y=0++++++ + + x=012345 + +Two things are notable about this master
[PATCH 07/12] opensm: Make torus-2QoS always use OSM_LOG_INFO, never LOG_INFO.
Signed-off-by: Jim Schutt jasc...@sandia.gov
---
opensm/opensm/osm_ucast_torus.c | 10 +-
1 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/opensm/opensm/osm_ucast_torus.c b/opensm/opensm/osm_ucast_torus.c
index 0306af9..61e0bf3 100644
--- a/opensm/opensm/osm_ucast_torus.c
+++ b/opensm/opensm/osm_ucast_torus.c
@@ -7971,7 +7971,7 @@ void torus_update_osm_sl2vl(void *context, osm_port_t
*sw_mgmt_port,
guid_t guid;
guid = osm_node_get_node_guid(sw_mgmt_port-p_node);
- OSM_LOG(log, LOG_INFO,
+ OSM_LOG(log, OSM_LOG_INFO,
Error: osm_port (GUID 0x%04llx)
not in our fabric description\n, ntohllu(guid));
return;
@@ -8527,7 +8527,7 @@ uint8_t torus_path_sl(void *context, uint8_t path_sl_hint,
sport = osm_port_relink_endpoint(osm_sport);
if (!sport) {
guid = osm_node_get_node_guid(osm_sport-p_node);
- OSM_LOG(log, LOG_INFO,
+ OSM_LOG(log, OSM_LOG_INFO,
Error: osm_sport (GUID 0x%04llx)
not in our fabric description\n,
ntohllu(guid));
@@ -8539,7 +8539,7 @@ uint8_t torus_path_sl(void *context, uint8_t path_sl_hint,
dport = osm_port_relink_endpoint(osm_dport);
if (!dport) {
guid = osm_node_get_node_guid(osm_dport-p_node);
- OSM_LOG(log, LOG_INFO,
+ OSM_LOG(log, OSM_LOG_INFO,
Error: osm_dport (GUID 0x%04llx)
not in our fabric description\n,
ntohllu(guid));
@@ -8552,14 +8552,14 @@ uint8_t torus_path_sl(void *context, uint8_t
path_sl_hint,
*/
if (sport-type != SRCSINK) {
guid = osm_node_get_node_guid(osm_sport-p_node);
- OSM_LOG(log, LOG_INFO,
+ OSM_LOG(log, OSM_LOG_INFO,
Error: osm_sport (GUID 0x%04llx)
not a data src/sink port\n, ntohllu(guid));
goto out;
}
if (dport-type != SRCSINK) {
guid = osm_node_get_node_guid(osm_dport-p_node);
- OSM_LOG(log, LOG_INFO,
+ OSM_LOG(log, OSM_LOG_INFO,
Error: osm_dport (GUID 0x%04llx)
not a data src/sink port\n, ntohllu(guid));
goto out;
--
1.5.6.GIT
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 06/12] opensm: Remove redundant function names in torus-2QoS logging.
Signed-off-by: Jim Schutt jasc...@sandia.gov
---
opensm/opensm/osm_ucast_torus.c | 14 +++---
1 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/opensm/opensm/osm_ucast_torus.c b/opensm/opensm/osm_ucast_torus.c
index b740f93..0306af9 100644
--- a/opensm/opensm/osm_ucast_torus.c
+++ b/opensm/opensm/osm_ucast_torus.c
@@ -8437,7 +8437,7 @@ bool get_lid(struct port_grp *pg, unsigned p,
osm_port = ep-osm_port;
if (!(osm_port osm_port-priv == ep)) {
OSM_LOG(pg-sw-torus-osm-log, OSM_LOG_ERROR,
- Error: get_lid: ep-osm_port-priv != ep
+ Error: ep-osm_port-priv != ep
for sw 0x%04llu port %d\n,
ntohllu(((struct t_switch *)(ep-sw))-n_id), ep-port);
return false;
@@ -8528,8 +8528,8 @@ uint8_t torus_path_sl(void *context, uint8_t path_sl_hint,
if (!sport) {
guid = osm_node_get_node_guid(osm_sport-p_node);
OSM_LOG(log, LOG_INFO,
- Error: get_torus_sl: osm_sport (GUID
- 0x%04llx) not in our fabric description\n,
+ Error: osm_sport (GUID 0x%04llx)
+ not in our fabric description\n,
ntohllu(guid));
goto out;
}
@@ -8540,8 +8540,8 @@ uint8_t torus_path_sl(void *context, uint8_t path_sl_hint,
if (!dport) {
guid = osm_node_get_node_guid(osm_dport-p_node);
OSM_LOG(log, LOG_INFO,
- Error: get_torus_sl: osm_dport (GUID
- 0x%04llx) not in our fabric description\n,
+ Error: osm_dport (GUID 0x%04llx)
+ not in our fabric description\n,
ntohllu(guid));
goto out;
}
@@ -8553,14 +8553,14 @@ uint8_t torus_path_sl(void *context, uint8_t
path_sl_hint,
if (sport-type != SRCSINK) {
guid = osm_node_get_node_guid(osm_sport-p_node);
OSM_LOG(log, LOG_INFO,
- Error: get_torus_sl: osm_sport (GUID 0x%04llx)
+ Error: osm_sport (GUID 0x%04llx)
not a data src/sink port\n, ntohllu(guid));
goto out;
}
if (dport-type != SRCSINK) {
guid = osm_node_get_node_guid(osm_dport-p_node);
OSM_LOG(log, LOG_INFO,
- Error: get_torus_sl: osm_dport (GUID 0x%04llx)
+ Error: osm_dport (GUID 0x%04llx)
not a data src/sink port\n, ntohllu(guid));
goto out;
}
--
1.5.6.GIT
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 05/12] opensm: Enforce torus-2QoS link ordering convention.
The function ring_next_sw() used by torus-2QoS to build LFTs relies on the ordering convention that the 1 end of a link is in the positive coordinate direction WRT the 0 end. Previously the links were always built this way, but nothing enforced the convention. This commit adds code to enforce the convention, including code needed to label switches as they are installed into the torus, rather than after all the torus switches are found. Signed-off-by: Jim Schutt jasc...@sandia.gov --- I've attached the patch as a compressed file, as otherwise it is too large to make it through the list. -- Jim opensm/opensm/osm_ucast_torus.c | 433 +-- 1 files changed, 237 insertions(+), 196 deletions(-) 0005-opensm-Enforce-torus-2QoS-link-ordering-convention.patch.bz2 Description: application/bzip
Re: [PATCH 03/12] opensm: Remove unused port specification from torus-2QoS config file parsing.
The patch this email replies to changes the format of the torus-2QoS.conf file. The attached file works with the new format, and replaces the example file sent in reply to the original torus-2QoS patch series. -- Jim # We want the torus routing engine to attempt to find a # 5x5x5 torus in the fabric: torus 5 5 5 # We need to tell the routing engine what directions we # want the torus coordinate directions to be, by specifing # the endpoints (switch GUID only) of a link in each # direction. Here we specify positive coordinate directions: xp_link 0x20 0x200019 # S_0_0_0 - S_1_0_0 yp_link 0x20 0x25 # S_0_0_0 - S_0_1_0 zp_link 0x20 0x21 # S_0_0_0 - S_0_0_1 # If one of the above switches were to fail, the routing # engine would not have sufficient information to locate the # torus in the fabric. Specify a backup origin here: next_origin xp_link 0x20001f 0x200038 # S_1_1_1 - S_2_1_1 yp_link 0x20001f 0x200024 # S_1_1_1 - S_1_2_1 zp_link 0x20001f 0x200020 # S_1_1_1 - S_1_1_2 # The torus routing engine uses the concept of a dateline, # where a coordinate wraps from its maximum back to zero, # in order to compute path SL values that provide routing # that is free from credit loops. # # If it is forced by a failed switch to use the backup # origin specification, that would cause the datelines # to move, which would change many path SL values, which # defeats one of the main benefits of this routing engine. # So, describe the position of the original datelines # relative to the backup origin as follows: x_dateline -1 y_dateline -1 z_dateline -1 # You can specify as many backup origins as you like, but # in practice, the torus routing engine is only guaranteed # to be able to route around a single failed switch without # introducing credit loops, so one backup origin is enough.
