Re: [PATCH] opensm: make loopback console compile on by default.
On Jul 10, 2011, at 2:14 AM, Alex Netes wrote: Hi Ira, On 15:54 Wed 06 Jul , Ira Weiny wrote: The console is very useful for debugging and should be available in opensm.conf as an option. Generic socket is still an option which is off for security reasons. Signed-off-by: Ira Weiny wei...@llnl.gov --- I was digging a little in a history and one concern that was issued while socket support was introduced is that it requires libwrap devel package, so any one who lacks this package, opensm compilation will fail. My intention was to disable console_looback if libwrap was not available. But as I look at the configure.in I think there may be a bug in that logic. I don't have a system without libwrap readily available so give me some time to fix this. Ira -- Alex -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
I'm not quite sure I understand the use for enum ibv_extension_type.
Where/how is this used?
Ira
On Jun 3, 2011, at 5:34 PM, Hefty, Sean wrote:
In order to support OFED or vendor specific calls, define a
generic extension mechanism. This allows OFED, an RDMA vendor,
or another registered 3rd party (for example, the librdmacm)
to define RDMA extensions.
Users which make use extensions are aware that they are not
only using an extended call, but are given information regarding
how widely the extension by be supported. Support for extended
functions, data structures, and enums are defined.
Extensions are referenced by name. There is an assumption that
extension names are prefixed relative to the supporting party.
Until an extension has been incorporated into libibverbs, it
should be defined in an appropriate external header file.
For example, OFA could provide a header file with their definition
for XRC extensions. A partial view of such a header file might
look something similar to:
#ifndef OFA_XRC_H
#define OFA_XRC_H
#include infiniband/verbs.h
#define OFA_XRC_OPS ofa-xrc
/* Extend IBV_QP_TYPE for XRC */
#define OFA_QPT_XRC ((enum ibv_qp_type) \
(IBV_EXTENSION_OFA IBV_EXTENSION_BASE_SHIFT) + 6)
struct ofa_xrcd {
struct ibv_context *context;
};
struct ofa_xrc_ops {
struct ofa_xrcd * (*open_xrcd)(struct ibv_context *context,
inf fd, int oflags);
int * (*close_xrcd)(struct ofa_xrcd *xrcd);
/* other functions left as exercise to the reader */
};
#endif /* OFA_XRC_H */
Driver libraries that support extensions are given a new
registration call, ibv_register_device_ext(). Use of this call
indicates to libibverbs that the library allocates extended
versions of struct ibv_device and struct ibv_context.
The following new APIs are added to libibverbs to applications
to use to determine if an extension is supported and to obtain the
extended function calls.
ibv_have_ext_ops - returns true if an extension is supported
ibv_get_device_ext_ops - return extended operations for a device
ibv_get_ext_ops - return extended operations for an open context
To maintain backwards compatibility with existing applications,
internally, the library uses the last byte of the device name
to record if the device was registered with extension support.
Signed-off-by: Sean Hefty sean.he...@intel.com
---
Compile tested only at this point. I'm still working on writing
an XRC sample program.
include/infiniband/driver.h |1 +
include/infiniband/verbs.h | 40 +++-
src/device.c| 18 ++
src/ibverbs.h | 18 ++
src/init.c | 17 -
src/libibverbs.map |5 +
src/verbs.c |9 +
7 files changed, 106 insertions(+), 2 deletions(-)
diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h
index 9a81416..e48abfd 100644
--- a/include/infiniband/driver.h
+++ b/include/infiniband/driver.h
@@ -57,6 +57,7 @@ typedef struct ibv_device *(*ibv_driver_init_func)(const
char *uverbs_sys_path,
int abi_version);
void ibv_register_driver(const char *name, ibv_driver_init_func init_func);
+void ibv_register_driver_ext(const char *name, ibv_driver_init_func
init_func);
int ibv_cmd_get_context(struct ibv_context *context, struct ibv_get_context
*cmd,
size_t cmd_size, struct ibv_get_context_resp *resp,
size_t resp_size);
diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index 0f1cb2e..b82cd3a 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -55,6 +55,15 @@
BEGIN_C_DECLS
+enum ibv_extension_type {
+ IBV_EXTENSION_COMMON,
+ IBV_EXTENSION_VENDOR,
+ IBV_EXTENSION_OFA,
+ IBV_EXTENSION_RDMA_CM
+};
+#define IBV_EXTENSION_BASE_SHIFT 24
+#define IBV_EXTENSION_MASK 0xFF00
+
union ibv_gid {
uint8_t raw[16];
struct {
@@ -92,7 +101,8 @@ enum ibv_device_cap_flags {
IBV_DEVICE_SYS_IMAGE_GUID = 1 11,
IBV_DEVICE_RC_RNR_NAK_GEN = 1 12,
IBV_DEVICE_SRQ_RESIZE = 1 13,
- IBV_DEVICE_N_NOTIFY_CQ = 1 14
+ IBV_DEVICE_N_NOTIFY_CQ = 1 14,
+ IBV_DEVICE_EXTENSIONS = 1 (IBV_EXTENSION_BASE_SHIFT - 1)
};
enum ibv_atomic_cap {
@@ -623,6 +633,13 @@ struct ibv_device {
chardev_path[IBV_SYSFS_PATH_MAX];
/* Path to infiniband class device in sysfs */
charibdev_path[IBV_SYSFS_PATH_MAX];
+
+ /* Following fields only available if device supports extensions */
+ void *private;
+ int (*have_ext_ops)(struct ibv_device
[PATCH v2] Return EPROTONOSUPPORT when an RDMA device lacks QP0
On May 20, 2011, at 4:23 PM, Hefty, Sean wrote: mad_agent_priv-agent.mr = ib_get_dma_mr(port_priv-qp_info[qpn].qp- pd, IB_ACCESS_LOCAL_WRITE); in which case it may be safer to check for the NULL pointer. Can you confirm if this was the spot? Yes it was that spot. I did think of that today after I sent the patch. Do you think it would be safer just to check for both pointers QP0 and 1 (depending on the registration)? Yes, it seems safer and easier to maintain if we just validated the pointer. - Sean New patch attached. Ira 0001-Return-EPROTONOSUPPORT-when-an-RDMA-device-lacks-the.patch Description: Binary data
Re: [PATCH v2] Return EPROTONOSUPPORT when an RDMA device lacks QP0
On May 22, 2011, at 9:05 PM, Roland Dreier wrote: @@ -2738,6 +2745,7 @@ static void init_mad_qp(struct ib_mad_port_private *port_priv, spin_lock_init(qp_info-snoop_lock); qp_info-snoop_table = NULL; qp_info-snoop_table_size = 0; +qp_info-qp = NULL; atomic_set(qp_info-snoop_count, 0); } This bit isn't needed, is it? qp_info is in memory allocated with kzalloc. (We could get rid of the other = 0 / = NULL lines too) Yes, sorry, revised attached, Ira 0001-Return-EPROTONOSUPPORT-when-an-RDMA-device-lacks-the.patch Description: Binary data
[PATCH] Return EPROTONOSUPPORT when an RDMA device lacks QP0
We had a script which was looping through the devices returned from ibstat and attempted to register a SMI agent on an ethernet device. This caused a kernel panic. We have fixed our script. This patch prevents the panic. 0001-Return-EPROTONOSUPPORT-when-an-RDMA-device-lacks-QP0.patch Description: Binary data
Re: opensm: file routing engine
On Apr 22, 2011, at 1:37 PM, Paul Monday (Parallel Scientific) wrote:
Thank you, your detail is greatly appreciated :)
I have one other strange question ... is it possible to carve a single
physical switch into two logical switches (put a cable between ports
16/17 and modify the routing tables ... this seems like it wouldn't work
as the Unicast LID / Switch: guid rows in the respective files below
serve as keys so the single switch would be identified twice).
Not that I am aware of. When you say you have a single switch I assume you
mean a switch based on a single switch ASIC? Like a 24 or 36 port pizza
box switch.
Yes, a 36 port Mellanox pizza box with a single crossbar ... based on
how I read these files, it looks like they key off a single GUID that
identifies the switch ... which would probably make the subnet manager
unhappy if I arbitrarily tried to mock it up being two switches somehow
The file formats seem to be:
opensm-lfts.dump (later becomes -U [file])
- Contains all discovered ports (powered on), their function (Switch vs.
Channel Adapter), their LID and some extra information. This is
essentially the physical network (if all machines are powered on) ...
the format is:
Unicast lids [0-x] of switch Lid LID# guidGUID ('switch description'):
LID 0x SwitchPort ZZZ #Channel Adapter | Switch portguid
GUID: 'Descirption'
I assume this file grows with all of the Channel Adapters and switches.
Given a switch-switch connection a row would look like
0x0019 005 # Switch portguid 0x003 'MF3:switch-my:MTS3600/U1'
Yes this file grows with more nodes in the system. But the line above is
not a connection but rather a linear forwarding table entry. In general,
this is saying that for the given lid 0x0019 route out port 5 of that
switch (the switch given by the Unicast lids [... line. The information
after '#' is more information about the node with lid=0x0019. This is _not_
the other end of the link on port 5.
Ahhh, I see ... so this table could get quite large ... if I have 1,000
nodes in a subnet, each with a LID assigned, this table would become
quite large as each LID would be listed for each switch if I have my
forwarding thoughts in my head ... maybe I need to wander around and
steal another switch from someone ;-)
Another option would be to use ibsim:
git://git.openfabrics.org/~alexnetes/ibsim.git
You could simulate more switches in your network.
Ira
The topology of the physical connections are shown in opensm-subnet.lst.
Ahhh, but the opensm-subnet.lst is not handed to the file routing
algorithm ... this must be derived at runtime each run I'm guessing
and then dumped to /var/log. Very helpful! Thank you for the pointer.
You could essentially use this file to map the entire physical network,
you would end up with a graph ... but no information for how to traverse
it efficiently, does that sound right?
No this is not mapping the physical network. It is a dump of the port
forwarding which was programed into each switch by opensm.
Changing this file is what allows you to change the routing and then feed it
back into opensm.
opensm-lid-matrix.dump
- Looks like it contains the hop information ... but it's a bit more
cryptic since I have only one switch :( It should contain a list of all
switches, the LID for the switch and then hop information. The hop
information is what I'm a bit puzzled about here, as well as what port
guid information is tacked on. The format of the file is:
Switch: guid 0xx
LID 0x 00 ff ffhops for all ports # portguid 0x000
That is the switch to switch hop count information. Probably not of much use
with only 1 switch.
Ugh ... I need another switch or .dump files from someone ... I haven't
found any stray .dump files out on the network, but then, Google knows
all and someone must have posted a couple somewhere to play with.
Thank you so much again Ira, I wasn't too far off and mostly it seems
I'm off in places that having only a single switch wouldn't let me see.
The semantic correction of opensm-lfts.dump was critical.
Cheers, have a wonderful weekend.
Paul Monday
Parallel Scientific, LLC
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] infiniband-diags: iblinkinfo; remove non-supported -c option from man page
0001-infiniband-diags-iblinkinfo-remove-non-supported-c-o.patch Description: Binary data
[PATCH] libibmad: Remove libibnetdisc specific define
0001-libibmad-Remove-libibnetdisc-specific-define.patch Description: Binary data
[PATCH] infiniband-diags: improve backwards/standalone compatibility
0002-infiniband-diags-improve-backwards-standalone-compat.patch Description: Binary data
Re: [PATCH] infiniband-diags/ibnetdiscover: Make --ports option use remap_node_name()
On Apr 25, 2011, at 10:29 AM, Jim Schutt wrote: From: Marcus Epperson mrep...@sandia.gov Signed-off-by: Marcus Epperson mrep...@sandia.gov Signed-off-by: Jim Schutt jasc...@sandia.gov --- Thanks applied, Ira -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: opensm: file routing engine
On Apr 22, 2011, at 7:41 AM, Paul Monday (Parallel Scientific) wrote:
I've been toying with the file routing engine implementation for some
work I'm doing, but I'm finding very little documentation on it. I only
have one switch to experiment with at the moment as well so some of the
information in the lid / lfts files that are generated are not obvious
for how they expand to a multiple switch environment. Perhaps there is
a document around since I'm a RTFM type of person?
At any rate, here's what I've gathered with 4. being the big question.
1. The easiest way to get started with the file routing engine is to
generate the lid / lfts using a different routing engine. I went ahead
and did the following: opensm -D 0x40 -R ftree
2. Once run, copy the /var/log/opensm-lfts.dump and
/var/log/opensm-lid-matrix.dump files elsewhere for use
3. I've tried to generalize the file contents below
4. Modify the opensm-lid-matrix.dump file to implement or tweak the
routing algorithm over the physical network?
5. Run opensm -R file -M new-lid-matrix.dump -U new-lfts.dump
I think this is the general method yes.
I have one other strange question ... is it possible to carve a single
physical switch into two logical switches (put a cable between ports
16/17 and modify the routing tables ... this seems like it wouldn't work
as the Unicast LID / Switch: guid rows in the respective files below
serve as keys so the single switch would be identified twice).
Not that I am aware of. When you say you have a single switch I assume you
mean a switch based on a single switch ASIC? Like a 24 or 36 port pizza box
switch.
The file formats seem to be:
opensm-lfts.dump (later becomes -U [file])
- Contains all discovered ports (powered on), their function (Switch vs.
Channel Adapter), their LID and some extra information. This is
essentially the physical network (if all machines are powered on) ...
the format is:
Unicast lids [0-x] of switch Lid LID# guid GUID ('switch description'):
LID 0x SwitchPort ZZZ # Channel Adapter | Switch portguid
GUID: 'Descirption'
I assume this file grows with all of the Channel Adapters and switches.
Given a switch-switch connection a row would look like
0x0019 005 # Switch portguid 0x003 'MF3:switch-my:MTS3600/U1'
Yes this file grows with more nodes in the system. But the line above is not a
connection but rather a linear forwarding table entry. In general, this is
saying that for the given lid 0x0019 route out port 5 of that switch (the
switch given by the Unicast lids [... line. The information after '#' is
more information about the node with lid=0x0019. This is _not_ the other end of
the link on port 5.
The topology of the physical connections are shown in opensm-subnet.lst.
You could essentially use this file to map the entire physical network,
you would end up with a graph ... but no information for how to traverse
it efficiently, does that sound right?
No this is not mapping the physical network. It is a dump of the port
forwarding which was programed into each switch by opensm.
Changing this file is what allows you to change the routing and then feed it
back into opensm.
opensm-lid-matrix.dump
- Looks like it contains the hop information ... but it's a bit more
cryptic since I have only one switch :( It should contain a list of all
switches, the LID for the switch and then hop information. The hop
information is what I'm a bit puzzled about here, as well as what port
guid information is tacked on. The format of the file is:
Switch: guid 0xx
LID 0x 00 ff ff hops for all ports # portguid 0x000
That is the switch to switch hop count information. Probably not of much use
with only 1 switch.
Ira
I know ... it's a detailed question but I figured I would write enough
so someone else wouldn't have to reverse engineer using the file routing
engine if this is basically right.
Paul Monday
Parallel Scientific, LLC
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: opensm: switch incorrectly reports IB_PORT_CAP_HAS_MCAST_FDB_TOP ?
On Apr 22, 2011, at 11:19 AM, Jim Schutt wrote:
Hi,
I've been testing the current opensm development head
(commit 83b67527d16 from git://git.openfabrics.org/~alexnetes/opensm),
and I've been getting some messages that are new since version 3.3.7:
Apr 22 12:08:09 646534 [411CD940] 0x01 - log_rcv_cb_error: ERR 3111:
Received MAD with error status = 0x1C
SubnGetResp(SwitchInfo), attr_mod 0x0, TID 0x4802
Initial path: 0,1,1,4 Return path: 0,20,1,7
I get one of these messages for each switch in my fabric, on every
heavy sweep.
It appears these are caused by my switches incorrectly reporting
the capability IB_PORT_CAP_HAS_MCAST_FDB_TOP; i.e. this patch stops
the messages:
diff --git a/opensm/osm_mcast_mgr.c b/opensm/osm_mcast_mgr.c
index ea52bfe..63d2968 100644
--- a/opensm/osm_mcast_mgr.c
+++ b/opensm/osm_mcast_mgr.c
@@ -1041,7 +1041,7 @@ static void mcast_mgr_set_mfttop(IN osm_sm_t * sm, IN
osm_switch_t * p_sw)
p_path = osm_physp_get_dr_path_ptr(p_physp);
p_tbl = osm_switch_get_mcast_tbl_ptr(p_sw);
- if (p_physp-port_info.capability_mask IB_PORT_CAP_HAS_MCAST_FDB_TOP)
{
+ if (0 p_physp-port_info.capability_mask
IB_PORT_CAP_HAS_MCAST_FDB_TOP) {
/*
Set the top of the multicast forwarding table.
*/
IB_PORT_CAP_HAS_MCAST_FDB_TOP is bit 30 of the port capability mask,
which in at least IBA v1.2.1 was a reserved bit but apparently is
not anymore.
Yes these have been published as errata to the 1.2.1 specification.
smpquery portinfo lid
should show you if it is reporting that field. Also what does
smpquery switchinfo lid
say?
Ira
Should I file a bug report with my switch vendor about setting
a port capability bit for a capability they don't support, or
is there something else going on that I haven't figured out yet?
FWIW I think my switches have a base SP0; maybe it's got something
to do with that?
Thanks -- Jim
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Issues with patchwork?
I just checked patchwork and it looks like those patches which were submitted since ~2/25/2011 are no longer there? This includes 3 patches which were in my todo list yesterday. Does anyone know what the issue is? Thanks, Ira -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] opensm: perfmgr, only set orig_lid when we have a valid port. Otherwise leave it as 0
On Apr 17, 2011, at 8:21 AM, Hal Rosenstock wrote:
On 4/15/2011 6:17 PM, Ira Weiny wrote:
Subject: [PATCH] opensm: perfmgr, only set orig_lid when we have a valid
port. Otherwise leave it as 0
Signed-off-by: Ira Weiny wei...@llnl.gov
---
opensm/osm_perfmgr.c |4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/opensm/osm_perfmgr.c b/opensm/osm_perfmgr.c
index 6a1fa63..3e1575a 100644
--- a/opensm/osm_perfmgr.c
+++ b/opensm/osm_perfmgr.c
@@ -454,7 +454,9 @@ static void collect_guids(cl_map_item_t * p_map_item,
void *context)
ib_switch_info_is_enhanced_port0(node-sw-
switch_info));
for (port = mon_node-esp0 ? 0 : 1; port num_ports; port++) {
-mon_node-port[port].orig_lid = get_base_lid(node,
port);
+mon_node-port[port].orig_lid = 0;
+if (osm_physp_is_valid(node-physp_table[port]))
+mon_node-port[port].orig_lid =
get_base_lid(node, port);
mon_node-port[port].valid = TRUE;
}
There are several other calls to get_base_lid. Is it also an issue there
too or is port always valid there ? If it's not always valid for those
cases, why not just move this check into get_base_lid ?
I believe this is a special case because we are looping through the ports of a
CA node and not all of it's ports are valid on this fabric. Most of the other
places where get_base_lid is called I believe the port is known to be good,
therefore there is an assertion in there just to protect us.
I think the better way to fix this would probably be to change the perfmgr to
track ports rather than nodes.[*] Unfortunately this will take a significant
amount of coding effort. I think the best thing to do is just check if the
physical port is valid as I did above.
As to Alex's comment I think the port should be marked invalid as well. (I
should have CC'ed the list on my response to him.) I will resend the patch
with that change.
Ira
[*] At the time I coded the perfmgr it seemed to make more sense to track nodes
rather than ports. In hindsight this might have been a mistake but for this
small place I don't see a reason to re-architect it all yet.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] opensm: perfmgr, only set orig_lid when we have a valid port. Otherwise leave it as 0
On Apr 17, 2011, at 8:06 AM, Alex Netes wrote:
Hi Ira,
On 15:17 Fri 15 Apr , Ira Weiny wrote:
Subject: [PATCH] opensm: perfmgr, only set orig_lid when we have a valid
port. Otherwise leave it as 0
Signed-off-by: Ira Weiny wei...@llnl.gov
---
opensm/osm_perfmgr.c |4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/opensm/osm_perfmgr.c b/opensm/osm_perfmgr.c
index 6a1fa63..3e1575a 100644
--- a/opensm/osm_perfmgr.c
+++ b/opensm/osm_perfmgr.c
@@ -454,7 +454,9 @@ static void collect_guids(cl_map_item_t * p_map_item,
void *context)
ib_switch_info_is_enhanced_port0(node-sw-
switch_info));
for (port = mon_node-esp0 ? 0 : 1; port num_ports; port++) {
-mon_node-port[port].orig_lid = get_base_lid(node,
port);
+mon_node-port[port].orig_lid = 0;
+if (osm_physp_is_valid(node-physp_table[port]))
+mon_node-port[port].orig_lid =
get_base_lid(node, port);
mon_node-port[port].valid = TRUE;
Shouldn't this port marked with mon_node-port[port].valid = FALSE ?
New patch below.
Subject: [PATCH 1/5] opensm/perfmgr: set redirect orig_lid and valid flag only
when we have a valid port.
Signed-off-by: Ira Weiny wei...@llnl.gov
---
opensm/osm_perfmgr.c |8 ++--
1 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/opensm/osm_perfmgr.c b/opensm/osm_perfmgr.c
index 5758587..3578e13 100644
--- a/opensm/osm_perfmgr.c
+++ b/opensm/osm_perfmgr.c
@@ -454,8 +454,12 @@ static void collect_guids(cl_map_item_t * p_map_item, void
*context)
ib_switch_info_is_enhanced_port0(node-sw-
switch_info));
for (port = mon_node-esp0 ? 0 : 1; port num_ports; port++) {
- mon_node-port[port].orig_lid = get_base_lid(node,
port);
- mon_node-port[port].valid = TRUE;
+ mon_node-port[port].orig_lid = 0;
+ mon_node-port[port].valid = FALSE;
+ if (osm_physp_is_valid(node-physp_table[port])) {
+ mon_node-port[port].orig_lid =
get_base_lid(node, port);
+ mon_node-port[port].valid = TRUE;
+ }
}
cl_qmap_insert(pm-monitored_map, node_guid,
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
