debug sometimes slow performance
Hi all. I'm very happy with using srpt disk via ib_srp (linux 3.10).
Storage used by virtual machines (xen), target side (2 servers):
Software raid 10 with lvm on top of it.
scst.conf:
HANDLER vdisk_fileio {
DEVICE sas00_md127 {
filename /dev/md127
nv_cache 1
}
}
TARGET_DRIVER ib_srpt {
TARGET ib_srpt_target_0 {
enabled 1
io_grouping_type this_group_only
rel_tgt_id 1
LUN 0 sas00_md127
}
}
LVM (pv /dev/md127) exported to initiator node (xen).
On initiator node from both servers disks are goes to multipath.
On this pv i have many lv for each virtual machine.
For each vps i'm assemble raid1 from 2 lv (lv from different target nodes).
Sometimes all works fine, sometimes not. Which sysfs/procfs entries i
can check on initiator and target side to determine bottleneck? I'm
try to run blktrace to md device provide for vps, but not get needed
info.
Linux 3.10
SCST v3.0.0-pre2 (svn 4973)
Debian 7
Can somebody helps me?
--
Vasiliy Tolstov,
e-mail: v.tols...@selfip.ru
jabber: v...@selfip.ru
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] please pull infiniband.git
Hi Linus, Please pull from git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git tags/rdma-for-linus Main batch of InfiniBand/RDMA changes for 3.15: - The biggest change is core API extensions and mlx5 low-level driver support for handling DIF/DIX-style protection information, and the addition of PI support to the iSER initiator. Target support will be arriving shortly through the SCSI target tree. - A nice simplification to the umem memory pinning library now that we have chained sg lists. Kudos to Yishai Hadas for realizing our code didn't have to be so crazy. - Another nice simplification to the sg wrappers used by qib, ipath and ehca to handle their mapping of memory to adapter. - The usual batch of fixes to bugs found by static checkers etc. from intrepid people like Dan Carpenter and Yann Droneaud. - A large batch of cxgb4, ocrdma, qib driver updates. Alex Tabachnik (2): IB/iser: Introduce pi_enable, pi_guard module parameters IB/iser: Initialize T10-PI resources Ariel Nahum (1): IB/iser: Remove struct iscsi_iser_conn Bart Van Assche (7): IB/mlx4: Fix a sparse endianness warning scsi_transport_srp: Fix two kernel-doc warnings IB/srp: Add more logging IB/srp: Avoid duplicate connections IB/srp: Make writing into the add_target sysfs attribute interruptible IB/srp: Avoid that writing into add_target hangs due to a cable pull IB/srp: Fix a race condition between failing I/O and I/O completion CQ Tang (1): IB/qib: Change SDMA progression mode depending on single- or multi-rail Dan Carpenter (7): IB/qib: Remove duplicate check in get_a_ctxt() RDMA/nes: Clean up a condition RDMA/cxgb4: Fix underflows in c4iw_create_qp() RDMA/cxgb4: Fix four byte info leak in c4iw_create_cq() IB/qib: Cleanup qib_register_observer() mlx4_core: Fix some indenting in mlx4_ib_add() mlx4_core: Make buffer larger to avoid overflow warning Dennis Dalessandro (3): IB/qib: Fix potential buffer overrun in sending diag packet routine IB/ipath: Fix potential buffer overrun in sending diag packet routine IB/qib: Fix memory leak of recv context when driver fails to initialize. Devesh Sharma (9): RDMA/ocrdma: EQ full catastrophe avoidance RDMA/ocrdma: SQ and RQ doorbell offset clean up RDMA/ocrdma: Read ASIC_ID register to select asic_gen RDMA/ocrdma: Allow DPP QP creation RDMA/ocrdma: ABI versioning between ocrdma and be2net be2net: Add abi version between be2net and ocrdma RDMA/ocrdma: Update version string RDMA/ocrdma: Increment abi version count RDMA/ocrdma: Code clean-up Fabio Estevam (1): IB/usnic: Remove '0x' when using %pa format Mike Marciniszyn (7): IB/qib: Fix debugfs ordering issue with multiple HCAs IB/qib: Add percpu counter replacing qib_devdata int_counter IB/qib: Modify software pma counters to use percpu variables IB/qib: Remove ib_sg_dma_address() and ib_sg_dma_len() overloads IB/ipath: Remove ib_sg_dma_address() and ib_sg_dma_len() overloads IB/ehca: Remove ib_sg_dma_address() and ib_sg_dma_len() overloads IB/core: Remove overload in ib_sg_dma* Moni Shoua (1): IB/core: Don't resolve passive side RoCE L2 address in CMA REQ handler Or Gerlitz (3): IB/iser: Print QP information once connection is established IB/iser: Update Mellanox copyright note IB/iser: Bump driver version to 1.3 Prarit Bhargava (1): RDMA/ocrdma: Fix compiler warning Randy Dunlap (1): IB/iser: Fix sector_t format warning Roi Dayan (1): IB/iser: Drain the tx cq once before looping on the rx cq Roland Dreier (2): RDMA/ocrdma: Fix warnings about pointer - integer casts Merge branches 'core', 'cxgb4', 'ip-roce', 'iser', 'misc', 'mlx4', 'nes', 'ocrdma', 'qib', 'sgwrapper', 'srp' and 'usnic' into for-next Sagi Grimberg (23): IB/core: Introduce protected memory regions IB/core: Introduce signature verbs API mlx5: Implement create_mr and destroy_mr IB/mlx5: Initialize mlx5_ib_qp signature-related members IB/mlx5: Break up wqe handling into begin finish routines IB/mlx5: Remove MTT access mode from umr flags helper function IB/mlx5: Keep mlx5 MRs in a radix tree under device IB/mlx5: Support IB_WR_REG_SIG_MR IB/mlx5: Collect signature error completion IB/mlx5: Expose support for signature MR feature IB/iser: Suppress completions for fast registration work requests IB/iser: Avoid FRWR notation, use fastreg instead IB/iser: Push the decision what memory key to use into fast_reg_mr routine IB/iser: Move fast_reg_descriptor initialization to a function IB/iser: Keep IB device attributes under
RE: [PATCH librdmacm] cmtime: rework program to be multithread
Ping. I noticed this never got picked up. Was there a problem, or just overlooked? Sorry - but I think it just got dropped or lost. Can you resend? I don't see this available on patchworks, and it's not in my inbox any longer. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH librdmacm] cmtime: rework program to be multithread
On 4/3/2014 5:00 PM, Hefty, Sean wrote: Ping. I noticed this never got picked up. Was there a problem, or just overlooked? Sorry - but I think it just got dropped or lost. Can you resend? I don't see this available on patchworks, and it's not in my inbox any longer. I'll add it to the set that I'm currently working on. signature.asc Description: OpenPGP digital signature
[PATCH 03/14] ibacm: use sysfs (in acm_if_is_ib) rather than ioctl(... SIOCGIFHWADDR ...) to read interface type
From: Ira Weiny ira.we...@intel.com
Getting an interface type from sysfs is easier than using an ioctl when an
interface name is readily available (as is the case with netlink.)
In preparation for netlink support create a function which uses sysfs and use
it instead of ioctls.
Signed-off-by: Ira Weiny ira.we...@intel.com
---
linux/acme_linux.c | 34 +++---
1 files changed, 27 insertions(+), 7 deletions(-)
diff --git a/linux/acme_linux.c b/linux/acme_linux.c
index 201ff19..0bd1f75 100644
--- a/linux/acme_linux.c
+++ b/linux/acme_linux.c
@@ -38,6 +38,7 @@
#include netinet/in.h
#include sys/socket.h
#include sys/types.h
+#include errno.h
#include infiniband/verbs.h
@@ -101,6 +102,31 @@ get_sgid(char *ifname, union ibv_gid *sgid)
return ret;
}
+static int acm_if_is_ib(char *ifname)
+{
+ unsigned type;
+ char buf[128];
+ FILE *f;
+ int ret;
+
+ snprintf(buf, sizeof buf, //sys//class//net//%s//type, ifname);
+ f = fopen(buf, r);
+ if (!f) {
+ printf(failed to open %s\n, buf);
+ return 0;
+ }
+
+ if (fgets(buf, sizeof buf, f)) {
+ type = strtol(buf, NULL, 0);
+ ret = (type == ARPHRD_INFINIBAND);
+ } else {
+ ret = 0;
+ }
+
+ fclose(f);
+ return ret;
+}
+
static int
get_devaddr(char *ifname, int *dev_index, uint8_t *port, uint16_t *pkey)
{
@@ -189,13 +215,7 @@ int gen_addr_ip(FILE *f)
continue;
}
- ret = ioctl(s, SIOCGIFHWADDR, ifr[i]);
- if (ret) {
- printf(failed to get hw address %d\n, ret);
- continue;
- }
-
- if (ifr[i].ifr_hwaddr.sa_family != ARPHRD_INFINIBAND)
+ if (!acm_if_is_ib(ifr[i].ifr_name))
continue;
ret = get_devaddr(ifr[i].ifr_name, dev_index, port, pkey);
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 05/14] ibacm: Move calling of helper functions out of get_devaddr
From: Ira Weiny ira.we...@intel.com
Again this is in prep for sharing this code with ibacm and ib_acme
get_devaddr uses globals which are specific to ib_acme. Move toward this
function being ib_acme specific while the code in get_addr_ip becomes generic.
Signed-off-by: Ira Weiny ira.we...@intel.com
---
linux/acme_linux.c | 64 ++--
src/acm_util.h |5
2 files changed, 47 insertions(+), 22 deletions(-)
diff --git a/linux/acme_linux.c b/linux/acme_linux.c
index b246a9c..187507d 100644
--- a/linux/acme_linux.c
+++ b/linux/acme_linux.c
@@ -47,25 +47,13 @@ extern int dev_cnt;
extern int verbose;
static int
-get_devaddr(char *ifname, int *dev_index, uint8_t *port, uint16_t *pkey)
+get_devaddr(union ibv_gid *sgid, int *dev_index, uint8_t *port)
{
struct ibv_device_attr dev_attr;
struct ibv_port_attr port_attr;
- union ibv_gid sgid, gid;
+ union ibv_gid gid;
int ret, i;
- ret = acm_if_get_sgid(ifname, sgid);
- if (ret) {
- printf(unable to get sgid\n);
- return ret;
- }
-
- ret = acm_if_get_pkey(ifname, pkey);
- if (ret) {
- printf(unable to get pkey\n);
- return ret;
- }
-
for (*dev_index = 0; *dev_index dev_cnt; (*dev_index)++) {
ret = ibv_query_device(verbs[*dev_index], dev_attr);
if (ret)
@@ -81,7 +69,7 @@ get_devaddr(char *ifname, int *dev_index, uint8_t *port,
uint16_t *pkey)
if (ret || !gid.global.interface_id)
break;
- if (!memcmp(sgid.raw, gid.raw, sizeof gid))
+ if (!memcmp(sgid-raw, gid.raw, sizeof gid))
return 0;
}
}
@@ -89,14 +77,39 @@ get_devaddr(char *ifname, int *dev_index, uint8_t *port,
uint16_t *pkey)
return -1;
}
+static void iter_cb(char *ifname, union ibv_gid *gid, uint16_t pkey,
+ uint8_t addr_type, uint8_t *addr, size_t addr_len,
+ char *addr_name, void *ctx)
+{
+ FILE *f = (FILE *)ctx;
+ int ret;
+ int dev_index;
+ uint8_t port;
+
+ ret = get_devaddr(gid, dev_index, port);
+ if (ret) {
+ printf(Failed to find verbs device for %s\n, ifname);
+ return;
+ }
+
+ if (verbose)
+ printf(%s %s %d 0x%x\n, addr_name,
verbs[dev_index]-device-name, port, pkey);
+ fprintf(f, %s %s %d 0x%x\n, addr_name,
verbs[dev_index]-device-name, port, pkey);
+}
+
int gen_addr_ip(FILE *f)
{
+ return acm_if_iter_sys(iter_cb, (void *)f);
+}
+
+int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx)
+{
struct ifconf *ifc;
struct ifreq *ifr;
char ip[INET6_ADDRSTRLEN];
- int s, ret, dev_index, i, len;
+ int s, ret, i, len;
uint16_t pkey;
- uint8_t port;
+ union ibv_gid sgid;
s = socket(AF_INET6, SOCK_DGRAM, 0);
if (!s)
@@ -137,13 +150,19 @@ int gen_addr_ip(FILE *f)
if (!acm_if_is_ib(ifr[i].ifr_name))
continue;
- ret = get_devaddr(ifr[i].ifr_name, dev_index, port, pkey);
- if (ret)
+ ret = acm_if_get_sgid(ifr[i].ifr_name, sgid);
+ if (ret) {
+ printf(unable to get sgid\n);
+ continue;
+ }
+
+ ret = acm_if_get_pkey(ifr[i].ifr_name, pkey);
+ if (ret) {
+ printf(unable to get pkey\n);
continue;
+ }
- if (verbose)
- printf(%s %s %d 0x%x\n, ip,
verbs[dev_index]-device-name, port, pkey);
- fprintf(f, %s %s %d 0x%x\n, ip,
verbs[dev_index]-device-name, port, pkey);
+ cb(ifr[i].ifr_name, sgid, pkey, 0, NULL, 0, ip, ctx);
}
ret = 0;
@@ -152,4 +171,5 @@ out2:
out1:
close(s);
return ret;
+
}
diff --git a/src/acm_util.h b/src/acm_util.h
index 94aac9d..ea86133 100644
--- a/src/acm_util.h
+++ b/src/acm_util.h
@@ -36,4 +36,9 @@ int acm_if_is_ib(char *ifname);
int acm_if_get_pkey(char *ifname, uint16_t *pkey);
int acm_if_get_sgid(char *ifname, union ibv_gid *sgid);
+typedef void (*acm_if_iter_cb)(char *ifname, union ibv_gid *gid, uint16_t pkey,
+ uint8_t addr_type, uint8_t *addr, size_t
addr_len,
+ char *addr_name, void *ctx);
+int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx);
+
#endif /* ACM_IF_H */
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 02/14] ibacm: pass interface name rather than 'struct ifreq' to get_devaddr, get_sgid, and get_pkey
From: Ira Weiny ira.we...@intel.com
Signed-off-by: Ira Weiny ira.we...@intel.com
---
linux/acme_linux.c | 17 -
1 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/linux/acme_linux.c b/linux/acme_linux.c
index 6978db7..201ff19 100644
--- a/linux/acme_linux.c
+++ b/linux/acme_linux.c
@@ -47,13 +47,13 @@ extern int verbose;
static int
-get_pkey(struct ifreq *ifreq, uint16_t *pkey)
+get_pkey(char *ifname, uint16_t *pkey)
{
char buf[128], *end;
FILE *f;
int ret;
- snprintf(buf, sizeof buf, //sys//class//net//%s//pkey,
ifreq-ifr_name);
+ snprintf(buf, sizeof buf, //sys//class//net//%s//pkey, ifname);
f = fopen(buf, r);
if (!f) {
printf(failed to open %s\n, buf);
@@ -73,13 +73,13 @@ get_pkey(struct ifreq *ifreq, uint16_t *pkey)
}
static int
-get_sgid(struct ifreq *ifr, union ibv_gid *sgid)
+get_sgid(char *ifname, union ibv_gid *sgid)
{
char buf[128], *end;
FILE *f;
int i, p, ret;
- snprintf(buf, sizeof buf, //sys//class//net//%s//address,
ifr-ifr_name);
+ snprintf(buf, sizeof buf, //sys//class//net//%s//address, ifname);
f = fopen(buf, r);
if (!f) {
printf(failed to open %s\n, buf);
@@ -102,21 +102,20 @@ get_sgid(struct ifreq *ifr, union ibv_gid *sgid)
}
static int
-get_devaddr(struct ifreq *ifr,
- int *dev_index, uint8_t *port, uint16_t *pkey)
+get_devaddr(char *ifname, int *dev_index, uint8_t *port, uint16_t *pkey)
{
struct ibv_device_attr dev_attr;
struct ibv_port_attr port_attr;
union ibv_gid sgid, gid;
int ret, i;
- ret = get_sgid(ifr, sgid);
+ ret = get_sgid(ifname, sgid);
if (ret) {
printf(unable to get sgid\n);
return ret;
}
- ret = get_pkey(ifr, pkey);
+ ret = get_pkey(ifname, pkey);
if (ret) {
printf(unable to get pkey\n);
return ret;
@@ -199,7 +198,7 @@ int gen_addr_ip(FILE *f)
if (ifr[i].ifr_hwaddr.sa_family != ARPHRD_INFINIBAND)
continue;
- ret = get_devaddr(ifr[i], dev_index, port, pkey);
+ ret = get_devaddr(ifr[i].ifr_name, dev_index, port, pkey);
if (ret)
continue;
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 01/14] ibacm: get_devaddr remove unused variable from signature
From: Ira Weiny ira.we...@intel.com
---
linux/acme_linux.c |4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/linux/acme_linux.c b/linux/acme_linux.c
index ef5b107..6978db7 100644
--- a/linux/acme_linux.c
+++ b/linux/acme_linux.c
@@ -102,7 +102,7 @@ get_sgid(struct ifreq *ifr, union ibv_gid *sgid)
}
static int
-get_devaddr(int s, struct ifreq *ifr,
+get_devaddr(struct ifreq *ifr,
int *dev_index, uint8_t *port, uint16_t *pkey)
{
struct ibv_device_attr dev_attr;
@@ -199,7 +199,7 @@ int gen_addr_ip(FILE *f)
if (ifr[i].ifr_hwaddr.sa_family != ARPHRD_INFINIBAND)
continue;
- ret = get_devaddr(s, ifr[i], dev_index, port, pkey);
+ ret = get_devaddr(ifr[i], dev_index, port, pkey);
if (ret)
continue;
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 04/14] ibacm: move sysfs helper functions to acm_util 'module'
From: Ira Weiny ira.we...@intel.com
This is in preparation for these functions to be included ib both ibacm and
ib_acme build.
Also rename to more global appropriate name acm_if_*
Signed-off-by: Ira Weiny ira.we...@intel.com
---
Makefile.am|7 ++-
linux/acme_linux.c | 87 +---
src/acm_util.c | 112
src/acm_util.h | 39 ++
src/acme.c |1 +
5 files changed, 159 insertions(+), 87 deletions(-)
create mode 100644 src/acm_util.c
create mode 100644 src/acm_util.h
diff --git a/Makefile.am b/Makefile.am
index 57021e0..539ef83 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,11 +1,12 @@
-AM_CPPFLAGS = -I$(srcdir)/include -I$(srcdir)/linux
+AM_CPPFLAGS = -I$(srcdir)/include -I$(srcdir)/linux -I$(srcdir)/src
AM_CFLAGS = -g -Wall -D_GNU_SOURCE -DSYSCONFDIR=\$(sysconfdir)\
-DBINDIR=\$(bindir)\ -DRDMADIR=\@rdmadir@\
bin_PROGRAMS = util/ib_acme
sbin_PROGRAMS = svc/ibacm
svc_ibacm_SOURCES = src/acm.c
-util_ib_acme_SOURCES = src/acme.c linux/acme_linux.c src/libacm.c
linux/libacm_linux.c src/parse.c
+util_ib_acme_SOURCES = src/acme.c linux/acme_linux.c src/libacm.c
linux/libacm_linux.c \
+ src/parse.c src/acm_util.c
svc_ibacm_CFLAGS = $(AM_CFLAGS)
util_ib_acme_CFLAGS = $(AM_CFLAGS)
@@ -18,7 +19,7 @@ man_MANS = \
man/ibacm.1 \
man/ibacm.7
-EXTRA_DIST = src/acm_mad.h src/libacm.h ibacm.init.in \
+EXTRA_DIST = src/acm_util.h src/acm_mad.h src/libacm.h ibacm.init.in \
linux/osd.h linux/dlist.h ibacm.spec.in $(man_MANS)
ibacm_hosts.data
install-exec-hook:
diff --git a/linux/acme_linux.c b/linux/acme_linux.c
index 0bd1f75..b246a9c 100644
--- a/linux/acme_linux.c
+++ b/linux/acme_linux.c
@@ -34,99 +34,18 @@
#include arpa/inet.h
#include sys/ioctl.h
#include net/if.h
-#include net/if_arp.h
#include netinet/in.h
#include sys/socket.h
#include sys/types.h
#include errno.h
#include infiniband/verbs.h
+#include acm_util.h
extern struct ibv_context **verbs;
extern int dev_cnt;
extern int verbose;
-
-static int
-get_pkey(char *ifname, uint16_t *pkey)
-{
- char buf[128], *end;
- FILE *f;
- int ret;
-
- snprintf(buf, sizeof buf, //sys//class//net//%s//pkey, ifname);
- f = fopen(buf, r);
- if (!f) {
- printf(failed to open %s\n, buf);
- return -1;
- }
-
- if (fgets(buf, sizeof buf, f)) {
- *pkey = strtol(buf, end, 16);
- ret = 0;
- } else {
- printf(failed to read pkey\n);
- ret = -1;
- }
-
- fclose(f);
- return ret;
-}
-
-static int
-get_sgid(char *ifname, union ibv_gid *sgid)
-{
- char buf[128], *end;
- FILE *f;
- int i, p, ret;
-
- snprintf(buf, sizeof buf, //sys//class//net//%s//address, ifname);
- f = fopen(buf, r);
- if (!f) {
- printf(failed to open %s\n, buf);
- return -1;
- }
-
- if (fgets(buf, sizeof buf, f)) {
- for (i = 0, p = 12; i 16; i++, p += 3) {
- buf[p + 2] = '\0';
- sgid-raw[i] = (uint8_t) strtol(buf + p, end, 16);
- }
- ret = 0;
- } else {
- printf(failed to read sgid\n);
- ret = -1;
- }
-
- fclose(f);
- return ret;
-}
-
-static int acm_if_is_ib(char *ifname)
-{
- unsigned type;
- char buf[128];
- FILE *f;
- int ret;
-
- snprintf(buf, sizeof buf, //sys//class//net//%s//type, ifname);
- f = fopen(buf, r);
- if (!f) {
- printf(failed to open %s\n, buf);
- return 0;
- }
-
- if (fgets(buf, sizeof buf, f)) {
- type = strtol(buf, NULL, 0);
- ret = (type == ARPHRD_INFINIBAND);
- } else {
- ret = 0;
- }
-
- fclose(f);
- return ret;
-}
-
static int
get_devaddr(char *ifname, int *dev_index, uint8_t *port, uint16_t *pkey)
{
@@ -135,13 +54,13 @@ get_devaddr(char *ifname, int *dev_index, uint8_t *port,
uint16_t *pkey)
union ibv_gid sgid, gid;
int ret, i;
- ret = get_sgid(ifname, sgid);
+ ret = acm_if_get_sgid(ifname, sgid);
if (ret) {
printf(unable to get sgid\n);
return ret;
}
- ret = get_pkey(ifname, pkey);
+ ret = acm_if_get_pkey(ifname, pkey);
if (ret) {
printf(unable to get pkey\n);
return ret;
diff --git a/src/acm_util.c b/src/acm_util.c
new file mode 100644
index 000..519735a
--- /dev/null
+++ b/src/acm_util.c
@@ -0,0 +1,112 @@
+/*
+ * Copyright (c) 2014 Intel Corporation. All rights reserved.
+ *
+ * This software is available to you under the OpenFabrics.org BSD license
+ * below:
+ *
+ * Redistribution and use in source and binary
[PATCH 0/14 V2] Add Dynamic IP processing to ibacm.
This is a new version of the entire patch series. In addition to fixing comments from Bart and Sean I removed the additional thread which removed the need for additional locking. [PATCH 01/14] ibacm: get_devaddr remove unused variable from signature [PATCH 02/14] ibacm: pass interface name rather than 'struct ifreq' to get_devaddr, get_sgid, and get_pkey [PATCH 03/14] ibacm: use sysfs (in acm_if_is_ib) rather than ioctl(... SIOCGIFHWADDR ...) to read interface type [PATCH 04/14] ibacm: move sysfs helper functions to acm_util 'module' [PATCH 05/14] ibacm: Move calling of helper functions out of get_devaddr [PATCH 06/14] ibacm: move acm_if_iter_sys to acm_util.c [PATCH 07/14] ibacm: convert logging in acm_util.c to acm_log [PATCH 08/14] ibacm: add separate acm_ep_insert_addr function [PATCH 09/14] ibacm: read system IP's into endpoints at startup [PATCH 10/14] ibacm: Add Netlink socket to monitor IP address changes [PATCH 11/14] ibacm: add/remove addr's in EP's when added/removed from the system. [PATCH 12/14] ibacm: fix handling of aliased IPoIB devices [PATCH 13/14] ibacm: ib_acme remove IP addresses from ibacm_addr.cfg file generation [PATCH 14/14] ibacm: by default remove processing of IP's from ibacm_addr.cfg -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 12/14] ibacm: fix handling of aliased IPoIB devices
From: Ira Weiny ira.we...@intel.com ibX:Y devices don't have sysfs files directly. They use the base interface of ibX. The ioctl calls however include the full aliased name. Netlink does not have this problem as the interface name is reported as it appears in sysfs. Signed-off-by: Ira Weiny ira.we...@intel.com --- src/acm_util.c |7 +++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/src/acm_util.c b/src/acm_util.c index c3c5b69..da6d058 100644 --- a/src/acm_util.c +++ b/src/acm_util.c @@ -134,6 +134,7 @@ int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx) uint8_t addr_type; uint8_t addr[ACM_MAX_ADDRESS]; size_t addr_len; + char *alias_sep; s = socket(AF_INET6, SOCK_DGRAM, 0); if (!s) @@ -177,6 +178,12 @@ int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx) continue; } + acm_log(2, %s\n, ifr[i].ifr_name); + + alias_sep = strchr(ifr[i].ifr_name, ':'); + if (alias_sep) + *alias_sep = '\0'; + if (!acm_if_is_ib(ifr[i].ifr_name)) continue; -- 1.7.1 -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 06/14] ibacm: move acm_if_iter_sys to acm_util.c
From: Ira Weiny ira.we...@intel.com
Signed-off-by: Ira Weiny ira.we...@intel.com
---
linux/acme_linux.c | 80 ---
src/acm_util.c | 81
2 files changed, 81 insertions(+), 80 deletions(-)
diff --git a/linux/acme_linux.c b/linux/acme_linux.c
index 187507d..33fafd2 100644
--- a/linux/acme_linux.c
+++ b/linux/acme_linux.c
@@ -30,14 +30,6 @@
#include stdio.h
#include stdlib.h
#include string.h
-#include unistd.h
-#include arpa/inet.h
-#include sys/ioctl.h
-#include net/if.h
-#include netinet/in.h
-#include sys/socket.h
-#include sys/types.h
-#include errno.h
#include infiniband/verbs.h
#include acm_util.h
@@ -101,75 +93,3 @@ int gen_addr_ip(FILE *f)
{
return acm_if_iter_sys(iter_cb, (void *)f);
}
-
-int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx)
-{
- struct ifconf *ifc;
- struct ifreq *ifr;
- char ip[INET6_ADDRSTRLEN];
- int s, ret, i, len;
- uint16_t pkey;
- union ibv_gid sgid;
-
- s = socket(AF_INET6, SOCK_DGRAM, 0);
- if (!s)
- return -1;
-
- len = sizeof(*ifc) + sizeof(*ifr) * 64;
- ifc = malloc(len);
- if (!ifc) {
- ret = -1;
- goto out1;
- }
-
- memset(ifc, 0, len);
- ifc-ifc_len = len;
- ifc-ifc_req = (struct ifreq *) (ifc + 1);
-
- ret = ioctl(s, SIOCGIFCONF, ifc);
- if (ret 0) {
- printf(ioctl ifconf error %d\n, ret);
- goto out2;
- }
-
- ifr = ifc-ifc_req;
- for (i = 0; i ifc-ifc_len / sizeof(struct ifreq); i++) {
- switch (ifr[i].ifr_addr.sa_family) {
- case AF_INET:
- inet_ntop(ifr[i].ifr_addr.sa_family,
- ((struct sockaddr_in *)
ifr[i].ifr_addr)-sin_addr, ip, sizeof ip);
- break;
- case AF_INET6:
- inet_ntop(ifr[i].ifr_addr.sa_family,
- ((struct sockaddr_in6 *)
ifr[i].ifr_addr)-sin6_addr, ip, sizeof ip);
- break;
- default:
- continue;
- }
-
- if (!acm_if_is_ib(ifr[i].ifr_name))
- continue;
-
- ret = acm_if_get_sgid(ifr[i].ifr_name, sgid);
- if (ret) {
- printf(unable to get sgid\n);
- continue;
- }
-
- ret = acm_if_get_pkey(ifr[i].ifr_name, pkey);
- if (ret) {
- printf(unable to get pkey\n);
- continue;
- }
-
- cb(ifr[i].ifr_name, sgid, pkey, 0, NULL, 0, ip, ctx);
- }
- ret = 0;
-
-out2:
- free(ifc);
-out1:
- close(s);
- return ret;
-
-}
diff --git a/src/acm_util.c b/src/acm_util.c
index 519735a..922eceb 100644
--- a/src/acm_util.c
+++ b/src/acm_util.c
@@ -30,6 +30,15 @@
#include stdio.h
#include stdlib.h
#include net/if_arp.h
+#include string.h
+#include unistd.h
+#include arpa/inet.h
+#include sys/ioctl.h
+#include net/if.h
+#include netinet/in.h
+#include sys/socket.h
+#include sys/types.h
+#include errno.h
#include acm_util.h
@@ -110,3 +119,75 @@ int acm_if_get_sgid(char *ifname, union ibv_gid *sgid)
fclose(f);
return ret;
}
+
+int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx)
+{
+ struct ifconf *ifc;
+ struct ifreq *ifr;
+ char ip[INET6_ADDRSTRLEN];
+ int s, ret, i, len;
+ uint16_t pkey;
+ union ibv_gid sgid;
+
+ s = socket(AF_INET6, SOCK_DGRAM, 0);
+ if (!s)
+ return -1;
+
+ len = sizeof(*ifc) + sizeof(*ifr) * 64;
+ ifc = malloc(len);
+ if (!ifc) {
+ ret = -1;
+ goto out1;
+ }
+
+ memset(ifc, 0, len);
+ ifc-ifc_len = len;
+ ifc-ifc_req = (struct ifreq *) (ifc + 1);
+
+ ret = ioctl(s, SIOCGIFCONF, ifc);
+ if (ret 0) {
+ printf(ioctl ifconf error %d\n, ret);
+ goto out2;
+ }
+
+ ifr = ifc-ifc_req;
+ for (i = 0; i ifc-ifc_len / sizeof(struct ifreq); i++) {
+ switch (ifr[i].ifr_addr.sa_family) {
+ case AF_INET:
+ inet_ntop(ifr[i].ifr_addr.sa_family,
+ ((struct sockaddr_in *)
ifr[i].ifr_addr)-sin_addr, ip, sizeof ip);
+ break;
+ case AF_INET6:
+ inet_ntop(ifr[i].ifr_addr.sa_family,
+ ((struct sockaddr_in6 *)
ifr[i].ifr_addr)-sin6_addr, ip, sizeof ip);
+ break;
+ default:
+ continue;
+ }
+
+ if (!acm_if_is_ib(ifr[i].ifr_name))
+ continue;
+
+ ret = acm_if_get_sgid(ifr[i].ifr_name, sgid);
+
[PATCH 10/14] ibacm: Add Netlink socket to monitor IP address changes
From: Ira Weiny ira.we...@intel.com
Currently only reports events to the log.
Signed-off-by: Ira Weiny ira.we...@intel.com
---
src/acm.c | 99 +
1 files changed, 99 insertions(+), 0 deletions(-)
diff --git a/src/acm.c b/src/acm.c
index af1c66e..45f3ed9 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -49,6 +49,9 @@
#include net/if.h
#include sys/ioctl.h
#include net/if_arp.h
+#include netinet/in.h
+#include linux/netlink.h
+#include linux/rtnetlink.h
#include acm_mad.h
#include acm_util.h
@@ -214,6 +217,7 @@ static event_t timeout_event;
static atomic_t wait_cnt;
static SOCKET listen_socket;
+static SOCKET ip_mon_socket;
static struct acm_client client[FD_SETSIZE - 1];
static FILE *flog;
@@ -2389,6 +2393,92 @@ out:
acm_disconnect_client(client);
}
+static int acm_ipnl_create(void)
+{
+ struct sockaddr_nl addr;
+
+ if ((ip_mon_socket = socket(PF_NETLINK, SOCK_RAW | SOCK_NONBLOCK,
NETLINK_ROUTE)) == -1) {
+ acm_log(0, Failed to open NETLINK_ROUTE socket);
+ return EIO;
+ }
+
+ memset(addr, 0, sizeof(addr));
+ addr.nl_family = AF_NETLINK;
+ addr.nl_groups = RTMGRP_LINK | RTMGRP_IPV4_IFADDR;
+
+ if (bind(ip_mon_socket, (struct sockaddr *)addr, sizeof(addr)) == -1) {
+ acm_log(0, Failed to bind NETLINK_ROUTE socket);
+ return EIO;
+ }
+
+ return 0;
+}
+
+#define NL_MSG_BUF_SIZE 4096
+static void acm_ipnl_handler(void)
+{
+ int len;
+ char buffer[NL_MSG_BUF_SIZE];
+ struct nlmsghdr *nlh;
+ char name[IFNAMSIZ];
+ char ip_str[INET6_ADDRSTRLEN];
+
+ while ((len = recv(ip_mon_socket, buffer, NL_MSG_BUF_SIZE, 0)) 0) {
+ nlh = (struct nlmsghdr *)buffer;
+ while ((NLMSG_OK(nlh, len)) (nlh-nlmsg_type != NLMSG_DONE))
{
+ struct ifaddrmsg *ifa = (struct ifaddrmsg *)
NLMSG_DATA(nlh);
+ struct ifinfomsg *ifi = (struct ifinfomsg *)
NLMSG_DATA(nlh);
+ struct rtattr *rth = IFA_RTA(ifa);
+ int rtl = IFA_PAYLOAD(nlh);
+
+ switch (nlh-nlmsg_type) {
+ case RTM_NEWADDR:
+ {
+ if_indextoname(ifa-ifa_index, name);
+ while (rtl RTA_OK(rth, rtl)) {
+ if (rth-rta_type == IFA_LOCAL) {
+ acm_log(0, Address added %s :
%s\n,
+ name,
inet_ntop(ifa-ifa_family, RTA_DATA(rth),
+ ip_str,
sizeof(ip_str)));
+ }
+ rth = RTA_NEXT(rth, rtl);
+ }
+ break;
+ }
+ case RTM_DELADDR:
+ {
+ if_indextoname(ifa-ifa_index, name);
+ while (rtl RTA_OK(rth, rtl)) {
+ if (rth-rta_type == IFA_LOCAL) {
+ acm_log(0, Address deleted %s
: %s\n,
+ name,
inet_ntop(ifa-ifa_family, RTA_DATA(rth),
+ ip_str,
sizeof(ip_str)));
+ }
+ rth = RTA_NEXT(rth, rtl);
+ }
+ break;
+ }
+ case RTM_NEWLINK:
+ {
+ acm_log(2, Link added : %s\n,
if_indextoname(ifi-ifi_index, name));
+ break;
+ }
+ case RTM_DELLINK:
+ {
+ acm_log(2, Link removed : %s\n,
if_indextoname(ifi-ifi_index, name));
+ break;
+ }
+ default:
+ acm_log(2, unknown netlink message\n);
+ break;
+ }
+ nlh = NLMSG_NEXT(nlh, len);
+ }
+ }
+ if (len 0 errno == ENOBUFS)
+ acm_log(0, ENOBUFS returned from netlink... Resyncing all
IP's\n);
+}
+
static void acm_server(void)
{
fd_set readfds;
@@ -2407,6 +2497,9 @@ static void acm_server(void)
FD_ZERO(readfds);
FD_SET(listen_socket, readfds);
+ n = max(n, (int) ip_mon_socket);
+ FD_SET(ip_mon_socket, readfds);
+
for (i = 0; i FD_SETSIZE - 1; i++) {
if (client[i].sock !=
[PATCH 07/14] ibacm: convert logging in acm_util.c to acm_log
From: Ira Weiny ira.we...@intel.com
ib_acme build defines ACME_PRINTS which overrides acm_log to printf
Signed-off-by: Ira Weiny ira.we...@intel.com
---
Makefile.am|2 +-
src/acm.c |6 ++
src/acm_util.c | 21 +
src/acm_util.h | 12
4 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index 539ef83..703030d 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -8,7 +8,7 @@ svc_ibacm_SOURCES = src/acm.c
util_ib_acme_SOURCES = src/acme.c linux/acme_linux.c src/libacm.c
linux/libacm_linux.c \
src/parse.c src/acm_util.c
svc_ibacm_CFLAGS = $(AM_CFLAGS)
-util_ib_acme_CFLAGS = $(AM_CFLAGS)
+util_ib_acme_CFLAGS = $(AM_CFLAGS) -DACME_PRINTS
ibacmincludedir = $(includedir)/infiniband
diff --git a/src/acm.c b/src/acm.c
index 82f9431..5dc2684 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -47,6 +47,7 @@
#include dlist.h
#include search.h
#include acm_mad.h
+#include acm_util.h
#define src_out data[0]
@@ -245,10 +246,7 @@ static uint8_t min_rate = IBV_RATE_10_GBPS;
static enum acm_route_preload route_preload;
static enum acm_addr_preload addr_preload;
-#define acm_log(level, format, ...) \
- acm_write(level, %s: format, __func__, ## __VA_ARGS__)
-
-static void acm_write(int level, const char *format, ...)
+void acm_write(int level, const char *format, ...)
{
va_list args;
struct timeval tv;
diff --git a/src/acm_util.c b/src/acm_util.c
index 922eceb..7a84ac7 100644
--- a/src/acm_util.c
+++ b/src/acm_util.c
@@ -52,7 +52,7 @@ int acm_if_is_ib(char *ifname)
snprintf(buf, sizeof buf, //sys//class//net//%s//type, ifname);
f = fopen(buf, r);
if (!f) {
- printf(failed to open %s\n, buf);
+ acm_log(0, failed to open %s\n, buf);
return 0;
}
@@ -60,6 +60,7 @@ int acm_if_is_ib(char *ifname)
type = strtol(buf, NULL, 0);
ret = (type == ARPHRD_INFINIBAND);
} else {
+ acm_log(0, failed to read interface type\n);
ret = 0;
}
@@ -76,7 +77,7 @@ int acm_if_get_pkey(char *ifname, uint16_t *pkey)
snprintf(buf, sizeof buf, //sys//class//net//%s//pkey, ifname);
f = fopen(buf, r);
if (!f) {
- printf(failed to open %s\n, buf);
+ acm_log(0, failed to open %s\n, buf);
return -1;
}
@@ -84,7 +85,7 @@ int acm_if_get_pkey(char *ifname, uint16_t *pkey)
*pkey = strtol(buf, end, 16);
ret = 0;
} else {
- printf(failed to read pkey\n);
+ acm_log(0, failed to read pkey\n);
ret = -1;
}
@@ -101,7 +102,7 @@ int acm_if_get_sgid(char *ifname, union ibv_gid *sgid)
snprintf(buf, sizeof buf, //sys//class//net//%s//address, ifname);
f = fopen(buf, r);
if (!f) {
- printf(failed to open %s\n, buf);
+ acm_log(0, failed to open %s\n, buf);
return -1;
}
@@ -112,7 +113,7 @@ int acm_if_get_sgid(char *ifname, union ibv_gid *sgid)
}
ret = 0;
} else {
- printf(failed to read sgid\n);
+ acm_log(0, failed to read sgid\n);
ret = -1;
}
@@ -146,7 +147,7 @@ int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx)
ret = ioctl(s, SIOCGIFCONF, ifc);
if (ret 0) {
- printf(ioctl ifconf error %d\n, ret);
+ acm_log(0, ioctl ifconf error %d\n, ret);
goto out2;
}
@@ -169,16 +170,12 @@ int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx)
continue;
ret = acm_if_get_sgid(ifr[i].ifr_name, sgid);
- if (ret) {
- printf(unable to get sgid\n);
+ if (ret)
continue;
- }
ret = acm_if_get_pkey(ifr[i].ifr_name, pkey);
- if (ret) {
- printf(unable to get pkey\n);
+ if (ret)
continue;
- }
cb(ifr[i].ifr_name, sgid, pkey, 0, NULL, 0, ip, ctx);
}
diff --git a/src/acm_util.h b/src/acm_util.h
index ea86133..4f111b6 100644
--- a/src/acm_util.h
+++ b/src/acm_util.h
@@ -32,6 +32,18 @@
#include infiniband/verbs.h
+#ifdef ACME_PRINTS
+
+#define acm_log(level, format, ...) \
+ printf(format, ## __VA_ARGS__)
+
+#else /* !ACME_PRINTS */
+#define acm_log(level, format, ...) \
+ acm_write(level, %s: format, __func__, ## __VA_ARGS__)
+
+void acm_write(int level, const char *format, ...);
+#endif /* ACME_PRINTS */
+
int acm_if_is_ib(char *ifname);
int acm_if_get_pkey(char *ifname, uint16_t *pkey);
int acm_if_get_sgid(char *ifname, union ibv_gid *sgid);
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
[PATCH 09/14] ibacm: read system IP's into endpoints at startup
From: Ira Weiny ira.we...@intel.com
ibacm_addr.cfg is read after system is read which can add endpoints which are
not active at start up.
ibacm_addr.cfg can still specify names for end points
Signed-off-by: Ira Weiny ira.we...@intel.com
---
Makefile.am|2 +-
src/acm.c | 66
src/acm_util.c | 13 ++-
3 files changed, 79 insertions(+), 2 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index 703030d..cab1ca1 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -4,7 +4,7 @@ AM_CFLAGS = -g -Wall -D_GNU_SOURCE
-DSYSCONFDIR=\$(sysconfdir)\ -DBINDIR=\$(b
bin_PROGRAMS = util/ib_acme
sbin_PROGRAMS = svc/ibacm
-svc_ibacm_SOURCES = src/acm.c
+svc_ibacm_SOURCES = src/acm.c src/acm_util.c
util_ib_acme_SOURCES = src/acme.c linux/acme_linux.c src/libacm.c
linux/libacm_linux.c \
src/parse.c src/acm_util.c
svc_ibacm_CFLAGS = $(AM_CFLAGS)
diff --git a/src/acm.c b/src/acm.c
index 6bbc1e5..af1c66e 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -46,6 +46,9 @@
#include infiniband/verbs.h
#include dlist.h
#include search.h
+#include net/if.h
+#include sys/ioctl.h
+#include net/if_arp.h
#include acm_mad.h
#include acm_util.h
@@ -2922,6 +2925,67 @@ acm_ep_insert_addr(struct acm_ep *ep, uint8_t *addr,
size_t addr_len, uint8_t ad
return ret;
}
+static struct acm_device *
+acm_get_device_from_gid(union ibv_gid *sgid, uint8_t *port)
+{
+ DLIST_ENTRY *dev_entry;
+ struct acm_device *dev;
+ struct ibv_device_attr dev_attr;
+ struct ibv_port_attr port_attr;
+ union ibv_gid gid;
+ int ret, i;
+
+ for (dev_entry = dev_list.Next; dev_entry != dev_list;
+dev_entry = dev_entry-Next) {
+
+ dev = container_of(dev_entry, struct acm_device, entry);
+
+ ret = ibv_query_device(dev-verbs, dev_attr);
+ if (ret)
+ continue;
+
+ for (*port = 1; *port = dev_attr.phys_port_cnt; (*port)++) {
+ ret = ibv_query_port(dev-verbs, *port, port_attr);
+ if (ret)
+ continue;
+
+ for (i = 0; i port_attr.gid_tbl_len; i++) {
+ ret = ibv_query_gid(dev-verbs, *port, i, gid);
+ if (ret || !gid.global.interface_id)
+ break;
+
+ if (!memcmp(sgid-raw, gid.raw, sizeof gid))
+ return dev;
+ }
+ }
+ }
+ return NULL;
+}
+
+static void acm_ep_ip_iter_cb(char *ifname, union ibv_gid *gid, uint16_t pkey,
+ uint8_t addr_type, uint8_t *addr, size_t addr_len,
+ char *addr_name, void *ctx)
+{
+ uint8_t port_num;
+ struct acm_device *dev;
+ struct acm_ep *ep = (struct acm_ep *)ctx;
+
+ dev = acm_get_device_from_gid(gid, port_num);
+ if (dev ep-port-dev == dev
+ep-port-port_num == port_num ep-pkey == pkey) {
+ if (!acm_ep_insert_addr(ep, addr, addr_len, addr_type)) {
+ acm_log(0, Added %s %s %d 0x%x from %s\n, addr_name,
+ dev-verbs-device-name, port_num, pkey,
+ ifname);
+ }
+ }
+}
+
+static int acm_get_system_ips(struct acm_ep *ep)
+{
+ return acm_if_iter_sys(acm_ep_ip_iter_cb, (void *)ep);
+}
+
static int acm_assign_ep_names(struct acm_ep *ep)
{
FILE *faddr;
@@ -2938,6 +3002,8 @@ static int acm_assign_ep_names(struct acm_ep *ep)
acm_log(1, device %s, port %d, pkey 0x%x\n,
dev_name, ep-port-port_num, ep-pkey);
+ acm_get_system_ips(ep);
+
if (!(faddr = acm_open_addr_file())) {
acm_log(0, ERROR - address file not found\n);
return -1;
diff --git a/src/acm_util.c b/src/acm_util.c
index 7a84ac7..c3c5b69 100644
--- a/src/acm_util.c
+++ b/src/acm_util.c
@@ -40,6 +40,8 @@
#include sys/types.h
#include errno.h
+#include infiniband/acm.h
+#include acm_mad.h
#include acm_util.h
int acm_if_is_ib(char *ifname)
@@ -129,6 +131,9 @@ int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx)
int s, ret, i, len;
uint16_t pkey;
union ibv_gid sgid;
+ uint8_t addr_type;
+ uint8_t addr[ACM_MAX_ADDRESS];
+ size_t addr_len;
s = socket(AF_INET6, SOCK_DGRAM, 0);
if (!s)
@@ -155,10 +160,16 @@ int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx)
for (i = 0; i ifc-ifc_len / sizeof(struct ifreq); i++) {
switch (ifr[i].ifr_addr.sa_family) {
case AF_INET:
+ addr_type = ACM_ADDRESS_IP;
+ memcpy(addr, ((struct sockaddr_in *)
ifr[i].ifr_addr)-sin_addr, sizeof addr);
+ addr_len = 4;
[PATCH 11/14] ibacm: add/remove addr's in EP's when added/removed from the system.
From: Ira Weiny ira.we...@intel.com
This builds on the previous patch by reacting to the IP address changes
monitored there.
Signed-off-by: Ira Weiny ira.we...@intel.com
---
src/acm.c | 188 +++-
1 files changed, 184 insertions(+), 4 deletions(-)
diff --git a/src/acm.c b/src/acm.c
index 45f3ed9..a6e24a9 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -796,6 +796,27 @@ out:
lock_release(ep-lock);
}
+static void acm_mark_addr_invalid(struct acm_ep *ep,
+ struct acm_ep_addr_data *data)
+{
+ int i;
+
+ lock_acquire(ep-lock);
+ for (i = 0; i MAX_EP_ADDR; i++) {
+ if (ep-addr_type[i] != data-type)
+ continue;
+
+ if ((data-type == ACM_ADDRESS_NAME
+ !strnicmp((char *) ep-addr[i].name,
+ (char *) data-info.addr, ACM_MAX_ADDRESS)) ||
+!memcmp(ep-addr[i].addr, data-info.addr,
ACM_MAX_ADDRESS)) {
+ ep-addr_type[i] = ACM_ADDRESS_INVALID;
+ break;
+ }
+ }
+ lock_release(ep-lock);
+}
+
static int acm_addr_index(struct acm_ep *ep, uint8_t *addr, uint8_t addr_type)
{
int i;
@@ -2393,6 +2414,85 @@ out:
acm_disconnect_client(client);
}
+static struct acm_device *
+acm_get_device_from_gid(union ibv_gid *sgid, uint8_t *port);
+static struct acm_ep *acm_find_ep(struct acm_port *port, uint16_t pkey);
+static int
+acm_ep_insert_addr(struct acm_ep *ep, uint8_t *addr, size_t addr_len, uint8_t
addr_type);
+
+static int acm_nl_to_addr_data(struct acm_ep_addr_data *ad,
+ int af_family, uint8_t *addr, size_t addr_len)
+{
+ if (addr_len ACM_MAX_ADDRESS)
+ return EINVAL;
+
+ /* find the ep associated with this address if any */
+ switch (af_family) {
+ case AF_INET:
+ ad-type = ACM_ADDRESS_IP;
+ break;
+ case AF_INET6:
+ ad-type = ACM_ADDRESS_IP6;
+ break;
+ default:
+ return EINVAL;
+ }
+ memcpy(ad-info.addr, addr, addr_len);
+ return 0;
+}
+
+static void acm_add_ep_ip(struct acm_ep_addr_data *data, char *ifname)
+{
+ struct acm_ep *ep;
+ struct acm_device *dev;
+ uint8_t port_num;
+ uint16_t pkey;
+ union ibv_gid sgid;
+
+ ep = acm_get_ep(data);
+ if (ep) {
+ acm_format_name(1, log_data, sizeof log_data,
+ data-type, data-info.addr, sizeof
data-info.addr);
+ acm_log(1, Address '%s' already available\n, log_data);
+ return;
+ }
+
+ if (acm_if_get_sgid(ifname, sgid))
+ return;
+
+ dev = acm_get_device_from_gid(sgid, port_num);
+ if (!dev)
+ return;
+
+ if (acm_if_get_pkey(ifname, pkey))
+ return;
+
+ acm_format_name(0, log_data, sizeof log_data,
+ data-type, data-info.addr, sizeof data-info.addr);
+ acm_log(0, %s\n, log_data);
+
+ ep = acm_find_ep(dev-port[port_num-1], pkey);
+ if (ep) {
+ if (acm_ep_insert_addr(ep, data-info.addr, sizeof
data-info.addr, data-type))
+ acm_log(0, Failed to add '%s' to EP\n, log_data);
+ } else {
+ acm_log(0, Failed to add '%s' no EP for pkey\n, log_data);
+ }
+}
+
+static void acm_rm_ep_ip(struct acm_ep_addr_data *data)
+{
+ struct acm_ep *ep;
+
+ ep = acm_get_ep(data);
+ if (ep) {
+ acm_format_name(0, log_data, sizeof log_data,
+ data-type, data-info.addr, sizeof
data-info.addr);
+ acm_log(0, %s\n, log_data);
+ acm_mark_addr_invalid(ep, data);
+ }
+}
+
static int acm_ipnl_create(void)
{
struct sockaddr_nl addr;
@@ -2414,6 +2514,72 @@ static int acm_ipnl_create(void)
return 0;
}
+static void acm_ip_iter_cb(char *ifname, union ibv_gid *gid, uint16_t pkey,
+ uint8_t addr_type, uint8_t *addr, size_t addr_len,
+ char *addr_name, void *ctx)
+{
+ int ret = EINVAL;
+ struct acm_device *dev = NULL;
+ struct acm_ep *ep = NULL;
+ uint8_t port_num;
+ char gid_str[INET6_ADDRSTRLEN];
+
+ dev = acm_get_device_from_gid(gid, port_num);
+ if (dev)
+ ep = acm_find_ep(dev-port[port_num-1], pkey);
+
+ if (ep)
+ ret = acm_ep_insert_addr(ep, addr, addr_len, addr_type);
+
+ if (ret) {
+ acm_format_name(2, log_data, sizeof log_data,
+ addr_type, addr, addr_len);
+ inet_ntop(AF_INET6, gid-raw, gid_str, sizeof(gid_str));
+ acm_log(0, Failed to add '%s' (gid %s; pkey 0x%x)\n,
+ log_data, gid_str, pkey);
+ }
+}
+
+/* Netlink
[PATCH 08/14] ibacm: add separate acm_ep_insert_addr function
From: Ira Weiny ira.we...@intel.com
This is in preparation for netlink support which will do this dynamically.
Signed-off-by: Ira Weiny ira.we...@intel.com
---
src/acm.c | 73
1 files changed, 58 insertions(+), 15 deletions(-)
diff --git a/src/acm.c b/src/acm.c
index 5dc2684..6bbc1e5 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -2875,6 +2875,53 @@ static void acm_parse_hosts_file(struct acm_ep *ep)
fclose(f);
}
+static int
+acm_ep_insert_addr(struct acm_ep *ep, uint8_t *addr, size_t addr_len, uint8_t
addr_type)
+{
+ int i;
+ int ret = ENOMEM;
+ uint8_t tmp[ACM_MAX_ADDRESS];
+ char name_str[INET6_ADDRSTRLEN];
+
+ if (addr_len ACM_MAX_ADDRESS)
+ return EINVAL;
+
+ memset(tmp, 0, sizeof tmp);
+ memcpy(tmp, addr, addr_len);
+
+ lock_acquire(ep-lock);
+ if (acm_addr_index(ep, tmp, addr_type) 0) {
+ for (i = 0; i MAX_EP_ADDR; i++) {
+ if (ep-addr_type[i] == ACM_ADDRESS_INVALID) {
+
+ ep-addr_type[i] = addr_type;
+ memcpy(ep-addr[i].addr, tmp, ACM_MAX_ADDRESS);
+
+ switch (addr_type) {
+ case ACM_ADDRESS_IP:
+ inet_ntop(AF_INET, addr, name_str,
sizeof name_str);
+ strncpy(ep-name[i], name_str,
ACM_MAX_ADDRESS);
+ break;
+ case ACM_ADDRESS_IP6:
+ inet_ntop(AF_INET6, addr, name_str,
sizeof name_str);
+ strncpy(ep-name[i], name_str,
ACM_MAX_ADDRESS);
+ break;
+ case ACM_ADDRESS_NAME:
+ strncpy(ep-name[i], (const char
*)addr, ACM_MAX_ADDRESS);
+ break;
+ }
+
+ ret = 0;
+ break;
+ }
+ }
+ } else {
+ ret = 0;
+ }
+ lock_release(ep-lock);
+ return ret;
+}
+
static int acm_assign_ep_names(struct acm_ep *ep)
{
FILE *faddr;
@@ -2883,8 +2930,9 @@ static int acm_assign_ep_names(struct acm_ep *ep)
char dev[32], addr[INET6_ADDRSTRLEN], pkey_str[8];
uint16_t pkey;
uint8_t type;
- int port, index = 0;
+ int port, ret = 0;
struct in6_addr ip_addr;
+ size_t addr_len;
dev_name = ep-port-dev-verbs-device-name;
acm_log(1, device %s, port %d, pkey 0x%x\n,
@@ -2903,12 +2951,16 @@ static int acm_assign_ep_names(struct acm_ep *ep)
continue;
acm_log(2, %s, s);
- if (inet_pton(AF_INET, addr, ip_addr) 0)
+ if (inet_pton(AF_INET, addr, ip_addr) 0) {
type = ACM_ADDRESS_IP;
- else if (inet_pton(AF_INET6, addr, ip_addr) 0)
+ addr_len = 4;
+ } else if (inet_pton(AF_INET6, addr, ip_addr) 0) {
type = ACM_ADDRESS_IP6;
- else
+ addr_len = ACM_MAX_ADDRESS;
+ } else {
type = ACM_ADDRESS_NAME;
+ addr_len = strlen(addr);
+ }
if (stricmp(pkey_str, default)) {
if (sscanf(pkey_str, %hx, pkey) != 1) {
@@ -2922,17 +2974,8 @@ static int acm_assign_ep_names(struct acm_ep *ep)
if (!stricmp(dev_name, dev) (ep-port-port_num == (uint8_t)
port)
(ep-pkey == pkey)) {
- ep-addr_type[index] = type;
acm_log(1, assigning %s\n, addr);
- strncpy(ep-name[index], addr, ACM_MAX_ADDRESS);
- if (type == ACM_ADDRESS_IP)
- memcpy(ep-addr[index].addr, ip_addr, 4);
- else if (type == ACM_ADDRESS_IP6)
- memcpy(ep-addr[index].addr, ip_addr, sizeof
ip_addr);
- else
- strncpy((char *) ep-addr[index].addr, addr,
ACM_MAX_ADDRESS);
-
- if (++index == MAX_EP_ADDR) {
+ if ((ret = acm_ep_insert_addr(ep, (uint8_t *)ip_addr,
addr_len, type)) != 0) {
acm_log(1, maximum number of names assigned to
EP\n);
break;
}
@@ -2940,7 +2983,7 @@ static int acm_assign_ep_names(struct acm_ep *ep)
}
fclose(faddr);
- return !index;
+ return ret;
}
/*
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to
[PATCH 13/14] ibacm: ib_acme remove IP addresses from ibacm_addr.cfg file generation
From: Ira Weiny ira.we...@intel.com
Signed-off-by: Ira Weiny ira.we...@intel.com
---
Makefile.am|3 +-
linux/acme_linux.c | 95
man/ib_acme.1 |3 +-
src/acme.c | 12 ++-
4 files changed, 5 insertions(+), 108 deletions(-)
delete mode 100644 linux/acme_linux.c
diff --git a/Makefile.am b/Makefile.am
index cab1ca1..9070a5a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -5,8 +5,7 @@ AM_CFLAGS = -g -Wall -D_GNU_SOURCE
-DSYSCONFDIR=\$(sysconfdir)\ -DBINDIR=\$(b
bin_PROGRAMS = util/ib_acme
sbin_PROGRAMS = svc/ibacm
svc_ibacm_SOURCES = src/acm.c src/acm_util.c
-util_ib_acme_SOURCES = src/acme.c linux/acme_linux.c src/libacm.c
linux/libacm_linux.c \
- src/parse.c src/acm_util.c
+util_ib_acme_SOURCES = src/acme.c src/libacm.c linux/libacm_linux.c src/parse.c
svc_ibacm_CFLAGS = $(AM_CFLAGS)
util_ib_acme_CFLAGS = $(AM_CFLAGS) -DACME_PRINTS
diff --git a/linux/acme_linux.c b/linux/acme_linux.c
deleted file mode 100644
index 33fafd2..000
--- a/linux/acme_linux.c
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Copyright (c) 2009 Intel Corporation. All rights reserved.
- *
- * This software is available to you under the OpenIB.org BSD license
- * below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- *copyright notice, this list of conditions and the following
- *disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- *copyright notice, this list of conditions and the following
- *disclaimer in the documentation and/or other materials
- *provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include stdio.h
-#include stdlib.h
-#include string.h
-
-#include infiniband/verbs.h
-#include acm_util.h
-
-extern struct ibv_context **verbs;
-extern int dev_cnt;
-extern int verbose;
-
-static int
-get_devaddr(union ibv_gid *sgid, int *dev_index, uint8_t *port)
-{
- struct ibv_device_attr dev_attr;
- struct ibv_port_attr port_attr;
- union ibv_gid gid;
- int ret, i;
-
- for (*dev_index = 0; *dev_index dev_cnt; (*dev_index)++) {
- ret = ibv_query_device(verbs[*dev_index], dev_attr);
- if (ret)
- continue;
-
- for (*port = 1; *port = dev_attr.phys_port_cnt; (*port)++) {
- ret = ibv_query_port(verbs[*dev_index], *port,
port_attr);
- if (ret)
- continue;
-
- for (i = 0; i port_attr.gid_tbl_len; i++) {
- ret = ibv_query_gid(verbs[*dev_index], *port,
i, gid);
- if (ret || !gid.global.interface_id)
- break;
-
- if (!memcmp(sgid-raw, gid.raw, sizeof gid))
- return 0;
- }
- }
- }
- return -1;
-}
-
-static void iter_cb(char *ifname, union ibv_gid *gid, uint16_t pkey,
- uint8_t addr_type, uint8_t *addr, size_t addr_len,
- char *addr_name, void *ctx)
-{
- FILE *f = (FILE *)ctx;
- int ret;
- int dev_index;
- uint8_t port;
-
- ret = get_devaddr(gid, dev_index, port);
- if (ret) {
- printf(Failed to find verbs device for %s\n, ifname);
- return;
- }
-
- if (verbose)
- printf(%s %s %d 0x%x\n, addr_name,
verbs[dev_index]-device-name, port, pkey);
- fprintf(f, %s %s %d 0x%x\n, addr_name,
verbs[dev_index]-device-name, port, pkey);
-}
-
-int gen_addr_ip(FILE *f)
-{
- return acm_if_iter_sys(iter_cb, (void *)f);
-}
diff --git a/man/ib_acme.1 b/man/ib_acme.1
index b00b315..baa1dd6 100644
--- a/man/ib_acme.1
+++ b/man/ib_acme.1
@@ -55,8 +55,7 @@ performance of ACM cache lookups. Defaults to 1.
\-A [addr_file]
With this option, the ib_acme utility automatically generates the address
configuration file ibacm_addr.cfg. The generated file is
-constructed using the system host name and any IP addresses that are
-assigned to IPoIB device instances.
+constructed using the system host name.
.TP
\-O [opt_file]
With this option, the ib_acme utility automatically generates the option
[PATCH 14/14] ibacm: by default remove processing of IP's from ibacm_addr.cfg
From: Ira Weiny ira.we...@intel.com
support_ips_in_addr_cfg option in config file can turn this back on if wanted.
Signed-off-by: Ira Weiny ira.we...@intel.com
---
src/acm.c | 12
src/acme.c |4
2 files changed, 16 insertions(+), 0 deletions(-)
diff --git a/src/acm.c b/src/acm.c
index a6e24a9..02272e3 100644
--- a/src/acm.c
+++ b/src/acm.c
@@ -252,6 +252,7 @@ static uint8_t min_mtu = IBV_MTU_2048;
static uint8_t min_rate = IBV_RATE_10_GBPS;
static enum acm_route_preload route_preload;
static enum acm_addr_preload addr_preload;
+static int support_ips_in_addr_cfg = 0;
void acm_write(int level, const char *format, ...)
{
@@ -3294,9 +3295,17 @@ static int acm_assign_ep_names(struct acm_ep *ep)
acm_log(2, %s, s);
if (inet_pton(AF_INET, addr, ip_addr) 0) {
+ if (!support_ips_in_addr_cfg) {
+ acm_log(0, ERROR - IP's are not configured to
be read from ibacm_addr.cfg\n);
+ continue;
+ }
type = ACM_ADDRESS_IP;
addr_len = 4;
} else if (inet_pton(AF_INET6, addr, ip_addr) 0) {
+ if (!support_ips_in_addr_cfg) {
+ acm_log(0, ERROR - IP's are not configured to
be read from ibacm_addr.cfg\n);
+ continue;
+ }
type = ACM_ADDRESS_IP6;
addr_len = ACM_MAX_ADDRESS;
} else {
@@ -3873,6 +3882,8 @@ static void acm_set_options(void)
addr_preload = acm_convert_addr_preload(value);
else if (!stricmp(addr_data_file, opt))
strcpy(addr_data_file, value);
+ else if (!stricmp(support_ips_in_addr_cfg, opt))
+ support_ips_in_addr_cfg = atoi(value);
}
fclose(f);
@@ -3900,6 +3911,7 @@ static void acm_log_options(void)
acm_log(0, route data file %s\n, route_data_file);
acm_log(0, address preload %d\n, addr_preload);
acm_log(0, address data file %s\n, addr_data_file);
+ acm_log(0, support IP's in ibacm_addr.cfg %d\n,
support_ips_in_addr_cfg);
}
static FILE *acm_open_log(void)
diff --git a/src/acme.c b/src/acme.c
index d83e56f..c93ee1b 100644
--- a/src/acme.c
+++ b/src/acme.c
@@ -280,6 +280,10 @@ static void gen_opts_temp(FILE *f)
fprintf(f, # Default is ACM_CONF_DIR/ibacm_hosts.data\n);
fprintf(f, # addr_data_file /etc/rdma/ibacm_hosts.data\n);
fprintf(f, \n);
+ fprintf(f, # support_ips_in_addr_cfg:\n);
+ fprintf(f, # If 1 continue to read IP addresses from
ibacm_addr.cfg\n);
+ fprintf(f, # Default is 0 \no\\n);
+ fprintf(f, # support_ips_in_addr_cfg 0\n);
}
static int open_dir(void)
--
1.7.1
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
