Re: [PATCH libibverbs] XRC - Sample application issues
On 8/19/2013 10:12 PM, Jason Gunthorpe wrote: On Sun, Aug 18, 2013 at 12:05:48PM +0300, Yishai Hadas wrote: On 8/16/2013 06:11 PM, Sean Hefty wrote: @@ -884,6 +884,13 @@ int main(int argc, char *argv[]) if (ctx.use_event) ibv_ack_cq_events(ctx.recv_cq, num_cq_events); + /* Process should wait before closing its resources to make sure + * latest daemon's response sent via its target QP destined to an XSRQ + * created by another client won't be lost. + * Failure to do so will cause the client to wait for that sent message forever. + * See comment on pp_post_send. + */ + sleep(1); I dislike adding sleep calls into code. Isn't there a more robust way to handle this? In general I agree this sleep is a workaround that comes to solve a synchronization hole in this sample application. For that reason I put 5 lines comment to describe the problem and the reason for that sleep statement. Long term you could think of synchronizing all the processes through the sockets mechanism to assure they terminate when all packets are received. This example is very close to the only code that people will have access to when trying to work with XRC. Latest perftest package also contains a use case of XRC. No sleep. Ido It should be complete and correct under all cases. So no sleeps, IMHO. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH perftest] Allow specification of pkey index
Thanks Susan,
We (me and Gil) will apply it soon.
Ido
On 8/20/2013 12:54 AM, Coulter, Susan K wrote:
Signed-off-by: Susan K. Coulter
---
src/perftest_parameters.c |7 ++-
src/perftest_parameters.h |3 +++
src/perftest_resources.c |2 +-
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/src/perftest_parameters.c b/src/perftest_parameters.c
index 084f16f..0350a54 100755
--- a/src/perftest_parameters.c
+++ b/src/perftest_parameters.c
@@ -267,6 +267,8 @@ static void usage(const char *argv0,VerbType verb,TestType
tst) {
if (tst == BW)
printf(" -q, --qp= Num of qp's(default
%d)\n",DEF_NUM_QPS);
+ printf(" -Y, --pkey= PKey index to use for QP\n");
+
putchar('\n');
}
/**
@@ -345,6 +347,7 @@ static void init_perftest_params(struct perftest_parameters
*user_param) {
user_param->limit_bw = 0;
user_param->is_limit_msgrate = OFF;
user_param->limit_msgrate = 0;
+ user_param->pkey= 0;
if (user_param->tst == LAT) {
user_param->r_flag->unsorted = OFF;
@@ -941,9 +944,10 @@ int parser(struct perftest_parameters *user_param,char
*argv[], int argc) {
{ .name = "run_infinitely", .has_arg = 0, .flag =
&run_inf_flag, .val = 1 },
{ .name = "report_gbits", .has_arg = 0, .flag =
&report_fmt_flag, .val = 1},
{ .name = "use-srq",.has_arg = 0, .flag =
&srq_flag, .val = 1},
+ { .name = "pkey", .has_arg = 1, .val = 'Y' },
{ 0 }
};
-c =
getopt_long(argc,argv,"w:y:p:d:i:m:s:n:t:u:S:x:c:q:I:o:M:r:Q:A:l:D:f:B:T:E:J:j:K:k:aFegzRvhbNVCHUOZP",long_options,NULL);
+c =
getopt_long(argc,argv,"w:y:p:d:i:m:s:n:t:u:S:x:c:q:I:o:M:r:Q:A:l:D:f:B:T:E:J:j:K:k:Y:aFegzRvhbNVCHUOZP",long_options,NULL);
if (c == -1)
break;
@@ -1143,6 +1147,7 @@ int parser(struct perftest_parameters *user_param,char
*argv[], int argc) {
case 'P': user_param->machine = CLIENT; break;
case 'Z': user_param->machine = SERVER; break;
case 'v': user_param->mac_fwd = ON; break;
+ case 'Y':
CHECK_VALUE(user_param->pkey,int,MIN_PKEY_IDX,MAX_PKEY_IDX," PKey Index");
break;
case 0: break; // required for long options to work.
default:
fprintf(stderr," Invalid Command or flag.\n");
diff --git a/src/perftest_parameters.h b/src/perftest_parameters.h
index 42f6966..2110066 100755
--- a/src/perftest_parameters.h
+++ b/src/perftest_parameters.h
@@ -135,6 +135,8 @@
#define MIN_CQ_MOD(1)
#define MAX_CQ_MOD(1024)
#define MAX_INLINE(912)
+#define MIN_PKEY_IDX (0)
+#define MAX_PKEY_IDX (127)
// Raw etherent defines
#define RAWETH_MIN_MSG_SIZE (64)
@@ -291,6 +293,7 @@ struct perftest_parameters {
int work_rdma_cm;
char*user_mgid;
int buff_size;
+ int pkey;
// New test params format pilot. will be used in all flags soon,.
enum ctx_test_methodtest_method;
enum ibv_transport_type transport_type;
diff --git a/src/perftest_resources.c b/src/perftest_resources.c
index 2d1..4323734 100755
--- a/src/perftest_resources.c
+++ b/src/perftest_resources.c
@@ -619,7 +619,7 @@ int ctx_modify_qp_to_init(struct ibv_qp *qp,struct
perftest_parameters *user_par
memset(&attr, 0, sizeof(struct ibv_qp_attr));
attr.qp_state= IBV_QPS_INIT;
- attr.pkey_index = 0;
+ attr.pkey_index = user_param->pkey;
if ( user_param->use_xrc && (user_param->duplex || user_param->tst == LAT)) {
num_of_qps /= 2;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH] Add new verb: uv_query_port_max_datagram()
> > This breaks the libibverbs ABI. You can't modify ibv_context_ops because it > changes struct ibv_context. > > Any suggestions on how one adds a new driver call without breaking ABI? It could be built on the verbs extension mechanism. Is it necessary to call into a provider library, versus simply dropping into the kernel? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Add new verb: uv_query_port_max_datagram()
On Aug 19, 2013, at 6:36 PM, "Hefty, Sean" wrote: > This breaks the libibverbs ABI. You can't modify ibv_context_ops because it > changes struct ibv_context. Any suggestions on how one adds a new driver call without breaking ABI? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH] Add new verb: uv_query_port_max_datagram()
> diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
> index 4b1ab57..a1d6c3d 100644
> --- a/include/infiniband/verbs.h
> +++ b/include/infiniband/verbs.h
> @@ -712,6 +712,7 @@ struct ibv_context_ops {
> int (*detach_mcast)(struct ibv_qp *qp, const union
> ibv_gid
> *gid,
> uint16_t lid);
> void(*async_event)(struct ibv_async_event *event);
> + int (*query_port_max_datagram)(struct ibv_context
> *context,
> uint8_t port_num, uint32_t *max_datagram);
> };
This breaks the libibverbs ABI. You can't modify ibv_context_ops because it
changes struct ibv_context.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Add new verb: uv_query_port_max_datagram()
On Aug 19, 2013, at 6:07 PM, "Hefty, Sean" wrote: >> It doesn't *break* the ABI, but it does add a new downcall into the kernel. >> That requires bumping the ABI version to 7, no? > > No - adding a new command is fine. Older kernels will return ENOSYS if that > command is not supported. In that case, you can handle things like Jason > suggested. Gotcha. I'll adjust the patch. Any other feedback? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH perftest] Allow specification of pkey index
Signed-off-by: Susan K. Coulter
---
src/perftest_parameters.c |7 ++-
src/perftest_parameters.h |3 +++
src/perftest_resources.c |2 +-
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/src/perftest_parameters.c b/src/perftest_parameters.c
index 084f16f..0350a54 100755
--- a/src/perftest_parameters.c
+++ b/src/perftest_parameters.c
@@ -267,6 +267,8 @@ static void usage(const char *argv0,VerbType verb,TestType
tst) {
if (tst == BW)
printf(" -q, --qp= Num of qp's(default
%d)\n",DEF_NUM_QPS);
+ printf(" -Y, --pkey= PKey index to use for QP\n");
+
putchar('\n');
}
/**
@@ -345,6 +347,7 @@ static void init_perftest_params(struct perftest_parameters
*user_param) {
user_param->limit_bw = 0;
user_param->is_limit_msgrate = OFF;
user_param->limit_msgrate = 0;
+ user_param->pkey= 0;
if (user_param->tst == LAT) {
user_param->r_flag->unsorted = OFF;
@@ -941,9 +944,10 @@ int parser(struct perftest_parameters *user_param,char
*argv[], int argc) {
{ .name = "run_infinitely", .has_arg = 0, .flag =
&run_inf_flag, .val = 1 },
{ .name = "report_gbits", .has_arg = 0, .flag =
&report_fmt_flag, .val = 1},
{ .name = "use-srq",.has_arg = 0, .flag =
&srq_flag, .val = 1},
+ { .name = "pkey", .has_arg = 1, .val = 'Y' },
{ 0 }
};
-c =
getopt_long(argc,argv,"w:y:p:d:i:m:s:n:t:u:S:x:c:q:I:o:M:r:Q:A:l:D:f:B:T:E:J:j:K:k:aFegzRvhbNVCHUOZP",long_options,NULL);
+c =
getopt_long(argc,argv,"w:y:p:d:i:m:s:n:t:u:S:x:c:q:I:o:M:r:Q:A:l:D:f:B:T:E:J:j:K:k:Y:aFegzRvhbNVCHUOZP",long_options,NULL);
if (c == -1)
break;
@@ -1143,6 +1147,7 @@ int parser(struct perftest_parameters *user_param,char
*argv[], int argc) {
case 'P': user_param->machine = CLIENT; break;
case 'Z': user_param->machine = SERVER; break;
case 'v': user_param->mac_fwd = ON; break;
+ case 'Y':
CHECK_VALUE(user_param->pkey,int,MIN_PKEY_IDX,MAX_PKEY_IDX," PKey Index");
break;
case 0: break; // required for long options to work.
default:
fprintf(stderr," Invalid Command or flag.\n");
diff --git a/src/perftest_parameters.h b/src/perftest_parameters.h
index 42f6966..2110066 100755
--- a/src/perftest_parameters.h
+++ b/src/perftest_parameters.h
@@ -135,6 +135,8 @@
#define MIN_CQ_MOD(1)
#define MAX_CQ_MOD(1024)
#define MAX_INLINE(912)
+#define MIN_PKEY_IDX (0)
+#define MAX_PKEY_IDX (127)
// Raw etherent defines
#define RAWETH_MIN_MSG_SIZE(64)
@@ -291,6 +293,7 @@ struct perftest_parameters {
int work_rdma_cm;
char*user_mgid;
int buff_size;
+ int pkey;
// New test params format pilot. will be used in all flags soon,.
enum ctx_test_methodtest_method;
enum ibv_transport_type transport_type;
diff --git a/src/perftest_resources.c b/src/perftest_resources.c
index 2d1..4323734 100755
--- a/src/perftest_resources.c
+++ b/src/perftest_resources.c
@@ -619,7 +619,7 @@ int ctx_modify_qp_to_init(struct ibv_qp *qp,struct
perftest_parameters *user_par
memset(&attr, 0, sizeof(struct ibv_qp_attr));
attr.qp_state= IBV_QPS_INIT;
- attr.pkey_index = 0;
+ attr.pkey_index = user_param->pkey;
if ( user_param->use_xrc && (user_param->duplex || user_param->tst ==
LAT)) {
num_of_qps /= 2;
--
1.7.1
Susan Coulter
HPC-3 Network/Infrastructure
505-667-8425
Increase the Peace...
An eye for an eye leaves the whole world blind
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH] Add new verb: uv_query_port_max_datagram()
> It doesn't *break* the ABI, but it does add a new downcall into the kernel. > That requires bumping the ABI version to 7, no? No - adding a new command is fine. Older kernels will return ENOSYS if that command is not supported. In that case, you can handle things like Jason suggested. - Sean -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Add new verb: uv_query_port_max_datagram()
On Aug 19, 2013, at 5:18 PM, "Hefty, Sean" wrote: >> Bumped the ABI version to 7 (the new verb will return -ENOSYS if >> abi_verb is < 7). > > How does this break the ABI? It doesn't *break* the ABI, but it does add a new downcall into the kernel. That requires bumping the ABI version to 7, no? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH] Add new verb: uv_query_port_max_datagram()
> Bumped the ABI version to 7 (the new verb will return -ENOSYS if > abi_verb is < 7). How does this break the ABI? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH libibverbs] Add new verb: uv_query_port_max_datagram()
On Aug 19, 2013, at 4:19 PM, Jason Gunthorpe wrote: > What about doing query port in this case and returning that value, > decoded to an enum? Otherwise apps have to include that logic anyhow. > > I'm assuming the kernel will do basically the same? > > Bascially, the only failure for this call should be due to a bad port > number.. Sure, can do. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH libibverbs] Add new verb: uv_query_port_max_datagram()
(re-sending because I forgot to include "libibverbs" in the subject)
Per lengthy discussion on the linux-rdma list, add a new verb to get
max datagram size (in bytes) since the methods for retrieving MTU
values are limited to a finite enum set, and are difficult to change
for backwards compatibility reasons.
Also add corresponding command: uv_cmd_query_port_max_datagram().
Since this is a new verb, there was no need to add a _V2 enum for the
command macro, which required adding a UB_INIT_CMD_RESP() macro.
Bumped the ABI version to 7 (the new verb will return -ENOSYS if
abi_verb is < 7).
Note that the name for this verb was chosen with the following
rationale:
* After discussion with Roland, use the prefix "uv" instead of "ibv",
since this verb is generic to both Ethernet, InfiniBand, and
whatever other transports are underneath.
* "query" was used (vs. "get") because it invokes a command (vs. a
struct lookup)
If the community likes this approach, I'll send the corresponding
kernel patch.
Signed-off-by: Jeff Squyres
---
Makefile.am | 3 +-
examples/devinfo.c | 7 +
include/infiniband/driver.h | 4 +++
include/infiniband/kern-abi.h| 19 +++--
include/infiniband/verbs.h | 7 +
man/uv_query_port_max_datagram.3 | 60
src/cmd.c| 25 +
src/ibverbs.h| 8 ++
src/libibverbs.map | 2 ++
src/verbs.c | 10 +++
10 files changed, 142 insertions(+), 3 deletions(-)
create mode 100644 man/uv_query_port_max_datagram.3
diff --git a/Makefile.am b/Makefile.am
index 40e83be..51fe5d5 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -54,7 +54,8 @@ man_MANS = man/ibv_asyncwatch.1 man/ibv_devices.1
man/ibv_devinfo.1 \
man/ibv_post_srq_recv.3 man/ibv_query_device.3 man/ibv_query_gid.3 \
man/ibv_query_pkey.3 man/ibv_query_port.3 man/ibv_query_qp.3 \
man/ibv_query_srq.3 man/ibv_rate_to_mult.3 man/ibv_reg_mr.3
\
-man/ibv_req_notify_cq.3 man/ibv_resize_cq.3 man/ibv_rate_to_mbps.3
+man/ibv_req_notify_cq.3 man/ibv_resize_cq.3 man/ibv_rate_to_mbps.3 \
+man/uv_query_port_max_datagram.3
DEBIAN = debian/changelog debian/compat debian/control debian/copyright \
debian/ibverbs-utils.install debian/libibverbs1.install \
diff --git a/examples/devinfo.c b/examples/devinfo.c
index ff078e4..f51620b 100644
--- a/examples/devinfo.c
+++ b/examples/devinfo.c
@@ -209,6 +209,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t
ib_port)
struct ibv_port_attr port_attr;
int rc = 0;
uint8_t port;
+ uint32_t max_datagram;
char buf[256];
ctx = ibv_open_device(ib_dev);
@@ -298,6 +299,11 @@ static int print_hca_cap(struct ibv_device *ib_dev,
uint8_t ib_port)
fprintf(stderr, "Failed to query port %u props\n",
port);
goto cleanup;
}
+ rc = uv_query_port_max_datagram(ctx, port, &max_datagram);
+ if (rc) {
+ fprintf(stderr, "Failed to query port %u max datagram
size\n", port);
+ goto cleanup;
+ }
printf("\t\tport:\t%d\n", port);
printf("\t\t\tstate:\t\t\t%s (%d)\n",
port_state_str(port_attr.state), port_attr.state);
@@ -305,6 +311,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t
ib_port)
mtu_str(port_attr.max_mtu), port_attr.max_mtu);
printf("\t\t\tactive_mtu:\t\t%s (%d)\n",
mtu_str(port_attr.active_mtu), port_attr.active_mtu);
+ printf("\t\t\tmax_datagram_size:\t%u\n", max_datagram);
printf("\t\t\tsm_lid:\t\t\t%d\n", port_attr.sm_lid);
printf("\t\t\tport_lid:\t\t%d\n", port_attr.lid);
printf("\t\t\tport_lmc:\t\t0x%02x\n", port_attr.lmc);
diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h
index 9a81416..6e1236c 100644
--- a/include/infiniband/driver.h
+++ b/include/infiniband/driver.h
@@ -67,6 +67,10 @@ int ibv_cmd_query_device(struct ibv_context *context,
int ibv_cmd_query_port(struct ibv_context *context, uint8_t port_num,
struct ibv_port_attr *port_attr,
struct ibv_query_port *cmd, size_t cmd_size);
+int uv_cmd_query_port_max_datagram(struct ibv_context *context, uint8_t
port_num,
+ uint32_t *max_datagram,
+ struct uv_query_port_max_datagram *cmd,
+ size_t cmd_size);
int ibv_cmd_query_gid(struct ibv_context *context, uint8_t port_num,
int index, union ibv_gid *gid);
int ibv_cmd_query_pkey(struct ibv_context *context, uint8_t port_num,
diff --git a/include/infiniband/kern-abi.h b/include/infi
Re: [PATCH] Add new verb: uv_query_port_max_datagram()
On Mon, Aug 19, 2013 at 01:05:04PM -0700, Jeff Squyres wrote:
> +int uv_cmd_query_port_max_datagram(struct ibv_context *context, uint8_t
> port_num,
> +uint32_t *max_datagram,
> +struct uv_query_port_max_datagram *cmd,
> +size_t cmd_size)
> +{
> + struct uv_query_port_max_datagram_resp resp;
> +
> + if (abi_ver < 7)
> + return -ENOSYS;
What about doing query port in this case and returning that value,
decoded to an enum? Otherwise apps have to include that logic anyhow.
I'm assuming the kernel will do basically the same?
Bascially, the only failure for this call should be due to a bad port
number..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Add new verb: uv_query_port_max_datagram()
Per lengthy discussion on the linux-rdma list, add a new verb to get
max datagram size (in bytes) since the methods for retrieving MTU
values are limited to a finite enum set, and are difficult to change
for backwards compatibility reasons.
Also add corresponding command: uv_cmd_query_port_max_datagram().
Since this is a new verb, there was no need to add a _V2 enum for the
command macro, which required adding a UB_INIT_CMD_RESP() macro.
Bumped the ABI version to 7 (the new verb will return -ENOSYS if
abi_verb is < 7).
Note that the name for this verb was chosen with the following
rationale:
* After discussion with Roland, use the prefix "uv" instead of "ibv",
since this verb is generic to both Ethernet, InfiniBand, and
whatever other transports are underneath.
* "query" was used (vs. "get") because it invokes a command (vs. a
struct lookup)
If the community likes this approach, I'll send the corresponding
kernel patch.
Signed-off-by: Jeff Squyres
---
Makefile.am | 3 +-
examples/devinfo.c | 7 +
include/infiniband/driver.h | 4 +++
include/infiniband/kern-abi.h| 19 +++--
include/infiniband/verbs.h | 7 +
man/uv_query_port_max_datagram.3 | 60
src/cmd.c| 25 +
src/ibverbs.h| 8 ++
src/libibverbs.map | 2 ++
src/verbs.c | 10 +++
10 files changed, 142 insertions(+), 3 deletions(-)
create mode 100644 man/uv_query_port_max_datagram.3
diff --git a/Makefile.am b/Makefile.am
index 40e83be..51fe5d5 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -54,7 +54,8 @@ man_MANS = man/ibv_asyncwatch.1 man/ibv_devices.1
man/ibv_devinfo.1 \
man/ibv_post_srq_recv.3 man/ibv_query_device.3 man/ibv_query_gid.3 \
man/ibv_query_pkey.3 man/ibv_query_port.3 man/ibv_query_qp.3 \
man/ibv_query_srq.3 man/ibv_rate_to_mult.3 man/ibv_reg_mr.3
\
-man/ibv_req_notify_cq.3 man/ibv_resize_cq.3 man/ibv_rate_to_mbps.3
+man/ibv_req_notify_cq.3 man/ibv_resize_cq.3 man/ibv_rate_to_mbps.3 \
+man/uv_query_port_max_datagram.3
DEBIAN = debian/changelog debian/compat debian/control debian/copyright \
debian/ibverbs-utils.install debian/libibverbs1.install \
diff --git a/examples/devinfo.c b/examples/devinfo.c
index ff078e4..f51620b 100644
--- a/examples/devinfo.c
+++ b/examples/devinfo.c
@@ -209,6 +209,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t
ib_port)
struct ibv_port_attr port_attr;
int rc = 0;
uint8_t port;
+ uint32_t max_datagram;
char buf[256];
ctx = ibv_open_device(ib_dev);
@@ -298,6 +299,11 @@ static int print_hca_cap(struct ibv_device *ib_dev,
uint8_t ib_port)
fprintf(stderr, "Failed to query port %u props\n",
port);
goto cleanup;
}
+ rc = uv_query_port_max_datagram(ctx, port, &max_datagram);
+ if (rc) {
+ fprintf(stderr, "Failed to query port %u max datagram
size\n", port);
+ goto cleanup;
+ }
printf("\t\tport:\t%d\n", port);
printf("\t\t\tstate:\t\t\t%s (%d)\n",
port_state_str(port_attr.state), port_attr.state);
@@ -305,6 +311,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t
ib_port)
mtu_str(port_attr.max_mtu), port_attr.max_mtu);
printf("\t\t\tactive_mtu:\t\t%s (%d)\n",
mtu_str(port_attr.active_mtu), port_attr.active_mtu);
+ printf("\t\t\tmax_datagram_size:\t%u\n", max_datagram);
printf("\t\t\tsm_lid:\t\t\t%d\n", port_attr.sm_lid);
printf("\t\t\tport_lid:\t\t%d\n", port_attr.lid);
printf("\t\t\tport_lmc:\t\t0x%02x\n", port_attr.lmc);
diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h
index 9a81416..6e1236c 100644
--- a/include/infiniband/driver.h
+++ b/include/infiniband/driver.h
@@ -67,6 +67,10 @@ int ibv_cmd_query_device(struct ibv_context *context,
int ibv_cmd_query_port(struct ibv_context *context, uint8_t port_num,
struct ibv_port_attr *port_attr,
struct ibv_query_port *cmd, size_t cmd_size);
+int uv_cmd_query_port_max_datagram(struct ibv_context *context, uint8_t
port_num,
+ uint32_t *max_datagram,
+ struct uv_query_port_max_datagram *cmd,
+ size_t cmd_size);
int ibv_cmd_query_gid(struct ibv_context *context, uint8_t port_num,
int index, union ibv_gid *gid);
int ibv_cmd_query_pkey(struct ibv_context *context, uint8_t port_num,
diff --git a/include/infiniband/kern-abi.h b/include/infiniband/kern-abi.h
index 619ea7e..951108e 100644
--- a/include/infiniba
Re: [PATCH libibverbs] XRC - Sample application issues
On Sun, Aug 18, 2013 at 12:05:48PM +0300, Yishai Hadas wrote: > On 8/16/2013 06:11 PM, Sean Hefty wrote: > >>@@ -884,6 +884,13 @@ int main(int argc, char *argv[]) > >>if (ctx.use_event) > >>ibv_ack_cq_events(ctx.recv_cq, num_cq_events); > >> > >>+ /* Process should wait before closing its resources to make sure > >>+ * latest daemon's response sent via its target QP destined to an XSRQ > >>+ * created by another client won't be lost. > >>+ * Failure to do so will cause the client to wait for that sent message > >>forever. > >>+ * See comment on pp_post_send. > >>+ */ > >>+ sleep(1); > >I dislike adding sleep calls into code. Isn't there a more robust way to > >handle this? > > In general I agree this sleep is a workaround that comes to solve a > synchronization hole in this sample application. For that reason I > put 5 lines comment to describe the problem and the reason for that > sleep statement. Long term you could think of synchronizing all the > processes through the sockets mechanism to assure they terminate > when all packets are received. This example is very close to the only code that people will have access to when trying to work with XRC. It should be complete and correct under all cases. So no sleeps, IMHO. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello My Dear friend,
Hello My Dear friend, How are you doing I am a lady,Josephine is my name i saw your profile today on(www.nlptoday.com/) and bei will like to be your friend, please send your email address to my inbox (josephin4...@live.com) so i will send you my beautiful Photos -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [ceph-users] Help needed porting Ceph to RSockets
Can you see if the patch below fixes the hang?
Signed-off-by: Sean Hefty
---
src/rsocket.c | 11 ++-
1 files changed, 10 insertions(+), 1 deletions(-)
diff --git a/src/rsocket.c b/src/rsocket.c
index d544dd0..e45b26d 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -2948,10 +2948,12 @@ static int rs_poll_events(struct pollfd *rfds, struct
pollfd *fds, nfds_t nfds)
rs = idm_lookup(&idm, fds[i].fd);
if (rs) {
+ fastlock_acquire(&rs->cq_wait_lock);
if (rs->type == SOCK_STREAM)
rs_get_cq_event(rs);
else
ds_get_cq_event(rs);
+ fastlock_release(&rs->cq_wait_lock);
fds[i].revents = rs_poll_rs(rs, fds[i].events, 1,
rs_poll_all);
} else {
fds[i].revents = rfds[i].revents;
@@ -3098,7 +3100,8 @@ int rselect(int nfds, fd_set *readfds, fd_set *writefds,
/*
* For graceful disconnect, notify the remote side that we're
- * disconnecting and wait until all outstanding sends complete.
+ * disconnecting and wait until all outstanding sends complete, provided
+ * that the remote side has not sent a disconnect message.
*/
int rshutdown(int socket, int how)
{
@@ -3138,6 +3141,12 @@ int rshutdown(int socket, int how)
if (rs->state & rs_connected)
rs_process_cq(rs, 0, rs_conn_all_sends_done);
+ if (rs->state & rs_disconnected) {
+ /* Generate event by flushing receives to unblock rpoll */
+ ibv_req_notify_cq(rs->cm_id->recv_cq, 0);
+ rdma_disconnect(rs->cm_id);
+ }
+
if ((rs->fd_flags & O_NONBLOCK) && (rs->state & rs_connected))
rs_set_nonblocking(rs, rs->fd_flags);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH libibverbs] XRC - Sample application issues
> -Original Message- > From: linux-rdma-ow...@vger.kernel.org > [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of > Hefty, Sean > Sent: Friday, August 16, 2013 10:11 AM > To: Yishai Hadas; linux-rdma@vger.kernel.org; rol...@purestorage.com > Cc: ogerl...@mellanox.com; tza...@mellanox.com; Sternberg, Jay E; Eli Cohen > Subject: RE: [PATCH libibverbs] XRC - Sample application issues > > > @@ -884,6 +884,13 @@ int main(int argc, char *argv[]) > > if (ctx.use_event) > > ibv_ack_cq_events(ctx.recv_cq, num_cq_events); > > > > + /* Process should wait before closing its resources to make sure > > + * latest daemon's response sent via its target QP destined to an XSRQ > > + * created by another client won't be lost. > > + * Failure to do so will cause the client to wait for that sent message > > forever. > > + * See comment on pp_post_send. > > + */ > > + sleep(1); > > I dislike adding sleep calls into code. Isn't there a more robust way to > handle this? Perhaps you could synchronize between the processes using the TCP socket used to exchange the QP info... Steve. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] IB/iser: fix possible memory leak in iser_create_frwr_pool
where desc is not being freed in error flows.
Signed-off-by: Roi Dayan
Signed-off-by: Sagi Grimberg
---
drivers/infiniband/ulp/iser/iser_verbs.c | 10 +++---
1 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c
b/drivers/infiniband/ulp/iser/iser_verbs.c
index 28badac..afe9567 100644
--- a/drivers/infiniband/ulp/iser/iser_verbs.c
+++ b/drivers/infiniband/ulp/iser/iser_verbs.c
@@ -305,7 +305,7 @@ int iser_create_frwr_pool(struct iser_conn *ib_conn,
unsigned cmds_max)
if (IS_ERR(desc->data_frpl)) {
ret = PTR_ERR(desc->data_frpl);
iser_err("Failed to allocate ib_fast_reg_page_list
err=%d\n", ret);
- goto err;
+ goto fast_reg_page_failure;
}
desc->data_mr = ib_alloc_fast_reg_mr(device->pd,
@@ -313,8 +313,7 @@ int iser_create_frwr_pool(struct iser_conn *ib_conn,
unsigned cmds_max)
if (IS_ERR(desc->data_mr)) {
ret = PTR_ERR(desc->data_mr);
iser_err("Failed to allocate ib_fast_reg_mr err=%d\n",
ret);
- ib_free_fast_reg_page_list(desc->data_frpl);
- goto err;
+ goto fast_reg_mr_failure;
}
desc->valid = true;
list_add_tail(&desc->list, &ib_conn->fastreg.frwr.pool);
@@ -322,6 +321,11 @@ int iser_create_frwr_pool(struct iser_conn *ib_conn,
unsigned cmds_max)
}
return 0;
+
+fast_reg_mr_failure:
+ ib_free_fast_reg_page_list(desc->data_frpl);
+fast_reg_page_failure:
+ kfree(desc);
err:
iser_free_frwr_pool(ib_conn);
return ret;
--
1.7.8.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] osm_port_info_rcv.c Issue a log message if we cannot read the MKey of a port
On 08/16/13 15:47, Hal Rosenstock wrote:
On 8/14/2013 6:26 AM, Line Holen wrote:
Signed-off-by: Line Holen
---
diff --git a/opensm/osm_port_info_rcv.c b/opensm/osm_port_info_rcv.c
index 7dcd15e..961b376 100644
--- a/opensm/osm_port_info_rcv.c
+++ b/opensm/osm_port_info_rcv.c
@@ -85,7 +85,7 @@ static void pi_rcv_process_endport(IN osm_sm_t * sm, IN
osm_physp_t * p_physp,
osm_madw_context_t context;
ib_api_status_t status;
ib_net64_t port_guid;
- uint8_t rate, mtu;
+ uint8_t rate, mtu, mpb;
unsigned data_vls;
cl_qmap_t *p_sm_tbl;
osm_remote_sm_t *p_sm;
@@ -126,6 +126,14 @@ static void pi_rcv_process_endport(IN osm_sm_t * sm, IN
osm_physp_t * p_physp,
}
}
+ /* Check M_Key vs M_Key protect, can we control the port ? */
+ mpb = ib_port_info_get_mpb(p_pi);
+ if (mpb> 0&& p_pi->m_key == 0) {
+ OSM_LOG(sm->p_log, OSM_LOG_INFO,
+ "Port 0x%" PRIx64 " has unknown M_Key, protection level
%u\n",
+ cl_ntoh64(port_guid), mpb);
+ }
+
It looks to me like the only case here is when protect bits is 1 for
gets; all others fail. Is it more than that ?
You are probably right - have to admit I haven't tried a higher
protection level.
Also, would this spam the OpenSM log ?
It would print one additional message per heavy sweep.
But if you have a system with unknown MKeys configured you would get
many error
messages as it is. With protection level 2 every MAD operation will generate
an error I guess (either 3111 or 3120). And with protection level 1 set
operations
will fail, but this new message will let you know why it failed.
Line
-- Hal
if (port_guid != sm->p_subn->sm_port_guid) {
p_sm_tbl =&sm->p_subn->sm_guid_tbl;
if (p_pi->capability_mask& IB_PORT_CAP_IS_SM) {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
