Re: [openib-general] Fwd: issues in ipoib
>Here's a patch for you to play with. I'll drop this for now. Thanks - I'll add the userspace SA support above this and update it with any changes. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Sean Hefty <[EMAIL PROTECTED]>:
> Subject: RE: Fwd: issues in ipoib
>
> >Yes. But if the user already has to keep track of when the deregister
> >of its SA client starts, then what is gained by taking a reference
> >when a query starts?
>
> It seems cleaner to me. When a user calls query(), they provide a pointer to
> their sa_client. We take a reference to that pointer and store it in the
> sa_query structure. We now expect that pointer to be valid at a later point,
> so
> we can increment the reference count on it. Why not increment the reference
> count when we take the actual reference and save off the pointer?
>
> The benefit is that when the user later tries to deregister, deregister will
> block while there's an outstanding query. This eliminates the need for
> clients
> to track their queries, cancel all of them, then wait for them to complete
> before calling deregister - which would involve another reference count and
> completion structure on the part of the client.
>
> Thinking about this more, I can see where a user would want to create one
> struct
> ib_sa_client per device to simplify their life handling device removal.
>
> - Sean
>
Here's a patch for you to play with. I'll drop this for now.
--
Require registration with SA module, to prevent module text from going away
while sa query callback is still running, and update all users.
Signed-off-by: Michael S. Tsirkin <[EMAIL PROTECTED]>
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index d6f99d5..bf668b3 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -60,6 +60,10 @@ static struct ib_client cma_client = {
.remove = cma_remove_one
};
+static struct ib_sa_client cma_sa_client = {
+ .name = "cma"
+};
+
static LIST_HEAD(dev_list);
static LIST_HEAD(listen_any_list);
static DEFINE_MUTEX(lock);
@@ -1140,7 +1144,7 @@ static int cma_query_ib_route(struct rdm
path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(addr));
path_rec.numb_path = 1;
- id_priv->query_id = ib_sa_path_rec_get(id_priv->id.device,
+ id_priv->query_id = ib_sa_path_rec_get(&cma_sa_client,
id_priv->id.device,
id_priv->id.port_num, &path_rec,
IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID |
IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH,
@@ -1910,6 +1914,8 @@ static int cma_init(void)
ret = ib_register_client(&cma_client);
if (ret)
goto err;
+
+ ib_sa_register_client(&cma_sa_client);
return 0;
err:
@@ -1919,6 +1925,7 @@ err:
static void cma_cleanup(void)
{
+ ib_sa_unregister_client(&cma_sa_client);
ib_unregister_client(&cma_client);
destroy_workqueue(cma_wq);
idr_destroy(&sdp_ps);
diff --git a/drivers/infiniband/core/sa_query.c
b/drivers/infiniband/core/sa_query.c
index aeda484..43b0323 100644
--- a/drivers/infiniband/core/sa_query.c
+++ b/drivers/infiniband/core/sa_query.c
@@ -78,6 +78,7 @@ struct ib_sa_query {
struct ib_sa_port *port;
struct ib_mad_send_buf *mad_buf;
struct ib_sa_sm_ah *sm_ah;
+ struct ib_sa_client*client;
int id;
};
@@ -532,6 +533,20 @@ retry:
return ret ? ret : id;
}
+static inline void ib_sa_client_get(struct ib_sa_query *query,
+ struct ib_sa_client *client)
+{
+ query->client = client;
+ atomic_inc(&client->users);
+}
+
+static inline void ib_sa_client_put(struct ib_sa_query *query)
+{
+ struct ib_sa_client *client = query->client;
+ if (atomic_dec_and_test(&client->users))
+ complete(&client->completion);
+}
+
static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query,
int status,
struct ib_sa_mad *mad)
@@ -547,6 +562,7 @@ static void ib_sa_path_rec_callback(stru
query->callback(status, &rec, query->context);
} else
query->callback(status, NULL, query->context);
+ ib_sa_client_put(sa_query);
}
static void ib_sa_path_rec_release(struct ib_sa_query *sa_query)
@@ -556,6 +572,7 @@ static void ib_sa_path_rec_release(struc
/**
* ib_sa_path_rec_get - Start a Path get query
+ * @client:client object used to track the query
* @device:device to send query on
* @port_num: port number to send query on
* @rec:Path Record to send in query
@@ -578,7 +595,8 @@ static void ib_sa_path_rec_release(struc
* error code. Otherwise it is a query ID that can be used to cancel
* the query.
*/
-int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
+int ib_sa_path_rec_get(struct ib_sa_client *client,
+ struct ib_device *device, u8 port_num,
struct ib_sa_path_rec *rec,
ib_sa_comp_mask comp_mask,
Re: [openib-general] Fwd: issues in ipoib
>Yes. But if the user already has to keep track of when the deregister >of its SA client starts, then what is gained by taking a reference >when a query starts? It seems cleaner to me. When a user calls query(), they provide a pointer to their sa_client. We take a reference to that pointer and store it in the sa_query structure. We now expect that pointer to be valid at a later point, so we can increment the reference count on it. Why not increment the reference count when we take the actual reference and save off the pointer? The benefit is that when the user later tries to deregister, deregister will block while there's an outstanding query. This eliminates the need for clients to track their queries, cancel all of them, then wait for them to complete before calling deregister - which would involve another reference count and completion structure on the part of the client. Thinking about this more, I can see where a user would want to create one struct ib_sa_client per device to simplify their life handling device removal. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: Fwd: issues in ipoib > > >> We don't know that sa_query->client is still valid here unless a > >> reference was taken earlier. > > > >Yes we do. > >Client must wait till all queries complete before unregistering. > > > >> I like this approach, but would like to see it expanded to track the > >requests, > >> to avoid duplicating this work in every client. Unregistration would then > >> cancel all outstanding queries issued by the user. > > > >I would not object to this on principle, but let's go there by small > >steps - for now, let's get the API fixed and solve the race with module > >unloading that we have for 2.6.18. OK? > > I don't see the need to rush for 2.6.18. No one is hitting this problem, Well, I know OFED 1.0 shipped with a different fix for this race so that's one reason people do not complain :) I don't think we should leave races that we know about and that are easy to fix in development kernels - these are not -stable rules. When there's a module unload crash at customer's site, I do not want to spend time trying to puzzle out whether this could or could not be related to this window - I want to have it covered, and think about something else. > and it > still applies to the ib_cm, ib_addr, and rdma_cm. > This race only applies to ib_addr - CM cleans up after itself flushing out callbacks when CM ID is destroyed. As you can see the patch is a trivial fix - it's just too late in the night here to code more, but I will do this on Sunday Roland, what is your stance? Can this fix be merged for 2.6.18? > re-working the patch in a week. Tracking queries will require changes to the > structure that are best hidden from the users. Hiding those changes requires > reworking the proposed ib_sa_register_client API to create and return struct > ib_sa_client, rather than it being provided by the caller. In kernel API's need not be stable. Since the code is not even yet written, why try to anticipate its needs now? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Sean> I don't see the need to rush for 2.6.18. No one is hitting Sean> this problem, and it still applies to the ib_cm, ib_addr, Sean> and rdma_cm. I have to agree -- this is a pretty big change, and as far as I know there is no evidence that anyone has ever hit this race in the real world. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Sean> I'm not seeing the problem yet. A user can't issue a query Sean> function at the same time they call deregister. There's Sean> nothing we can ever do against that. Is this what you're Sean> referring to? Yes. But if the user already has to keep track of when the deregister of its SA client starts, then what is gained by taking a reference when a query starts? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
>Incrementing the count inside sa_query when a query function is >called. No matter what you do, there's always that window between >"started to call a query function" and "bumped the reference count" >when another thread might get you in trouble. I'm not seeing the problem yet. A user can't issue a query function at the same time they call deregister. There's nothing we can ever do against that. Is this what you're referring to? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Sean> Can you clarify which way you meant with "this" way? Are Sean> you referring to the patch Michael submitted, or Sean> incrementing the count when a query function is called? Incrementing the count inside sa_query when a query function is called. No matter what you do, there's always that window between "started to call a query function" and "bumped the reference count" when another thread might get you in trouble. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: [openib-general] Fwd: issues in ipoib > > >Is this for some kind of diagnostic/management tool? > >It seems ultimately a wrong thing to expose to unpriviledged users. > > It's required in order to establish a connection. How would you propose > userspace applications get their path records? > CMA? Or something along these lines ... -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
>There's a similar unfixable race with trying to count queries this >way: there's a window between when a client module calls the query >function and when the sa_query module actually increments the >reference count. So I think it's better to keep track of what we can >control in the sa_query module, and leave clients responsible for >making sure they wait for all queries that they started. Can you clarify which way you meant with "this" way? Are you referring to the patch Michael submitted, or incrementing the count when a query function is called? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
>> We don't know that sa_query->client is still valid here unless a >> reference was taken earlier. > >Yes we do. >Client must wait till all queries complete before unregistering. > >> I like this approach, but would like to see it expanded to track the >requests, >> to avoid duplicating this work in every client. Unregistration would then >> cancel all outstanding queries issued by the user. > >I would not object to this on principle, but let's go there by small >steps - for now, let's get the API fixed and solve the race with module >unloading that we have for 2.6.18. OK? I don't see the need to rush for 2.6.18. No one is hitting this problem, and it still applies to the ib_cm, ib_addr, and rdma_cm. I'm working on the userspace SA interface now, which means that I'll end up re-working the patch in a week. Tracking queries will require changes to the structure that are best hidden from the users. Hiding those changes requires reworking the proposed ib_sa_register_client API to create and return struct ib_sa_client, rather than it being provided by the caller. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
>Is this for some kind of diagnostic/management tool? >It seems ultimately a wrong thing to expose to unpriviledged users. It's required in order to establish a connection. How would you propose userspace applications get their path records? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: [openib-general] Fwd: issues in ipoib > > >I like this approach, but would like to see it expanded to track the > >requests, > >to avoid duplicating this work in every client. Unregistration would then > >cancel all outstanding queries issued by the user. > > To add to this, I'm trying to expose the SA interfaces in userspace, and it > would be ideal to have each userspace application contained to a single > ib_sa_client. Is this for some kind of diagnostic/management tool? It seems ultimately a wrong thing to expose to unpriviledged users. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Michael> We can do that, I don't have a strong opinion either way. Michael> Roland? There's a similar unfixable race with trying to count queries this way: there's a window between when a client module calls the query function and when the sa_query module actually increments the reference count. So I think it's better to keep track of what we can control in the sa_query module, and leave clients responsible for making sure they wait for all queries that they started. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 0/6] Tranport Neutral Verbs Proposal.
On Tue, Aug 01, 2006 at 01:25:44PM +0200, Christoph Hellwig wrote:
> Exactly. Please don't even try to put brand names (especially if
> they're as stupid as this) in. We don't call our wireless stack
> centrino just because intel contributed to it either.
Centrino: Intel-only brand name
WiFi: trade association brand-name, not joined by all players
802.11{a,b,g}: technical name of technologies
wireless: an overly generic name that people might think should
include bluetooth, wireless usb, etc etc.
OpenFabrics is not a single company brand name, it is the name of
the community that's actually implementing this software stack,
like 'Gnome' or 'KDE'.
BTW, I've had meetings with about 5 startups that began like, "We have
an rdma device, but it's not actually RDMA as defined by that IEEE
Committee." And these devices don't work like that definition. So
there's considerable difference of opinion as to what RDMA means.
-- greg
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: Fwd: issues in ipoib > > > static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query, > > int status, > > struct ib_sa_mad *mad) > >@@ -539,6 +551,7 @@ static void ib_sa_path_rec_callback(stru > > struct ib_sa_path_query *query = > > container_of(sa_query, struct ib_sa_path_query, sa_query); > > > >+ib_sa_client_get(sa_query->client); > > It makes more sense to me to increment the reference count when the query is > initiated. We can do that, I don't have a strong opinion either way. Roland? > We don't know that sa_query->client is still valid here unless a > reference was taken earlier. Yes we do. Client must wait till all queries complete before unregistering. > I like this approach, but would like to see it expanded to track the requests, > to avoid duplicating this work in every client. Unregistration would then > cancel all outstanding queries issued by the user. I would not object to this on principle, but let's go there by small steps - for now, let's get the API fixed and solve the race with module unloading that we have for 2.6.18. OK? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
>I like this approach, but would like to see it expanded to track the requests, >to avoid duplicating this work in every client. Unregistration would then >cancel all outstanding queries issued by the user. To add to this, I'm trying to expose the SA interfaces in userspace, and it would be ideal to have each userspace application contained to a single ib_sa_client. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
> static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query, > int status, > struct ib_sa_mad *mad) >@@ -539,6 +551,7 @@ static void ib_sa_path_rec_callback(stru > struct ib_sa_path_query *query = > container_of(sa_query, struct ib_sa_path_query, sa_query); > >+ ib_sa_client_get(sa_query->client); It makes more sense to me to increment the reference count when the query is initiated. We don't know that sa_query->client is still valid here unless a reference was taken earlier. I like this approach, but would like to see it expanded to track the requests, to avoid duplicating this work in every client. Unregistration would then cancel all outstanding queries issued by the user. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: Fwd: issues in ipoib
>
> Michael> No, I think there's no problem - user callback has
> Michael> finished running and that's all we care about.
>
> I don't think so -- because then the unregister call can return and
> the client module free the client struct before the wake_up() runs,
> which leads to use-after-free.
OK, here's a version using completion which should address this issue.
How does this look?
--
Require registration with SA module, to prevent module text from going away
while sa query callback is still running, and update all users.
Signed-off-by: Michael S. Tsirkin <[EMAIL PROTECTED]>
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index d6f99d5..bf668b3 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -60,6 +60,10 @@ static struct ib_client cma_client = {
.remove = cma_remove_one
};
+static struct ib_sa_client cma_sa_client = {
+ .name = "cma"
+};
+
static LIST_HEAD(dev_list);
static LIST_HEAD(listen_any_list);
static DEFINE_MUTEX(lock);
@@ -1140,7 +1144,7 @@ static int cma_query_ib_route(struct rdm
path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(addr));
path_rec.numb_path = 1;
- id_priv->query_id = ib_sa_path_rec_get(id_priv->id.device,
+ id_priv->query_id = ib_sa_path_rec_get(&cma_sa_client,
id_priv->id.device,
id_priv->id.port_num, &path_rec,
IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID |
IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH,
@@ -1910,6 +1914,8 @@ static int cma_init(void)
ret = ib_register_client(&cma_client);
if (ret)
goto err;
+
+ ib_sa_register_client(&cma_sa_client);
return 0;
err:
@@ -1919,6 +1925,7 @@ err:
static void cma_cleanup(void)
{
+ ib_sa_unregister_client(&cma_sa_client);
ib_unregister_client(&cma_client);
destroy_workqueue(cma_wq);
idr_destroy(&sdp_ps);
diff --git a/drivers/infiniband/core/sa_query.c
b/drivers/infiniband/core/sa_query.c
index aeda484..ea03677 100644
--- a/drivers/infiniband/core/sa_query.c
+++ b/drivers/infiniband/core/sa_query.c
@@ -78,6 +78,7 @@ struct ib_sa_query {
struct ib_sa_port *port;
struct ib_mad_send_buf *mad_buf;
struct ib_sa_sm_ah *sm_ah;
+ struct ib_sa_client*client;
int id;
};
@@ -532,6 +533,17 @@ retry:
return ret ? ret : id;
}
+static inline void ib_sa_client_get(struct ib_sa_client *client)
+{
+ atomic_inc(&client->users);
+}
+
+static inline void ib_sa_client_put(struct ib_sa_client *client)
+{
+ if (atomic_dec_and_test(&client->users))
+ complete(&client->completion);
+}
+
static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query,
int status,
struct ib_sa_mad *mad)
@@ -539,6 +551,7 @@ static void ib_sa_path_rec_callback(stru
struct ib_sa_path_query *query =
container_of(sa_query, struct ib_sa_path_query, sa_query);
+ ib_sa_client_get(sa_query->client);
if (mad) {
struct ib_sa_path_rec rec;
@@ -547,6 +560,7 @@ static void ib_sa_path_rec_callback(stru
query->callback(status, &rec, query->context);
} else
query->callback(status, NULL, query->context);
+ ib_sa_client_put(sa_query->client);
}
static void ib_sa_path_rec_release(struct ib_sa_query *sa_query)
@@ -556,6 +570,7 @@ static void ib_sa_path_rec_release(struc
/**
* ib_sa_path_rec_get - Start a Path get query
+ * @client:client object used to track the query
* @device:device to send query on
* @port_num: port number to send query on
* @rec:Path Record to send in query
@@ -578,7 +593,8 @@ static void ib_sa_path_rec_release(struc
* error code. Otherwise it is a query ID that can be used to cancel
* the query.
*/
-int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
+int ib_sa_path_rec_get(struct ib_sa_client *client,
+ struct ib_device *device, u8 port_num,
struct ib_sa_path_rec *rec,
ib_sa_comp_mask comp_mask,
int timeout_ms, gfp_t gfp_mask,
@@ -619,6 +635,7 @@ int ib_sa_path_rec_get(struct ib_device
mad = query->sa_query.mad_buf->mad;
init_mad(mad, agent);
+ query->sa_query.client = client;
query->sa_query.callback = callback ? ib_sa_path_rec_callback : NULL;
query->sa_query.release = ib_sa_path_rec_release;
query->sa_query.port = port;
@@ -653,6 +670,7 @@ static void ib_sa_service_rec_callback(s
struct ib_sa_service_query *query =
container_of(sa_query, struct ib_sa_service_query, sa_query);
+ ib_sa_client_get(sa_query->clie
[openib-general] [PATCH v4 5/7] AMSO1100 Message Queues.
---
drivers/infiniband/hw/amso1100/c2_mq.c | 175
drivers/infiniband/hw/amso1100/c2_mq.h | 107
2 files changed, 282 insertions(+), 0 deletions(-)
diff --git a/drivers/infiniband/hw/amso1100/c2_mq.c
b/drivers/infiniband/hw/amso1100/c2_mq.c
new file mode 100644
index 000..96bbe9a
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_mq.c
@@ -0,0 +1,175 @@
+/*
+ * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#include "c2.h"
+#include "c2_mq.h"
+
+void *c2_mq_alloc(struct c2_mq *q)
+{
+ BUG_ON(q->magic != C2_MQ_MAGIC);
+ BUG_ON(q->type != C2_MQ_ADAPTER_TARGET);
+
+ if (c2_mq_full(q)) {
+ return NULL;
+ } else {
+#ifdef DEBUG
+ struct c2wr_hdr *m =
+ (struct c2wr_hdr *) (q->msg_pool.host + q->priv *
q->msg_size);
+#ifdef CCMSGMAGIC
+ BUG_ON(m->magic != be32_to_cpu(~CCWR_MAGIC));
+ m->magic = cpu_to_be32(CCWR_MAGIC);
+#endif
+ return m;
+#else
+ return q->msg_pool.host + q->priv * q->msg_size;
+#endif
+ }
+}
+
+void c2_mq_produce(struct c2_mq *q)
+{
+ BUG_ON(q->magic != C2_MQ_MAGIC);
+ BUG_ON(q->type != C2_MQ_ADAPTER_TARGET);
+
+ if (!c2_mq_full(q)) {
+ q->priv = (q->priv + 1) % q->q_size;
+ q->hint_count++;
+ /* Update peer's offset. */
+ __raw_writew(cpu_to_be16(q->priv), &q->peer->shared);
+ }
+}
+
+void *c2_mq_consume(struct c2_mq *q)
+{
+ BUG_ON(q->magic != C2_MQ_MAGIC);
+ BUG_ON(q->type != C2_MQ_HOST_TARGET);
+
+ if (c2_mq_empty(q)) {
+ return NULL;
+ } else {
+#ifdef DEBUG
+ struct c2wr_hdr *m = (struct c2wr_hdr *)
+ (q->msg_pool.host + q->priv * q->msg_size);
+#ifdef CCMSGMAGIC
+ BUG_ON(m->magic != be32_to_cpu(CCWR_MAGIC));
+#endif
+ return m;
+#else
+ return q->msg_pool.host + q->priv * q->msg_size;
+#endif
+ }
+}
+
+void c2_mq_free(struct c2_mq *q)
+{
+ BUG_ON(q->magic != C2_MQ_MAGIC);
+ BUG_ON(q->type != C2_MQ_HOST_TARGET);
+
+ if (!c2_mq_empty(q)) {
+
+#ifdef CCMSGMAGIC
+ {
+ struct c2wr_hdr __iomem *m = (struct c2wr_hdr __iomem *)
+ (q->msg_pool.adapter + q->priv * q->msg_size);
+ __raw_writel(cpu_to_be32(~CCWR_MAGIC), &m->magic);
+ }
+#endif
+ q->priv = (q->priv + 1) % q->q_size;
+ /* Update peer's offset. */
+ __raw_writew(cpu_to_be16(q->priv), &q->peer->shared);
+ }
+}
+
+
+void c2_mq_lconsume(struct c2_mq *q, u32 wqe_count)
+{
+ BUG_ON(q->magic != C2_MQ_MAGIC);
+ BUG_ON(q->type != C2_MQ_ADAPTER_TARGET);
+
+ while (wqe_count--) {
+ BUG_ON(c2_mq_empty(q));
+ *q->shared = cpu_to_be16((be16_to_cpu(*q->shared)+1) %
q->q_size);
+ }
+}
+
+
+u32 c2_mq_count(struct c2_mq *q)
+{
+ s32 count;
+
+ if (q->type == C2_MQ_HOST_TARGET) {
+ count = be16_to_cpu(*q->shared) - q->priv;
+ } else {
+ count = q->priv - be16_to_cpu(*q->shared);
+ }
+
+ if (count < 0) {
+ count += q->q_size;
+ }
+
+ return (u32) count;
+}
+
+void c2_mq_req_init(struct c2_mq *q, u32 index, u32 q_size, u32 msg_size,
+ u8 __iomem *pool_start, u16 __iomem *peer, u32 type)
+{
+ BUG_ON(!q->
[openib-general] [PATCH v4 6/7] AMSO1100: Privileged Verbs Queues.
---
drivers/infiniband/hw/amso1100/c2_vq.c | 260
drivers/infiniband/hw/amso1100/c2_vq.h | 63
2 files changed, 323 insertions(+), 0 deletions(-)
diff --git a/drivers/infiniband/hw/amso1100/c2_vq.c
b/drivers/infiniband/hw/amso1100/c2_vq.c
new file mode 100644
index 000..445b1ed
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_vq.c
@@ -0,0 +1,260 @@
+/*
+ * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#include
+#include
+
+#include "c2_vq.h"
+#include "c2_provider.h"
+
+/*
+ * Verbs Request Objects:
+ *
+ * VQ Request Objects are allocated by the kernel verbs handlers.
+ * They contain a wait object, a refcnt, an atomic bool indicating that the
+ * adapter has replied, and a copy of the verb reply work request.
+ * A pointer to the VQ Request Object is passed down in the context
+ * field of the work request message, and reflected back by the adapter
+ * in the verbs reply message. The function handle_vq() in the interrupt
+ * path will use this pointer to:
+ * 1) append a copy of the verbs reply message
+ * 2) mark that the reply is ready
+ * 3) wake up the kernel verbs handler blocked awaiting the reply.
+ *
+ *
+ * The kernel verbs handlers do a "get" to put a 2nd reference on the
+ * VQ Request object. If the kernel verbs handler exits before the adapter
+ * can respond, this extra reference will keep the VQ Request object around
+ * until the adapter's reply can be processed. The reason we need this is
+ * because a pointer to this object is stuffed into the context field of
+ * the verbs work request message, and reflected back in the reply message.
+ * It is used in the interrupt handler (handle_vq()) to wake up the appropriate
+ * kernel verb handler that is blocked awaiting the verb reply.
+ * So handle_vq() will do a "put" on the object when it's done accessing it.
+ * NOTE: If we guarantee that the kernel verb handler will never bail before
+ *getting the reply, then we don't need these refcnts.
+ *
+ *
+ * VQ Request objects are freed by the kernel verbs handlers only
+ * after the verb has been processed, or when the adapter fails and
+ * does not reply.
+ *
+ *
+ * Verbs Reply Buffers:
+ *
+ * VQ Reply bufs are local host memory copies of a
+ * outstanding Verb Request reply
+ * message. The are always allocated by the kernel verbs handlers, and _may_
be
+ * freed by either the kernel verbs handler -or- the interrupt handler. The
+ * kernel verbs handler _must_ free the repbuf, then free the vq request object
+ * in that order.
+ */
+
+int vq_init(struct c2_dev *c2dev)
+{
+ sprintf(c2dev->vq_cache_name, "c2-vq:dev%c",
+ (char) ('0' + c2dev->devnum));
+ c2dev->host_msg_cache =
+ kmem_cache_create(c2dev->vq_cache_name, c2dev->rep_vq.msg_size, 0,
+ SLAB_HWCACHE_ALIGN, NULL, NULL);
+ if (c2dev->host_msg_cache == NULL) {
+ return -ENOMEM;
+ }
+ return 0;
+}
+
+void vq_term(struct c2_dev *c2dev)
+{
+ kmem_cache_destroy(c2dev->host_msg_cache);
+}
+
+/* vq_req_alloc - allocate a VQ Request Object and initialize it.
+ * The refcnt is set to 1.
+ */
+struct c2_vq_req *vq_req_alloc(struct c2_dev *c2dev)
+{
+ struct c2_vq_req *r;
+
+ r = kmalloc(sizeof(struct c2_vq_req), GFP_KERNEL);
+ if (r) {
+ init_waitqueue_head(&r->wait_object);
+ r->reply_msg = (u64) NULL;
+ r->event = 0;
+ r->cm_id = NU
[openib-general] [PATCH v4 7/7] AMSO1100 Makefiles and Kconfig changes.
--- drivers/infiniband/Kconfig |1 + drivers/infiniband/Makefile|1 + drivers/infiniband/hw/amso1100/Kbuild | 10 ++ drivers/infiniband/hw/amso1100/Kconfig | 15 +++ drivers/infiniband/hw/amso1100/README | 11 +++ 5 files changed, 38 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig index 69a53d4..47f1de5 100644 --- a/drivers/infiniband/Kconfig +++ b/drivers/infiniband/Kconfig @@ -36,6 +36,7 @@ config INFINIBAND_ADDR_TRANS source "drivers/infiniband/hw/mthca/Kconfig" source "drivers/infiniband/hw/ipath/Kconfig" +source "drivers/infiniband/hw/amso1100/Kconfig" source "drivers/infiniband/ulp/ipoib/Kconfig" diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile index c7ff58c..ec330d5 100644 --- a/drivers/infiniband/Makefile +++ b/drivers/infiniband/Makefile @@ -1,6 +1,7 @@ obj-$(CONFIG_INFINIBAND) += core/ obj-$(CONFIG_INFINIBAND_MTHCA) += hw/mthca/ obj-$(CONFIG_IPATH_CORE) += hw/ipath/ +obj-$(CONFIG_INFINIBAND_AMSO1100) += hw/amso1100/ obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/ obj-$(CONFIG_INFINIBAND_ISER) += ulp/iser/ diff --git a/drivers/infiniband/hw/amso1100/Kbuild b/drivers/infiniband/hw/amso1100/Kbuild new file mode 100644 index 000..e1f10ab --- /dev/null +++ b/drivers/infiniband/hw/amso1100/Kbuild @@ -0,0 +1,10 @@ +EXTRA_CFLAGS += -Idrivers/infiniband/include + +ifdef CONFIG_INFINIBAND_AMSO1100_DEBUG +EXTRA_CFLAGS += -DDEBUG +endif + +obj-$(CONFIG_INFINIBAND_AMSO1100) += iw_c2.o + +iw_c2-y := c2.o c2_provider.o c2_rnic.o c2_alloc.o c2_mq.o c2_ae.o c2_vq.o \ + c2_intr.o c2_cq.o c2_qp.o c2_cm.o c2_mm.o c2_pd.o diff --git a/drivers/infiniband/hw/amso1100/Kconfig b/drivers/infiniband/hw/amso1100/Kconfig new file mode 100644 index 000..809cb14 --- /dev/null +++ b/drivers/infiniband/hw/amso1100/Kconfig @@ -0,0 +1,15 @@ +config INFINIBAND_AMSO1100 + tristate "Ammasso 1100 HCA support" + depends on PCI && INET && INFINIBAND + ---help--- + This is a low-level driver for the Ammasso 1100 host + channel adapter (HCA). + +config INFINIBAND_AMSO1100_DEBUG + bool "Verbose debugging output" + depends on INFINIBAND_AMSO1100 + default n + ---help--- + This option causes the amso1100 driver to produce a bunch of + debug messages. Select this if you are developing the driver + or trying to diagnose a problem. diff --git a/drivers/infiniband/hw/amso1100/README b/drivers/infiniband/hw/amso1100/README new file mode 100644 index 000..1331353 --- /dev/null +++ b/drivers/infiniband/hw/amso1100/README @@ -0,0 +1,11 @@ +This is the OpenFabrics provider driver for the +AMSO1100 1Gb RNIC adapter. + +This adapter is available in limited quantities +for development purposes from Open Grid Computing. + +This driver requires the IWCM and CMA mods necessary +to support iWARP. + +Contact [EMAIL PROTECTED] for more information. + ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v4 4/7] AMSO1100 Memory Management.
---
drivers/infiniband/hw/amso1100/c2_alloc.c | 144 +++
drivers/infiniband/hw/amso1100/c2_mm.c| 375 +
2 files changed, 519 insertions(+), 0 deletions(-)
diff --git a/drivers/infiniband/hw/amso1100/c2_alloc.c
b/drivers/infiniband/hw/amso1100/c2_alloc.c
new file mode 100644
index 000..013b152
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_alloc.c
@@ -0,0 +1,144 @@
+/*
+ * Copyright (c) 2004 Topspin Communications. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include
+#include
+#include
+
+#include "c2.h"
+
+static int c2_alloc_mqsp_chunk(struct c2_dev *c2dev, gfp_t gfp_mask,
+ struct sp_chunk **head)
+{
+ int i;
+ struct sp_chunk *new_head;
+
+ new_head = (struct sp_chunk *) __get_free_page(gfp_mask);
+ if (new_head == NULL)
+ return -ENOMEM;
+
+ new_head->dma_addr = dma_map_single(c2dev->ibdev.dma_device, new_head,
+ PAGE_SIZE, DMA_FROM_DEVICE);
+ pci_unmap_addr_set(new_head, mapping, new_head->dma_addr);
+
+ new_head->next = NULL;
+ new_head->head = 0;
+
+ /* build list where each index is the next free slot */
+ for (i = 0;
+i < (PAGE_SIZE - sizeof(struct sp_chunk) -
+ sizeof(u16)) / sizeof(u16) - 1;
+i++) {
+ new_head->shared_ptr[i] = i + 1;
+ }
+ /* terminate list */
+ new_head->shared_ptr[i] = 0x;
+
+ *head = new_head;
+ return 0;
+}
+
+int c2_init_mqsp_pool(struct c2_dev *c2dev, gfp_t gfp_mask,
+ struct sp_chunk **root)
+{
+ return c2_alloc_mqsp_chunk(c2dev, gfp_mask, root);
+}
+
+void c2_free_mqsp_pool(struct c2_dev *c2dev, struct sp_chunk *root)
+{
+ struct sp_chunk *next;
+
+ while (root) {
+ next = root->next;
+ dma_unmap_single(c2dev->ibdev.dma_device,
+pci_unmap_addr(root, mapping), PAGE_SIZE,
+DMA_FROM_DEVICE);
+ __free_page((struct page *) root);
+ root = next;
+ }
+}
+
+u16 *c2_alloc_mqsp(struct c2_dev *c2dev, struct sp_chunk *head,
+ dma_addr_t *dma_addr, gfp_t gfp_mask)
+{
+ u16 mqsp;
+
+ while (head) {
+ mqsp = head->head;
+ if (mqsp != 0x) {
+ head->head = head->shared_ptr[mqsp];
+ break;
+ } else if (head->next == NULL) {
+ if (c2_alloc_mqsp_chunk(c2dev, gfp_mask, &head->next) ==
+ 0) {
+ head = head->next;
+ mqsp = head->head;
+ head->head = head->shared_ptr[mqsp];
+ break;
+ } else
+ return NULL;
+ } else
+ head = head->next;
+ }
+ if (head) {
+ *dma_addr = head->dma_addr +
+ ((unsigned long) &(head->shared_ptr[mqsp]) -
+(unsigned long) head);
+ pr_debug("%s addr %p dma_addr %llx\n", __FUNCTION__,
+&(head->shared_ptr[mqsp]), (u64)*dma_addr);
+ return &(head->shared_ptr[mqsp]);
+ }
+ return NULL;
+}
+
+void c2_free_mqsp(u16 * mqsp)
+{
+ struct sp_chunk *head;
+ u16 idx;
+
+ /* The c
[openib-general] [PATCH v4 2/7] AMSO1100 WR / Event Definitions.
---
drivers/infiniband/hw/amso1100/c2_ae.h | 108 ++
drivers/infiniband/hw/amso1100/c2_status.h | 158 +++
drivers/infiniband/hw/amso1100/c2_wr.h | 1520
3 files changed, 1786 insertions(+), 0 deletions(-)
diff --git a/drivers/infiniband/hw/amso1100/c2_ae.h
b/drivers/infiniband/hw/amso1100/c2_ae.h
new file mode 100644
index 000..3a065c3
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_ae.h
@@ -0,0 +1,108 @@
+/*
+ * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#ifndef _C2_AE_H_
+#define _C2_AE_H_
+
+/*
+ * WARNING: If you change this file, also bump C2_IVN_BASE
+ * in common/include/clustercore/c2_ivn.h.
+ */
+
+/*
+ * Asynchronous Event Identifiers
+ *
+ * These start at 0x80 only so it's obvious from inspection that
+ * they are not work-request statuses. This isn't critical.
+ *
+ * NOTE: these event id's must fit in eight bits.
+ */
+enum c2_event_id {
+ CCAE_REMOTE_SHUTDOWN = 0x80,
+ CCAE_ACTIVE_CONNECT_RESULTS,
+ CCAE_CONNECTION_REQUEST,
+ CCAE_LLP_CLOSE_COMPLETE,
+ CCAE_TERMINATE_MESSAGE_RECEIVED,
+ CCAE_LLP_CONNECTION_RESET,
+ CCAE_LLP_CONNECTION_LOST,
+ CCAE_LLP_SEGMENT_SIZE_INVALID,
+ CCAE_LLP_INVALID_CRC,
+ CCAE_LLP_BAD_FPDU,
+ CCAE_INVALID_DDP_VERSION,
+ CCAE_INVALID_RDMA_VERSION,
+ CCAE_UNEXPECTED_OPCODE,
+ CCAE_INVALID_DDP_QUEUE_NUMBER,
+ CCAE_RDMA_READ_NOT_ENABLED,
+ CCAE_RDMA_WRITE_NOT_ENABLED,
+ CCAE_RDMA_READ_TOO_SMALL,
+ CCAE_NO_L_BIT,
+ CCAE_TAGGED_INVALID_STAG,
+ CCAE_TAGGED_BASE_BOUNDS_VIOLATION,
+ CCAE_TAGGED_ACCESS_RIGHTS_VIOLATION,
+ CCAE_TAGGED_INVALID_PD,
+ CCAE_WRAP_ERROR,
+ CCAE_BAD_CLOSE,
+ CCAE_BAD_LLP_CLOSE,
+ CCAE_INVALID_MSN_RANGE,
+ CCAE_INVALID_MSN_GAP,
+ CCAE_IRRQ_OVERFLOW,
+ CCAE_IRRQ_MSN_GAP,
+ CCAE_IRRQ_MSN_RANGE,
+ CCAE_IRRQ_INVALID_STAG,
+ CCAE_IRRQ_BASE_BOUNDS_VIOLATION,
+ CCAE_IRRQ_ACCESS_RIGHTS_VIOLATION,
+ CCAE_IRRQ_INVALID_PD,
+ CCAE_IRRQ_WRAP_ERROR,
+ CCAE_CQ_SQ_COMPLETION_OVERFLOW,
+ CCAE_CQ_RQ_COMPLETION_ERROR,
+ CCAE_QP_SRQ_WQE_ERROR,
+ CCAE_QP_LOCAL_CATASTROPHIC_ERROR,
+ CCAE_CQ_OVERFLOW,
+ CCAE_CQ_OPERATION_ERROR,
+ CCAE_SRQ_LIMIT_REACHED,
+ CCAE_QP_RQ_LIMIT_REACHED,
+ CCAE_SRQ_CATASTROPHIC_ERROR,
+ CCAE_RNIC_CATASTROPHIC_ERROR
+/* WARNING If you add more id's, make sure their values fit in eight bits. */
+};
+
+/*
+ * Resource Indicators and Identifiers
+ */
+enum c2_resource_indicator {
+ C2_RES_IND_QP = 1,
+ C2_RES_IND_EP,
+ C2_RES_IND_CQ,
+ C2_RES_IND_SRQ,
+};
+
+#endif /* _C2_AE_H_ */
diff --git a/drivers/infiniband/hw/amso1100/c2_status.h
b/drivers/infiniband/hw/amso1100/c2_status.h
new file mode 100644
index 000..6ee4aa9
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_status.h
@@ -0,0 +1,158 @@
+/*
+ * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that
[openib-general] [PATCH v4 0/7] Ammasso 1100 iWARP Driver
Roland, here is the updated Ammasso 1100 iWARP Driver. This patchset implements the iWARP provider driver for the Ammasso 1100 RNIC. It is dependent on the "iWARP Core Support" patch set. The patchset consists of 7 patches: 1 - Low-level device interface and native stack support 2 - Work request definitions 3 - Provider interface 4 - Memory management 5 - User mode message queue implementation 6 - Verbs queue implementation 7 - Kconfig and Makefile Signed-off-by: Tom Tucker <[EMAIL PROTECTED]> Signed-off-by: Steve Wise <[EMAIL PROTECTED]> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v5 1/2] iWARP Connection Manager.
This patch provides the new files implementing the iWARP Connection
Manager.
This module is a logical instance of the xx_cm where xx is the transport
type (ib or iw). The symbols exported are used by the transport
independent rdma_cm module, and are available also for transport
dependent ULPs.
---
drivers/infiniband/core/iwcm.c | 1008
include/rdma/iw_cm.h | 255 ++
include/rdma/iw_cm_private.h | 63 +++
3 files changed, 1326 insertions(+), 0 deletions(-)
diff --git a/drivers/infiniband/core/iwcm.c b/drivers/infiniband/core/iwcm.c
new file mode 100644
index 000..fe43c00
--- /dev/null
+++ b/drivers/infiniband/core/iwcm.c
@@ -0,0 +1,1008 @@
+/*
+ * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved.
+ * Copyright (c) 2004 Topspin Corporation. All rights reserved.
+ * Copyright (c) 2004, 2005 Voltaire Corporation. All rights reserved.
+ * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ * Copyright (c) 2005 Network Appliance, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ */
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+MODULE_AUTHOR("Tom Tucker");
+MODULE_DESCRIPTION("iWARP CM");
+MODULE_LICENSE("Dual BSD/GPL");
+
+static struct workqueue_struct *iwcm_wq;
+struct iwcm_work {
+ struct work_struct work;
+ struct iwcm_id_private *cm_id;
+ struct list_head list;
+ struct iw_cm_event event;
+ struct list_head free_list;
+};
+
+/*
+ * The following services provide a mechanism for pre-allocating iwcm_work
+ * elements. The design pre-allocates them based on the cm_id type:
+ * LISTENING IDS: Get enough elements preallocated to handle the
+ * listen backlog.
+ * ACTIVE IDS: 4: CONNECT_REPLY, ESTABLISHED, DISCONNECT, CLOSE
+ * PASSIVE IDS:3: ESTABLISHED, DISCONNECT, CLOSE
+ *
+ * Allocating them in connect and listen avoids having to deal
+ * with allocation failures on the event upcall from the provider (which
+ * is called in the interrupt context).
+ *
+ * One exception is when creating the cm_id for incoming connection requests.
+ * There are two cases:
+ * 1) in the event upcall, cm_event_handler(), for a listening cm_id. If
+ *the backlog is exceeded, then no more connection request events will
+ *be processed. cm_event_handler() returns -ENOMEM in this case. Its up
+ *to the provider to reject the connectino request.
+ * 2) in the connection request workqueue handler, cm_conn_req_handler().
+ *If work elements cannot be allocated for the new connect request cm_id,
+ *then IWCM will call the provider reject method. This is ok since
+ *cm_conn_req_handler() runs in the workqueue thread context.
+ */
+
+static struct iwcm_work *get_work(struct iwcm_id_private *cm_id_priv)
+{
+ struct iwcm_work *work;
+
+ if (list_empty(&cm_id_priv->work_free_list))
+ return NULL;
+ work = list_entry(cm_id_priv->work_free_list.next, struct iwcm_work,
+ free_list);
+ list_del_init(&work->free_list);
+ return work;
+}
+
+static void put_work(struct iwcm_work *work)
+{
+ list_add(&work->free_list, &work->cm_id->work_free_list);
+}
+
+static void dealloc_work_entries(struct iwcm_id_private *cm_id_priv)
+{
+ struct list_head *e, *tmp;
+
+ list_for_each_safe(e, tmp, &cm_id_priv->work_free_list)
+ kfree(list_entry(e, struct i
[openib-general] [PATCH v5 2/2] iWARP Core Changes.
Modifications to the existing rdma header files, core files, drivers,
and ulp files to support iWARP.
---
drivers/infiniband/core/Makefile |4
drivers/infiniband/core/addr.c | 18 +
drivers/infiniband/core/cache.c |7 -
drivers/infiniband/core/cm.c |3
drivers/infiniband/core/cma.c| 355 +++---
drivers/infiniband/core/device.c |6
drivers/infiniband/core/mad.c| 11 +
drivers/infiniband/core/sa_query.c |5
drivers/infiniband/core/smi.c| 18 +
drivers/infiniband/core/sysfs.c | 18 +
drivers/infiniband/core/ucm.c|5
drivers/infiniband/core/user_mad.c |9 -
drivers/infiniband/hw/ipath/ipath_verbs.c|2
drivers/infiniband/hw/mthca/mthca_provider.c |2
drivers/infiniband/ulp/ipoib/ipoib_main.c|8 +
drivers/infiniband/ulp/srp/ib_srp.c |2
include/rdma/ib_addr.h | 17 +
include/rdma/ib_verbs.h | 39 ++-
18 files changed, 439 insertions(+), 90 deletions(-)
diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index 68e73ec..163d991 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -1,7 +1,7 @@
infiniband-$(CONFIG_INFINIBAND_ADDR_TRANS) := ib_addr.o rdma_cm.o
obj-$(CONFIG_INFINIBAND) +=ib_core.o ib_mad.o ib_sa.o \
- ib_cm.o $(infiniband-y)
+ ib_cm.o iw_cm.o $(infiniband-y)
obj-$(CONFIG_INFINIBAND_USER_MAD) += ib_umad.o
obj-$(CONFIG_INFINIBAND_USER_ACCESS) +=ib_uverbs.o ib_ucm.o
@@ -14,6 +14,8 @@ ib_sa-y :=sa_query.o
ib_cm-y := cm.o
+iw_cm-y := iwcm.o
+
rdma_cm-y := cma.o
ib_addr-y := addr.o
diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
index d294bbc..399151a 100644
--- a/drivers/infiniband/core/addr.c
+++ b/drivers/infiniband/core/addr.c
@@ -60,12 +60,15 @@ static LIST_HEAD(req_list);
static DECLARE_WORK(work, process_req, NULL);
static struct workqueue_struct *addr_wq;
-static int copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev,
-unsigned char *dst_dev_addr)
+int rdma_copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev,
+const unsigned char *dst_dev_addr)
{
switch (dev->type) {
case ARPHRD_INFINIBAND:
- dev_addr->dev_type = IB_NODE_CA;
+ dev_addr->dev_type = RDMA_NODE_IB_CA;
+ break;
+ case ARPHRD_ETHER:
+ dev_addr->dev_type = RDMA_NODE_RNIC;
break;
default:
return -EADDRNOTAVAIL;
@@ -77,6 +80,7 @@ static int copy_addr(struct rdma_dev_add
memcpy(dev_addr->dst_dev_addr, dst_dev_addr, MAX_ADDR_LEN);
return 0;
}
+EXPORT_SYMBOL(rdma_copy_addr);
int rdma_translate_ip(struct sockaddr *addr, struct rdma_dev_addr *dev_addr)
{
@@ -88,7 +92,7 @@ int rdma_translate_ip(struct sockaddr *a
if (!dev)
return -EADDRNOTAVAIL;
- ret = copy_addr(dev_addr, dev, NULL);
+ ret = rdma_copy_addr(dev_addr, dev, NULL);
dev_put(dev);
return ret;
}
@@ -160,7 +164,7 @@ static int addr_resolve_remote(struct so
/* If the device does ARP internally, return 'done' */
if (rt->idev->dev->flags & IFF_NOARP) {
- copy_addr(addr, rt->idev->dev, NULL);
+ rdma_copy_addr(addr, rt->idev->dev, NULL);
goto put;
}
@@ -180,7 +184,7 @@ static int addr_resolve_remote(struct so
src_in->sin_addr.s_addr = rt->rt_src;
}
- ret = copy_addr(addr, neigh->dev, neigh->ha);
+ ret = rdma_copy_addr(addr, neigh->dev, neigh->ha);
release:
neigh_release(neigh);
put:
@@ -244,7 +248,7 @@ static int addr_resolve_local(struct soc
if (ZERONET(src_ip)) {
src_in->sin_family = dst_in->sin_family;
src_in->sin_addr.s_addr = dst_ip;
- ret = copy_addr(addr, dev, dev->dev_addr);
+ ret = rdma_copy_addr(addr, dev, dev->dev_addr);
} else if (LOOPBACK(src_ip)) {
ret = rdma_translate_ip((struct sockaddr *)dst_in, addr);
if (!ret)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index e05ca2c..425a218 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -32,7 +32,7 @@
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
- * $Id: cache.c 1349 2004-12-16 21:09:43Z roland $
+ * $Id: cache.c 6885 2006-05-03 18:22:02Z sean.hefty $
*/
#include
@@ -62,12 +62,13 @@ struct ib_update_work {
static i
[openib-general] [PATCH v5 0/2] iWARP Core Support
With Seans comments resolved... I also changed iw_addr_get_dgid() to memcpy the gid like the other functions do. This patchset defines the modifications to the Linux infiniband subsystem to support iWARP devices. The patchset consists of 2 patches: 1 - New iWARP CM implementation. 2 - Core changes to support iWARP. Signed-off-by: Tom Tucker <[EMAIL PROTECTED]> Signed-off-by: Steve Wise <[EMAIL PROTECTED]> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Roland Dreier <[EMAIL PROTECTED]>: > Subject: Re: Fwd: issues in ipoib > > Michael> No, I think there's no problem - user callback has > Michael> finished running and that's all we care about. > > I don't think so -- because then the unregister call can return and > the client module free the client struct before the wake_up() runs, > which leads to use-after-free. > > Michael> But I'd like to point out that using rwsem gets all these > Michael> corners right automatically. Reconsider that approach? > > Again, I don't think so -- rwsem just hides the race from you. There > is a race with semaphores (access semaphore after waking up waiter) > which is the original reason for creating struct completion. Hmm, you are right. So I need to replace the wait queue with completion. Thanks. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Michael> No, I think there's no problem - user callback has Michael> finished running and that's all we care about. I don't think so -- because then the unregister call can return and the client module free the client struct before the wake_up() runs, which leads to use-after-free. Michael> But I'd like to point out that using rwsem gets all these Michael> corners right automatically. Reconsider that approach? Again, I don't think so -- rwsem just hides the race from you. There is a race with semaphores (access semaphore after waking up waiter) which is the original reason for creating struct completion. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH report] IB/ipoib fix flush/start xmit race (from code review)
Seems to work fine here. Pls consider for 2.6.18. --- Prevent flush task from freeing the ipoib_neigh pointer, while ipoib_start_xmit is accessing the ipoib_neigh through the pointer is has loaded from the hardware address. Signed-off-by: Michael S. Tsirkin <[EMAIL PROTECTED]> diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index cf71d2a..31c4b05 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -336,7 +336,8 @@ void ipoib_flush_paths(struct net_device struct ipoib_path *path, *tp; LIST_HEAD(remove_list); - spin_lock_irq(&priv->lock); + spin_lock_irq(&priv->tx_lock); + spin_lock(&priv->lock); list_splice(&priv->path_list, &remove_list); INIT_LIST_HEAD(&priv->path_list); @@ -352,7 +353,8 @@ void ipoib_flush_paths(struct net_device path_free(dev, path); spin_lock_irq(&priv->lock); } - spin_unlock_irq(&priv->lock); + spin_unlock(&priv->lock); + spin_unlock_irq(&priv->tx_lock); } static void path_rec_completion(int status, -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
Quoting r. Roland Dreier <[EMAIL PROTECTED]>: > Subject: Re: Fwd: issues in ipoib > > > + if (atomic_dec_and_test(&client->users)) > > + wake_up(&client->wait); > > > + wait_event(client->wait, atomic_read(&client->users) == 0); > > I think this is vulnerable to the race we fixed up all over the place > a few months ago, where wait_event runs between the > atomic_dec_and_test() and the wake_up(). No, I think there's no problem - user callback has finished running and that's all we care about. But I'd like to point out that using rwsem gets all these corners right automatically. Reconsider that approach? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v2 1/2] sa_query: add generic query interfaces capable of supporting RMPP
Sean Hefty wrote: > +int ib_sa_send_mad(struct ib_device *device, u8 port_num, > +int method, void *attr, int attr_id, > +ib_sa_comp_mask comp_mask, > +int timeout_ms, int retries, gfp_t gfp_mask, > +void (*callback)(int status, > + struct ib_mad_recv_wc *mad_recv_wc, > + void *context), > +void *context, struct ib_sa_query **query); Now that I've just updated and posted this... I think it makes sense for this call to take a packed attribute as input, versus an unpacked one. Attr_id can then be passed in as be16, which lets this interface, the callback, and it_sa_iter calls all deal with data in wire format. My longer term intent is to create a libibsa to support easier userspace SA interactions. It may be easier to handle packing / unpacking of attributes from userspace. Comments? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v4 2/2] iWARP Core Changes.
Roland/Sean,
I'll fix all these and retest, then resubmit...
Comments below...
Steve.
On Thu, 2006-08-03 at 11:11 -0700, Sean Hefty wrote:
> Steve Wise wrote:
> > diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
> > index d294bbc..83f84ef 100644
> > --- a/drivers/infiniband/core/addr.c
> > +++ b/drivers/infiniband/core/addr.c
> > @@ -32,6 +32,7 @@ #include
> > #include
> > #include
> > #include
> > +#include
>
> File is included 3 lines up.
>
Oops. I'll fix this.
> > diff --git a/drivers/infiniband/core/cache.c
> > b/drivers/infiniband/core/cache.c
> > index e05ca2c..061858c 100644
> > #include
> > #include
> > #include
> > -#include/* INIT_WORK, schedule_work(),
> > flush_scheduled_work() */
>
> I'm guessing that the include isn't currently needed, since none of the other
> changes to the file should have removed its dependency. Should this be put
> into
> a separate patch?
>
And this.
> > +static int iw_conn_req_handler(struct iw_cm_id *cm_id,
> > + struct iw_cm_event *iw_event)
> > +{
> > + struct rdma_cm_id *new_cm_id;
> > + struct rdma_id_private *listen_id, *conn_id;
> > + struct sockaddr_in *sin;
> > + struct net_device *dev = NULL;
> > + int ret;
> > +
> > + listen_id = cm_id->context;
> > + atomic_inc(&listen_id->dev_remove);
> > + if (!cma_comp(listen_id, CMA_LISTEN)) {
> > + ret = -ECONNABORTED;
> > + goto out;
> > + }
> > +
> > + /* Create a new RDMA id for the new IW CM ID */
> > + new_cm_id = rdma_create_id(listen_id->id.event_handler,
> > + listen_id->id.context,
> > + RDMA_PS_TCP);
> > + if (!new_cm_id) {
> > + ret = -ENOMEM;
> > + goto out;
> > + }
> > + conn_id = container_of(new_cm_id, struct rdma_id_private, id);
> > + atomic_inc(&conn_id->dev_remove);
>
> This is not released in error cases. See below.
>
> > + conn_id->state = CMA_CONNECT;
> > +
> > + dev = ip_dev_find(iw_event->local_addr.sin_addr.s_addr);
> > + if (!dev) {
> > + ret = -EADDRNOTAVAIL;
>
> cma_release_remove(conn_id);
>
> > + rdma_destroy_id(new_cm_id);
> > + goto out;
> > + }
> > + ret = rdma_copy_addr(&conn_id->id.route.addr.dev_addr, dev, NULL);
> > + if (ret) {
>
> cma_release_remove(conn_id);
>
> > + rdma_destroy_id(new_cm_id);
> > + goto out;
> > + }
> > +
> > + ret = cma_acquire_dev(conn_id);
> > + if (ret) {
>
> cma_release_remove(conn_id);
>
I'll fix these too.
> > + rdma_destroy_id(new_cm_id);
> > + goto out;
> > + }
> > +
> > + conn_id->cm_id.iw = cm_id;
> > + cm_id->context = conn_id;
> > + cm_id->cm_handler = cma_iw_handler;
> > +
> > + sin = (struct sockaddr_in *) &new_cm_id->route.addr.src_addr;
> > + *sin = iw_event->local_addr;
> > + sin = (struct sockaddr_in *) &new_cm_id->route.addr.dst_addr;
> > + *sin = iw_event->remote_addr;
> > +
> > + ret = cma_notify_user(conn_id, RDMA_CM_EVENT_CONNECT_REQUEST, 0,
> > + iw_event->private_data,
> > + iw_event->private_data_len);
> > + if (ret) {
> > + /* User wants to destroy the CM ID */
> > + conn_id->cm_id.iw = NULL;
> > + cma_exch(conn_id, CMA_DESTROYING);
> > + cma_release_remove(conn_id);
> > + rdma_destroy_id(&conn_id->id);
> > + }
> > +
> > +out:
> > + if (!dev)
> > + dev_put(dev);
>
> Shouldn't this be: if (dev)?
>
yup. This was added (incorrecty by me) during the review process...
> > + cma_release_remove(listen_id);
> > + return ret;
> > +}
> > @@ -1357,8 +1552,8 @@ static int cma_resolve_loopback(struct r
> > ib_addr_set_dgid(&id_priv->id.route.addr.dev_addr, &gid);
> >
> > if (cma_zero_addr(&id_priv->id.route.addr.src_addr)) {
> > - src_in = (struct sockaddr_in *)&id_priv->id.route.addr.src_addr;
> > - dst_in = (struct sockaddr_in *)&id_priv->id.route.addr.dst_addr;
> > + src_in = (struct sockaddr_in *)
> > &id_priv->id.route.addr.src_addr;
> > + dst_in = (struct sockaddr_in *)
> > &id_priv->id.route.addr.dst_addr;
>
> trivial spacing change only
>
right.
> > +static inline void iw_addr_get_sgid(struct rdma_dev_addr* rda,
> > + union ib_gid *gid)
> > +{
> > + memcpy(gid, rda->src_dev_addr, sizeof *gid);
> > +}
> > +
> > +static inline union ib_gid* iw_addr_get_dgid(struct rdma_dev_addr* rda)
> > +{
> > + return (union ib_gid *) rda->dst_dev_addr;
> > +}
>
> Minor personal nit: for consistency with the rest of the file, can you use
> dev_addr in place of rda?
>
okay.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v4 0/2][RFC] iWARP Core Support
On Thu, 2006-08-03 at 10:40 -0700, Roland Dreier wrote: > Steve> Here is the iWARP Core Support patchset merged to your > Steve> latest for-2.6.19 branch. It has gone through 3 reviews on > Steve> lklm and netdev a while ago, and I think its ready to be > Steve> pulled in. > > I agree. I'll read this over and queue it for 2.6.19 unless someone > objects. > > What's the status of the amasso driver? do you think that's ready to > merge too? > Yea, I'll re-merge it and post it soon. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v4 2/2] iWARP Core Changes.
Steve Wise wrote:
> diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
> index d294bbc..83f84ef 100644
> --- a/drivers/infiniband/core/addr.c
> +++ b/drivers/infiniband/core/addr.c
> @@ -32,6 +32,7 @@ #include
> #include
> #include
> #include
> +#include
File is included 3 lines up.
> diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
> index e05ca2c..061858c 100644
> #include
> #include
> #include
> -#include /* INIT_WORK, schedule_work(),
> flush_scheduled_work() */
I'm guessing that the include isn't currently needed, since none of the other
changes to the file should have removed its dependency. Should this be put
into
a separate patch?
> +static int iw_conn_req_handler(struct iw_cm_id *cm_id,
> +struct iw_cm_event *iw_event)
> +{
> + struct rdma_cm_id *new_cm_id;
> + struct rdma_id_private *listen_id, *conn_id;
> + struct sockaddr_in *sin;
> + struct net_device *dev = NULL;
> + int ret;
> +
> + listen_id = cm_id->context;
> + atomic_inc(&listen_id->dev_remove);
> + if (!cma_comp(listen_id, CMA_LISTEN)) {
> + ret = -ECONNABORTED;
> + goto out;
> + }
> +
> + /* Create a new RDMA id for the new IW CM ID */
> + new_cm_id = rdma_create_id(listen_id->id.event_handler,
> +listen_id->id.context,
> +RDMA_PS_TCP);
> + if (!new_cm_id) {
> + ret = -ENOMEM;
> + goto out;
> + }
> + conn_id = container_of(new_cm_id, struct rdma_id_private, id);
> + atomic_inc(&conn_id->dev_remove);
This is not released in error cases. See below.
> + conn_id->state = CMA_CONNECT;
> +
> + dev = ip_dev_find(iw_event->local_addr.sin_addr.s_addr);
> + if (!dev) {
> + ret = -EADDRNOTAVAIL;
cma_release_remove(conn_id);
> + rdma_destroy_id(new_cm_id);
> + goto out;
> + }
> + ret = rdma_copy_addr(&conn_id->id.route.addr.dev_addr, dev, NULL);
> + if (ret) {
cma_release_remove(conn_id);
> + rdma_destroy_id(new_cm_id);
> + goto out;
> + }
> +
> + ret = cma_acquire_dev(conn_id);
> + if (ret) {
cma_release_remove(conn_id);
> + rdma_destroy_id(new_cm_id);
> + goto out;
> + }
> +
> + conn_id->cm_id.iw = cm_id;
> + cm_id->context = conn_id;
> + cm_id->cm_handler = cma_iw_handler;
> +
> + sin = (struct sockaddr_in *) &new_cm_id->route.addr.src_addr;
> + *sin = iw_event->local_addr;
> + sin = (struct sockaddr_in *) &new_cm_id->route.addr.dst_addr;
> + *sin = iw_event->remote_addr;
> +
> + ret = cma_notify_user(conn_id, RDMA_CM_EVENT_CONNECT_REQUEST, 0,
> + iw_event->private_data,
> + iw_event->private_data_len);
> + if (ret) {
> + /* User wants to destroy the CM ID */
> + conn_id->cm_id.iw = NULL;
> + cma_exch(conn_id, CMA_DESTROYING);
> + cma_release_remove(conn_id);
> + rdma_destroy_id(&conn_id->id);
> + }
> +
> +out:
> + if (!dev)
> + dev_put(dev);
Shouldn't this be: if (dev)?
> + cma_release_remove(listen_id);
> + return ret;
> +}
> @@ -1357,8 +1552,8 @@ static int cma_resolve_loopback(struct r
> ib_addr_set_dgid(&id_priv->id.route.addr.dev_addr, &gid);
>
> if (cma_zero_addr(&id_priv->id.route.addr.src_addr)) {
> - src_in = (struct sockaddr_in *)&id_priv->id.route.addr.src_addr;
> - dst_in = (struct sockaddr_in *)&id_priv->id.route.addr.dst_addr;
> + src_in = (struct sockaddr_in *)
> &id_priv->id.route.addr.src_addr;
> + dst_in = (struct sockaddr_in *)
> &id_priv->id.route.addr.dst_addr;
trivial spacing change only
> +static inline void iw_addr_get_sgid(struct rdma_dev_addr* rda,
> + union ib_gid *gid)
> +{
> + memcpy(gid, rda->src_dev_addr, sizeof *gid);
> +}
> +
> +static inline union ib_gid* iw_addr_get_dgid(struct rdma_dev_addr* rda)
> +{
> + return (union ib_gid *) rda->dst_dev_addr;
> +}
Minor personal nit: for consistency with the rest of the file, can you use
dev_addr in place of rda?
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [GIT PULL] please pull infiniband.git
Linus, please pull from
master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus
This tree is also available from kernel.org mirrors at:
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git
for-linus
to get a few fixes:
Ishai Rabinovitz:
IB/srp: Fix crash in srp_reconnect_target
IB/srp: Work around data corruption bug on Mellanox targets
Jack Morgenstein:
IB/uverbs: Avoid a crash on device hot remove
Michael S. Tsirkin:
IB/mthca: Fix mthca_array_clear() thinko
Or Gerlitz:
IB/ipoib: Remove broken link from Kconfig and documentation
Roland Dreier:
IB/mthca: Clean up mthca array index mask
Sean Hefty:
IB/cm: Fix error handling in ib_send_cm_req
Documentation/infiniband/ipoib.txt|2 --
drivers/infiniband/core/cm.c |4 +++-
drivers/infiniband/core/uverbs.h |2 ++
drivers/infiniband/core/uverbs_main.c |8 +++-
drivers/infiniband/hw/mthca/mthca_allocator.c | 15 ---
drivers/infiniband/ulp/ipoib/Kconfig |3 +--
drivers/infiniband/ulp/srp/ib_srp.c | 19 +--
7 files changed, 38 insertions(+), 15 deletions(-)
diff --git a/Documentation/infiniband/ipoib.txt
b/Documentation/infiniband/ipoib.txt
index 1870355..864ff32 100644
--- a/Documentation/infiniband/ipoib.txt
+++ b/Documentation/infiniband/ipoib.txt
@@ -51,8 +51,6 @@ Debugging Information
References
- IETF IP over InfiniBand (ipoib) Working Group
-http://ietf.org/html.charters/ipoib-charter.html
Transmission of IP over InfiniBand (IPoIB) (RFC 4391)
http://ietf.org/rfc/rfc4391.txt
IP over InfiniBand (IPoIB) Architecture (RFC 4392)
diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
index f85c97f..0de335b 100644
--- a/drivers/infiniband/core/cm.c
+++ b/drivers/infiniband/core/cm.c
@@ -975,8 +975,10 @@ int ib_send_cm_req(struct ib_cm_id *cm_i
cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv->
id.local_id);
- if (IS_ERR(cm_id_priv->timewait_info))
+ if (IS_ERR(cm_id_priv->timewait_info)) {
+ ret = PTR_ERR(cm_id_priv->timewait_info);
goto out;
+ }
ret = cm_init_av_by_path(param->primary_path, &cm_id_priv->av);
if (ret)
diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index bb9bee5..102a59c 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -42,6 +42,7 @@ #define UVERBS_H
#include
#include
#include
+#include
#include
#include
@@ -69,6 +70,7 @@ #include
struct ib_uverbs_device {
struct kref ref;
+ struct completion comp;
int devnum;
struct cdev*dev;
struct class_device*class_dev;
diff --git a/drivers/infiniband/core/uverbs_main.c
b/drivers/infiniband/core/uverbs_main.c
index e725ccc..4e16314 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -122,7 +122,7 @@ static void ib_uverbs_release_dev(struct
struct ib_uverbs_device *dev =
container_of(ref, struct ib_uverbs_device, ref);
- kfree(dev);
+ complete(&dev->comp);
}
void ib_uverbs_release_ucq(struct ib_uverbs_file *file,
@@ -740,6 +740,7 @@ static void ib_uverbs_add_one(struct ib_
return;
kref_init(&uverbs_dev->ref);
+ init_completion(&uverbs_dev->comp);
spin_lock(&map_lock);
uverbs_dev->devnum = find_first_zero_bit(dev_map,
IB_UVERBS_MAX_DEVICES);
@@ -793,6 +794,8 @@ err_cdev:
err:
kref_put(&uverbs_dev->ref, ib_uverbs_release_dev);
+ wait_for_completion(&uverbs_dev->comp);
+ kfree(uverbs_dev);
return;
}
@@ -812,7 +815,10 @@ static void ib_uverbs_remove_one(struct
spin_unlock(&map_lock);
clear_bit(uverbs_dev->devnum, dev_map);
+
kref_put(&uverbs_dev->ref, ib_uverbs_release_dev);
+ wait_for_completion(&uverbs_dev->comp);
+ kfree(uverbs_dev);
}
static int uverbs_event_get_sb(struct file_system_type *fs_type, int flags,
diff --git a/drivers/infiniband/hw/mthca/mthca_allocator.c
b/drivers/infiniband/hw/mthca/mthca_allocator.c
index 9ba3211..25157f5 100644
--- a/drivers/infiniband/hw/mthca/mthca_allocator.c
+++ b/drivers/infiniband/hw/mthca/mthca_allocator.c
@@ -108,14 +108,15 @@ void mthca_alloc_cleanup(struct mthca_al
* serialize access to the array.
*/
+#define MTHCA_ARRAY_MASK (PAGE_SIZE / sizeof (void *) - 1)
+
void *mthca_array_get(struct mthca_array *array, int index)
{
int p = (index * sizeof (void *)) >> PAGE_SHIFT;
- if (array->page_list[p].page) {
- int i = index & (PAGE_SIZE / sizeof (void *) - 1);
Re: [openib-general] hotplug support in mthca
OK, I applied this for now. Let's try to revisit the revoke method later though. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fwd: issues in ipoib
> +if (atomic_dec_and_test(&client->users)) > +wake_up(&client->wait); > +wait_event(client->wait, atomic_read(&client->users) == 0); I think this is vulnerable to the race we fixed up all over the place a few months ago, where wait_event runs between the atomic_dec_and_test() and the wake_up(). - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/ipoib: remove broken link from Kconfig and documentation
thanks, applied ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v2 2/2] local_sa: use SA iterator routines to walk attributes in RMPP response
Convert local SA to use the new SA iterator routines for walking a
list of attributes in an RMPP response returned by the SA. This
replaces a local SA specific implementation.
Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
---
Index: core/local_sa.c
===
--- core/local_sa.c (revision 8647)
+++ core/local_sa.c (working copy)
@@ -107,16 +107,6 @@ struct sa_db_device {
struct sa_db_port port[0];
};
-/* Define path record format to enable needed checks against MAD data. */
-struct ib_path_rec {
- u8 reserved[8];
- u8 dgid[16];
- u8 sgid[16];
- __be16 dlid;
- __be16 slid;
- u8 reserved2[20];
-};
-
struct ib_sa_cursor {
struct ib_sa_cursor *next;
};
@@ -194,60 +184,27 @@ static int insert_attr(struct index_root
static void update_path_rec(struct sa_db_port *port,
struct ib_mad_recv_wc *mad_recv_wc)
{
- struct ib_mad_recv_buf *recv_buf;
- struct ib_sa_mad *mad = (void *) mad_recv_wc->recv_buf.mad;
+ struct ib_sa_iter *iter;
struct ib_path_rec_info *path_info;
- struct ib_path_rec ib_path, *path = NULL;
- int i, attr_size, left, offset = 0;
+ void *attr;
- attr_size = be16_to_cpu(mad->sa_hdr.attr_offset) * 8;
- if (attr_size < sizeof ib_path)
+ iter = ib_sa_iter_create(mad_recv_wc);
+ if (IS_ERR(iter))
return;
down_write(&lock);
port->update++;
- list_for_each_entry(recv_buf, &mad_recv_wc->rmpp_list, list) {
- for (i = 0; i < IB_MGMT_SA_DATA;) {
- mad = (struct ib_sa_mad *) recv_buf->mad;
-
- left = IB_MGMT_SA_DATA - i;
- if (left < sizeof ib_path) {
- /* copy first piece of the attribute */
- memcpy(&ib_path, &mad->data[i], left);
- path = &ib_path;
- offset = left;
- break;
- } else if (offset) {
- /* copy the second piece of the attribute */
- memcpy((void*) path + offset, &mad->data[i],
- sizeof ib_path - offset);
- i += attr_size - offset;
- offset = 0;
- } else {
- path = (void *) &mad->data[i];
- i += attr_size;
- }
-
- if (!path->slid)
- goto unlock;
-
- path_info = kmalloc(sizeof *path_info, GFP_KERNEL);
- if (!path_info)
- goto unlock;
-
- ib_sa_unpack_attr(&path_info->rec, path,
- IB_SA_ATTR_PATH_REC);
-
- if (insert_attr(&port->index, port->update,
- path_info->rec.dgid.raw,
- &path_info->cursor)) {
- kfree(path_info);
- goto unlock;
- }
- }
+ while ((attr = ib_sa_iter_next(iter)) &&
+ (path_info = kmalloc(sizeof *path_info, GFP_KERNEL))) {
+
+ ib_sa_unpack_attr(&path_info->rec, attr, IB_SA_ATTR_PATH_REC);
+ if (insert_attr(&port->index, port->update,
+ path_info->rec.dgid.raw,
+ &path_info->cursor))
+ break;
}
-unlock:
up_write(&lock);
+ ib_sa_iter_free(iter);
}
static void recv_handler(struct ib_mad_agent *mad_agent,
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v2 1/2] sa_query: add generic query interfaces capable of supporting RMPP
The following patch adds a generic interface to send MADs to the SA.
The primary motivation of adding these calls is to expand the SA query
interface to include RMPP responses for users wanting more than a
single attribute returned from a query (e.g. multipath record queries).
The implementation of existing SA query routines was layered on top of
the generic query interface.
Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
---
Notes from v1:
- "cursor" was renamed to "iter"
- ib_sa_iter_next() returns a pointer to the packed attribute
- The interface turned out being easier to use having a single function
call, ib_sa_iter_next(), to walk through the attributes.
Index: include/rdma/ib_sa.h
===
--- include/rdma/ib_sa.h(revision 8647)
+++ include/rdma/ib_sa.h(working copy)
@@ -254,6 +254,71 @@ struct ib_sa_query;
void ib_sa_cancel_query(int id, struct ib_sa_query *query);
+struct ib_sa_iter;
+
+/**
+ * ib_sa_iter_create - Create an iterator that may be used to walk through
+ * a list of returned SA records.
+ * @mad_recv_wc: A received response from the SA.
+ *
+ * This call allocates an iterator that is used to walk through a list of
+ * SA records. Users must free the iterator by calling ib_sa_iter_free.
+ */
+struct ib_sa_iter *ib_sa_iter_create(struct ib_mad_recv_wc *mad_recv_wc);
+
+/**
+ * ib_sa_iter_free - Release an iterator.
+ * @iter: The iterator to free.
+ */
+void ib_sa_iter_free(struct ib_sa_iter *iter);
+
+/**
+ * ib_sa_iter_next - Move an iterator to reference the next attribute and
+ * return the attribute.
+ * @iter: The iterator to move.
+ *
+ * The referenced attribute will be in wire format. The funtion returns NULL
+ * if there are no more attributes to return.
+ */
+void *ib_sa_iter_next(struct ib_sa_iter *iter);
+
+/**
+ * ib_sa_send_mad - Send a MAD to the SA.
+ * @device:device to send query on
+ * @port_num: port number to send query on
+ * @method:MAD method to use in the send.
+ * @attr:Reference to attribute to send in MAD.
+ * @attr_id:Attribute type identifier.
+ * @comp_mask:component mask to send in MAD
+ * @timeout_ms:time to wait for response, if one is expected
+ * @retries:number of times to retry request
+ * @gfp_mask:GFP mask to use for internal allocations
+ * @callback:function called when query completes, times out or is
+ * canceled
+ * @context:opaque user context passed to callback
+ * @sa_query:query context, used to cancel query
+ *
+ * Send a message to the SA. If a response is expected (timeout_ms is
+ * non-zero), the callback function will be called when the query completes.
+ * Status is 0 for a successful response, -EINTR if the query
+ * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error
+ * occurred sending the query. Mad_recv_wc will reference any returned
+ * response from the SA. It is the responsibility of the caller to free
+ * mad_recv_wc by call ib_free_recv_mad() if it is non-NULL.
+ *
+ * If the return value of ib_sa_send_mad() is negative, it is an
+ * error code. Otherwise it is a query ID that can be used to cancel
+ * the query.
+ */
+int ib_sa_send_mad(struct ib_device *device, u8 port_num,
+ int method, void *attr, int attr_id,
+ ib_sa_comp_mask comp_mask,
+ int timeout_ms, int retries, gfp_t gfp_mask,
+ void (*callback)(int status,
+ struct ib_mad_recv_wc *mad_recv_wc,
+ void *context),
+ void *context, struct ib_sa_query **query);
+
int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
struct ib_sa_path_rec *rec,
ib_sa_comp_mask comp_mask,
Index: core/sa_query.c
===
--- core/sa_query.c (revision 8647)
+++ core/sa_query.c (working copy)
@@ -72,30 +72,42 @@ struct ib_sa_device {
};
struct ib_sa_query {
- void (*callback)(struct ib_sa_query *, int, struct ib_sa_mad *);
- void (*release)(struct ib_sa_query *);
+ void (*callback)(int, struct ib_mad_recv_wc *, void *);
struct ib_sa_port *port;
struct ib_mad_send_buf *mad_buf;
struct ib_sa_sm_ah *sm_ah;
+ void *context;
int id;
};
struct ib_sa_service_query {
void (*callback)(int, struct ib_sa_service_rec *, void *);
void *context;
- struct ib_sa_query sa_query;
+ struct ib_sa_query *sa_query;
};
struct ib_sa_path_query {
void (*callback)(int, struct ib_sa_path_rec *, void *);
void *context;
- struct ib_sa_query sa_query;
+ struct ib_sa_query *sa_query;
};
struct ib_sa_mcmember_query {
void (*callback)(int, struct ib_sa_mcmember_rec *, void *);
void *context;
- struct ib_sa_query sa_query;
+ struct ib_sa_qu
Re: [openib-general] [PATCH v4 0/2][RFC] iWARP Core Support
Steve> Here is the iWARP Core Support patchset merged to your Steve> latest for-2.6.19 branch. It has gone through 3 reviews on Steve> lklm and netdev a while ago, and I think its ready to be Steve> pulled in. I agree. I'll read this over and queue it for 2.6.19 unless someone objects. What's the status of the amasso driver? do you think that's ready to merge too? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] RFC: srp filesystem data corruption problem/work-around
Thanks, applied. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH/RFC] libibverbs and libmthca fork support
On Thu, Aug 03, 2006 at 09:40:09AM -0700, Roland Dreier wrote:
> Gleb> We can also provide environment variable to control
> Gleb> libibverbs behaviour. This way if programmer made a wrong
> Gleb> assumption user will be able to fix it.
>
> OK, I added this little snippet:
>
> if (getenv("RDMAV_FORK_SAFE") || getenv("IBV_FORK_SAFE"))
> if (ibv_fork_init())
> fprintf(stderr, PFX "Warning: fork()-safety requested "
> "but init failed\n");
>
> so that setting either RDMAV_FORK_SAFE or IBV_FORK_SAFE environment
> variables forces the library to try to be fork()-safe.
>
Looks good to me. Thanks.
--
Gleb.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH repost] libmthca: fix compilation on SLES10
Thanks, applied. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH repost] libmthca: stricter checks in mthca_create_srq
thanks, applied ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v3 0/6] Tranport Neutral Verbs Proposal.
On Thu, 3 Aug 2006, Krishna Kumar wrote: > 2. Changed rdma_ to rdmav_. This also enabled to retain rdma_create_qp() and >rdma_destroy_qp() routines. I'm glad to see that the generic RDMA verbs are becoming a reality. Exporting QP create/destroy functions from both the RDMA CM library and RDMA verbs library is going to confusion new developers. The names, rdmav_create_qp() and rdma_create_qp(), only differ by 1 character and their arguments only differ by 1 parameter (the RDMA CM's version takes a cma id). Are the rdmav_ versions intended to be generic or are they intended for use with the native communications managers (IB CM and iWARP CM)? Is there a way that the differences could be made clearer? Could one be eliminated? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH/RFC] libibverbs and libmthca fork support
Gleb> We can also provide environment variable to control
Gleb> libibverbs behaviour. This way if programmer made a wrong
Gleb> assumption user will be able to fix it.
OK, I added this little snippet:
if (getenv("RDMAV_FORK_SAFE") || getenv("IBV_FORK_SAFE"))
if (ibv_fork_init())
fprintf(stderr, PFX "Warning: fork()-safety requested "
"but init failed\n");
so that setting either RDMAV_FORK_SAFE or IBV_FORK_SAFE environment
variables forces the library to try to be fork()-safe.
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [libibcm] does the libibcm support multithreaded applications?
Dotan Barak wrote: > I'm trying to use the libibcm in a multithreaded test and i get weird > failures (instead of RTU event i get a DREQ event). This is possible sequence. If the RTU is lost, or the connecting client aborts before sending the RTU, a DREQ can occur. > Does the libibcm supports multi threading applications? > (every thread have it's own CM device and each one of them listen is > using a different service ID) There's nothing (other than an unknown bug) that should prevent a multi-threaded application from running. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RDMA_CM_EVENT_UNREACHABLE(-ETIMEDOUT)
Or Gerlitz wrote: > Is it correct that with the gen2 code, the remote **CM** will reconnect > on that case? I don't think so. The QP needs to move into timewait, so a new connection request is needed with a different QPN. > I see in cm.c :: cm_rej_handler() that when the state is IB_CM_REQ_SENT > and the reject reason is IB_CM_REJ_STALE_CONN you just move the cm_id > into timewait state, which will cause a retry on the REQ, correct? The cm_id moves into timewait, but that shouldn't cause a retry. The CM should notify the ULP of the reject. The QP cannot be re-used until the cm_id exits the timewait state. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] rdma cm process hang
On Wed, 2006-08-02 at 11:57 -0400, Pete Wyckoff wrote: > [EMAIL PROTECTED] wrote on Wed, 02 Aug 2006 10:09 -0500: > > This hang is due to 2 things: > > > > 1) the amso card will _never_ timeout a connection that is awaiting an > > MP reply. That is exactly what is happening here. The fix for this > > (timeout mpa connection setup stalls) is a firmware fix and we don't > > have the firmware src. > > > > 2) the IWCM holds a reference on the QP until connection setup either > > succeeds or fails. So that's where we get the stall. The amso driver > > is waiting for the reference on the qp to go to zero, and it never will > > because the amso firmware will never timeout the stalled mpa connection > > setup. > > > > Lemme look more at the amso driver and see if this can be avoided. > > Perhaps the amso driver can blow away the qp and stop the stall. I > > thought thats what it did, but I'll look... > > Thanks for looking. I'd just come to the conclusion that it was > waiting on the qp refcnt, but didn't get much farther when your mail > arrived. > I don't know when, or if I'll have time to address this limitation in the ammasso firmware. But there is a way (if anyone wants to implement it): 1) add a timer to the c2_qp struct and start it when c2_llp_connect() is called. 2) if the timer fires, generate a CONNECT_REPLY upcall to the IWCM with status TIMEDOUT. Mark in the qp that the connect timed out. 3) deal with the rare condition that the timer fires at or about the same time the connection really does get established: if the adapter passes up a CCAE_ACTIVE_CONNECT_RESULTS -after- the timer fires but before the qp is destroyed by the consumer, then you must squelch this event and probably destroy the HWQP at least from the adapter's perspective... > Testing on mthca would be a bit more difficult here, but hopefully > that's not an issue now. There's no need. This is an Ammaso-only issue. Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [libibcm] linking object files with the libibcm failed on latest driver
please ignore this email, this failure occured because of internal code modifications. thanks Dotan On Thursday 03 August 2006 12:15, Dotan Barak wrote: > Hi. > > I tried to add a (basic) support of libibcm to one of our tests and i got > linking problems with latest driver. > (several days ago, i didn't get this error). > > Here are the machine and driver props: > * > Host Name : sw087 > Host Architecture : x86_64 > Linux Distribution: Fedora Core release 4 (Stentz) > Kernel Version: 2.6.11-1.1369_FC4smp > Memory size : 4071672 kB > Driver Version: gen2_linux-20060803-0800 (REV=8813) > HCA ID(s) : mthca0 > HCA model(s) : 23108 > FW version(s) : 3.4.000 > Board(s) : MT_003001 > * > > Here are the compilation errors: > > gcc main.o cmd_pars.o common.o qp_test.o connect_qp.o data_operation.o -o > qp_test -libverbs -libcm -lvl > /usr/local/lib64/libibcm.so: undefined reference to `_dlist_mark_move' > /usr/local/lib64/libibcm.so: undefined reference to `dlist_destroy' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_get_mnt_path' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_close_class_device' > /usr/local/lib64/libibcm.so: undefined reference to `dlist_push' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_open_class' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_get_classdev_attr' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_read_attribute' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_open_class_device' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_get_class_devices' > /usr/local/lib64/libibcm.so: undefined reference to `dlist_start' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_close_attribute' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_close_class' > /usr/local/lib64/libibcm.so: undefined reference to `dlist_new' > /usr/local/lib64/libibcm.so: undefined reference to `sysfs_open_attribute' > collect2: ld returned 1 exit status > make: *** [qp_test] Error 1 > > > Did you notice this error before? > Thanks > Dotan > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH/RFC] libibverbs and libmthca fork support
On Wed, Aug 02, 2006 at 08:08:48AM -0700, Roland Dreier wrote: > > Roland> But one last call for comments: in particular, does anyone > Roland> object to libibverbs being fork-unsafe by default unless > Roland> ibv_fork_init is called? > > Gleb> I am not sure about this one. The library like MPI will have > Gleb> to always call this anyway. On the other side if some > Gleb> library calls fork() without application knowing it? > Gleb> Suddenly programmer should care about such details. Perhaps > Gleb> opt out is better the opt in and libibverbs should skip > Gleb> ibv_fork_init() only if application ask this explicitly? > > But then what should libibverbs do on a kernel that doesn't support > the required madvise(MADV_DONTFORK) call? > If your kernel doesn't support MADV_DONTFORK then you are SOL and I think that what you are doing now (disabling it) is perfect. It should be clearly documented starting from what kernel version fork() is supported. > It's fine if MPI calls ibv_fork_init() always -- at least then it has > a hint about whether fork() will work. There is nothing MPI can do with this hint. It is not going to use fork by itself but because library never knows what application is going to do it will have to call ibv_fork_init() just in case. > > Do you know if there are libraries that call fork()? > Honestly I can think of anything right now :) But I haven't looked at enough of them. By the way in multi threaded program simple system() from inside the library will be enough to screw up. We can also provide environment variable to control libibverbs behaviour. This way if programmer made a wrong assumption user will be able to fix it. -- Gleb. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] some issues related to when/why IPoIB calls netif_carrier_on() etc
Roland Dreier wrote: > > 1) what is the exact reason that ib0 is running here, is it as of this > >"magic" configuration of the IPv6 addr that caused it to join to > >the IPv4 and IPv6 broascast groups? > > No, ipv6 autoconf has nothing to do with it. I think it's because you > did ifconfig ib0 up, which called ipoib_open(), which calls > ipoib_ib_dev_up(), which joins the ipv4 broadcast group. > > Bringing the interface up the starts ipv6 autoconf but that is just a > side issue. You could build a kernel without ipv6 and see what happens. I have verified your assumption to be correct, building the kernel without ipv6 support and doing ifconfig up ib0 i see that the port associated with ib0 joined the ipv4 broadcast group and ib0 is in "running" state. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [libibcm] does the libibcm support multithreaded applications?
Title: Message Hi. I'm trying to use the libibcm in a multithreaded test and i get weird failures (instead of RTU event i get a DREQ event). Does the libibcm supports multi threading applications? (every thread have it's own CM device and each one of them listen is using a different service ID) thanks Dotan Barak Software Verification Engineer Mellanox Technologies Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245 P.O. Box 86 Yokneam 20692 ISRAEL. Home: +972-77-8841095 Cell: 052-4222383 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [libibcm] linking object files with the libibcm failed on latest driver
Hi. I tried to add a (basic) support of libibcm to one of our tests and i got linking problems with latest driver. (several days ago, i didn't get this error). Here are the machine and driver props: * Host Name : sw087 Host Architecture : x86_64 Linux Distribution: Fedora Core release 4 (Stentz) Kernel Version: 2.6.11-1.1369_FC4smp Memory size : 4071672 kB Driver Version: gen2_linux-20060803-0800 (REV=8813) HCA ID(s) : mthca0 HCA model(s) : 23108 FW version(s) : 3.4.000 Board(s) : MT_003001 * Here are the compilation errors: gcc main.o cmd_pars.o common.o qp_test.o connect_qp.o data_operation.o -o qp_test -libverbs -libcm -lvl /usr/local/lib64/libibcm.so: undefined reference to `_dlist_mark_move' /usr/local/lib64/libibcm.so: undefined reference to `dlist_destroy' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_get_mnt_path' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_close_class_device' /usr/local/lib64/libibcm.so: undefined reference to `dlist_push' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_open_class' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_get_classdev_attr' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_read_attribute' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_open_class_device' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_get_class_devices' /usr/local/lib64/libibcm.so: undefined reference to `dlist_start' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_close_attribute' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_close_class' /usr/local/lib64/libibcm.so: undefined reference to `dlist_new' /usr/local/lib64/libibcm.so: undefined reference to `sysfs_open_attribute' collect2: ld returned 1 exit status make: *** [qp_test] Error 1 Did you notice this error before? Thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [openfabrics-ewg] Multicast traffic performace of OFED 1.0 ipoib
Mike, On 8/2/06, Michael Krause <[EMAIL PROTECTED]> wrote: > > > Is the performance being measured on an identical topology and hardware set > as before? Multicast by its very nature is sensitive to topology, hardware > components used (buffer depth, latency, etc.) and workload occurring within > the fabric. Loss occurs as a function of congestion or lack of forward > progress resulting in a timeout and thus a toss of a packet. If the > hardware is different or the settings chosen are changed, then the results > would be expected to change. > > It is not clear what you hope to achieve with such tests as there will be > other workloads flowing over the fabric which will create random HOL > blocking which can result in packet loss. Multicast workloads should be > tolerant of such loss. > > Mike I'm sorry about not beeing clear. My intention in the last sentance was that we got the better (120k-140k PPS) results with our proprietary IB stack and not with a previous openib snapshot. The tests were run on the same setup, which by the way was dedicated only to that traffic. I' m aware of the network implications of the test, I was looking for hints of improvements needed in the ipoib implementation. -- Moni > > > > > At 04:30 AM 8/2/2006, Moni Levy wrote: > > Hi, > we are doing some performance testing of multicast traffic over > ipoib. The tests are performed by using iperf on dual 1.6G AMD PCI-X > servers with PCI-X Tavor cards with 3.4.FW. Below are the command the > may be used to run the test. > > Iperf server: > route add -net 224.0.0.0 netmask 240.0.0.0 dev ib0 > /home/qa/testing-tools/iperf-2.0.2/iperf -us -B 224.4.4.4 -i 1 > > Iperf client: > route add -net 224.0.0.0 netmask 240.0.0.0 dev ib0 > /home/qa/testing-tools/iperf-2.0.2/iperf -uc 224.4.4.4 -i 1 -b 100M -t > 400 -l 100 > > We are looking for the max PPT rate (100 byte packets size) without > losses, by changing the BW parameter and looking at the point where we > get no losses reported. The best results we received were around 50k > PPS. I remember that we got some 120k-140k packets of the same size > running without losses. > > We are going to look into it and try to see where is the time spent, > but any ideas are welcome. > > Best regards, > Moni > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > > > > At 04:30 AM 8/2/2006, Moni Levy wrote: > > Hi, > we are doing some performance testing of multicast traffic over > ipoib. The tests are performed by using iperf on dual 1.6G AMD PCI-X > servers with PCI-X Tavor cards with 3.4.FW. Below are the command the > may be used to run the test. > > Iperf server: > route add -net 224.0.0.0 netmask 240.0.0.0 dev ib0 > /home/qa/testing-tools/iperf-2.0.2/iperf -us -B 224.4.4.4 -i 1 > > Iperf client: > route add -net 224.0.0.0 netmask 240.0.0.0 dev ib0 > /home/qa/testing-tools/iperf-2.0.2/iperf -uc 224.4.4.4 -i 1 -b 100M -t > 400 -l 100 > > We are looking for the max PPT rate (100 byte packets size) without > losses, by changing the BW parameter and looking at the point where we > get no losses reported. The best results we received were around 50k > PPS. I remember that we got some 120k-140k packets of the same size > running without losses. > > We are going to look into it and try to see where is the time spent, > but any ideas are welcome. > > Best regards, > Moni > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > ___ > openfabrics-ewg mailing list > [EMAIL PROTECTED] > http://openib.org/mailman/listinfo/openfabrics-ewg > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v3 3/6] libibverbs configuration files changes.
Configuration/Makefiles to build libibverbs with the new API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/libibverbs/Makefile.am NEW/libibverbs/Makefile.am
--- ORG/libibverbs/Makefile.am 2006-07-30 21:18:16.0 -0700
+++ NEW/libibverbs/Makefile.am 2006-08-03 17:15:33.0 -0700
@@ -9,7 +9,7 @@ AM_CFLAGS = -g -Wall -D_GNU_SOURCE
src_libibverbs_la_CFLAGS = -g -Wall -D_GNU_SOURCE
-DDRIVER_PATH=\"$(libdir)/infiniband\"
if HAVE_LD_VERSION_SCRIPT
-libibverbs_version_script =
-Wl,--version-script=$(srcdir)/src/libibverbs.map
+libibverbs_version_script =
-Wl,--version-script=$(srcdir)/src/librdmaverbs.map
else
libibverbs_version_script =
endif
@@ -18,7 +18,7 @@ src_libibverbs_la_SOURCES = src/cmd.c sr
src/memory.c src/sysfs.c src/verbs.c
src_libibverbs_la_LDFLAGS = -version-info 2 -export-dynamic \
$(libibverbs_version_script)
-src_libibverbs_la_DEPENDENCIES = $(srcdir)/src/libibverbs.map
+src_libibverbs_la_DEPENDENCIES = $(srcdir)/src/librdmaverbs.map
bin_PROGRAMS = examples/ibv_devices examples/ibv_devinfo \
examples/ibv_asyncwatch examples/ibv_rc_pingpong examples/ibv_uc_pingpong \
@@ -42,7 +42,8 @@ libibverbsincludedir = $(includedir)/inf
libibverbsinclude_HEADERS = include/infiniband/arch.h
include/infiniband/driver.h \
include/infiniband/kern-abi.h include/infiniband/opcode.h
include/infiniband/verbs.h \
-include/infiniband/sa-kern-abi.h include/infiniband/sa.h
include/infiniband/marshall.h
+include/infiniband/sa-kern-abi.h include/infiniband/sa.h
include/infiniband/marshall.h \
+include/infiniband/deprecate.h
man_MANS = man/ibv_asyncwatch.1 man/ibv_devices.1 man/ibv_devinfo.1 \
man/ibv_rc_pingpong.1 man/ibv_uc_pingpong.1 man/ibv_ud_pingpong.1 \
@@ -56,8 +57,9 @@ DEBIAN = debian/changelog debian/compat
EXTRA_DIST = include/infiniband/driver.h include/infiniband/kern-abi.h \
include/infiniband/opcode.h include/infiniband/verbs.h
include/infiniband/marshall.h \
include/infiniband/sa-kern-abi.h include/infiniband/sa.h \
-src/ibverbs.h examples/pingpong.h \
-src/libibverbs.map libibverbs.spec.in $(man_MANS)
+include/infiniband/deprecate.h \
+src/rdmaverbs.h examples/pingpong.h \
+src/librdmaverbs.map libibverbs.spec.in $(man_MANS)
dist-hook: libibverbs.spec
cp libibverbs.spec $(distdir)
diff -ruNp ORG/libibverbs/configure.in NEW/libibverbs/configure.in
--- ORG/libibverbs/configure.in 2006-07-30 21:18:16.0 -0700
+++ NEW/libibverbs/configure.in 2006-08-02 18:24:49.0 -0700
@@ -2,7 +2,7 @@ dnl Process this file with autoconf to p
AC_PREREQ(2.57)
AC_INIT(libibverbs, 1.1-pre1, openib-general@openib.org)
-AC_CONFIG_SRCDIR([src/ibverbs.h])
+AC_CONFIG_SRCDIR([src/rdmaverbs.h])
AC_CONFIG_AUX_DIR(config)
AM_CONFIG_HEADER(config.h)
AM_INIT_AUTOMAKE(libibverbs, 1.1-pre1)
@@ -33,5 +33,5 @@ AC_CACHE_CHECK(whether ld accepts --vers
AM_CONDITIONAL(HAVE_LD_VERSION_SCRIPT, test "$ac_cv_version_script" = "yes")
-AC_CONFIG_FILES([Makefile libibverbs.spec])
+AC_CONFIG_FILES([Makefile librdmaverbs.spec])
AC_OUTPUT
diff -ruNp ORG/libibverbs/libibverbs.spec.in NEW/libibverbs/libibverbs.spec.in
--- ORG/libibverbs/libibverbs.spec.in 2006-07-30 21:18:16.0 -0700
+++ NEW/libibverbs/libibverbs.spec.in 1969-12-31 16:00:00.0 -0800
@@ -1,106 +0,0 @@
-# $Id: libibverbs.spec.in 7484 2006-05-24 21:12:21Z roland $
-
-%define ver @VERSION@
-
-Name: libibverbs
-Version: 1.1
-Release: 0.1.pre1%{?dist}
-Summary: A library for direct userspace use of InfiniBand
-
-Group: System Environment/Libraries
-License: GPL/BSD
-Url: http://openib.org/
-Source: http://openib.org/downloads/libibverbs-1.1-pre1.tar.gz
-BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
-
-%description
-libibverbs is a library that allows userspace processes to use
-InfiniBand "verbs" as described in the InfiniBand Architecture
-Specification. This includes direct hardware access for fast path
-operations.
-
-For this library to be useful, a device-specific plug-in module should
-also be installed.
-
-%package devel
-Summary: Development files for the libibverbs library
-Group: System Environment/Libraries
-
-%description devel
-Static libraries and header files for the libibverbs verbs library.
-
-%package utils
-Summary: Examples for the libibverbs library
-Group: System Environment/Libraries
-Requires: %{name} = %{version}-%{release}
-
-%description utils
-Useful libibverbs1 example programs such as ibv_devinfo, which
-displays information about InfiniBand devices.
-
-%prep
-%setup -q -n %{name}-%{ver}
-
-%build
-%configure
-make %{?_smp_mflags}
-
-%install
-rm -rf $RPM_BUILD_ROOT
-%makeinstall
-# remove unpackaged files from the buildroot
-rm -f $RPM_BUILD_ROOT%{_libdir}/*.la
-
-%clean
-rm -rf $RPM_BUILD_ROOT
-
-%post -p /sbin/ldconfig
-%postun -p /sbin/ldconfig
-
-%files
-%defattr(-,root,root,-)
-%{_libdir}/libibverbs*.so.*
-%doc AU
[openib-general] [PATCH v3 6/6] librdmacm examples changes.
Convert librdmacm examples to use the new API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/librdmacm/examples/cmatose.c NEW/librdmacm/examples/cmatose.c
--- ORG/librdmacm/examples/cmatose.c2006-07-30 21:18:17.0 -0700
+++ NEW/librdmacm/examples/cmatose.c2006-08-03 17:32:36.0 -0700
@@ -62,9 +62,9 @@ struct cmatest_node {
int id;
struct rdma_cm_id *cma_id;
int connected;
- struct ibv_pd *pd;
- struct ibv_cq *cq;
- struct ibv_mr *mr;
+ struct rdmav_pd *pd;
+ struct rdmav_cq *cq;
+ struct rdmav_mr *mr;
void*mem;
};
@@ -100,8 +100,8 @@ static int create_message(struct cmatest
printf("failed message allocation\n");
return -1;
}
- node->mr = ibv_reg_mr(node->pd, node->mem, message_size,
-IBV_ACCESS_LOCAL_WRITE);
+ node->mr = rdmav_reg_mr(node->pd, node->mem, message_size,
+RDMAV_ACCESS_LOCAL_WRITE);
if (!node->mr) {
printf("failed to reg MR\n");
goto err;
@@ -114,10 +114,10 @@ err:
static int init_node(struct cmatest_node *node)
{
- struct ibv_qp_init_attr init_qp_attr;
+ struct rdmav_qp_init_attr init_qp_attr;
int cqe, ret;
- node->pd = ibv_alloc_pd(node->cma_id->verbs);
+ node->pd = rdmav_alloc_pd(node->cma_id->verbs);
if (!node->pd) {
ret = -ENOMEM;
printf("cmatose: unable to allocate PD\n");
@@ -125,7 +125,7 @@ static int init_node(struct cmatest_node
}
cqe = message_count ? message_count * 2 : 2;
- node->cq = ibv_create_cq(node->cma_id->verbs, cqe, node, 0, 0);
+ node->cq = rdmav_create_cq(node->cma_id->verbs, cqe, node, 0, 0);
if (!node->cq) {
ret = -ENOMEM;
printf("cmatose: unable to create CQ\n");
@@ -139,7 +139,7 @@ static int init_node(struct cmatest_node
init_qp_attr.cap.max_recv_sge = 1;
init_qp_attr.qp_context = node;
init_qp_attr.sq_sig_all = 1;
- init_qp_attr.qp_type = IBV_QPT_RC;
+ init_qp_attr.qp_type = RDMAV_QPT_RC;
init_qp_attr.send_cq = node->cq;
init_qp_attr.recv_cq = node->cq;
ret = rdma_create_qp(node->cma_id, node->pd, &init_qp_attr);
@@ -159,8 +159,8 @@ out:
static int post_recvs(struct cmatest_node *node)
{
- struct ibv_recv_wr recv_wr, *recv_failure;
- struct ibv_sge sge;
+ struct rdmav_recv_wr recv_wr, *recv_failure;
+ struct rdmav_sge sge;
int i, ret = 0;
if (!message_count)
@@ -176,7 +176,7 @@ static int post_recvs(struct cmatest_nod
sge.addr = (uintptr_t) node->mem;
for (i = 0; i < message_count && !ret; i++ ) {
- ret = ibv_post_recv(node->cma_id->qp, &recv_wr, &recv_failure);
+ ret = rdmav_post_recv(node->cma_id->qp, &recv_wr,
&recv_failure);
if (ret) {
printf("failed to post receives: %d\n", ret);
break;
@@ -187,8 +187,8 @@ static int post_recvs(struct cmatest_nod
static int post_sends(struct cmatest_node *node)
{
- struct ibv_send_wr send_wr, *bad_send_wr;
- struct ibv_sge sge;
+ struct rdmav_send_wr send_wr, *bad_send_wr;
+ struct rdmav_sge sge;
int i, ret = 0;
if (!node->connected || !message_count)
@@ -197,7 +197,7 @@ static int post_sends(struct cmatest_nod
send_wr.next = NULL;
send_wr.sg_list = &sge;
send_wr.num_sge = 1;
- send_wr.opcode = IBV_WR_SEND;
+ send_wr.opcode = RDMAV_WR_SEND;
send_wr.send_flags = 0;
send_wr.wr_id = (unsigned long)node;
@@ -206,7 +206,7 @@ static int post_sends(struct cmatest_nod
sge.addr = (uintptr_t) node->mem;
for (i = 0; i < message_count && !ret; i++) {
- ret = ibv_post_send(node->cma_id->qp, &send_wr, &bad_send_wr);
+ ret = rdmav_post_send(node->cma_id->qp, &send_wr, &bad_send_wr);
if (ret)
printf("failed to post sends: %d\n", ret);
}
@@ -350,15 +350,15 @@ static void destroy_node(struct cmatest_
rdma_destroy_qp(node->cma_id);
if (node->cq)
- ibv_destroy_cq(node->cq);
+ rdmav_destroy_cq(node->cq);
if (node->mem) {
- ibv_dereg_mr(node->mr);
+ rdmav_dereg_mr(node->mr);
free(node->mem);
}
if (node->pd)
- ibv_dealloc_pd(node->pd);
+ rdmav_dealloc_pd(node->pd);
/* Destroy the RDMA ID after all device resources */
rdma_destroy_id(node->cma_id);
@@ -404,7 +404,7 @@ static void destroy_nodes(void)
static int poll_cqs(void)
{
- struct ibv_wc wc[8];
[openib-general] [PATCH v3 5/6] librdmacm source file changes.
Convert librdmacm source files to use the new libibverbs API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/librdmacm/configure.in NEW/librdmacm/configure.in
--- ORG/librdmacm/configure.in 2006-07-30 21:18:17.0 -0700
+++ NEW/librdmacm/configure.in 2006-08-03 00:02:57.0 -0700
@@ -25,8 +25,8 @@ AC_CHECK_SIZEOF(long)
dnl Checks for libraries
if test "$disable_libcheck" != "yes"
then
-AC_CHECK_LIB(ibverbs, ibv_get_device_list, [],
-AC_MSG_ERROR([ibv_get_device_list() not found. librdmacm requires
libibverbs.]))
+AC_CHECK_LIB(ibverbs, rdmav_get_device_list, [],
+AC_MSG_ERROR([rdmav_get_device_list() not found. librdmacm requires
libibverbs.]))
fi
dnl Checks for header files.
diff -ruNp ORG/librdmacm/src/cma.c NEW/librdmacm/src/cma.c
--- ORG/librdmacm/src/cma.c 2006-07-30 21:18:17.0 -0700
+++ NEW/librdmacm/src/cma.c 2006-08-03 17:23:08.0 -0700
@@ -103,7 +103,7 @@ do {
} while (0)
struct cma_device {
- struct ibv_context *verbs;
+ struct rdmav_context *verbs;
uint64_tguid;
int port_cnt;
};
@@ -130,7 +130,7 @@ static void ucma_cleanup(void)
{
if (cma_dev_cnt) {
while (cma_dev_cnt)
- ibv_close_device(cma_dev_array[--cma_dev_cnt].verbs);
+ rdmav_close_device(cma_dev_array[--cma_dev_cnt].verbs);
free(cma_dev_array);
cma_dev_cnt = 0;
@@ -141,7 +141,7 @@ static int check_abi_version(void)
{
char value[8];
- if (ibv_read_sysfs_file(ibv_get_sysfs_path(),
+ if (rdmav_read_sysfs_file(rdmav_get_sysfs_path(),
"class/misc/rdma_cm/abi_version",
value, sizeof value) < 0) {
/*
@@ -167,9 +167,9 @@ static int check_abi_version(void)
static int ucma_init(void)
{
- struct ibv_device **dev_list = NULL;
+ struct rdmav_device **dev_list = NULL;
struct cma_device *cma_dev;
- struct ibv_device_attr attr;
+ struct rdmav_device_attr attr;
int i, ret;
pthread_mutex_lock(&mut);
@@ -180,7 +180,7 @@ static int ucma_init(void)
if (ret)
goto err;
- dev_list = ibv_get_device_list(&cma_dev_cnt);
+ dev_list = rdmav_get_device_list(&cma_dev_cnt);
if (!dev_list) {
printf("CMA: unable to get RDMA device list\n");
ret = -ENODEV;
@@ -196,15 +196,15 @@ static int ucma_init(void)
for (i = 0; dev_list[i]; ++i) {
cma_dev = &cma_dev_array[i];
- cma_dev->guid = ibv_get_device_guid(dev_list[i]);
- cma_dev->verbs = ibv_open_device(dev_list[i]);
+ cma_dev->guid = rdmav_get_device_guid(dev_list[i]);
+ cma_dev->verbs = rdmav_open_device(dev_list[i]);
if (!cma_dev->verbs) {
printf("CMA: unable to open RDMA device\n");
ret = -ENODEV;
goto err;
}
- ret = ibv_query_device(cma_dev->verbs, &attr);
+ ret = rdmav_query_device(cma_dev->verbs, &attr);
if (ret) {
printf("CMA: unable to query RDMA device\n");
goto err;
@@ -219,13 +219,13 @@ err:
ucma_cleanup();
pthread_mutex_unlock(&mut);
if (dev_list)
- ibv_free_device_list(dev_list);
+ rdmav_free_device_list(dev_list);
return ret;
}
-struct ibv_context **rdma_get_devices(int *num_devices)
+struct rdmav_context **rdma_get_devices(int *num_devices)
{
- struct ibv_context **devs = NULL;
+ struct rdmav_context **devs = NULL;
int i;
if (!cma_dev_cnt && ucma_init())
@@ -244,7 +244,7 @@ out:
return devs;
}
-void rdma_free_devices(struct ibv_context **list)
+void rdma_free_devices(struct rdmav_context **list)
{
free(list);
}
@@ -479,7 +479,7 @@ static int ucma_query_route(struct rdma_
id->route.num_paths = resp->num_paths;
for (i = 0; i < resp->num_paths; i++)
- ibv_copy_path_rec_from_kern(&id->route.path_rec[i],
+ rdmav_copy_path_rec_from_kern(&id->route.path_rec[i],
&resp->ib_route[i]);
}
@@ -578,11 +578,11 @@ int rdma_resolve_route(struct rdma_cm_id
return 0;
}
-static int rdma_init_qp_attr(struct rdma_cm_id *id, struct ibv_qp_attr
*qp_attr,
+static int rdma_init_qp_attr(struct rdma_cm_id *id, struct rdmav_qp_attr
*qp_attr,
int *qp_attr_mask)
{
struct ucma_abi_init_qp_attr *cmd;
- struct ibv_kern_qp_attr *resp;
+ struct rdmav_kern_qp_attr *resp;
struct cma_id_private *id_priv;
void *msg;
int ret, size;
@@
[openib-general] [PATCH v3 4/6] librdmacm include file changes.
Convert librdmacm include files to use the new libibverbs API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/librdmacm/include/rdma/rdma_cma.h
NEW/librdmacm/include/rdma/rdma_cma.h
--- ORG/librdmacm/include/rdma/rdma_cma.h 2006-07-30 21:18:17.0
-0700
+++ NEW/librdmacm/include/rdma/rdma_cma.h 2006-08-03 17:22:26.0
-0700
@@ -68,8 +68,8 @@ enum {
};
struct ib_addr {
- union ibv_gid sgid;
- union ibv_gid dgid;
+ union rdmav_gid sgid;
+ union rdmav_gid dgid;
uint16_tpkey;
};
@@ -83,7 +83,7 @@ struct rdma_addr {
struct rdma_route {
struct rdma_addr addr;
- struct ibv_sa_path_rec *path_rec;
+ struct rdmav_sa_path_rec*path_rec;
int num_paths;
};
@@ -92,10 +92,10 @@ struct rdma_event_channel {
};
struct rdma_cm_id {
- struct ibv_context *verbs;
+ struct rdmav_context*verbs;
struct rdma_event_channel *channel;
void*context;
- struct ibv_qp *qp;
+ struct rdmav_qp *qp;
struct rdma_routeroute;
enum rdma_port_space ps;
uint8_t port_num;
@@ -191,8 +191,8 @@ int rdma_resolve_route(struct rdma_cm_id
* QPs allocated to an rdma_cm_id will automatically be transitioned by the CMA
* through their states.
*/
-int rdma_create_qp(struct rdma_cm_id *id, struct ibv_pd *pd,
- struct ibv_qp_init_attr *qp_init_attr);
+int rdma_create_qp(struct rdma_cm_id *id, struct rdmav_pd *pd,
+ struct rdmav_qp_init_attr *qp_init_attr);
/**
* rdma_destroy_qp - Deallocate the QP associated with the specified RDMA
@@ -214,7 +214,7 @@ struct rdma_conn_param {
/* Fields below ignored if a QP is created on the rdma_cm_id. */
uint8_t srq;
uint32_t qp_num;
- enum ibv_qp_type qp_type;
+ enum rdmav_qp_type qp_type;
};
/**
@@ -341,11 +341,11 @@ static inline uint16_t rdma_get_dst_port
* across multiple rdma_cm_id's.
* The array must be released by calling rdma_free_devices().
*/
-struct ibv_context **rdma_get_devices(int *num_devices);
+struct rdmav_context **rdma_get_devices(int *num_devices);
/**
* rdma_free_devices - Frees the list of devices returned by
rdma_get_devices().
*/
-void rdma_free_devices(struct ibv_context **list);
+void rdma_free_devices(struct rdmav_context **list);
#endif /* RDMA_CMA_H */
diff -ruNp ORG/librdmacm/include/rdma/rdma_cma_abi.h
NEW/librdmacm/include/rdma/rdma_cma_abi.h
--- ORG/librdmacm/include/rdma/rdma_cma_abi.h 2006-07-30 21:18:17.0
-0700
+++ NEW/librdmacm/include/rdma/rdma_cma_abi.h 2006-08-03 17:22:32.0
-0700
@@ -123,7 +123,7 @@ struct ucma_abi_query_route {
struct ucma_abi_query_route_resp {
__u64 node_guid;
- struct ibv_kern_path_rec ib_route[2];
+ struct rdmav_kern_path_rec ib_route[2];
struct sockaddr_in6 src_addr;
struct sockaddr_in6 dst_addr;
__u32 num_paths;
@@ -194,7 +194,7 @@ struct ucma_abi_leave_mcast {
struct ucma_abi_dst_attr_resp {
__u32 remote_qpn;
__u32 remote_qkey;
- struct ibv_kern_ah_attr ah_attr;
+ struct rdmav_kern_ah_attr ah_attr;
};
struct ucma_abi_get_dst_attr {
diff -ruNp ORG/librdmacm/include/rdma/rdma_cma_ib.h
NEW/librdmacm/include/rdma/rdma_cma_ib.h
--- ORG/librdmacm/include/rdma/rdma_cma_ib.h2006-07-30 21:18:17.0
-0700
+++ NEW/librdmacm/include/rdma/rdma_cma_ib.h2006-08-03 17:22:39.0
-0700
@@ -34,7 +34,7 @@
/* IB specific option names for get/set. */
enum {
- IB_PATH_OPTIONS = 1,/* struct ibv_kern_path_rec */
+ IB_PATH_OPTIONS = 1,/* struct rdmav_kern_path_rec */
IB_CM_REQ_OPTIONS = 2 /* struct ib_cm_req_opt */
};
@@ -56,7 +56,7 @@ struct ib_cm_req_opt {
* Users must have called rdma_connect() to resolve the destination
information.
*/
int rdma_get_dst_attr(struct rdma_cm_id *id, struct sockaddr *addr,
- struct ibv_ah_attr *ah_attr, uint32_t *remote_qpn,
+ struct rdmav_ah_attr *ah_attr, uint32_t *remote_qpn,
uint32_t *remote_qkey);
#endif /* RDMA_CMA_IB_H */
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v3 3/6] libibverbs configuration files changes.
Configuration/Makefiles to build libibverbs with the new API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/libibverbs/Makefile.am NEW/libibverbs/Makefile.am
--- ORG/libibverbs/Makefile.am 2006-07-30 21:18:16.0 -0700
+++ NEW/libibverbs/Makefile.am 2006-08-03 17:15:33.0 -0700
@@ -9,7 +9,7 @@ AM_CFLAGS = -g -Wall -D_GNU_SOURCE
src_libibverbs_la_CFLAGS = -g -Wall -D_GNU_SOURCE
-DDRIVER_PATH=\"$(libdir)/infiniband\"
if HAVE_LD_VERSION_SCRIPT
-libibverbs_version_script =
-Wl,--version-script=$(srcdir)/src/libibverbs.map
+libibverbs_version_script =
-Wl,--version-script=$(srcdir)/src/librdmaverbs.map
else
libibverbs_version_script =
endif
@@ -18,7 +18,7 @@ src_libibverbs_la_SOURCES = src/cmd.c sr
src/memory.c src/sysfs.c src/verbs.c
src_libibverbs_la_LDFLAGS = -version-info 2 -export-dynamic \
$(libibverbs_version_script)
-src_libibverbs_la_DEPENDENCIES = $(srcdir)/src/libibverbs.map
+src_libibverbs_la_DEPENDENCIES = $(srcdir)/src/librdmaverbs.map
bin_PROGRAMS = examples/ibv_devices examples/ibv_devinfo \
examples/ibv_asyncwatch examples/ibv_rc_pingpong examples/ibv_uc_pingpong \
@@ -42,7 +42,8 @@ libibverbsincludedir = $(includedir)/inf
libibverbsinclude_HEADERS = include/infiniband/arch.h
include/infiniband/driver.h \
include/infiniband/kern-abi.h include/infiniband/opcode.h
include/infiniband/verbs.h \
-include/infiniband/sa-kern-abi.h include/infiniband/sa.h
include/infiniband/marshall.h
+include/infiniband/sa-kern-abi.h include/infiniband/sa.h
include/infiniband/marshall.h \
+include/infiniband/deprecate.h
man_MANS = man/ibv_asyncwatch.1 man/ibv_devices.1 man/ibv_devinfo.1 \
man/ibv_rc_pingpong.1 man/ibv_uc_pingpong.1 man/ibv_ud_pingpong.1 \
@@ -56,8 +57,9 @@ DEBIAN = debian/changelog debian/compat
EXTRA_DIST = include/infiniband/driver.h include/infiniband/kern-abi.h \
include/infiniband/opcode.h include/infiniband/verbs.h
include/infiniband/marshall.h \
include/infiniband/sa-kern-abi.h include/infiniband/sa.h \
-src/ibverbs.h examples/pingpong.h \
-src/libibverbs.map libibverbs.spec.in $(man_MANS)
+include/infiniband/deprecate.h \
+src/rdmaverbs.h examples/pingpong.h \
+src/librdmaverbs.map libibverbs.spec.in $(man_MANS)
dist-hook: libibverbs.spec
cp libibverbs.spec $(distdir)
diff -ruNp ORG/libibverbs/configure.in NEW/libibverbs/configure.in
--- ORG/libibverbs/configure.in 2006-07-30 21:18:16.0 -0700
+++ NEW/libibverbs/configure.in 2006-08-02 18:24:49.0 -0700
@@ -2,7 +2,7 @@ dnl Process this file with autoconf to p
AC_PREREQ(2.57)
AC_INIT(libibverbs, 1.1-pre1, openib-general@openib.org)
-AC_CONFIG_SRCDIR([src/ibverbs.h])
+AC_CONFIG_SRCDIR([src/rdmaverbs.h])
AC_CONFIG_AUX_DIR(config)
AM_CONFIG_HEADER(config.h)
AM_INIT_AUTOMAKE(libibverbs, 1.1-pre1)
@@ -33,5 +33,5 @@ AC_CACHE_CHECK(whether ld accepts --vers
AM_CONDITIONAL(HAVE_LD_VERSION_SCRIPT, test "$ac_cv_version_script" = "yes")
-AC_CONFIG_FILES([Makefile libibverbs.spec])
+AC_CONFIG_FILES([Makefile librdmaverbs.spec])
AC_OUTPUT
diff -ruNp ORG/libibverbs/libibverbs.spec.in NEW/libibverbs/libibverbs.spec.in
--- ORG/libibverbs/libibverbs.spec.in 2006-07-30 21:18:16.0 -0700
+++ NEW/libibverbs/libibverbs.spec.in 1969-12-31 16:00:00.0 -0800
@@ -1,106 +0,0 @@
-# $Id: libibverbs.spec.in 7484 2006-05-24 21:12:21Z roland $
-
-%define ver @VERSION@
-
-Name: libibverbs
-Version: 1.1
-Release: 0.1.pre1%{?dist}
-Summary: A library for direct userspace use of InfiniBand
-
-Group: System Environment/Libraries
-License: GPL/BSD
-Url: http://openib.org/
-Source: http://openib.org/downloads/libibverbs-1.1-pre1.tar.gz
-BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
-
-%description
-libibverbs is a library that allows userspace processes to use
-InfiniBand "verbs" as described in the InfiniBand Architecture
-Specification. This includes direct hardware access for fast path
-operations.
-
-For this library to be useful, a device-specific plug-in module should
-also be installed.
-
-%package devel
-Summary: Development files for the libibverbs library
-Group: System Environment/Libraries
-
-%description devel
-Static libraries and header files for the libibverbs verbs library.
-
-%package utils
-Summary: Examples for the libibverbs library
-Group: System Environment/Libraries
-Requires: %{name} = %{version}-%{release}
-
-%description utils
-Useful libibverbs1 example programs such as ibv_devinfo, which
-displays information about InfiniBand devices.
-
-%prep
-%setup -q -n %{name}-%{ver}
-
-%build
-%configure
-make %{?_smp_mflags}
-
-%install
-rm -rf $RPM_BUILD_ROOT
-%makeinstall
-# remove unpackaged files from the buildroot
-rm -f $RPM_BUILD_ROOT%{_libdir}/*.la
-
-%clean
-rm -rf $RPM_BUILD_ROOT
-
-%post -p /sbin/ldconfig
-%postun -p /sbin/ldconfig
-
-%files
-%defattr(-,root,root,-)
-%{_libdir}/libibverbs*.so.*
-%doc AU
[openib-general] [PATCH v3 0/6] Tranport Neutral Verbs Proposal.
[REPOSTING - as 2 patches in my first set didn't seem to have got to openib
mailing list though I checked after 2 hours. Apologies in advance since people
will get duplicates of atleast 4 of the successful patches]
This patchset is a proposal to create new API's and data structures with
transport neutral names. The idea is to remove the old API once all
libraries/applications/examples are gradually converted to use the new API.
Patch 1/6 - Changes to libibverbs configuration file to build the libibverbs
with the new API.
Patch 2/6 - Additions to include files in libibverbs for the new API.
Patch 3/6 - Source files in libibverbs defining the new API.
Patch 4/6 - Convert librdmacm examples to use the new API.
Patch 5/6 - Convert librdmacm include files to use the new libibverbs API.
Patch 6/6 - Convert librdmacm source files to use the new libibverbs API.
Changes from previous (v2) round :
--
1. #defined most data structures as suggested by Sean. Created a deprecate.h
which can be removed once all apps are converted to new API.
2. Changed rdma_ to rdmav_. This also enabled to retain rdma_create_qp() and
rdma_destroy_qp() routines.
(The last suggestion to not convert IB specific types to generic types was not
done at this time, since my previous note explained that it is not clear how to
retain some names while changing others. Eg, ibv_event_type or
ibv_qp_attr_mask, which have enums for both IB and generic types, etc).
Testing :
-
1. Compile tested libibverbs, librdmacm, libmthca, libsdp, libibcm,
libipathverbs and dapl. No warnings or failures in any of these. The
only warning in libmthca was regarding multiple ibv_read_sysfs_file()'s,
which is not a compile issue (and also can be removed).
2. Tested rping, ibv_devices & ibv_devinfo utils.
Information notes found during the changes :
1. Added LIBRDMAVERBS_DRIVER_PATH and also use old OPENIB_DRIVER_PATH_ENV for
backwards compatibility, but have not set user_path to include
OPENIB_DRIVER_PATH_ENV results.
2. Currently ibv_driver_init is implemented in all drivers. But the function
returns a "struct ibv_driver *", while we expect "struct rdma_driver *". In
reality this is fine as they are both pointers pointing to identical objects.
Otherwise each driver has to be changed now. Once all drivers are changed to
use rdma_* API's, this will not be an issue.
3. All names are changed to neutral names, even IB specific names as it is not
clear how to retain some names while changing others. Eg, ibv_event_type
or ibv_qp_attr_mask, etc.
4. Passing different pointer to verbs, though the end result is the same (no
warnings generated though as this is a link-time trick). Eg :
int rdma_query_device(struct rdma_context *context,
struct rdma_device_attr *device_attr)
{
return context->ops.query_device(context, device_attr);
}
However this will not be an issue once the drivers are changed to use the
new API. Eg :
int mthca_query_device(struct rdma_context *context,
struct rdma_device_attr *attr)
5. Makefile.am still makes libibverbs.* libraries so that other apps do not
break. librdmaverbs.spec.in also does the same.
6. Kept ibv_driver_init call as all libraries have implemented ibv_driver_init,
but this can be changed easily to new API (and then retired).
7. Prefix is kept as rdmav_ (rdv_ didn't have much takers) to be generic and
consistent enough.
8. [Missing] IBV_OPCODE() macro is not done (but no use ones it currently).
---
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v3 6/6] librdmacm examples changes.
Convert librdmacm examples to use the new API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/librdmacm/examples/cmatose.c NEW/librdmacm/examples/cmatose.c
--- ORG/librdmacm/examples/cmatose.c2006-07-30 21:18:17.0 -0700
+++ NEW/librdmacm/examples/cmatose.c2006-08-03 17:32:36.0 -0700
@@ -62,9 +62,9 @@ struct cmatest_node {
int id;
struct rdma_cm_id *cma_id;
int connected;
- struct ibv_pd *pd;
- struct ibv_cq *cq;
- struct ibv_mr *mr;
+ struct rdmav_pd *pd;
+ struct rdmav_cq *cq;
+ struct rdmav_mr *mr;
void*mem;
};
@@ -100,8 +100,8 @@ static int create_message(struct cmatest
printf("failed message allocation\n");
return -1;
}
- node->mr = ibv_reg_mr(node->pd, node->mem, message_size,
-IBV_ACCESS_LOCAL_WRITE);
+ node->mr = rdmav_reg_mr(node->pd, node->mem, message_size,
+RDMAV_ACCESS_LOCAL_WRITE);
if (!node->mr) {
printf("failed to reg MR\n");
goto err;
@@ -114,10 +114,10 @@ err:
static int init_node(struct cmatest_node *node)
{
- struct ibv_qp_init_attr init_qp_attr;
+ struct rdmav_qp_init_attr init_qp_attr;
int cqe, ret;
- node->pd = ibv_alloc_pd(node->cma_id->verbs);
+ node->pd = rdmav_alloc_pd(node->cma_id->verbs);
if (!node->pd) {
ret = -ENOMEM;
printf("cmatose: unable to allocate PD\n");
@@ -125,7 +125,7 @@ static int init_node(struct cmatest_node
}
cqe = message_count ? message_count * 2 : 2;
- node->cq = ibv_create_cq(node->cma_id->verbs, cqe, node, 0, 0);
+ node->cq = rdmav_create_cq(node->cma_id->verbs, cqe, node, 0, 0);
if (!node->cq) {
ret = -ENOMEM;
printf("cmatose: unable to create CQ\n");
@@ -139,7 +139,7 @@ static int init_node(struct cmatest_node
init_qp_attr.cap.max_recv_sge = 1;
init_qp_attr.qp_context = node;
init_qp_attr.sq_sig_all = 1;
- init_qp_attr.qp_type = IBV_QPT_RC;
+ init_qp_attr.qp_type = RDMAV_QPT_RC;
init_qp_attr.send_cq = node->cq;
init_qp_attr.recv_cq = node->cq;
ret = rdma_create_qp(node->cma_id, node->pd, &init_qp_attr);
@@ -159,8 +159,8 @@ out:
static int post_recvs(struct cmatest_node *node)
{
- struct ibv_recv_wr recv_wr, *recv_failure;
- struct ibv_sge sge;
+ struct rdmav_recv_wr recv_wr, *recv_failure;
+ struct rdmav_sge sge;
int i, ret = 0;
if (!message_count)
@@ -176,7 +176,7 @@ static int post_recvs(struct cmatest_nod
sge.addr = (uintptr_t) node->mem;
for (i = 0; i < message_count && !ret; i++ ) {
- ret = ibv_post_recv(node->cma_id->qp, &recv_wr, &recv_failure);
+ ret = rdmav_post_recv(node->cma_id->qp, &recv_wr,
&recv_failure);
if (ret) {
printf("failed to post receives: %d\n", ret);
break;
@@ -187,8 +187,8 @@ static int post_recvs(struct cmatest_nod
static int post_sends(struct cmatest_node *node)
{
- struct ibv_send_wr send_wr, *bad_send_wr;
- struct ibv_sge sge;
+ struct rdmav_send_wr send_wr, *bad_send_wr;
+ struct rdmav_sge sge;
int i, ret = 0;
if (!node->connected || !message_count)
@@ -197,7 +197,7 @@ static int post_sends(struct cmatest_nod
send_wr.next = NULL;
send_wr.sg_list = &sge;
send_wr.num_sge = 1;
- send_wr.opcode = IBV_WR_SEND;
+ send_wr.opcode = RDMAV_WR_SEND;
send_wr.send_flags = 0;
send_wr.wr_id = (unsigned long)node;
@@ -206,7 +206,7 @@ static int post_sends(struct cmatest_nod
sge.addr = (uintptr_t) node->mem;
for (i = 0; i < message_count && !ret; i++) {
- ret = ibv_post_send(node->cma_id->qp, &send_wr, &bad_send_wr);
+ ret = rdmav_post_send(node->cma_id->qp, &send_wr, &bad_send_wr);
if (ret)
printf("failed to post sends: %d\n", ret);
}
@@ -350,15 +350,15 @@ static void destroy_node(struct cmatest_
rdma_destroy_qp(node->cma_id);
if (node->cq)
- ibv_destroy_cq(node->cq);
+ rdmav_destroy_cq(node->cq);
if (node->mem) {
- ibv_dereg_mr(node->mr);
+ rdmav_dereg_mr(node->mr);
free(node->mem);
}
if (node->pd)
- ibv_dealloc_pd(node->pd);
+ rdmav_dealloc_pd(node->pd);
/* Destroy the RDMA ID after all device resources */
rdma_destroy_id(node->cma_id);
@@ -404,7 +404,7 @@ static void destroy_nodes(void)
static int poll_cqs(void)
{
- struct ibv_wc wc[8];
[openib-general] [PATCH v3 5/6] librdmacm source file changes.
Convert librdmacm source files to use the new libibverbs API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/librdmacm/configure.in NEW/librdmacm/configure.in
--- ORG/librdmacm/configure.in 2006-07-30 21:18:17.0 -0700
+++ NEW/librdmacm/configure.in 2006-08-03 00:02:57.0 -0700
@@ -25,8 +25,8 @@ AC_CHECK_SIZEOF(long)
dnl Checks for libraries
if test "$disable_libcheck" != "yes"
then
-AC_CHECK_LIB(ibverbs, ibv_get_device_list, [],
-AC_MSG_ERROR([ibv_get_device_list() not found. librdmacm requires
libibverbs.]))
+AC_CHECK_LIB(ibverbs, rdmav_get_device_list, [],
+AC_MSG_ERROR([rdmav_get_device_list() not found. librdmacm requires
libibverbs.]))
fi
dnl Checks for header files.
diff -ruNp ORG/librdmacm/src/cma.c NEW/librdmacm/src/cma.c
--- ORG/librdmacm/src/cma.c 2006-07-30 21:18:17.0 -0700
+++ NEW/librdmacm/src/cma.c 2006-08-03 17:23:08.0 -0700
@@ -103,7 +103,7 @@ do {
} while (0)
struct cma_device {
- struct ibv_context *verbs;
+ struct rdmav_context *verbs;
uint64_tguid;
int port_cnt;
};
@@ -130,7 +130,7 @@ static void ucma_cleanup(void)
{
if (cma_dev_cnt) {
while (cma_dev_cnt)
- ibv_close_device(cma_dev_array[--cma_dev_cnt].verbs);
+ rdmav_close_device(cma_dev_array[--cma_dev_cnt].verbs);
free(cma_dev_array);
cma_dev_cnt = 0;
@@ -141,7 +141,7 @@ static int check_abi_version(void)
{
char value[8];
- if (ibv_read_sysfs_file(ibv_get_sysfs_path(),
+ if (rdmav_read_sysfs_file(rdmav_get_sysfs_path(),
"class/misc/rdma_cm/abi_version",
value, sizeof value) < 0) {
/*
@@ -167,9 +167,9 @@ static int check_abi_version(void)
static int ucma_init(void)
{
- struct ibv_device **dev_list = NULL;
+ struct rdmav_device **dev_list = NULL;
struct cma_device *cma_dev;
- struct ibv_device_attr attr;
+ struct rdmav_device_attr attr;
int i, ret;
pthread_mutex_lock(&mut);
@@ -180,7 +180,7 @@ static int ucma_init(void)
if (ret)
goto err;
- dev_list = ibv_get_device_list(&cma_dev_cnt);
+ dev_list = rdmav_get_device_list(&cma_dev_cnt);
if (!dev_list) {
printf("CMA: unable to get RDMA device list\n");
ret = -ENODEV;
@@ -196,15 +196,15 @@ static int ucma_init(void)
for (i = 0; dev_list[i]; ++i) {
cma_dev = &cma_dev_array[i];
- cma_dev->guid = ibv_get_device_guid(dev_list[i]);
- cma_dev->verbs = ibv_open_device(dev_list[i]);
+ cma_dev->guid = rdmav_get_device_guid(dev_list[i]);
+ cma_dev->verbs = rdmav_open_device(dev_list[i]);
if (!cma_dev->verbs) {
printf("CMA: unable to open RDMA device\n");
ret = -ENODEV;
goto err;
}
- ret = ibv_query_device(cma_dev->verbs, &attr);
+ ret = rdmav_query_device(cma_dev->verbs, &attr);
if (ret) {
printf("CMA: unable to query RDMA device\n");
goto err;
@@ -219,13 +219,13 @@ err:
ucma_cleanup();
pthread_mutex_unlock(&mut);
if (dev_list)
- ibv_free_device_list(dev_list);
+ rdmav_free_device_list(dev_list);
return ret;
}
-struct ibv_context **rdma_get_devices(int *num_devices)
+struct rdmav_context **rdma_get_devices(int *num_devices)
{
- struct ibv_context **devs = NULL;
+ struct rdmav_context **devs = NULL;
int i;
if (!cma_dev_cnt && ucma_init())
@@ -244,7 +244,7 @@ out:
return devs;
}
-void rdma_free_devices(struct ibv_context **list)
+void rdma_free_devices(struct rdmav_context **list)
{
free(list);
}
@@ -479,7 +479,7 @@ static int ucma_query_route(struct rdma_
id->route.num_paths = resp->num_paths;
for (i = 0; i < resp->num_paths; i++)
- ibv_copy_path_rec_from_kern(&id->route.path_rec[i],
+ rdmav_copy_path_rec_from_kern(&id->route.path_rec[i],
&resp->ib_route[i]);
}
@@ -578,11 +578,11 @@ int rdma_resolve_route(struct rdma_cm_id
return 0;
}
-static int rdma_init_qp_attr(struct rdma_cm_id *id, struct ibv_qp_attr
*qp_attr,
+static int rdma_init_qp_attr(struct rdma_cm_id *id, struct rdmav_qp_attr
*qp_attr,
int *qp_attr_mask)
{
struct ucma_abi_init_qp_attr *cmd;
- struct ibv_kern_qp_attr *resp;
+ struct rdmav_kern_qp_attr *resp;
struct cma_id_private *id_priv;
void *msg;
int ret, size;
@@
[openib-general] [PATCH v3 4/6] librdmacm include file changes.
Convert librdmacm include files to use the new libibverbs API.
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
diff -ruNp ORG/librdmacm/include/rdma/rdma_cma.h
NEW/librdmacm/include/rdma/rdma_cma.h
--- ORG/librdmacm/include/rdma/rdma_cma.h 2006-07-30 21:18:17.0
-0700
+++ NEW/librdmacm/include/rdma/rdma_cma.h 2006-08-03 17:22:26.0
-0700
@@ -68,8 +68,8 @@ enum {
};
struct ib_addr {
- union ibv_gid sgid;
- union ibv_gid dgid;
+ union rdmav_gid sgid;
+ union rdmav_gid dgid;
uint16_tpkey;
};
@@ -83,7 +83,7 @@ struct rdma_addr {
struct rdma_route {
struct rdma_addr addr;
- struct ibv_sa_path_rec *path_rec;
+ struct rdmav_sa_path_rec*path_rec;
int num_paths;
};
@@ -92,10 +92,10 @@ struct rdma_event_channel {
};
struct rdma_cm_id {
- struct ibv_context *verbs;
+ struct rdmav_context*verbs;
struct rdma_event_channel *channel;
void*context;
- struct ibv_qp *qp;
+ struct rdmav_qp *qp;
struct rdma_routeroute;
enum rdma_port_space ps;
uint8_t port_num;
@@ -191,8 +191,8 @@ int rdma_resolve_route(struct rdma_cm_id
* QPs allocated to an rdma_cm_id will automatically be transitioned by the CMA
* through their states.
*/
-int rdma_create_qp(struct rdma_cm_id *id, struct ibv_pd *pd,
- struct ibv_qp_init_attr *qp_init_attr);
+int rdma_create_qp(struct rdma_cm_id *id, struct rdmav_pd *pd,
+ struct rdmav_qp_init_attr *qp_init_attr);
/**
* rdma_destroy_qp - Deallocate the QP associated with the specified RDMA
@@ -214,7 +214,7 @@ struct rdma_conn_param {
/* Fields below ignored if a QP is created on the rdma_cm_id. */
uint8_t srq;
uint32_t qp_num;
- enum ibv_qp_type qp_type;
+ enum rdmav_qp_type qp_type;
};
/**
@@ -341,11 +341,11 @@ static inline uint16_t rdma_get_dst_port
* across multiple rdma_cm_id's.
* The array must be released by calling rdma_free_devices().
*/
-struct ibv_context **rdma_get_devices(int *num_devices);
+struct rdmav_context **rdma_get_devices(int *num_devices);
/**
* rdma_free_devices - Frees the list of devices returned by
rdma_get_devices().
*/
-void rdma_free_devices(struct ibv_context **list);
+void rdma_free_devices(struct rdmav_context **list);
#endif /* RDMA_CMA_H */
diff -ruNp ORG/librdmacm/include/rdma/rdma_cma_abi.h
NEW/librdmacm/include/rdma/rdma_cma_abi.h
--- ORG/librdmacm/include/rdma/rdma_cma_abi.h 2006-07-30 21:18:17.0
-0700
+++ NEW/librdmacm/include/rdma/rdma_cma_abi.h 2006-08-03 17:22:32.0
-0700
@@ -123,7 +123,7 @@ struct ucma_abi_query_route {
struct ucma_abi_query_route_resp {
__u64 node_guid;
- struct ibv_kern_path_rec ib_route[2];
+ struct rdmav_kern_path_rec ib_route[2];
struct sockaddr_in6 src_addr;
struct sockaddr_in6 dst_addr;
__u32 num_paths;
@@ -194,7 +194,7 @@ struct ucma_abi_leave_mcast {
struct ucma_abi_dst_attr_resp {
__u32 remote_qpn;
__u32 remote_qkey;
- struct ibv_kern_ah_attr ah_attr;
+ struct rdmav_kern_ah_attr ah_attr;
};
struct ucma_abi_get_dst_attr {
diff -ruNp ORG/librdmacm/include/rdma/rdma_cma_ib.h
NEW/librdmacm/include/rdma/rdma_cma_ib.h
--- ORG/librdmacm/include/rdma/rdma_cma_ib.h2006-07-30 21:18:17.0
-0700
+++ NEW/librdmacm/include/rdma/rdma_cma_ib.h2006-08-03 17:22:39.0
-0700
@@ -34,7 +34,7 @@
/* IB specific option names for get/set. */
enum {
- IB_PATH_OPTIONS = 1,/* struct ibv_kern_path_rec */
+ IB_PATH_OPTIONS = 1,/* struct rdmav_kern_path_rec */
IB_CM_REQ_OPTIONS = 2 /* struct ib_cm_req_opt */
};
@@ -56,7 +56,7 @@ struct ib_cm_req_opt {
* Users must have called rdma_connect() to resolve the destination
information.
*/
int rdma_get_dst_attr(struct rdma_cm_id *id, struct sockaddr *addr,
- struct ibv_ah_attr *ah_attr, uint32_t *remote_qpn,
+ struct rdmav_ah_attr *ah_attr, uint32_t *remote_qpn,
uint32_t *remote_qkey);
#endif /* RDMA_CMA_IB_H */
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v3 0/6] Tranport Neutral Verbs Proposal.
This patchset is a proposal to create new API's and data structures with
transport neutral names. The idea is to remove the old API once all
libraries/applications/examples are gradually converted to use the new API.
Patch 1/6 - Changes to libibverbs configuration file to build the libibverbs
with the new API.
Patch 2/6 - Additions to include files in libibverbs for the new API.
Patch 3/6 - Source files in libibverbs defining the new API.
Patch 4/6 - Convert librdmacm examples to use the new API.
Patch 5/6 - Convert librdmacm include files to use the new libibverbs API.
Patch 6/6 - Convert librdmacm source files to use the new libibverbs API.
Changes from previous (v2) round :
--
1. #defined most data structures as suggested by Sean. Created a deprecate.h
which can be removed once all apps are converted to new API.
2. Changed rdma_ to rdmav_. This also enabled to retain rdma_create_qp() and
rdma_destroy_qp() routines.
(The last suggestion to not convert IB specific types to generic types was not
done at this time, since my previous note explained that it is not clear how to
retain some names while changing others. Eg, ibv_event_type or
ibv_qp_attr_mask, which have enums for both IB and generic types, etc).
Testing :
-
1. Compile tested libibverbs, librdmacm, libmthca, libsdp, libibcm,
libipathverbs and dapl. No warnings or failures in any of these. The
only warning in libmthca was regarding multiple ibv_read_sysfs_file()'s,
which is not a compile issue (and also can be removed).
2. Tested rping, ibv_devices & ibv_devinfo utils.
Information notes found during the changes :
1. Added LIBRDMAVERBS_DRIVER_PATH and also use old OPENIB_DRIVER_PATH_ENV for
backwards compatibility, but have not set user_path to include
OPENIB_DRIVER_PATH_ENV results.
2. Currently ibv_driver_init is implemented in all drivers. But the function
returns a "struct ibv_driver *", while we expect "struct rdma_driver *". In
reality this is fine as they are both pointers pointing to identical objects.
Otherwise each driver has to be changed now. Once all drivers are changed to
use rdma_* API's, this will not be an issue.
3. All names are changed to neutral names, even IB specific names as it is not
clear how to retain some names while changing others. Eg, ibv_event_type
or ibv_qp_attr_mask, etc.
4. Passing different pointer to verbs, though the end result is the same (no
warnings generated though as this is a link-time trick). Eg :
int rdma_query_device(struct rdma_context *context,
struct rdma_device_attr *device_attr)
{
return context->ops.query_device(context, device_attr);
}
However this will not be an issue once the drivers are changed to use the
new API. Eg :
int mthca_query_device(struct rdma_context *context,
struct rdma_device_attr *attr)
5. Makefile.am still makes libibverbs.* libraries so that other apps do not
break. librdmaverbs.spec.in also does the same.
6. Kept ibv_driver_init call as all libraries have implemented ibv_driver_init,
but this can be changed easily to new API (and then retired).
7. Prefix is kept as rdmav_ (rdv_ didn't have much takers) to be generic and
consistent enough.
8. [Missing] IBV_OPCODE() macro is not done (but no use ones it currently).
---
Signed-off-by: Krishna Kumar <[EMAIL PROTECTED]>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
