RE: [openib-general] How about ib_send_page() ?
On Tue, 7 Jun 2005, Michael Krause wrote: > At 09:28 AM 6/7/2005, Fab Tillier wrote: > > > From: Roland Dreier [mailto:[EMAIL PROTECTED] > > > Sent: Tuesday, June 07, 2005 8:38 AM > > > > > > Michael> Why not just use the IETF draft for RC / UC based IP over > > > Michael> IB and not worry about creating something new? > > > > > > I think we've come full circle. The original post was a suggestion on > > > how to handle the fact the the connected-mode IPoIB draft requires a > > > network stack to deal with different MTUs for different destinations > > > on the same logical link. Yes, the ipoib wg opinion is that a node should be able to communicate in any IB mode. (I've had to implement such a setup for tokenring in the past - e.g. if the peer was on a ring with smaller MTU ). I beleive this is an issue that needs furhter discussion on the ipoverib discussion list in ietf. Would you restart the discussion there? We can update the draft as needed after a consensus is reached in the wg. > > > >That's right - by implementing IP segmentation in the IPoIB driver when going > >over UD, the driver could expose a single MTU to the network stack, thereby > >removing all the issues related to having per-endpoint MTUs. > > > >Keeping a 2K MTU for RC mode doesn't really take advantage of IB's RC > >capabilities. I'd probably target 64K as the MTU. > > The draft should state a minimum for all RC / UC which should be the TCP > MSS. Whether one does SAR over a UD endpoint independent of the underlying > physical MTU can be done but it should not require end-to-end understanding > of the operation, i.e. the send side tells its local that the TCP MSS is X > while the receive side only posts 2-4 KB buffers. This has been done over > Ethernet for years. > > Mike I suggest we move this discussion to ipoverib@ietf.org (can continue to cc this list) to involve all since there have been suggestions for draft update. Vivek __ Vivek Kashyap Linux Technology Center, IBM ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] 2.6.11.11 NFS over IPoIB crash
Troy> We are running NFS over IPOIB, and are getting kernel panics Troy> under heavy NFS I/O. This is on a PowerMac G5, and the Troy> server is a dual opteron running 2.6.11 with the OpenIB code Troy> from subversion. It looks like a bug in nfs.. but we've only Troy> seen it using IPoIB... Is it worth trying to reproduce this Troy> over gigabit ethernet? I don't see anything obviously IPoIB related in the traceback, so it might be worth trying to get it to happen over ethernet. It's probably also worth running it past the Linux NFS mailing list. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query()
On 6/8/05, Tom Duffy <[EMAIL PROTECTED]> wrote: > I said: > > > I am putting this out there to see what people think about this sort of > > > change. This only does it for one of the functions in dat provider, but > > > I will change it for all if this is accepted. > Oops, I parsed the header and "sort of change" but missed "in dat provider", which did make the set clear. > Then Caitlin Bestler said: > > What I am not following is why you think DAT_IA_QUERY_FUNC > > is somehow different than all of the other functions in a DAT method > > table. Is this just a test run? If not, what is the criteria for deciding > > which methods are exposed and which have their method table > > underpinnings kept hidden? > > > Right, dat_ia_query() is not special. Just somewhere to start. > Anyway, one extra benefit of this change is that eliminates any risk of breaking kernel code using other existing DAT Providers. And it is easy for someone who has a large body of code using the function calls to use their own macro or inline function if they don't want to change their code. So consistency with other kernel code probably is the best guideline here. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query()
I said: > > I am putting this out there to see what people think about this sort of > > change. This only does it for one of the functions in dat provider, but > > I will change it for all if this is accepted. Then Caitlin Bestler said: > What I am not following is why you think DAT_IA_QUERY_FUNC > is somehow different than all of the other functions in a DAT method > table. Is this just a test run? If not, what is the criteria for deciding > which methods are exposed and which have their method table > underpinnings kept hidden? Right, dat_ia_query() is not special. Just somewhere to start. -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query()
On Wed, 2005-06-08 at 17:07 -0700, Grant Grundler wrote:
> > -ret = dat_ia_query (test_ptr->ia,
> > - NULL,
> > - &test_ptr->ia_attr,
> > - NULL);
> > +ret = test_ptr->ia->common.provider->dat_ia_query(test_ptr->ia, NULL,
> > + &test_ptr->ia_attr, NULL);
>
> Maybe not reference test_ptr in this line so often?
> It could read:
> ret = ia->common.provider->dat_ia_query(ia, NULL, &test_ptr->ia_attr,
> NULL);
Yup, cleaner. New patch attached.
> > - .ia_query_func = &dapl_ia_query,
> > + .dat_ia_query = &dapl_ia_query,
>
> Yes. I like this alot better too.
Or should the name in dat_provider just be ia_query? Any need for the
dat_?
Signed-off-by: Tom Duffy <[EMAIL PROTECTED]>
Index: linux-kernel-return/test/dapltest/test/dapl_performance_util.c
===
--- linux-kernel-return/test/dapltest/test/dapl_performance_util.c
(revision 2573)
+++ linux-kernel-return/test/dapltest/test/dapl_performance_util.c
(working copy)
@@ -64,10 +64,7 @@ DT_Performance_Test_Create (
test_ptr->ia = ia;
test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd;
-ret = dat_ia_query (test_ptr->ia,
- NULL,
- &test_ptr->ia_attr,
- NULL);
+ret = ia->common.provider->dat_ia_query(ia, NULL, &test_ptr->ia_attr,
NULL);
if ( DAT_SUCCESS != ret)
{
DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ia_query error: %s\n",
Index: linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c
===
--- linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (revision 2573)
+++ linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (working copy)
@@ -165,10 +165,8 @@ int DT_queryinfo_basic (Params_t *params
{
if (result_wanted == DAT_SUCCESS)
{
- rc = dat_ia_query (ia,
- &evd,
- &ia_attributes,
- &provider_attributes);
+ rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes,
+ &provider_attributes);
}
else if (result_wanted == DAT_INVALID_PARAMETER)
{
@@ -177,17 +175,13 @@ int DT_queryinfo_basic (Params_t *params
* NULL out ia_attr and for the DAT_IA_ATTR_MASK to
* have values
*/
- rc = dat_ia_query (ia,
- &evd,
- NULL,
- &provider_attributes);
+ rc = ia->common.provider->dat_ia_query(ia, &evd, NULL,
+ &provider_attributes);
}
else if (result_wanted == DAT_INVALID_HANDLE)
{
- rc = dat_ia_query (ia,
- &evd,
- &ia_attributes,
- &provider_attributes);
+ rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes,
+ &provider_attributes);
}
}
Index: linux-kernel-return/test/dapltest/test/dapl_test_util.c
===
--- linux-kernel-return/test/dapltest/test/dapl_test_util.c (revision 2573)
+++ linux-kernel-return/test/dapltest/test/dapl_test_util.c (working copy)
@@ -44,10 +44,9 @@ DT_query ( Per_Test_Data_t *pt_ptr,
phead = pt_ptr->Params.phead;
/* Query the IA */
-ret = dat_ia_query (ia,
-&async_evd_hdl,
-&pt_ptr->ia_attr,
-&pt_ptr->provider_attr);
+ret = ia->common.provider->dat_ia_query(ia, &async_evd_hdl,
+ &pt_ptr->ia_attr,
+ &pt_ptr->provider_attr);
if (ret != DAT_SUCCESS)
{
DT_Tdep_PT_Printf (phead, "%s: dat_ia_query error: %s\n",
Index: linux-kernel-return/dat-provider/dapl_provider.c
===
--- linux-kernel-return/dat-provider/dapl_provider.c(revision 2573)
+++ linux-kernel-return/dat-provider/dapl_provider.c(working copy)
@@ -67,7 +67,7 @@ static struct dat_provider g_dapl_provid
.extension = NULL,
.ia_open_func = &dapl_ia_open,
- .ia_query_func = &dapl_ia_query,
+ .dat_ia_query = &dapl_ia_query,
.ia_close_func = &dapl_ia_close,
.ia_memtype_hint_func = &dapl_ia_memtype_hint,
Index: linux-kernel-return/dat/api.c
===
--- linux-kernel-return/dat/api.c (revision 2573)
+++ linux-kernel-return/dat/api.c
Re: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query()
On 6/8/05, Tom Duffy <[EMAIL PROTECTED]> wrote: > I am putting this out there to see what people think about this sort of > change. This only does it for one of the functions in dat provider, but > I will change it for all if this is accepted. > > Basically, instead of using a redirected function wrapper, it calls the > function directly from the struct (how the rest of the kernel does it). > > And it gets rid of the typedef DAT_IA_QUERY_FUNC. > Hiding the fact that there was a method table was definitely a cosmetic decision to appeal to the biases of user mode C programmers. I agree that anyone working in the kernel is already quite familiar with calling a function from a struct. A strong argument can be made that a kernel programmer should know that they are making a call through a method table. What I am not following is why you think DAT_IA_QUERY_FUNC is somehow different than all of the other functions in a DAT method table. Is this just a test run? If not, what is the criteria for deciding which methods are exposed and which have their method table underpinnings kept hidden? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query()
On Wed, Jun 08, 2005 at 03:54:34PM -0700, Tom Duffy wrote: > I am putting this out there to see what people think about this sort of > change. This only does it for one of the functions in dat provider, but > I will change it for all if this is accepted. I agree it's the right direction. I would not have guessed dat_ia_query() is really a macro that does pointer chasing. > -ret = dat_ia_query (test_ptr->ia, > - NULL, > - &test_ptr->ia_attr, > - NULL); > +ret = test_ptr->ia->common.provider->dat_ia_query(test_ptr->ia, NULL, > + &test_ptr->ia_attr, NULL); Maybe not reference test_ptr in this line so often? It could read: ret = ia->common.provider->dat_ia_query(ia, NULL, &test_ptr->ia_attr, NULL); > - .ia_query_func = &dapl_ia_query, > + .dat_ia_query = &dapl_ia_query, Yes. I like this alot better too. thanks, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest
On Wed, Jun 08, 2005 at 04:40:57PM -0700, Grant Grundler wrote:
> Index: rdma_bw.c
> ===
> --- rdma_bw.c (revision 2570)
> +++ rdma_bw.c (working copy)
> @@ -61,10 +61,6 @@
...
Sorry - just realized I forgot the:
Signed-off-by: Grant Grundler <[EMAIL PROTECTED]>
thanks,
grant
>
> static int page_size;
>
> -struct report_options {
> - int cycles; /* report delta's in cycles, not microsec's */
> -};
> -
> struct pingpong_context {
> struct ibv_context *context;
> struct ibv_pd *pd;
> @@ -422,19 +418,14 @@ static void usage(const char *argv0)
> printf(" -t, --tx-depth= size of tx queue (default 100)\n");
> printf(" -n, --iters=number of exchanges (at least 2,
> default 1000)\n");
> printf(" -b, --bidirectionalmeasure bidirectional bandwidth
> (default unidirectional)\n");
> - printf(" -C, --report-cyclesreport times in cpu cycle units
> (default seconds)\n");
> - printf(" -H, --report-histogram print out all results (default print
> summary only)\n");
> - printf(" -U, --report-unsorted (implies -H) print out unsorted
> results (default sorted)\n");
> }
>
> -static void print_report(struct report_options * options,
> - unsigned int iters, double size, int duplex,
> +static void print_report(unsigned int iters, int size, int duplex,
>cycles_t *tposted, cycles_t *tcompleted)
> {
> double cycles_to_units;
> - double tsize; /* Transferred size, in megabytes */
> + unsigned long tsize;/* Transferred size, in megabytes */
> int i, j;
> - const char* units;
> int opt_posted = 0, opt_completed = 0;
> cycles_t opt_delta;
> cycles_t t;
> @@ -453,19 +444,21 @@ static void print_report(struct report_o
> }
> }
>
> - if (options->cycles) {
> - cycles_to_units = 1;
> - units = "cycles";
> - } else {
> - cycles_to_units = get_cpu_mhz() * 100;
> - units = "sec";
> - }
> + cycles_to_units = get_cpu_mhz() * 100;
>
> tsize = duplex ? 2 : 1;
> - tsize = tsize * size / 0x10;
> + tsize = tsize * size / 1024;
>
> - printf("Bandwidth peak (#%d to #%d): %g MByte/%s\n", opt_posted,
> opt_completed, tsize * cycles_to_units / opt_delta, units);
> - printf("Bandwidth average: %g MByte/%s\n", tsize * iters *
> cycles_to_units / (tcompleted[iters - 1] - tposted[0]), units);
> + printf("Bandwidth peak (#%d to #%d): %g MB/sec\n",
> + opt_posted, opt_completed,
> + tsize * cycles_to_units / opt_delta / 1024);
> + printf("Bandwidth average: %g MB/sec\n",
> + tsize * iters * cycles_to_units / (tcompleted[iters -
> 1] - tposted[0]) / 1024);
> +
> + printf("Service Demand peak (#%d to #%d): %ld cycles/KB\n",
> + opt_posted, opt_completed, opt_delta/tsize);
> + printf("Service Demand Avg : %ld cycles/KB\n",
> + (tcompleted[iters - 1] - tposted[0])/(tsize * iters));
> }
>
>
> @@ -480,14 +473,13 @@ int main(int argc, char *argv[])
> char*servername = NULL;
> int port = 18515;
> int ib_port = 1;
> - int size = 1;
> + int size = 4 * 1024;
> int tx_depth = 100;
> int iters = 1000;
> int scnt, ccnt;
> int sockfd;
> int duplex = 0;
> struct ibv_qp *qp;
> - struct report_optionsreport = {};
>
> cycles_t*tposted;
> cycles_t*tcompleted;
> @@ -504,11 +496,10 @@ int main(int argc, char *argv[])
> { .name = "iters", .has_arg = 1, .val = 'n' },
> { .name = "tx-depth", .has_arg = 1, .val = 't' },
> { .name = "bidirectional", .has_arg = 0, .val = 'b' },
> - { .name = "report-cycles", .has_arg = 0, .val = 'C' },
> { 0 }
> };
>
> - c = getopt_long(argc, argv, "p:d:i:s:n:t:bC", long_options,
> NULL);
> + c = getopt_long(argc, argv, "p:d:i:s:n:t:b", long_options,
> NULL);
> if (c == -1)
> break;
>
> @@ -565,10 +556,6 @@ int main(int argc, char *argv[])
> duplex = 1;
> break;
>
> - case 'C':
> - report.cycles = 1;
> - break;
> -
> default:
> usage(argv[0]);
> return 1;
> @@ -751,7 +738,7 @@ int main(int argc, char *argv[])
> write(sockfd, "done", sizeof "done");
> close(sockfd);
>
>
Re: [openib-general] [PATCH] rdma_lat-09 and results
On Wed, Jun 01, 2005 at 05:52:28PM -0700, Grant Grundler wrote:
> Michael,
>
> Good news:
> My next cleanup of rdma_lat.c is working and patch is appended.
Michael,
Did you want to comment on this patch or apply it?
I can resend privately it you don't have the original mail.
> Summary of changes below.
>
> Bad News:
> perf is about ~15 cycles slower since the last time I tested.
> (Hrm...maybe it's time to cycle power on the TS90 switch again.)
Seems with discussion on perf was distracting from the patch.
I'll seperate those next time.
thanks,
grant
> Here's with the new rdma_lat.c:
> [EMAIL PROTECTED]:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C
>local address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr
> 0x60014001
> remote address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr
> 0x60014001
> Latency typical: 7140 cycles
> Latency best : 6915 cycles
> Latency worst : 52915.5 cycles
> [EMAIL PROTECTED]:/usr/src/openib_gen2/src/userspace/perftest$
>
> And the "client" side:
> [EMAIL PROTECTED]:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C
> 10.0.0.51
>local address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr
> 0x60014001
> remote address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr
> 0x60014001
> Latency typical: 7140 cycles
> Latency best : 6907 cycles
> Latency worst : 94920 cycles
>
>
> The previous set of rdma_lat results are here:
> http://openib.org/pipermail/openib-general/2005-May/006721.html
>
> I'll guess the previous SVN verion was no older than r2229.
>
>
> I get 7140 to 7151 for the original rdma_lat. Usually 7147.5.
> I get 7132 to 7155 with my version of rdma_lat. Usually 7140.
> No statistically significant differences.
> Both essentially agree on the higher result.
> Using "-n 1" gave more consistent results *
>
> I use "taskset" to bind the rdma_lat test to a CPU.
> But it didn't matter which CPU I bound the task to - results
> where basically the same. I suspect the "stream" mode just
> does not depend on or generating that many interrupts.
>
>
> diffstat rdma_lat.c-09-diff
> rdma_lat.c | 395
> +
> 1 files changed, 188 insertions(+), 207 deletions(-)
>
> Commit Log entry/Summary of changes:
> o move device lookup from main() to pp_find_dev()
> o move sockfd handling code to pp_open_port()
> o consolidate server/client "key exchange" code path
> o enumerate return values in main()
> o fixed nit: pp_*_exch_dest was called twice.
> Each time it would malloc a new "rem_dest".
> Code in pp_open_port() now free()'s the first one.
>
> Signed-off-by: Grant Grundler <[EMAIL PROTECTED]>
>
> thanks,
> grant
>
>
>
> Index: rdma_lat.c
> ===
> --- rdma_lat.c(revision 2519)
> +++ rdma_lat.c(working copy)
> @@ -103,30 +103,40 @@ static uint16_t pp_get_local_lid(struct
> return attr.lid;
> }
>
> -static int pp_client_connect(const char *servername, int port)
> +
> +static int pp_connect_sock(const char *servername, int port)
> {
> struct addrinfo *res, *t;
> struct addrinfo hints = {
> + .ai_flags= AI_PASSIVE, /* Server only? */
> .ai_family = AF_UNSPEC,
> .ai_socktype = SOCK_STREAM
> };
> char *service;
> - int n;
> int sockfd = -1;
> + int n;
>
> asprintf(&service, "%d", port);
> n = getaddrinfo(servername, service, &hints, &res);
>
> if (n < 0) {
> - fprintf(stderr, "%s for %s:%d\n", gai_strerror(n), servername,
> port);
> + fprintf(stderr, "%s for %s:%d\n", gai_strerror(n),
> + servername ? servername : "NULL", port);
> return n;
> }
>
> for (t = res; t; t = t->ai_next) {
> sockfd = socket(t->ai_family, t->ai_socktype, t->ai_protocol);
> if (sockfd >= 0) {
> - if (!connect(sockfd, t->ai_addr, t->ai_addrlen))
> - break;
> + if (servername) {
> + if (!connect(sockfd, t->ai_addr, t->ai_addrlen))
> + break;
> + } else {
> + setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
> &n, sizeof n);
> + if (!bind(sockfd, t->ai_addr, t->ai_addrlen))
> + break;
> + }
> +
> close(sockfd);
> sockfd = -1;
> }
> @@ -134,147 +144,100 @@ static int pp_client_connect(const char
>
> freeaddrinfo(res);
>
> - if (sockfd < 0) {
> - fprintf(stderr, "Couldn't connect to %s:%d\n", servername,
> por
[openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest
On Wed, Jun 08, 2005 at 08:50:30AM +0300, Michael S. Tsirkin wrote:
> > The numbers (MByte/cycles) don't mean anything to me.
> > Can we report this as cycles/MB (or KB) instead? (aka Service Demand)
>
> Sure. Patch?
Appended. Please apply.
> > The other metric I care about is MB/s.
>
> Dont pass -C then.
Doh! Of course.
-C makes no sense in this context.
It's removed in the patch below.
Here is a sample output:
[EMAIL PROTECTED]:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw -b -n
1 -s 32768 10.0.0.51
local address: LID 0x06, QPN 0x280406, PSN 0x929ad8 RKey 0xf00434 VAddr
0x6001c000
remote address: LID 0x04, QPN 0x270406, PSN 0x61c246, RKey 0xea0434 VAddr
0x6001c000
Bandwidth peak (#0 to #7080): 722.421 MB/sec
Bandwidth average: 721.997 MB/sec
Service Demand peak (#0 to #7080): 2027 cycles/KB
Service Demand Avg : 2028 cycles/KB
thanks,
grant
Index: rdma_bw.c
===
--- rdma_bw.c (revision 2570)
+++ rdma_bw.c (working copy)
@@ -61,10 +61,6 @@
static int page_size;
-struct report_options {
- int cycles; /* report delta's in cycles, not microsec's */
-};
-
struct pingpong_context {
struct ibv_context *context;
struct ibv_pd *pd;
@@ -422,19 +418,14 @@ static void usage(const char *argv0)
printf(" -t, --tx-depth= size of tx queue (default 100)\n");
printf(" -n, --iters=number of exchanges (at least 2,
default 1000)\n");
printf(" -b, --bidirectionalmeasure bidirectional bandwidth
(default unidirectional)\n");
- printf(" -C, --report-cyclesreport times in cpu cycle units
(default seconds)\n");
- printf(" -H, --report-histogram print out all results (default print
summary only)\n");
- printf(" -U, --report-unsorted (implies -H) print out unsorted
results (default sorted)\n");
}
-static void print_report(struct report_options * options,
-unsigned int iters, double size, int duplex,
+static void print_report(unsigned int iters, int size, int duplex,
cycles_t *tposted, cycles_t *tcompleted)
{
double cycles_to_units;
- double tsize; /* Transferred size, in megabytes */
+ unsigned long tsize;/* Transferred size, in megabytes */
int i, j;
- const char* units;
int opt_posted = 0, opt_completed = 0;
cycles_t opt_delta;
cycles_t t;
@@ -453,19 +444,21 @@ static void print_report(struct report_o
}
}
- if (options->cycles) {
- cycles_to_units = 1;
- units = "cycles";
- } else {
- cycles_to_units = get_cpu_mhz() * 100;
- units = "sec";
- }
+ cycles_to_units = get_cpu_mhz() * 100;
tsize = duplex ? 2 : 1;
- tsize = tsize * size / 0x10;
+ tsize = tsize * size / 1024;
- printf("Bandwidth peak (#%d to #%d): %g MByte/%s\n", opt_posted,
opt_completed, tsize * cycles_to_units / opt_delta, units);
- printf("Bandwidth average: %g MByte/%s\n", tsize * iters *
cycles_to_units / (tcompleted[iters - 1] - tposted[0]), units);
+ printf("Bandwidth peak (#%d to #%d): %g MB/sec\n",
+opt_posted, opt_completed,
+tsize * cycles_to_units / opt_delta / 1024);
+ printf("Bandwidth average: %g MB/sec\n",
+tsize * iters * cycles_to_units / (tcompleted[iters -
1] - tposted[0]) / 1024);
+
+ printf("Service Demand peak (#%d to #%d): %ld cycles/KB\n",
+opt_posted, opt_completed, opt_delta/tsize);
+ printf("Service Demand Avg : %ld cycles/KB\n",
+(tcompleted[iters - 1] - tposted[0])/(tsize * iters));
}
@@ -480,14 +473,13 @@ int main(int argc, char *argv[])
char*servername = NULL;
int port = 18515;
int ib_port = 1;
- int size = 1;
+ int size = 4 * 1024;
int tx_depth = 100;
int iters = 1000;
int scnt, ccnt;
int sockfd;
int duplex = 0;
struct ibv_qp *qp;
- struct report_optionsreport = {};
cycles_t*tposted;
cycles_t*tcompleted;
@@ -504,11 +496,10 @@ int main(int argc, char *argv[])
{ .name = "iters", .has_arg = 1, .val = 'n' },
{ .name = "tx-depth", .has_arg = 1, .val = 't' },
{ .name = "bidirectional", .has_arg = 0, .val = 'b' },
- { .name = "report-cycles", .has_arg = 0, .val = 'C' },
{ 0 }
};
- c = getopt_long(arg
[openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query()
I am putting this out there to see what people think about this sort of
change. This only does it for one of the functions in dat provider, but
I will change it for all if this is accepted.
Basically, instead of using a redirected function wrapper, it calls the
function directly from the struct (how the rest of the kernel does it).
And it gets rid of the typedef DAT_IA_QUERY_FUNC.
Signed-off-by: Tom Duffy <[EMAIL PROTECTED]>
dat-provider/dapl_provider.c |2 +-
dat/api.c |4 ++--
dat/dat.h | 15 ++-
test/dapltest/test/dapl_fft_queryinfo.c| 18 ++
test/dapltest/test/dapl_performance_util.c |6 ++
test/dapltest/test/dapl_test_util.c|7 +++
6 files changed, 16 insertions(+), 36 deletions(-)
Index: linux-kernel-return/test/dapltest/test/dapl_performance_util.c
===
--- linux-kernel-return/test/dapltest/test/dapl_performance_util.c
(revision 2573)
+++ linux-kernel-return/test/dapltest/test/dapl_performance_util.c
(working copy)
@@ -64,10 +64,8 @@ DT_Performance_Test_Create (
test_ptr->ia = ia;
test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd;
-ret = dat_ia_query (test_ptr->ia,
- NULL,
- &test_ptr->ia_attr,
- NULL);
+ret = test_ptr->ia->common.provider->dat_ia_query(test_ptr->ia, NULL,
+ &test_ptr->ia_attr, NULL);
if ( DAT_SUCCESS != ret)
{
DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ia_query error: %s\n",
Index: linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c
===
--- linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (revision 2573)
+++ linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (working copy)
@@ -165,10 +165,8 @@ int DT_queryinfo_basic (Params_t *params
{
if (result_wanted == DAT_SUCCESS)
{
- rc = dat_ia_query (ia,
- &evd,
- &ia_attributes,
- &provider_attributes);
+ rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes,
+ &provider_attributes);
}
else if (result_wanted == DAT_INVALID_PARAMETER)
{
@@ -177,17 +175,13 @@ int DT_queryinfo_basic (Params_t *params
* NULL out ia_attr and for the DAT_IA_ATTR_MASK to
* have values
*/
- rc = dat_ia_query (ia,
- &evd,
- NULL,
- &provider_attributes);
+ rc = ia->common.provider->dat_ia_query(ia, &evd, NULL,
+ &provider_attributes);
}
else if (result_wanted == DAT_INVALID_HANDLE)
{
- rc = dat_ia_query (ia,
- &evd,
- &ia_attributes,
- &provider_attributes);
+ rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes,
+ &provider_attributes);
}
}
Index: linux-kernel-return/test/dapltest/test/dapl_test_util.c
===
--- linux-kernel-return/test/dapltest/test/dapl_test_util.c (revision 2573)
+++ linux-kernel-return/test/dapltest/test/dapl_test_util.c (working copy)
@@ -44,10 +44,9 @@ DT_query ( Per_Test_Data_t *pt_ptr,
phead = pt_ptr->Params.phead;
/* Query the IA */
-ret = dat_ia_query (ia,
-&async_evd_hdl,
-&pt_ptr->ia_attr,
-&pt_ptr->provider_attr);
+ret = ia->common.provider->dat_ia_query(ia, &async_evd_hdl,
+ &pt_ptr->ia_attr,
+ &pt_ptr->provider_attr);
if (ret != DAT_SUCCESS)
{
DT_Tdep_PT_Printf (phead, "%s: dat_ia_query error: %s\n",
Index: linux-kernel-return/dat-provider/dapl_provider.c
===
--- linux-kernel-return/dat-provider/dapl_provider.c(revision 2573)
+++ linux-kernel-return/dat-provider/dapl_provider.c(working copy)
@@ -67,7 +67,7 @@ static struct dat_provider g_dapl_provid
.extension = NULL,
.ia_open_func = &dapl_ia_open,
- .ia_query_func = &dapl_ia_query,
+ .dat_ia_query = &dapl_ia_query,
.ia_close_func = &dapl_ia_close,
.ia_memtype_hint_func = &dapl_ia_memtype_hint,
Index: linux-kernel-return/dat/api.c
===
--- linux-kernel-return/dat/api.c
Re: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own
On Wed, 8 Jun 2005, Sean Hefty wrote: Hal Rosenstock wrote: On Wed, 2005-06-08 at 11:44, James Lentini wrote: We interpreted the above to mean "give the connection protocol as much time as it needs to establish a connection, but don't mask errors (no path to the remove node, etc.)". For that reason we changed the variable name to DAT_TIMEOUT_MAX. But if the REQ is lost, the timeout is really really long (longer than most will wait for an error). Transaction test also appears to be using this as well as the quit test. My interpretation was that this is a DAPL level timeout and did not necessarily relate to a timeout for a single CM REQ. That is, there could still be a different timeout specified to the CM, but the number of retries could be infinite. If there are kernel users in need of an truly inifinite timeout, we could do that. Note that I'm not saying that an infinite timeout makes sense, but the use of TIMEOUT_MAX seems reasonable. To me that indicates that DAPL decides how long is needed to establish a timeout, and it manages all retries. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
On Wed, 8 Jun 2005, Tom Duffy wrote:
On Wed, 2005-06-08 at 22:41 +0200, Bernhard Fischer wrote:
On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote:
On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote:
'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally
would appreciate it, since these variants are harder to read for me.
What do you want to see instead?
if (this) {
/* do stuff */
}
yes, this.
I think James would take these types of patches.
I would.
In any event, going through the code with fine toothed comb will happen
after all the major upheavals, I think. *I* want to fix all the major
architectural issues that are holding up acceptance before nit-picking.
If I have the time or inclination, I will clean up these things.
-tduffy
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own
On Wed, 8 Jun 2005, Hal Rosenstock wrote: halr> On Wed, 2005-06-08 at 11:44, James Lentini wrote: halr> > We interpreted the above to mean "give the connection protocol as halr> > much time as it needs to establish a connection, but don't mask halr> > errors (no path to the remove node, etc.)". For that reason we changed halr> > the variable name to DAT_TIMEOUT_MAX. halr> halr> But if the REQ is lost, the timeout is really really long (longer than halr> most will wait for an error). If a user doesn't want to wait DAT_TIMEOUT_MAX time, it can pass a smaller amount of time to dat_ep_connect. Does this satisfy your requirements? halr> Transaction test also appears to be using this as well as the halr> quit test. halr> halr> -- Hal halr> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] shutdown issue with SDP
On 6/8/05, Libor Michalek <[EMAIL PROTECTED]> wrote: > On Wed, Jun 08, 2005 at 01:29:42PM -0400, William Jordan wrote: > > I'm trying to run netperf over SDP using libsdp. I'm having > > intermittent success, but usually, I get a failure. The client does a > > shutdown (SHUT_WR), but the server gets an ECONNRESET, "Connection > > reset by peer", on the recv instead of a zero length recv. > > > > I'm running 2568. > > > > Anyone seen similar/different results? > > Bill, > > Can you try the patch I sent out yesterday? It significantly simplifies > the connection state machine within SDP. I ran it successfully with a > number of the netperf tests. (TCP_CC, TCP_CRR, TCP_RR, TCP_STREAM) > > Thanks. > > -Libor > That appears to have fixed it. Thanks. -- Bill Jordan SilverStorm Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own
That would be another way to interpret this: that each time the CM times out, DAPL should re-attempt to establish a connection. However, the original implementation didn't do this and the feedback I've received is that such a feature would not be appropriate for the kernel. If there are kernel applications that want this, we can add it. On Wed, 8 Jun 2005, Tom Duffy wrote: tduffy> On Wed, 2005-06-08 at 11:44 -0400, James Lentini wrote: tduffy> > We interpreted the above to mean "give the connection protocol as tduffy> > much time as it needs to establish a connection, but don't mask tduffy> > errors (no path to the remove node, etc.)". For that reason we changed tduffy> > the variable name to DAT_TIMEOUT_MAX. tduffy> tduffy> Well, let's say the end node is not there yet. Should the CM keep tduffy> trying indefinitely waiting for somebody to show up and respond? tduffy> tduffy> -tduffy tduffy> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
Committed in revision 2573. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564)
The new problem that I'm seeing is below. I don't think that this patch caused this though, so I'll go ahead and commit your patch with my modifications. The oops I see is below. By my calculations, the crash is on line 450 of mthca_cq.c. That line is: entry->wr_id = (*cur_qp)->wrid[wqe_index]; and the resulting instruction that fails is b98: 8b 54 d8 04 mov0x4(%eax,%ebx,8),%edx Does anyone know which part of the C statement that is? Unable to handle kernel paging request at virtual address 2014 printing eip: e0a65008 *pde = 1814d067 Oops: [#1] Modules linked in: kdapltest ib_dat_provider dat ib_cm ib_at ib_ipoib ib_sa md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc i2c_piix4 i2c_core ib_mthca ib_mad ib_core e100 mii floppy sg aic7xxx sd_mod scsi_mod CPU:0 EIP:0060:[]Not tainted VLI EFLAGS: 00010046 (2.6.11-openib) EIP is at mthca_poll_cq+0x368/0x760 [ib_mthca] eax: ebx: 0402 ecx: edx: dd6ea000 esi: cdbf7000 edi: cdbf7058 ebp: c0469cc8 esp: c0469c5c ds: 007b es: 007b ss: 0068 Process swapper (pid: 0, threadinfo=c0469000 task=c03bfc20) Stack: c046cf1f 0030 cdbf5040 c0469d04 0086 dd8af000 c0469d04 0001 d1a8d9c0 0046 0001 cdbf7000 Call Trace: [] show_stack+0x7a/0x90 [] show_registers+0x149/0x1c0 [] die+0x126/0x2a0 [] do_page_fault+0x45e/0x644 [] error_code+0x2b/0x30 [] dapl_ib_completion_poll+0x3c/0xd1 [ib_dat_provider] [] dapl_evd_cq_poll_to_event+0x17/0x3b [ib_dat_provider] [] dapl_evd_dequeue+0x277/0x34b [ib_dat_provider] [] dapl_evd_upcall_trigger+0x34/0x66 [ib_dat_provider] [] dapl_evd_dto_callback+0xd4/0xea [ib_dat_provider] [] mthca_cq_event+0x33/0x80 [ib_mthca] [] mthca_eq_int+0x3a4/0x580 [ib_mthca] [] mthca_tavor_interrupt+0x81/0x350 [ib_mthca] [] handle_IRQ_event+0x35/0x70 [] __do_IRQ+0x108/0x340 [] do_IRQ+0x96/0xa0 [] common_interrupt+0x1a/0x20 [] handle_IRQ_event+0x35/0x70 [] __do_IRQ+0x108/0x340 [] do_IRQ+0x5a/0xa0 === [] common_interrupt+0x1a/0x20 [] cpu_idle+0x57/0x60 [] rest_init+0x19/0x20 [] start_kernel+0x17a/0x1f0 [] 0xc010019f Code: 00 00 00 8b 55 b4 0f b6 52 1d 81 e2 80 00 00 00 e9 d3 fd ff ff 8b 45 b4 8d 7e 58 8b 4f 34 8b 58 18 8b 86 e0 00 00 00 0f cb d3 eb <8b> 54 d8 04 8b 04 d8 e9 36 fe ff ff 8b 55 b4 8b 4d c4 8b 42 14 On Wed, 8 Jun 2005, Tom Duffy wrote: On Wed, 2005-06-08 at 13:44 -0700, Tom Duffy wrote: On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote: Do you see any additional stability problems after applying this? I'm updating my OpenIB tree to see if that is my problem. This just in! I think your patch is fine, at least it doesn't introduce any new bugs, because after a reboot of both machines, restarting the SM, the quit test and the transaction test work fine. -tduffy ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
On Wed, Jun 08, 2005 at 01:52:47PM -0700, Tom Duffy wrote:
>On Wed, 2005-06-08 at 22:41 +0200, Bernhard Fischer wrote:
>> On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote:
>> >On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote:
>>
>> >> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally
>> >> would appreciate it, since these variants are harder to read for me.
>> >
>> >What do you want to see instead?
>> >
>> >if (this) {
>> >/* do stuff */
>> >}
>>
>> yes, this.
>
>I think James would take these types of patches.
>
>In any event, going through the code with fine toothed comb will happen
>after all the major upheavals, I think. *I* want to fix all the major
>architectural issues that are holding up acceptance before nit-picking.
certainly.
>
>If I have the time or inclination, I will clean up these things.
Index: TODO
===
--- TODO(revision 2572)
+++ TODO(working copy)
@@ -23,3 +23,7 @@ IB DAT PROVIDER
*) cache remote peer's address so that it does not need to be retrieved in
dapl_ep_query.
*) Replace dapl linked lists with Linux linked lists
+*) referencing
+ http://openib.org/pipermail/openib-general/2005-January/003149.html
+ and only then replace the checks turned up by
+ 'grep -r -E "NULL.*!=|!=.*NULL" *' with if(foo)... for better readability
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564)
On Wed, 2005-06-08 at 13:44 -0700, Tom Duffy wrote: > On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote: > > Do you see any additional stability problems after applying this? I'm > > updating my OpenIB tree to see if that is my problem. > > This just in! I think your patch is fine, at least it doesn't introduce any new bugs, because after a reboot of both machines, restarting the SM, the quit test and the transaction test work fine. -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] 2.6.11.11 NFS over IPoIB crash
We are running NFS over IPOIB, and are getting kernel panics under heavy NFS I/O. This is on a PowerMac G5, and the server is a dual opteron running 2.6.11 with the OpenIB code from subversion. It looks like a bug in nfs.. but we've only seen it using IPoIB... Is it worth trying to reproduce this over gigabit ethernet? Brett Bode wrote: Well I finally got 2.6.11.11 running this morning, but crashed it too. I guess I will try updating to the latest OpenIB code in subversion? Here is the kernel panic: Oops: Kernel access of bad area, sig: 11 [#1] SMP NR_CPUS=2 POWERMAC Modules linked in: ib_ipoib ib_sa eth1394 ib_mthca ib_mad ib_core ohci1394 ieee1394 NIP: C0038B58 XER: LR: C00390C8 CTR: C003B340 REGS: c00254c845f0 TRAP: 0300 Not tainted (2.6.11.11-G5) MSR: 90001032 EE: 0 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 42028488 DAR: c0060045b218 DSISR: 4001 TASK: c19047d0[2004] 'IOTest.ppc.x' THREAD: c00254c84000 CPU: 1 GPR00: 00060010 C00254C84870 C045F280 C19047D0 GPR04: C04B4480 3B9A1A60 C04B4488 GPR08: C045B208 C00254C84000 C0328C40 C0328C40 GPR12: 28028428 C037E000 C043B170 C0470340 GPR16: 0001 C044EE20 GPR20: 0003 GPR24: CFCC4000 CFD0B030 0001 GPR28: C04B3B40 0001 C0391A90 C00254C84870 NIP [c0038b58] .resched_task+0x38/0xcc LR [c00390c8] .try_to_wake_up+0x304/0x324 Call Trace: [c00254c848f0] [c00390c8] .try_to_wake_up+0x304/0x324 [c00254c849d0] [c003b3e0] .__wake_up_common+0x84/0xe8 [c00254c84a90] [c003b4a0] .__wake_up+0x5c/0x94 [c00254c84b40] [c0057840] .__queue_work+0x80/0xb4 [c00254c84be0] [c0057908] .queue_work+0x94/0xb0 [c00254c84c60] [c0280df8] .rpc_make_runnable+0xe0/0x154 [c00254c84cf0] [c0281324] .__rpc_do_wake_up_task+0xc8/0x24c [c00254c84d80] [c0281608] .rpc_wake_up_task+0xb0/0xd8 [c00254c84e20] [c027e61c] .xprt_complete_rqst+0x90/0x188 [c00254c84ed0] [c027ec3c] .udp_data_ready+0x230/0x290 [c00254c84fa0] [c0236714] .udp_queue_rcv_skb+0x268/0x51c [c00254c85040] [c0236fc8] .udp_rcv+0x600/0x7f0 [c00254c85160] [c0209e8c] .ip_local_deliver_finish+0xd4/0x35c [c00254c851f0] [c01fa928] .nf_hook_slow+0x184/0x1bc [c00254c852d0] [c020a360] .ip_local_deliver+0x24c/0x3bc [c00254c85360] [c020a6f4] .ip_rcv_finish+0x224/0x378 [c00254c85410] [c01fa928] .nf_hook_slow+0x184/0x1bc [c00254c854f0] [c020ad30] .ip_rcv+0x4e8/0x610 [c00254c855a0] [c01eac68] .netif_receive_skb+0x2ec/0x3a0 [c00254c85650] [c01eae34] .process_backlog+0x118/0x278 [c00254c85730] [c01eb07c] .net_rx_action+0xe8/0x24c [c00254c857f0] [c0047ea8] .__do_softirq+0xdc/0x1b8 [c00254c858b0] [c004800c] .do_softirq+0x88/0x90 [c00254c85940] [c00480b4] .local_bh_enable+0xa0/0xa4 [c00254c859c0] [c01ea274] .dev_queue_xmit+0x2f4/0x3a0 [c00254c85a60] [c01f3308] .neigh_connected_output+0x104/0x1a0 [c00254c85b00] [c020e824] .ip_finish_output2+0x100/0x2dc [c00254c85bc0] [c020fa18] .ip_fragment+0x4d8/0x800 [c00254c85cd0] [c024f768] .ip_refrag+0xa4/0xb0 [c00254c85d70] [c01fa2a4] .nf_iterate+0x128/0x190 [c00254c85e30] [c01fa86c] .nf_hook_slow+0xc8/0x1bc [c00254c85f10] [c020ebd8] .ip_finish_output+0x1d8/0x36c [c00254c85fd0] [c020fa18] .ip_fragment+0x4d8/0x800 [c00254c860e0] [c020e1e4] .dst_output+0x4c/0x80 [c00254c86170] [c01fa928] .nf_hook_slow+0x184/0x1bc [c00254c86250] [c021162c] .ip_push_pending_frames+0x4ec/0x550 [c00254c86300] [c02353b8] .udp_push_pending_frames+0x16c/0x260 [c00254c863c0] [c0235af0] .udp_sendmsg+0x644/0x758 [c00254c86570] [c023fdd0] .inet_sendmsg+0x88/0xb8 [c00254c86610] [c01dba14] .sock_sendmsg+0xdc/0x13c [c00254c86810] [c01dbab0] .kernel_sendmsg+0x3c/0x64 [c00254c868a0] [c01dff94] .sock_no_sendpage+0x9c/0xb8 [c00254c86970] [c0235d88] .udp_sendpage+0x184/0x200 [c00254c86a60] [c023fe9c] .inet_sendpage+0x9c/0x108 [c00254c86b10] [c028cd24] .xdr_sendpages+0x280/0x34c [c00254c86c80] [c027fad0] .xprt_transmit+0x170/0x550 [c00254c86d50] [c027cfd4] .call_transmit+0x1cc/0x2bc [c00254c86e00] [c02822fc] .__rpc_execute+0xf4/0x524 [c00254c86f60] [c00ff690] .nfs_execute_write+0x50/0x88 [c00254c87000] [c00ffbc0] .nfs_flush_list+0x4f8/0x644 [c00254
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
On Wed, 2005-06-08 at 22:41 +0200, Bernhard Fischer wrote:
> On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote:
> >On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote:
>
> >> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally
> >> would appreciate it, since these variants are harder to read for me.
> >
> >What do you want to see instead?
> >
> >if (this) {
> > /* do stuff */
> >}
>
> yes, this.
I think James would take these types of patches.
In any event, going through the code with fine toothed comb will happen
after all the major upheavals, I think. *I* want to fix all the major
architectural issues that are holding up acceptance before nit-picking.
If I have the time or inclination, I will clean up these things.
-tduffy
signature.asc
Description: This is a digitally signed message part
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564)
On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote:
> Do you see any additional stability problems after applying this? I'm
> updating my OpenIB tree to see if that is my problem.
This just in! Don't know if was caused by your patches or mine, but I
ran the server on 192.168.0.26, and tried to connect with the client, it
didn't connect right away, so I did a control-c. Then, the kernel
panic'ed:
[EMAIL PROTECTED] ~]# ./kdapltest -T Q -d -s 192.168.0.26 -D mthca0a
Server Name: 192.168.0.26
Server Net Address: 192.168.0.26
DT_cs_Client: Starting Test ...
DT_cs_Client: IA mthca0a opened
DT_cs_Client: EP created
* DAPL Characteristics *
Provider: mthca0a Version 1.0 DAPL 1.2
Adapter: Generic InfiniBand HCA by Linux Version 0.0
Supporting:
64512 EPs with 65535 DTOs and 0 in RDMA/RDs and 0ut RDMA/RDs each
65408 EVDs of up to 65535 entries (default S/R size is 256/256)
IOVs of up to 28 elements
131056 LMRs (and 131056 RMRs) of up to 0x bytes
Maximum MTU 0x8000 bytes, RDMA 0x8000 bytes
Maximum Private data size 92 bytes
* * * * * *
DT_cs_Client: Posting 1 recv buffer
DT_cs_Client: Connect Endpoint
DT_cs_Client: Await connection ...
<--- I DID A CONTROL-C HERE --->
[EMAIL PROTECTED] ~]# dapl_path_comp_handler: path resolution failed -110 retry
1802201964!!!
dl_ibpa_atht:_c romeqp__henand:dl peren: d epff_pfftr81
000x65ab622b6bbe46b0 6bal6bre6bad6by
completed? status 3
dapl_path_comp_handler: path resolution failed -110 retry 1802201965!!!
dapl_path_comp_handler: ep_ptr 0x6b6b6b6b6b6b6b6b
general protection fault: [1] SMP
CPU 0
Modules linked in: kdapltest ib_dat_provider dat ib_at ib_ipoib ib_sdp ib_cm
md5 ipv6 parport_pc lp parport autofs4 nfs lockd rfcomm l2cap bluetooth pcmcia
yenta_socket rsrc_nonstatic pcmcia_core sunrpc ext3 jbd dm_mod video container
button battery ac ohci_hcd tpm_nsc tpm i2c_amd756 i2c_core ib_mthca ib_sa
ib_mad ib_core tg3 floppy xfs exportfs mptscsih mptbase sd_mod scsi_mod
Pid: 11881, comm: ib_at_wq/0 Not tainted 2.6.12-rc6openib
RIP: 0010:[]
{:ib_dat_provider:dapl_evd_connection_callback+67}
RSP: 0018:8100218b9dc8 EFLAGS: 00010296
RAX: 6b6b6b6b6b6b6b6b RBX: 81005a22be40 RCX: 4008
RDX: 810075baaaf8 RSI: 883141a0 RDI: 0048
RBP: 81005a22be70 R08: 6b6b6b6b6b6b6b6b R09:
R10: 0010 R11: 0010 R12: 81007d9b8c50
R13: 81005a22be40 R14: 0292 R15: 882f5280
FS: 2aad7d60() GS:804e7880() knlGS:
CS: 0010 DS: 0018 ES: 0018 CR0: 8005003b
CR2: 7fd1dce6 CR3: 769db000 CR4: 06e0
Process ib_at_wq/0 (pid: 11881, threadinfo 8100218b8000, task
81004167caa0)
Stack: 81004167caa0 6b6b6b6b6b6b6b6b 4008006e
810075baaaf8 81003fdc8f10 0003
8100218b9e48 88302d18
Call Trace:{:ib_dat_provider:dapl_path_comp_handler+416}
{:ib_at:req_comp_work+27}
{worker_thread+476}
{default_wake_function+0}
{keventd_create_kthread+0}
{worker_thread+0}
{keventd_create_kthread+0}
{kthread+217} {schedule_tail+64}
{child_rip+8}
{keventd_create_kthread+0}
{kthread+0} {child_rip+0}
Code: 48 8b 80 90 00 00 00 48 89 44 24 28 c7 44 24 30 00 00 00 00
RIP {:ib_dat_provider:dapl_evd_connection_callback+67} RSP
<<0>3>geSlneabra cl orprruotpteciotin:on s ftaaurtlt=f: ff00f80010 [072]5b
aa<4af>S8,MP l en
51CP2=
1R eon
e:M 0odx5ula2escf l07in1/ke0xd 5ain2c:f0 k71da.
teLastst iusb_erda: t_[_a]t(
daibpl_i_dpoesibtro iy_b_cms_dpid +0ibxb_c9/m0 xbmde 5[ ibip_dv6at
_pparorpviordter_p])c l
p<04>f0 p:a rp6bor t6 ba ut6bofs 64b n 6fsb l6bock 6db rf6bco mm6d l 62cba
p6b bl 6uebto 6otbh 6bpc m6ciba 6bye
aP_sreocv keobtj:<4 s> tarsrtrc=f_nffonf8st10at07ic5b aapc8emc0,ia l_cenor=5e1
2 nRrpedcz : ex0xt3170 jfcbd2a 5/dm0x_m17od0fc v2aid5.eo
Lcoasntt aiusneerr: [bu] (ohkmciem_h_acdllo
tc+pm0x_n61sc/0 xetp0 m[ xfi2s]c_)am
d750060: i2 0c_0co 0re0 i00b_ m0th0ca 00 i b_00sa 00ib _m00ad 0 i1b _c00or ec
0tg a32 fl0o6pp 0y0 xf00s 0ex0p
tf01s0 :mp 0ts0c si02h m00ptb 0as0e 00sd _m00od 0 s0c si00_mo 0d2
4>Pi 0d:0 <114>88 82,0 coa3mm : 06ib_ 0at0_w 0q/01 N00ot <
taiNentxted o 2bj.6:. s12ta-rrtc6=fopffenf8ib10
075RIbaP:ad 01001, 0:le[]x1 70<4fc><2af5f/ff0xff17ff0f88c230a59c.
>L{:asibt _dusater_p:
>ro[]nec(ktimeonm__calallolbc+ac0xk+6167/0}xe
0 R[xSPfs:] 0)01<48:>
f0f80010:0 5900c3 dd0c08 0 E0F LA00GS : 0 010029 06
0R0AX:<4 6> b601b6 b600b6b a6b06b f6be RB00X: f04fff 081000 05a02
e4010 0:RC X:00 00 000200 0 04 82b
00RD f 0ff0f 81 750b1aa 0af08 R00SI : cbfff 0ff5ff f0488 310041a 00
0RD
I: 0048
RBP: 81
Message from [EMAIL PROTECTED] at Wed 0Jun 8 13:36:54 2005 ...
sins-5stinger-10 kerneal: general protectio
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote:
>On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote:
>> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally
>> would appreciate it, since these variants are harder to read for me.
>
>What do you want to see instead?
>
>if (this) {
> /* do stuff */
>}
yes, this.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564)
On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote: > Do you see any additional stability problems after applying this? I'm > updating my OpenIB tree to see if that is my problem. Sorry, I just looked at the code, didn't actually test it. Lemme give it a whirl... -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH][SDP] state transition simplification.
Quoting r. Libor Michalek <[EMAIL PROTECTED]>: > Subject: [PATCH][SDP] state transition simplification. > > > This patch simplifies the state transitions for connection management > within SDP. Previously the connection state was maintained using two > variables 'istate' and 'state' with a combined total of 34 different > values. The patch merges the two into a single variable 'state' and > removes redundancy for a total of 16 state values. Also the file > sdp_wall.c is removed who's original purpose was to keep the two state > variables synchronized. > > The simplification also paves the way for the next step, which is to > use the CM event for transitions instead of the current incorrest use > of CM state. > > The CQ event pending flags were moved from the connection flags field > to a new field in the event lock. > > Finally, I needed to ifdef out one of the send_dreq calls, because > I was seeing a problem when two DREQs would cross in flight and both > sides responded with a DREP. > > 13 files changed, 404 insertions(+), 1045 deletions(-) > > -Libor Works fine for me so far. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564)
Do you see any additional stability problems after applying this? I'm updating my OpenIB tree to see if that is my problem. james On Wed, 8 Jun 2005, Tom Duffy wrote: On Wed, 2005-06-08 at 15:48 -0400, James Lentini wrote: I like this update Tom. I'd propse a few changes: - remove dat_cno because it is not part of the kDAPL API - remove DAT_HANDLE_TO_PROVIDER - we can provide the functionality of dat_set_consumer_context and dat_get_consumer_context very easily by creating a dat_common structure (see my patch). In that case we can remove the user_context member from the dapl_common structure. - I've added the symbols removed from dat.h to kdat.h for legacy support. Let me know what you think. This looks real good, James. I agree with all of your changes. -tduffy ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] openib-commits administrative
Hi, May i as ask if diffs sent to the -commits list may be changed to be generated with the --show-c-fun option to diff? Something along the lines of svn diff --diff-cmd diff -x -urNp .. works for me. Knowing the function the change was based against simply makes it easier to follow. PS: yes, OT. But as #openib is completely orphaned (which is a pity, one would expect that folks at least once a week would check the logs they gather from reading there, but well) i know no other addr i could send that plea too. Sorry to the not concerned majority who i had to mux. PPS: having a cia-bot in #openib would be great, too, but i have no illusion that anyone would be willing to setup this. Prove me wrong. thank you, ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote:
> Tom, just curious..
> Do you plan to iterate over those disturbing
> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally
> would appreciate it, since these variants are harder to read for me.
What do you want to see instead?
if (this) {
/* do stuff */
}
or were you thinking:
if (this != NULL) {
/* do stuff */
}
> But then i also tend to prefer '* 4711e-0815' over '/ 0x10'
Come again?
-tduffy
signature.asc
Description: This is a digitally signed message part
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564)
On Wed, 2005-06-08 at 15:48 -0400, James Lentini wrote: > I like this update Tom. > > I'd propse a few changes: > > - remove dat_cno because it is not part of the kDAPL API > > - remove DAT_HANDLE_TO_PROVIDER > > - we can provide the functionality of dat_set_consumer_context and >dat_get_consumer_context very easily by creating a dat_common >structure (see my patch). In that case we can remove >the user_context member from the dapl_common structure. > > - I've added the symbols removed from dat.h to kdat.h for legacy >support. > > Let me know what you think. This looks real good, James. I agree with all of your changes. -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
I had all day meetings Monday and Tuesday that left me 2 days behind on email. I've just finished going through this. On Wed, 8 Jun 2005, Sean Hefty wrote: Tom Duffy wrote: On Wed, 2005-06-08 at 12:35 -0400, James Lentini wrote: Committed in revision 2572. Updated patch against 2572. Signed-off-by: Tom Duffy <[EMAIL PROTECTED]> James, is there a specific hold-up for this patch? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
On Wed, Jun 08, 2005 at 11:10:03AM -0700, Sean Hefty wrote: >Tom Duffy wrote: >>On Wed, 2005-06-08 at 12:35 -0400, James Lentini wrote: >> >>>Committed in revision 2572. >> >>Updated patch against 2572. >> >>Signed-off-by: Tom Duffy <[EMAIL PROTECTED]> > >James, is there a specific hold-up for this patch? > Tom, just curious.. Do you plan to iterate over those disturbing 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally would appreciate it, since these variants are harder to read for me. But then i also tend to prefer '* 4711e-0815' over '/ 0x10' ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] shutdown issue with SDP
On Wed, Jun 08, 2005 at 01:29:42PM -0400, William Jordan wrote: > I'm trying to run netperf over SDP using libsdp. I'm having > intermittent success, but usually, I get a failure. The client does a > shutdown (SHUT_WR), but the server gets an ECONNRESET, "Connection > reset by peer", on the recv instead of a zero length recv. > > I'm running 2568. > > Anyone seen similar/different results? Bill, Can you try the patch I sent out yesterday? It significantly simplifies the connection state machine within SDP. I ran it successfully with a number of the netperf tests. (TCP_CC, TCP_CRR, TCP_RR, TCP_STREAM) Thanks. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572)
Tom Duffy wrote: On Wed, 2005-06-08 at 12:35 -0400, James Lentini wrote: Committed in revision 2572. Updated patch against 2572. Signed-off-by: Tom Duffy <[EMAIL PROTECTED]> James, is there a specific hold-up for this patch? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] shutdown issue with SDP
I'm trying to run netperf over SDP using libsdp. I'm having intermittent success, but usually, I get a failure. The client does a shutdown (SHUT_WR), but the server gets an ECONNRESET, "Connection reset by peer", on the recv instead of a zero length recv. I'm running 2568. Anyone seen similar/different results? -- Bill Jordan SilverStorm Technologies ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RMPP and timeouts/retries
Hal Rosenstock wrote: Let me clarify a little: The no timeout is set based on resp_expected flag which is set at a higher level (presumably method based). When I did have retries set, things did not work properly. Unfortunately I was unable to get IB traces yesterday of what was going on but will try again once the analyzer is straightened out. I did have retries on with the original traces I obtained and discussed on the list where there were bigger problems than the ones observed by Tom yesterday. If you could just describe what you did/saw, I will take a look at the code to see if I can find anything. Traces are helpful, but not necessary. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] kdapltest: fix pointer to pointer bug
Committed in revision 2572.
On Thu, 2 Jun 2005, Tom Duffy wrote:
tduffy> In my work going through trying to get rid of the opaque dat handles, I
tduffy> came across what looks like a bug in kdapltest. I don't think
tduffy> DT_Performance_Test_Create() and DT_Performance_Test_Client() should
tduffy> take a DAT_IA_HANDLE * as an argument as this would be a pointer to a
tduffy> pointer.
tduffy>
tduffy> Of course, the compiler didn't catch it until I changed it to a real
tduffy> struct pointers thus showing one of the problems with using opaques.
tduffy>
tduffy> Signed-off-by: Tom Duffy <[EMAIL PROTECTED]>
tduffy>
tduffy> Index: linux-kernel-clean/test/dapltest/test/dapl_performance_util.c
tduffy> ===
tduffy> --- linux-kernel-clean/test/dapltest/test/dapl_performance_util.c
(revision 2532)
tduffy> +++ linux-kernel-clean/test/dapltest/test/dapl_performance_util.c
(working copy)
tduffy> @@ -33,7 +33,7 @@
tduffy> boolean_t
tduffy> DT_Performance_Test_Create (
tduffy> Per_Test_Data_t*pt_ptr,
tduffy> -DAT_IA_HANDLE *ia_handle,
tduffy> +DAT_IA_HANDLE ia_handle,
tduffy> struct sockaddr * remote_ia_addr,
tduffy> boolean_t is_server,
tduffy> boolean_t is_remote_little_endian,
tduffy> Index: linux-kernel-clean/test/dapltest/test/dapl_performance_client.c
tduffy> ===
tduffy> --- linux-kernel-clean/test/dapltest/test/dapl_performance_client.c
(revision 2532)
tduffy> +++ linux-kernel-clean/test/dapltest/test/dapl_performance_client.c
(working copy)
tduffy> @@ -34,7 +34,7 @@ int
tduffy> DT_Performance_Test_Client (
tduffy> Params_t *params_ptr,
tduffy> Per_Test_Data_t*pt_ptr,
tduffy> -DAT_IA_HANDLE *ia_handle,
tduffy> +DAT_IA_HANDLE ia_handle,
tduffy> struct sockaddr * remote_ia_addr)
tduffy> {
tduffy> Performance_Test_t *test_ptr = NULL;
tduffy> Index: linux-kernel-clean/test/dapltest/include/dapl_proto.h
tduffy> ===
tduffy> --- linux-kernel-clean/test/dapltest/include/dapl_proto.h
(revision 2532)
tduffy> +++ linux-kernel-clean/test/dapltest/include/dapl_proto.h
(working copy)
tduffy> @@ -236,7 +236,7 @@ voidDT_Performance_Cmd_Endia
tduffy> /* dapl_performance_client.c */
tduffy> int DT_Performance_Test_Client ( Params_t
*params_ptr,
tduffy>Per_Test_Data_t * pt_ptr,
tduffy> - DAT_IA_HANDLE *
ia_handle,
tduffy> + DAT_IA_HANDLE ia_handle,
tduffy>struct sockaddr *remote);
tduffy>
tduffy> boolean_tDT_Performance_Test_Client_Connect (
tduffy> @@ -261,7 +261,7 @@ boolean_tDT_Performance_Test
tduffy>
tduffy> /* dapl_performance_util.c */
tduffy> boolean_tDT_Performance_Test_Create (Per_Test_Data_t *
pt_ptr,
tduffy> - DAT_IA_HANDLE * ia_handle,
tduffy> + DAT_IA_HANDLE ia_handle,
tduffy>struct sockaddr
*remote_ia_addr,
tduffy>boolean_t is_server,
tduffy>boolean_t
is_remote_little_endian,
tduffy>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] uverbs performance; ibv_pingpong poll vs sleep
Steven> I have compaired the data rates using ibv_pingpong with Steven> and without the -e option. Therefore, using polling and Steven> waiting the CQ events (sleeping). Steven> Is there any way to trade off the data rate with the CPU Steven> usage (I was thinking of some timeout from polling). I suppose you could have some sort of adaptive polling scheme that spins polling for a while and then sleeps waiting for an event. However, as Michael said, it's probably better to use pipelining and post multiple send work requests. This hides the latency of getting a completion event by keeping the HCA busy. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RMPP and timeouts/retries
Hal Rosenstock wrote: Setting timeout > 0 when sending indicates that a MAD should be received with the response bit set that is in reply to the request. If timeout = 0 but RMPP is marked active, the MAD will still invoke RMPP and not complete until all segments have been ACKed. (I.e. the completion of an RMPP send indicates that the MAD was received.) If timeout = 0, does RMPP still indicate a timeout if an ACK is lost and the other end does not reACK ? It should, yes. The send will complete with a timeout error. Retries is a little more complex. It indicates the number of times to send a request in hopes of receiving a response (if one is expected) or an ACK (if using RMPP). Any effect on ABORT and/or STOP ? I wouldn't think so, receiving an ABORT or STOP should fail the send immediately. I will try that. Using retry and timeout on the SA side is definitely problematic. Right now it is set to no retries and no timeout on the SA side. I will change and retest. This should work, but will not be able to recover from a lost or delayed MAD. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own
Hal Rosenstock wrote: On Wed, 2005-06-08 at 11:44, James Lentini wrote: We interpreted the above to mean "give the connection protocol as much time as it needs to establish a connection, but don't mask errors (no path to the remove node, etc.)". For that reason we changed the variable name to DAT_TIMEOUT_MAX. But if the REQ is lost, the timeout is really really long (longer than most will wait for an error). Transaction test also appears to be using this as well as the quit test. My interpretation was that this is a DAPL level timeout and did not necessarily relate to a timeout for a single CM REQ. That is, there could still be a different timeout specified to the CM, but the number of retries could be infinite. Note that I'm not saying that an infinite timeout makes sense, but the use of TIMEOUT_MAX seems reasonable. To me that indicates that DAPL decides how long is needed to establish a timeout, and it manages all retries. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own
On Wed, 2005-06-08 at 11:44, James Lentini wrote: > We interpreted the above to mean "give the connection protocol as > much time as it needs to establish a connection, but don't mask > errors (no path to the remove node, etc.)". For that reason we changed > the variable name to DAT_TIMEOUT_MAX. But if the REQ is lost, the timeout is really really long (longer than most will wait for an error). Transaction test also appears to be using this as well as the quit test. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own
On Wed, 2005-06-08 at 11:44 -0400, James Lentini wrote: > We interpreted the above to mean "give the connection protocol as > much time as it needs to establish a connection, but don't mask > errors (no path to the remove node, etc.)". For that reason we changed > the variable name to DAT_TIMEOUT_MAX. Well, let's say the end node is not there yet. Should the CM keep trying indefinitely waiting for somebody to show up and respond? -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own
On Tue, 7 Jun 2005, Hal Rosenstock wrote: On Tue, 2005-05-31 at 14:17, James Lentini wrote: Here's the specification's exact description: timeout: Duration of time, in microseconds, that a consumer waits for connection establishment. The value of DAT_TIMEOUT_INFINITE represents no timeout, indefinite wait. Values must be positive. What is the purpose of an infinite timeout (other than the obvious) ? The quit test uses this feature. Not sure if other tests do as well. What happens if the REQ is lost ? Why would someone want an infinite timeout ? We interpreted the above to mean "give the connection protocol as much time as it needs to establish a connection, but don't mask errors (no path to the remove node, etc.)". For that reason we changed the variable name to DAT_TIMEOUT_MAX. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [ib_at] oops in ib_at when running kdapltest
On Tue, 2005-06-07 at 11:56, Itamar Rabenstein wrote:
> attached is the output of /var/log/messages that contain ib_at oops with
> ib_at debug trun on.
> svn rev 2564 (ib + kdapl + at from users/jlentini)
> test is : ./kdapltest -T T -s 11.1.4.43 -D mthca0a -d -t 2 -w 8 -i 20 client
> SR server SR
> i run the test 4-5 times before the oops occur
> (In order to get the oops your kernel must be compiled with
> CONFIG_DEBUG_PAGEALLOC)
I've done this over 30 times and still not recreated this :-( My kernel
is compiled with CONFIG_DEBUG_PAGEALLOC ever since the last time... The
only difference now is my CM is instrumented which may change the timing
slightly if this is due to a timing window. Out of curiousity, what SM
are you using ?
The sequence appears the same as all the ones which work prior to this.
Also, in looking at resolve_path, I do not see a way for this to occur
as long at the pointer to the path_rec struct is valid other than some
scribbling problem (stack trashed somehow ?).
So, can you redo this with one more change as follows:
In resolve_path right before the call to ib_sa_path_rec_get insert the
line:
DEBUG("ib_sa_path_rec_get");
I want to rule in or out a possible theory. Thanks.
-- Hal
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: uverbs performance; ibv_pingpong poll vs sleep
Quoting r. Steven Wooding <[EMAIL PROTECTED]>: > Subject: uverbs performance; ibv_pingpong poll vs sleep > > Hi, > > I wonder if anyone could help me with uverbs performance. > > I have compaired the data rates using ibv_pingpong with and without the -e > option. Therefore, using polling and waiting the CQ events (sleeping). > > For a data message size of 16K I get the following results: > > Poll: 718 MB/s with CPU at 100% > Sleep: 479 MB/s with CPU at 16% What do you do for CPU utilisation measurement? > I suppose the decrease in throughput is due to the time it takes to get the CQ > event. Yes, pingpong is not really a bandwidth benchmark. It never has more than one outstanding transaction on a qp, so its bandwidth is very sensitive to roundtrip times. > Is there any way to trade off the data rate with the CPU usage (I was thinking > of some timeout from polling). > > Any suggests would be very welcome. > > Regards, > > Steve. Steve, I think an application should create some kind of pipelining, performing multiple sends before waiting for events. I have a working example of how this can be done, but unfortunately its not yet ready for publication. The idea basically is to have n outstanding buffers and tx of size n. Count the number of send requests sent and completed. Now basically replace each post send with while (send - completed < n) post send As n grows, you shall see different cpu utilization/bandwidth points. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] uverbs performance; ibv_pingpong poll vs sleep
Hi, I wonder if anyone could help me with uverbs performance. I have compaired the data rates using ibv_pingpong with and without the -e option. Therefore, using polling and waiting the CQ events (sleeping). For a data message size of 16K I get the following results: Poll: 718 MB/s with CPU at 100% Sleep: 479 MB/s with CPU at 16% I suppose the decrease in throughput is due to the time it takes to get the CQ event. Is there any way to trade off the data rate with the CPU usage (I was thinking of some timeout from polling). Any suggests would be very welcome. Regards, Steve. How much free photo storage do you get? Store your holiday snaps for FREE with Yahoo! Photos. Get Yahoo! Photos___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RMPP and timeouts/retries
On Wed, 2005-06-08 at 06:58, Hal Rosenstock wrote: > Right now it is set to no retries and no timeout on the SA > side. I will change and retest. Let me clarify a little: The no timeout is set based on resp_expected flag which is set at a higher level (presumably method based). When I did have retries set, things did not work properly. Unfortunately I was unable to get IB traces yesterday of what was going on but will try again once the analyzer is straightened out. I did have retries on with the original traces I obtained and discussed on the list where there were bigger problems than the ones observed by Tom yesterday. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RMPP and timeouts/retries
On Wed, 2005-06-08 at 01:04, Sean Hefty wrote: > >> I'm having some trouble using timeouts and retries with RMPP. The issue > >> I see is that what constitutes a response is different than normal MADs > >> for RMPP. With the current "definition" of response, is the > >> timeout/retry only usable on the SA client side where the normal > >> definition is more closely followed ? If so, is there still an issue > >> with SA GetTable as the initial request is not RMPP ? > > The SA client should set the timeout > 0 to indicate that a response MAD is > expected. It can set retries, but isn't required to do so. OK. > >Is the response not at the RMPP level ? So a response would be for dual > >ended RMPP ? > > Request/response is separate from RMPP. Request/response is defined at the > message level from the viewpoint of the user of the MAD layer. (I.e. is the > response bit set in the MAD header, regardless of the size of the MAD.) Understood. It's method based (response bit and trap repress). > Setting timeout > 0 when sending indicates that a MAD should be received with > the response bit set that is in reply to the request. If timeout = 0 but RMPP > is marked active, the MAD will still invoke RMPP and not complete until all > segments have been ACKed. (I.e. the completion of an RMPP send indicates that > the MAD was received.) If timeout = 0, does RMPP still indicate a timeout if an ACK is lost and the other end does not reACK ? > Retries is a little more complex. It indicates the number of times to send a > request in hopes of receiving a response (if one is expected) or an ACK (if > using RMPP). Any effect on ABORT and/or STOP ? > >> On the SA side, there does not appear to be a way to use this feature. > >> Is that correct ? > > For the SA side responding to a request, you would want to set retries > 1, > but > timeout = 0. Timeout of 0 indicates that no response is expected, since the > SA > is sending the response. Having a retry count would allow a retransmission if > an ACK were lost. I will try that. Using retry and timeout on the SA side is definitely problematic. Right now it is set to no retries and no timeout on the SA side. I will change and retest. > You should be able to mix RMPP and non-RMPP MADs without restriction: non-RMPP > request - RMPP response, RMPP request - non-RMPP response, RMPP request and > response, non-RMPP request and response. If one of these conditions doesn't > work, then there's a bug in the code. I'm pretty sure that I tested all of > these combinations, but that doesn't mean that it won't hit a bug talking with > another RMPP implementation. The main new one right now is non RMPP request - RMPP response. SA does not currently support any of the dual sided RMPP methods (so there is no RMPP request - RMPP response). (Non RMPP request - non RMPP response is what is mainly used). I'm not sure the other combination has a use. Thanks. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCH] [CM] add OS administered service IDs
Thanks Sean,
I will update the kdapl code soon.
-Itamar
> -Original Message-
> From: Sean Hefty [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, June 07, 2005 8:50 PM
> To: openib-general@openib.org
> Subject: [openib-general] [PATCH] [CM] add OS administered service IDs
>
>
> The following patch adds support for OS administered service IDs.
>
> Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
>
>
> Index: include/ib_cm.h
> ===
> --- include/ib_cm.h (revision 2563)
> +++ include/ib_cm.h (working copy)
> @@ -311,16 +311,20 @@ struct ib_cm_id *ib_create_cm_id(ib_cm_h
> */
> void ib_destroy_cm_id(struct ib_cm_id *cm_id);
>
> +#define IB_CM_ASSIGN_SERVICE_ID
> __constant_cpu_to_be64(0x0200ULL)
> +
> /**
> * ib_cm_listen - Initiates listening on the specified service ID for
> * connection and service ID resolution requests.
> * @cm_id: Connection identifier associated with the listen request.
> * @service_id: Service identifier matched against incoming
> connection
> * and service ID resolution requests. The service ID
> should be specified
> - * network-byte order.
> + * network-byte order. If set to IB_CM_ASSIGN_SERVICE_ID,
> the CM will
> + * assign a service ID to the caller.
> * @service_mask: Mask applied to service ID used to listen across a
> * range of service IDs. If set to 0, the service ID is matched
> - * exactly.
> + * exactly. This parameter is ignored if %service_id is set to
> + * IB_CM_ASSIGN_SERVICE_ID.
> */
> int ib_cm_listen(struct ib_cm_id *cm_id,
>u64 service_id,
> Index: core/cm.c
> ===
> --- core/cm.c (revision 2563)
> +++ core/cm.c (working copy)
> @@ -64,6 +64,7 @@ static struct ib_cm {
> struct list_head device_list;
> rwlock_t device_lock;
> struct rb_root listen_service_table;
> + u64 listen_service_id;
> /* struct rb_root peer_service_table; todo: fix peer to peer */
> struct rb_root remote_qp_table;
> struct rb_root remote_id_table;
> @@ -718,14 +719,23 @@ int ib_cm_listen(struct ib_cm_id *cm_id,
> unsigned long flags;
> int ret = 0;
>
> + if ((service_id & IB_CM_ASSIGN_SERVICE_ID) ==
> IB_CM_ASSIGN_SERVICE_ID &&
> + (service_id != IB_CM_ASSIGN_SERVICE_ID))
> + return -EINVAL;
> +
> cm_id_priv = container_of(cm_id, struct cm_id_private, id);
> BUG_ON(cm_id->state != IB_CM_IDLE);
>
> cm_id->state = IB_CM_LISTEN;
> - cm_id->service_id = service_id;
> - cm_id->service_mask = service_mask ? service_mask : ~0ULL;
>
> spin_lock_irqsave(&cm.lock, flags);
> + if (service_id == IB_CM_ASSIGN_SERVICE_ID) {
> + cm_id->service_id =
> __cpu_to_be64(cm.listen_service_id++);
> + cm_id->service_mask = ~0ULL;
> + } else {
> + cm_id->service_id = service_id;
> + cm_id->service_mask = service_mask ?
> service_mask : ~0ULL;
> + }
> cur_cm_id_priv = cm_insert_listen(cm_id_priv);
> spin_unlock_irqrestore(&cm.lock, flags);
>
> @@ -3251,6 +3261,7 @@ static int __init ib_cm_init(void)
> rwlock_init(&cm.device_lock);
> spin_lock_init(&cm.lock);
> cm.listen_service_table = RB_ROOT;
> + cm.listen_service_id =
> __constant_be64_to_cpu(IB_CM_ASSIGN_SERVICE_ID);
> cm.remote_id_table = RB_ROOT;
> cm.remote_qp_table = RB_ROOT;
> cm.remote_sidr_table = RB_ROOT;
>
>
>
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [patch][kdapl] enable kdapltest -T P
> > Current openib gen2 code is not reporting the max cq size > and i dont think > > that we should put > > a fix number . > > if we want to get the number we need Roland to fill this > number in mthca but > > as Roland said before > > "what real App will meed this number?" > > I see the initialization code you are refering to in > mthca_query_device of mthca_provider.c. > > I think a "real app" would use this number in exactly the same way > that dapltest's performance subtest uses it: > > pipeline_length = min(max_cqe, max_qp_wr) > > Why would a limit like the one above be unnecessary? you can try to printk the value and you will see it is not initialized. it is not enough to assign value to the output attr struct if this value is not initialized (;-) i dont mind if Roland will implement it but for now it is not implemented. Please ci the patch for now in order to enable kdapltest -T P and if Roland will implement it then unremark the lines. > >> > >> Was this also a problem in the transaction test? > >> > > > > Yes but in order to fall on the bug you need to alloc a > very small buffer > > (like 12 byte) > > and this is only in -T P . > > So is it only in the performance test or would it occur in the > transaction test if I specified a small buffer? It will fail for every small buffer (include in -T T ) > > Why do we need to translate the virtual address to a physical address > in all cases? Will this interact properly with the transaction test's > -M (memory type) option? Current openib gen2 register only physical memory. the code will work also for any -M option . -Itamar ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
