Re: [openib-general] [PATCH 1/7] IB/ipath - performance improvements via mmap of queues
I find it somehow hard to review your patch set as of two reasons: (a) all the patches having the same subject line a) I thought using the same subject line was the convention since it is essentially one patch. I split it due to size and the fact that each patch has a different owner. Nope, the convention is to have the subject line telling what it this patch role within the patch series very similarly to what you have stated in some beginning of most of the patches. Another useful practice is to have all the patches sent over the same thread. The bulk of the changes are to the InfiniPath kernel driver (ib_ipath) to support mmap'ing the CQ and receive queues (QP, SRQ) into the user level verbs library. The changes to the core IB were neccessary in order to allow additional information (i.e., the mmap offset) to be returned from the kernel driver to the user level verbs driver plugin. thanks for the clarification. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] where are the userspace sources after installing the ofed-1.0?
Thank you very much! On 8/13/06, Tziporet Koren [EMAIL PROTECTED] wrote: Ian Jiang wrote: I know the userspace sources could be got from OFED-1.0.tgz\SOURCES\openib-1.0\src\userspace. But I am wondering wheather these source will be installed when installing the OFED-1.0 RPMs. I cannot find them under /usr/local/ofed/ Thanks! In OFED 1.0 sources are not installed at all after the installation. In OFED 1.1 after installation the sources are located on: prefix/src/ Tziporet -- Ian Jiang ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCHv2] osm: OSM crash TRIVIAL bug fix
Hi Hal.This patch fixes an OSM crash when working with Cisco's stack.
Cisco's doesn't follow the same TID convention when generating
transaction id which in some bad flow revealed this bug in the get_madw lookup.The bug is in get_madw which does not cleanup old pointers to retrieved madw and also does not detect lookup of its reserved free entry of key==0.
(This better text replaces my previous patch:OSM crash when working with Cisco's TopSpin stack)
YevgenySigned-off-by:Yevgeny Kliteynik
[EMAIL PROTECTED]
Index: osm/libvendor/osm_vendor_ibumad.c===
--- osm/libvendor/osm_vendor_ibumad.c (revision 8614)
+++ osm/libvendor/osm_vendor_ibumad.c (working copy)@@ -141,12 +141,20 @@ get_madw(osm_vendor_t *p_vend, ib_net64_
ib_net64_t mtid = (*tid cl_ntoh64(0xllu));
osm_madw_t *res;
+ /*+ * Since mtid == 0 is the empty key we should not
+ * waste time looking for it+ */
+ if (mtid == 0)+ return 0;
+ cl_spinlock_acquire( p_vend-match_tbl_lock );
for (m = p_vend-mtbl.tbl, e = m + p_vend-mtbl.max; m e; m++) {
if (m-tid == mtid) { m-tid = 0;
*tid = mtid;
res = m-v;+ m-v = NULL; /* just make sure we do not point to free'd madw */
cl_spinlock_release( p_vend-match_tbl_lock );
return res; }
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] osm: TRIVIAL wrong description in log message
Hi Hal, Inspecting the log messages of the error flow in osm_vendor_send I have noticed that the terms request and response are reversed: If we are sending with response_expected it means we are sending our request... Yevgeny Signed-off-by: Yevgeny Kliteynik [EMAIL PROTECTED] Index: osm/libvendor/osm_vendor_ibumad.c === --- osm/libvendor/osm_vendor_ibumad.c (revision 8614) +++ osm/libvendor/osm_vendor_ibumad.c (working copy) @@ -1148,7 +1156,7 @@ Resp: osm_log(p_vend-p_log, OSM_LOG_DEBUG, osm_vendor_send: Completed sending %s p_madw = %p\n, - resp_expected ? response : request, p_madw); + resp_expected ? request : response, p_madw); Exit: OSM_LOG_EXIT( p_vend-p_log ); return( ret ); ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] Fix potential deadlock in mthca
On Fri, 2006-08-11 at 09:15 -0700, Arjan van de Ven wrote: Roland Dreier wrote: Here's a long-standing bug that lockdep found very nicely. Ingo/Arjan, can you confirm that the fix looks OK and I am using spin_lock_nested() properly? I couldn't find much documentation or many examples of it, so I'm not positive this is the right way to handle this fix. looks correct to me; Acked-by: Arjan van de Ven [EMAIL PROTECTED] looks good to me too. Acked-by: Ingo Molnar [EMAIL PROTECTED] btw., we could introduce a new spin-lock op: spin_lock_double(l1, l2); ... spin_unlock_double(l1, l2); because some other code, like kernel/sched.c, fs/dcache.c and kernel/futex.c uses quite similar locking. Ingo ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [opensm] the default behavior of the openSM causes problems (configure the PKey table)
Hi. I noticed that the behavior of the openSM was changed in the latest driver: in the past, every HCA was configured (by the FW) with 0x in the first entry. today, the PKey table is being configured by the openSM: the first entry is being set to 0x7fff (except for the host that the SM is being executed from) This behavior is very problemtic because not all of the users would like to change the default PKey table (for example: MPI users). Users that will try to use OFED 1.1 (in the same way they used OFED 1.0) will get unexplained failures, because the connectivity because the nodes will be broken. (even the perfquery started to fail after executing the SM) I think that the default behavior of the openSM should be: not to change the PKey table, unless the user provided a PKey table policy file. Here are the props of the machines * Host Architecture : x86_64 Linux Distribution: Red Hat Enterprise Linux AS release 4 (Nahant Update 3) Kernel Version: 2.6.9-34.ELsmp GCC Version : gcc (GCC) 3.4.5 20051201 (Red Hat 3.4.5-2) Memory size : 4039892 kB Driver Version: gen2_linux-20060813-1905 (REV=8916) HCA ID(s) : mthca0 HCA model(s) : 23108 FW version(s) : 3.4.927 Board(s) : MT_003001 * thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [opensm] the default behavior of the openSM causes problems (configure the PKey table)
Hi Dotan, On 14:36 Mon 14 Aug , Dotan Barak wrote: Hi. I noticed that the behavior of the openSM was changed in the latest driver: in the past, every HCA was configured (by the FW) with 0x in the first entry. today, the PKey table is being configured by the openSM: the first entry is being set to 0x7fff (except for the host that the SM is being executed from) This is OpenSM default behavior in the case where partition policy file exists (/etc/osm-partitions.conf is default name), even if it is empty. When the partition policy file does not exist default 0x pkey value (full membership) should be inserted for all end-ports. I am not able to reproduce the reported behavior with my setup. If you are please describe your scenario. Thanks. Sasha This behavior is very problemtic because not all of the users would like to change the default PKey table (for example: MPI users). Users that will try to use OFED 1.1 (in the same way they used OFED 1.0) will get unexplained failures, because the connectivity because the nodes will be broken. (even the perfquery started to fail after executing the SM) I think that the default behavior of the openSM should be: not to change the PKey table, unless the user provided a PKey table policy file. Here are the props of the machines * Host Architecture : x86_64 Linux Distribution: Red Hat Enterprise Linux AS release 4 (Nahant Update 3) Kernel Version: 2.6.9-34.ELsmp GCC Version : gcc (GCC) 3.4.5 20051201 (Red Hat 3.4.5-2) Memory size : 4039892 kB Driver Version: gen2_linux-20060813-1905 (REV=8916) HCA ID(s) : mthca0 HCA model(s) : 23108 FW version(s) : 3.4.927 Board(s) : MT_003001 * thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [opensm] the default behavior of the openSM causes problems (configure the PKey table)
Thanks for the quick response. On Monday 14 August 2006 15:17, Sasha Khapyorsky wrote: Hi Dotan, On 14:36 Mon 14 Aug , Dotan Barak wrote: Hi. I noticed that the behavior of the openSM was changed in the latest driver: in the past, every HCA was configured (by the FW) with 0x in the first entry. today, the PKey table is being configured by the openSM: the first entry is being set to 0x7fff (except for the host that the SM is being executed from) This is OpenSM default behavior in the case where partition policy file exists (/etc/osm-partitions.conf is default name), even if it is empty. You are right, this file was exist in the host with the following content: Default=0x7fff : ALL, SELF=full ; YetAnotherOne = 0x300 : ALL, SELF=full ; partition1 = 0x1 : 0x0002c9020020b1c9=full; When the partition policy file does not exist default 0x pkey value (full membership) should be inserted for all end-ports. I am not able to reproduce the reported behavior with my setup. If you are please describe your scenario. Thanks. I have 2 machines connected b2b (without any switch in the middle) connected using one cable from port 1 to port 1. I executed the SM from one machine, and in the other machine perfquery i got the failure. Why doesn't the SM print that this file was found? this way, users can know that this file was found in their machine and the SM is using those rules (instead of the default rules, as you described) Thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [opensm] the default behavior of the openSM causes problems (configure the PKey table)
On 15:36 Mon 14 Aug , Dotan Barak wrote: Thanks for the quick response. On Monday 14 August 2006 15:17, Sasha Khapyorsky wrote: Hi Dotan, On 14:36 Mon 14 Aug , Dotan Barak wrote: Hi. I noticed that the behavior of the openSM was changed in the latest driver: in the past, every HCA was configured (by the FW) with 0x in the first entry. today, the PKey table is being configured by the openSM: the first entry is being set to 0x7fff (except for the host that the SM is being executed from) This is OpenSM default behavior in the case where partition policy file exists (/etc/osm-partitions.conf is default name), even if it is empty. You are right, this file was exist in the host with the following content: Default=0x7fff : ALL, SELF=full ; YetAnotherOne = 0x300 : ALL, SELF=full ; partition1 = 0x1 : 0x0002c9020020b1c9=full; When the partition policy file does not exist default 0x pkey value (full membership) should be inserted for all end-ports. I am not able to reproduce the reported behavior with my setup. If you are please describe your scenario. Thanks. I have 2 machines connected b2b (without any switch in the middle) connected using one cable from port 1 to port 1. I executed the SM from one machine, and in the other machine perfquery i got the failure. OpenSM configures pkey tables as requested in osm-partitions.conf file - this is the reason. Just remove (or rename) this file if you don't need it. Why doesn't the SM print that this file was found? Yes, some prints may be helpful. Do you mean just log file or would prefer the message on stdout too? Sasha this way, users can know that this file was found in their machine and the SM is using those rules (instead of the default rules, as you described) Thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCHv2] osm: OSM crash TRIVIAL bug fix
Hi Yevgeny,
On Mon, 2006-08-14 at 04:13, Yevgeny Kliteynik wrote:
Hi Hal.
This patch fixes an OSM crash when working with Cisco's stack.
Cisco's doesn't follow the same TID convention when generating
transaction id which in some bad flow revealed this bug in the
get_madw lookup.
The bug is in get_madw which does not cleanup old pointers to
retrieved madw and also does not detect lookup of its reserved free
entry of key==0.
(This better text replaces my previous patch:
OSM crash when working with Cisco's TopSpin stack)
Yevgeny
Thanks. Good find.
Signed-off-by: Yevgeny Kliteynik [EMAIL PROTECTED]
Index: osm/libvendor/osm_vendor_ibumad.c
===
--- osm/libvendor/osm_vendor_ibumad.c (revision 8614)
+++ osm/libvendor/osm_vendor_ibumad.c (working copy)
@@ -141,12 +141,20 @@ get_madw(osm_vendor_t *p_vend, ib_net64_
ib_net64_t mtid = (*tid cl_ntoh64(0xllu));
osm_madw_t *res;
+/*
+ * Since mtid == 0 is the empty key we should not
+ * waste time looking for it
+ */
+if (mtid == 0)
+ return 0;
+
cl_spinlock_acquire( p_vend-match_tbl_lock );
for (m = p_vend-mtbl.tbl, e = m + p_vend-mtbl.max; m e;
m++) {
if (m-tid == mtid) {
m-tid = 0;
*tid = mtid;
res = m-v;
+ m-v = NULL; /* just make sure we do not point
to free'd madw */
This line wrapped so there is something wrong with your mailer.
Also, is this line really needed (and if so why) ? I know you did say
it cleans up old pointers to retrieved madw but this shouldn't be
accessed, right ? Also, if this is added here, there are other places
where the same thing should be done ?
cl_spinlock_release( p_vend-match_tbl_lock
);
return res;
}
Applied to trunk and 1.1 with the exception noted above.
-- Hal
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [opensm] the default behavior of the openSM causes problems (configure the PKey table)
On Monday 14 August 2006 16:09, Sasha Khapyorsky wrote: Why doesn't the SM print that this file was found? Yes, some prints may be helpful. Do you mean just log file or would prefer the message on stdout too? I believe that most of the users don't look at the log file, so a message in the stdout can be usefull. thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] osm: TRIVIAL wrong description in log message
Hi Yevgeny, On Mon, 2006-08-14 at 04:24, Yevgeny Kliteynik wrote: Hi Hal, Inspecting the log messages of the error flow in osm_vendor_send I have noticed that the terms request and response are reversed: If we are sending with response_expected it means we are sending our request... Yevgeny Signed-off-by: Yevgeny Kliteynik [EMAIL PROTECTED] Thanks. Applied to trunk and 1.1 (with minor cosmetic change). -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [openfabrics-ewg] OFED 1.1-rc1 is available
On Sun, 2006-08-13 at 09:14, Tziporet Koren wrote: Hal Rosenstock wrote: Target release date: 12-Sep Intermediate milestones: 1. Create 1.1 branch of user level: 27-Jul - done 2. RC1: 8-Aug - done 3. Feature freeze (RC2): 17-Aug What is the start build date for RC2 ? When do developers need to have their code in by to make RC2 ? We will start on Tue 15-Aug. Is this OK with you? Yes; I just needed to know when this needed to be done by. Thanks. -- Hal 4. Code freeze (rc-x): 6-Sep Is this 1 or 2 RCs beyond RC2 in order to make this ? I hope one but I guess it will be two more RCs. Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RDMA_READ SGE
[...snip...] Practically I don't see reporting the exact values as a priority - I think applications really can figure this out easier by attempting operating with relevant parameters and fallback to smaller values on failure. Perhaps it's not a priority, and it is certainly technically possible, however, I would say that this is quite a burden to place on every application to attempt to discover limits by submitting WR and checking when they fail. Who do they talk to while they're doing all this? It seems to me that it would be better to expand the set of attributes. Leave the current max_sge for backward compatibility, but let apps reliably query limits instead of attempting to discover them through trial and error. But assuming that applications really need this information - it seems we really should generalize this - maybe make the device provide a function mapping QP attributes and operation kinds to the max set of values allowed? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] osm: Dynamic verbosity control per file
Hi Yevgeny, On Sun, 2006-08-13 at 15:16, Yevgeny Kliteynik wrote: Hi Hal. Is it hard to find which file and line an opensm log message comes from ? Is this functionality really needed ? I'm guessing that your question refers to the second bullet - logging source code filename and line number. IMHO, this is a nice-to-have functionality - when debugging the SM that runs with a high verbosity, the log has a lot of information, and it's much easier to follow the SM flow when each message tells where exactly it came from. OK. I'll work on adding it but may not get to it for another day or so (so it won't be in OFED 1.1 rc2) and may have other comments on the specifics of the patch. -- Hal Regards, Yevgeny Kliteynik Mellanox Technologies LTD Tel: +972-4-909-7200 ext: 394 Fax: +972-4-959-3245 P.O. Box 586 Yokneam 20692 ISRAEL -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hal Rosenstock Sent: Tuesday, August 08, 2006 11:50 PM To: Yevgeny Kliteynik Cc: OPENIB Subject: Re: [openib-general] [PATCH] osm: Dynamic verbosity control per file Hi Yevgeny, On Wed, 2006-08-02 at 11:16, Yevgeny Kliteynik wrote: Hi Hal Just got back from vacation and am in the process of catching up. This patch adds new verbosity functionality. 1. Verbosity configuration file --- The user is able to set verbosity level per source code file by supplying verbosity configuration file using the following command line arguments: -b filename --verbosity_file filename By default, the OSM will use the following file: /etc/opensmlog.conf Verbosity configuration file should contain zero or more lines of the following pattern: filename verbosity_level where 'filename' is the name of the source code file that the 'verbosity_level' refers to, and the 'verbosity_level' itself should be specified as an integer number (decimal or hexadecimal). One reserved filename is 'all' - it represents general verbosity level, that is used for all the files that are not specified in the verbosity configuration file. If 'all' is not specified, the verbosity level set in the command line will be used instead. Note: The 'all' file verbosity level will override any other general level that was specified by the command line arguments. Sending a SIGHUP signal to the OSM will cause it to reload the verbosity configuration file. 2. Logging source code filename and line number --- If command line option -S or --log_source_info is specified, OSM will add source code filename and line number to every log message that is written to the log file. By default, the OSM will not log this additional info. Yevgeny Is it hard to find which file and line an opensm log message comes from ? Is this functionality really needed ? -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] include change
Include the header file we really want. Signed-off-by: James Lentini [EMAIL PROTECTED] Index: hw/mthca/mthca_cmd.c === --- hw/mthca/mthca_cmd.c(revision 8850) +++ hw/mthca/mthca_cmd.c(working copy) @@ -34,7 +34,7 @@ * $Id$ */ -#include linux/sched.h +#include linux/completion.h #include linux/pci.h #include linux/errno.h #include asm/io.h ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 1/7] IB/ipath - performance improvements via mmap of queues
int ibv_cmd_resize_cq(struct ibv_cq *cq, int cqe, - struct ibv_resize_cq *cmd, size_t cmd_size); + struct ibv_resize_cq *cmd, size_t cmd_size, + struct ibv_resize_cq_resp *resp, size_t resp_size); We can't make this change without a little more work -- as it stands this makes it impossible to have a low-level driver that works with both libibverbs 1.0 and 1.1, since there doesn't seem to be any autoconf way to check the number of parameters a function takes. I see two ways forward: either at a new ibv_cmd_resize_cq_resp() function (as you did originally), or add something like #define IBV_CMD_RESIZE_CQ_HAS_RESP_PARAMS and test that in libmthca and libehca. Personally I lean towards the second solution, although neither is very elegant. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Conflicting typedefs for ib_gid_t
In .../include/infiniband/mad.h, it is:
typedef uint8_t ib_gid_t[16];
In .../include/infiniband/iba/ib_types.h, it is:
#include complib/cl_packon.h
typedef union _ib_gid
{
uint8_t raw[16];
struct _ib_gid_unicast
{
ib_gid_prefix_t prefix;
ib_net64_t interface_id;
} PACK_SUFFIX unicast;
struct _ib_gid_multicast
{
uint8_t header[2];
uint8_t raw_group_id[14];
} PACK_SUFFIX multicast;
} PACK_SUFFIX ib_gid_t;
#include complib/cl_packoff.h
I need to include both files for a user space tool and I'm getting a compile
error due to the conflict. Is it not the norm for a user space application to
include both files?
Appreciate any thoughts on this.
Madhu Lakshmanan
Silverstorm Technologies, Inc.
[EMAIL PROTECTED]
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 4/7] IB/ipath - performance improvements via mmap of queues
struct ib_uverbs_resize_cq_resp {
__u32 cqe;
+__u32 reserved;
+__u64 driver_data[0];
};
I don't see any changes related to this in uverbs_cmd.c, and you don't
bump the ABI version. So as far as I can tell, ib_uverbs_resize_cq()
will silently corrupt the stack on an old libibverbs that passes in a
pointer to a 4-byte response structure.
In general I've resisted putting backwards compatibility stuff into
the kernel side of uverbs, so maybe an ABI bump is the answer in this
case too.
But then I have to do another libibverbs 1.0 release etc., which is
kind of a pain. So in this case it's probably OK to add a check in
ib_uverbs_resize_cq() for when out_len == 4, and not overflow the
response buffer in that case.
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 7/7] IB/ipath - performance improvements via mmap of queues
+/*
+ * ipath_vma_nopage - handle a VMA page fault.
+ */
+static struct page *ipath_vma_nopage(struct vm_area_struct *vma,
+ unsigned long address, int *type)
+{
+struct ipath_mmap_info *ip = vma-vm_private_data;
+unsigned long offset = address - vma-vm_start;
+struct page *page = NOPAGE_SIGBUS;
+void *pageptr;
+
+if (offset = ip-size)
+goto out; /* out of range */
+
+/*
+ * Convert the vmalloc address into a struct page.
+ */
+pageptr = (void *)(offset + (vma-vm_pgoff PAGE_SHIFT));
+page = vmalloc_to_page(pageptr);
+if (!page)
+goto out;
+
+/* Increment the reference count. */
+get_page(page);
+if (type)
+*type = VM_FAULT_MINOR;
+out:
+return page;
+}
+
+static struct vm_operations_struct ipath_vm_ops = {
+.open = ipath_vma_open,
+.close =ipath_vma_close,
+.nopage = ipath_vma_nopage,
+};
It seems that all this would be much simpler with vmalloc_user() and
remap_vmalloc_range(). And it would fix the information leak you
introduce here: I don't see anywhere that you clear the memory you
vmalloc and remap to userspace, so you're potentially exposing the
cached contents of /etc/passwd or something like that.
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 1/7] IB/ipath - performance improvements via mmap of queues
int ibv_cmd_resize_cq(struct ibv_cq *cq, int cqe, -struct ibv_resize_cq *cmd, size_t cmd_size); +struct ibv_resize_cq *cmd, size_t cmd_size, +struct ibv_resize_cq_resp *resp, size_t resp_size); We can't make this change without a little more work -- as it stands this makes it impossible to have a low-level driver that works with both libibverbs 1.0 and 1.1, since there doesn't seem to be any autoconf way to check the number of parameters a function takes. What we really need is a version number for the device library plug-in to libibverbs.so interface verses the existing kernel device driver to plug-in or application to libibverbs.so version. I see two ways forward: either at a new ibv_cmd_resize_cq_resp() function (as you did originally), or add something like #define IBV_CMD_RESIZE_CQ_HAS_RESP_PARAMS and test that in libmthca and libehca. Personally I lean towards the second solution, although neither is very elegant. - R. A #define won't help the plug-in know what parameters to pass, only a function name change will work if the semantics change. I can add another version argument to ibv_driver_init() if you agree. It seems to me that we have already made incompatible changes by moving to libibverbs.so.2. Couldn't we include this as part of the transition? Otherwise, I would vote for ibv_cmd_resize_cq_resp(). ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 4/7] IB/ipath - performance improvements via mmap of queues
struct ib_uverbs_resize_cq_resp {
__u32 cqe;
+ __u32 reserved;
+ __u64 driver_data[0];
};
I don't see any changes related to this in uverbs_cmd.c, and you don't
bump the ABI version. So as far as I can tell, ib_uverbs_resize_cq()
will silently corrupt the stack on an old libibverbs that passes in a
pointer to a 4-byte response structure.
In general I've resisted putting backwards compatibility stuff into
the kernel side of uverbs, so maybe an ABI bump is the answer in this
case too.
But then I have to do another libibverbs 1.0 release etc., which is
kind of a pain. So in this case it's probably OK to add a check in
ib_uverbs_resize_cq() for when out_len == 4, and not overflow the
response buffer in that case.
- R.
This doesn't break compatibility. uverbs_cmd.c ib_uverbs_resize_cq()
allocates a struct ib_uverbs_resize_cq_resp on the stack but
only reads the first element in.
The structure change isn't really needed at all since the INIT_UDATA()
macro gets the start of driver_data from the struct ib_uverbs_resize_cq.
The change to ib_uverbs_resize_cq_resp just matches the structure
change used by libipathverbs to initialize ib_uverbs_resize_cq.response.
If you want, I can remove this from the patch.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 7/7] IB/ipath - performance improvements via mmap of queues
+/*
+ * ipath_vma_nopage - handle a VMA page fault.
+ */
+static struct page *ipath_vma_nopage(struct vm_area_struct *vma,
+ unsigned long address, int *type)
+{
+ struct ipath_mmap_info *ip = vma-vm_private_data;
+ unsigned long offset = address - vma-vm_start;
+ struct page *page = NOPAGE_SIGBUS;
+ void *pageptr;
+
+ if (offset = ip-size)
+ goto out; /* out of range */
+
+ /*
+ * Convert the vmalloc address into a struct page.
+ */
+ pageptr = (void *)(offset + (vma-vm_pgoff PAGE_SHIFT));
+ page = vmalloc_to_page(pageptr);
+ if (!page)
+ goto out;
+
+ /* Increment the reference count. */
+ get_page(page);
+ if (type)
+ *type = VM_FAULT_MINOR;
+out:
+ return page;
+}
+
+static struct vm_operations_struct ipath_vm_ops = {
+ .open = ipath_vma_open,
+ .close =ipath_vma_close,
+ .nopage = ipath_vma_nopage,
+};
It seems that all this would be much simpler with vmalloc_user() and
remap_vmalloc_range(). And it would fix the information leak you
introduce here: I don't see anywhere that you clear the memory you
vmalloc and remap to userspace, so you're potentially exposing the
cached contents of /etc/passwd or something like that.
- R.
I was unaware of these functions. Looks like they were just
recently added (7/23/2006).
I will update the patch to use these.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [RFC] IPoIB high availability daemon
Hi,
The first version of the IPoIB high availability daemon can be found at:
https://openib.org/svn/trunk/contrib/mellanox/ipoibtools
The daemon is a perl script ipoib_ha.pl that should get the primary and
the backup IPoIB interfaces as a parameters (default values are ib0 as a
primary and ib1 as a backup).
The basic steps performed by IPoIB High Availability (HA) daemon:
- Get names of the IPoIB primary and backup interfaces.
- Get configuration of the primary interface from its standard place
(ifcfg-ibN from /etc/sysconfig/{network,network-scripts}).
- Run 'ip monitor link all' and parse its output to monitor IPoIB
primary interface.
- When NO-CARRIER occur, check if it is a primary IPoIB interface and
if yes
then migrate its IPoIB configuration to the backup IPoIB interface.
- Run 'arpingib' utility if configured to update neighbors with a new
MAC address
- Get the list of multicast groups from /proc/net/dev_mcast
that the primary IPoIB interface was registered to. Then register the
backup IPoIB interface to these multicast groups (using ipmaddr utility).
Currently there is an issue with join to IPoIB multicast group using
both ip and ipmaddr utilities.
This daemon is going to be added to the OFED-1.1 release.
Please comment.
Thanks,
Regards,
Vladimir
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Roland Dreier wrote: @@ -1502,7 +1506,7 @@ int mthca_tavor_post_send(struct ib_qp * int i; int size; int size0 = 0; - u32 f0 = 0; + u32 f0; This causes compile warnings for me that 'f0' might be used uninitialized. @@ -1843,7 +1849,7 @@ int mthca_arbel_post_send(struct ib_qp * int i; int size; int size0 = 0; - u32 f0 = 0; + u32 f0; Same here. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] OpenSM/osm_resp.c: In osm_resp_make_resp_smp, set direction bit only if direct routed class
OpenSM/osm_resp.c: In osm_resp_make_resp_smp, set direction bit only if
direct routed class
This change fixes two minor issues with osm_resp_make_resp_smp:
1. Get/Set responses always had direction bit set.
2. Trap represses never had direction bit set.
The direction bit needs setting in direct routed responses and it
doesn't exist in LID routed responses.
Signed-off-by: Hal Rosenstock [EMAIL PROTECTED]
Index: opensm/osm_resp.c
===
--- opensm/osm_resp.c (revision 8931)
+++ opensm/osm_resp.c (working copy)
@@ -127,7 +127,7 @@ osm_resp_make_resp_smp(
if (p_src_smp-method == IB_MAD_METHOD_GET ||
p_src_smp-method == IB_MAD_METHOD_SET ) {
p_dest_smp-method = IB_MAD_METHOD_GET_RESP;
-p_dest_smp-status = (ib_net16_t)(status | IB_SMP_DIRECTION);
+p_dest_smp-status = status;
}
else if (p_src_smp-method == IB_MAD_METHOD_TRAP)
{
@@ -143,6 +143,9 @@ osm_resp_make_resp_smp(
goto Exit;
}
+ if (p_src_smp-mgmt_class == IB_MCLASS_SUBN_DIR)
+p_dest_smp-status |= IB_SMP_DIRECTION;
+
p_dest_smp-dr_dlid = p_dest_smp-dr_slid;
p_dest_smp-dr_slid = p_dest_smp-dr_dlid;
memcpy( p_dest_smp-data, p_payload, IB_SMP_DATA_SIZE );
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
This is a similar but more mainstream example of the conflicts. A previous
one was reported last week in terms of CM. Still not sure of the best
resolution for this.
Do you really need both includes ?
The userspace tool shows a textual representation of a HCA port's capability
mask. So it
requires the port capability bit definitions in ib_types.h. And I require mad.h
for the MAD API.
-- Hal
Thanks,
Madhu
From: [EMAIL PROTECTED] on behalf of Lakshmanan, Madhu
Sent: Mon 8/14/2006 12:19 PM
To: openib-general@openib.org
Subject: [openib-general] Conflicting typedefs for ib_gid_t
In .../include/infiniband/mad.h, it is:
typedef uint8_t ib_gid_t[16];
In .../include/infiniband/iba/ib_types.h, it is:
#include complib/cl_packon.h
typedef union _ib_gid
{
uint8_t raw[16];
struct _ib_gid_unicast
{
ib_gid_prefix_t prefix;
ib_net64_t interface_id;
} PACK_SUFFIX unicast;
struct _ib_gid_multicast
{
uint8_t header[2];
uint8_t raw_group_id[14];
} PACK_SUFFIX multicast;
} PACK_SUFFIX ib_gid_t;
#include complib/cl_packoff.h
I need to include both files for a user space tool and I'm getting a compile
error due to the conflict. Is it not the norm for a user space application to
include both files?
Appreciate any thoughts on this.
Madhu Lakshmanan
Silverstorm Technologies, Inc.
[EMAIL PROTECTED]
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] IB mcast question
IB experts, I'm playing around with UD QPs over IB, and using the librdma services to join and send to multicast groups. I can run seans mckey example program and it works. I've written a simply program to echo stdin to a mcast address and another program to read mcast packets and echo to stout. I run this on two hosts connected p2p via IB. And it works as expected. However, if I run 2 instances of the app that reads mcasts and dumps them to stdout, I only get the mcast packets delivered to one of the applications. Namely the first one who joins the group seems to get the mcasts. I know for UDP/IP multicast, all applications bound to the same port and joined to the IP mcast addr will get a copy of incoming mcast packets. Is this not true for IB mcast? It appears not based on my tests... Thanks, Stevo. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
However, if I run 2 instances of the app that reads mcasts and dumps them to stdout, I only get the mcast packets delivered to one of the applications. Namely the first one who joins the group seems to get the mcasts. I know for UDP/IP multicast, all applications bound to the same port and joined to the IP mcast addr will get a copy of incoming mcast packets. Is this not true for IB mcast? It appears not based on my tests... My testing revealed the same issue, and I was unable to locate the root cause of the problem. I was not able to confirm that this configuration had ever been successfully tested. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 4/7] IB/ipath - performance improvements via mmap of queues
ralphc This doesn't break compatibility. uverbs_cmd.c ralphc ib_uverbs_resize_cq() allocates a struct ralphc ib_uverbs_resize_cq_resp on the stack but only reads the ralphc first element in. The structure change isn't really ralphc needed at all since the INIT_UDATA() macro gets the start ralphc of driver_data from the struct ib_uverbs_resize_cq. The ralphc change to ib_uverbs_resize_cq_resp just matches the ralphc structure change used by libipathverbs to initialize ralphc ib_uverbs_resize_cq.response. Am I missing something? Think about the case of old libibverbs, new kernel. libibverbs allocates a 4-byte response structure and passes the pointer to that to the kernel. The kernel allocates an 8-byte response structure and copies it back to userspace. And the 4 bytes after the userspace response structure get zeroed. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Sean This causes compile warnings for me that 'f0' might be used Sean uninitialized. Yes, but they're bogus. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
Vladimir Currently there is an issue with join to IPoIB multicast Vladimir group using both ip and ipmaddr utilities. What's the issue? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
Vladimir The daemon is a perl script ipoib_ha.pl that should get
Vladimir the primary and the backup IPoIB interfaces as a
Vladimir parameters (default values are ib0 as a primary and ib1
Vladimir as a backup).
Seems like a perl script that relies on the ip command is a little
heavyweight. Why not a standalone program that uses rtnetlink?
Vladimir - Get configuration of the primary interface from its
Vladimir standard place (ifcfg-ibN from
Vladimir /etc/sysconfig/{network,network-scripts}).
This is not standard on all distributions. It would be better to have
a more flexible method that worked on Debian/Ubuntu, etc.
- R.
Date: Mon, 14 Aug 2006 12:41:25 -0700
In-Reply-To: [EMAIL PROTECTED] (Vladimir Sokolovsky's message of Mon, 14 Aug
2006 20:44:10 +0300)
Message-ID: [EMAIL PROTECTED]
User-Agent: Gnus/5.1007 (Gnus v5.10.7) XEmacs/21.4.18 (linux)
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve However, if I run 2 instances of the app that reads mcasts Steve and dumps them to stdout, I only get the mcast packets Steve delivered to one of the applications. Namely the first one Steve who joins the group seems to get the mcasts. I know for Steve UDP/IP multicast, all applications bound to the same port Steve and joined to the IP mcast addr will get a copy of incoming Steve mcast packets. Is this not true for IB mcast? It appears Steve not based on my tests... This should work -- multicast packets should be replicated to all attached UD QPs. There is likely a bug in the librdma multicast support. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Sean My testing revealed the same issue, and I was unable to Sean locate the root cause of the problem. I was not able to Sean confirm that this configuration had ever been successfully Sean tested. Are you positive ibv_attach_mcast() is called on all the QPs, and that the MGID is passed correctly in to all calls? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Quoting r. [EMAIL PROTECTED] [EMAIL PROTECTED]: Subject: Re: Conflicting typedefs for ib_gid_t This is a similar but more mainstream example of the conflicts. A previous one was reported last week in terms of CM. Still not sure of the best resolution for this. Do you really need both includes ? The userspace tool shows a textual representation of a HCA port's capability mask. So it requires the port capability bit definitions in ib_types.h. And I require mad.h for the MAD API. I don't think the way forward is using iba/ in all applications. I see it mostly as a legacy header for opensm and related apps that want their own layer for portability between stacks. Wrt issue at hand, using ib_ prefix anywhere is a mistake which will always lead to conflicts between libraries. Let us start prefixing types libibumad defines with umad_, just like ib verbs library prefixes types by ibv_. For example we have union ibv_gid, so can't mad.h have umad_gid_t? Hal, what do you say? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Mon, 2006-08-14 at 12:43 -0700, Roland Dreier wrote: Sean My testing revealed the same issue, and I was unable to Sean locate the root cause of the problem. I was not able to Sean confirm that this configuration had ever been successfully Sean tested. Are you positive ibv_attach_mcast() is called on all the QPs, and that the MGID is passed correctly in to all calls? - R. I'll let you know... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Mon, 2006-08-14 at 12:31 -0700, Sean Hefty wrote: However, if I run 2 instances of the app that reads mcasts and dumps them to stdout, I only get the mcast packets delivered to one of the applications. Namely the first one who joins the group seems to get the mcasts. I know for UDP/IP multicast, all applications bound to the same port and joined to the IP mcast addr will get a copy of incoming mcast packets. Is this not true for IB mcast? It appears not based on my tests... My testing revealed the same issue, and I was unable to locate the root cause of the problem. I was not able to confirm that this configuration had ever been successfully tested. - Sean Hmm. Ok. I'll debug this. I need to get this working... Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Quoting r. Roland Dreier [EMAIL PROTECTED]: Subject: Re: [PATCH] mthca: make IB_SEND_FENCE work Sean This causes compile warnings for me that 'f0' might be used Sean uninitialized. Yes, but they're bogus. Yes, I see lots of such bogus warnings in kernel code too. And it seems obvious that since compiler isn't smart enough to figure out the initialization isn't needed, it will generate unecessary code if we *do* add initilization just to shut it up. Maybe compile with -Wno-uninitialized? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Roland Dreier wrote: Are you positive ibv_attach_mcast() is called on all the QPs, and that the MGID is passed correctly in to all calls? Yes - ibv_attach_mcast() is being called with the same MLID, MGID by both receiving processes. That doesn't necessarily mean that there's not a bug in ib_multicast or the RDMA CM; I just couldn't locate any. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
I agree with your view on iba/ib_types.h I'm not sure I understand what you mean in terms of libibumad. He's including libibmad rather than libibumad. So I suspect you mean changing this (ib_gid_t) to mad_gid_t ? -- Hal From: Michael S. Tsirkin [mailto:[EMAIL PROTECTED] Sent: Mon 8/14/2006 3:44 PM To: [EMAIL PROTECTED] Cc: openib-general@openib.org; Hal Rosenstock Subject: Re: Conflicting typedefs for ib_gid_t Quoting r. [EMAIL PROTECTED] [EMAIL PROTECTED]: Subject: Re: Conflicting typedefs for ib_gid_t This is a similar but more mainstream example of the conflicts. A previous one was reported last week in terms of CM. Still not sure of the best resolution for this. Do you really need both includes ? The userspace tool shows a textual representation of a HCA port's capability mask. So it requires the port capability bit definitions in ib_types.h. And I require mad.h for the MAD API. I don't think the way forward is using iba/ in all applications. I see it mostly as a legacy header for opensm and related apps that want their own layer for portability between stacks. Wrt issue at hand, using ib_ prefix anywhere is a mistake which will always lead to conflicts between libraries. Let us start prefixing types libibumad defines with umad_, just like ib verbs library prefixes types by ibv_. For example we have union ibv_gid, so can't mad.h have umad_gid_t? Hal, what do you say? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
Quoting r. Roland Dreier [EMAIL PROTECTED]: Subject: Re: [RFC] IPoIB high availability daemon Vladimir Currently there is an issue with join to IPoIB multicast Vladimir group using both ip and ipmaddr utilities. What's the issue? I think I heard that ip maddr shows trunkated addresses, and attempt to join also fails as join is done to he wrong address. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Mon, 2006-08-14 at 12:42 -0700, Roland Dreier wrote: Steve However, if I run 2 instances of the app that reads mcasts Steve and dumps them to stdout, I only get the mcast packets Steve delivered to one of the applications. Namely the first one Steve who joins the group seems to get the mcasts. I know for Steve UDP/IP multicast, all applications bound to the same port Steve and joined to the IP mcast addr will get a copy of incoming Steve mcast packets. Is this not true for IB mcast? It appears Steve not based on my tests... This should work -- multicast packets should be replicated to all attached UD QPs. There is likely a bug in the librdma multicast support. So is this replicating done in the mthca hca? Since one app is getting the mcast packet, can I assume the opensm code is doing the right thing switch/port wise? Should the SM get join requests for both applications that join the group on the same host? Or only the first one? Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Quoting r. Hal Rosenstock [EMAIL PROTECTED]: Subject: RE: Conflicting typedefs for ib_gid_t I agree with your view on iba/ib_types.h I'm not sure I understand what you mean in terms of libibumad. He's including libibmad rather than libibumad. So I suspect you mean changing this (ib_gid_t) to mad_gid_t ? Yes, or ib_mad_, or ibmad_. And same for other IB_ and ib_ names there. We really need to do something about names like ib_attr_t. rant I also would like the number of typedefs libibmad exposes reduced - its not like you'll be able to convert these structs to anything else transparently without breaking users - anyone sticks their data in there anyway. But this is more a matter of taste. /rant -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Michael S. Tsirkin wrote: Yes, I see lots of such bogus warnings in kernel code too. And it seems obvious that since compiler isn't smart enough to figure out the initialization isn't needed, it will generate unecessary code if we *do* add initilization just to shut it up. Can we relocate the setting for op0 and f0 outside of the for-loop (maybe before we enter the loop)? I'm really not familiar with this code, but on first glance, it looks like these are set based on the send_flag of the first posted wr. This could eliminate the warning, and remove an if statement from executing on each iteration. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve, IB only replicates once per node (and also not on the incoming port if there are any members). The SM tracks join states (full, non member, send only member) for a port. It doesn't matter whether the SM gets duplicated join requests for a port. It would just indicate that was OK and the node would still only get one packet per send from another port in that group. It's up to the HCA to replicate the multicast packet to all its QPs which are part of that group. -- Hal From: [EMAIL PROTECTED] on behalf of Steve Wise Sent: Mon 8/14/2006 4:18 PM To: Roland Dreier Cc: openib-general Subject: Re: [openib-general] IB mcast question On Mon, 2006-08-14 at 12:42 -0700, Roland Dreier wrote: Steve However, if I run 2 instances of the app that reads mcasts Steve and dumps them to stdout, I only get the mcast packets Steve delivered to one of the applications. Namely the first one Steve who joins the group seems to get the mcasts. I know for Steve UDP/IP multicast, all applications bound to the same port Steve and joined to the IP mcast addr will get a copy of incoming Steve mcast packets. Is this not true for IB mcast? It appears Steve not based on my tests... This should work -- multicast packets should be replicated to all attached UD QPs. There is likely a bug in the librdma multicast support. So is this replicating done in the mthca hca? Since one app is getting the mcast packet, can I assume the opensm code is doing the right thing switch/port wise? Should the SM get join requests for both applications that join the group on the same host? Or only the first one? Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
Quoting r. Roland Dreier [EMAIL PROTECTED]:
Subject: Re: [RFC] IPoIB high availability daemon
Vladimir The daemon is a perl script ipoib_ha.pl that should get
Vladimir the primary and the backup IPoIB interfaces as a
Vladimir parameters (default values are ib0 as a primary and ib1
Vladimir as a backup).
Seems like a perl script that relies on the ip command is a little
heavyweight. Why not a standalone program that uses rtnetlink?
We could go there yet, but there'd be a lot of issues to cover:
note how we need to also set IP addresses, read configuration ...
Further, the only system that might care about this I know about (bproc)
dislikes running daemons on endnodes anyway.
Finally - work is underway for kernel-level solutions
involving bonding etc.
So - let's see a working implementation first, optimize later.
Makes sense?
Vladimir - Get configuration of the primary interface from its
Vladimir standard place (ifcfg-ibN from
Vladimir /etc/sysconfig/{network,network-scripts}).
This is not standard on all distributions. It would be better to have
a more flexible method that worked on Debian/Ubuntu, etc.
Yea, but just adding our own configuration would mean configuration
overhead for the administrator, lack of convenient tools to do it ...
Using standard interfaces is good. An idea how to solve this?
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
I don't think the way forward is using iba/ in all applications. I see it mostly as a legacy header for opensm and related apps that want their own layer for portability between stacks. I agree with your view on iba/ib_types.h Does that imply that the definitions in iba/ib_types.h are not expected to be required or used by user-space applications other than those categories mentioned above? If iba/ib_types.h is only a legacy header, are the definitions also present in another header file? Madhu ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve Wise wrote: So is this replicating done in the mthca hca? As just an FYI, I didn't see anything wrong in the mthca driver either when I was looking at this problem. Since one app is getting the mcast packet, can I assume the opensm code is doing the right thing switch/port wise? That seems like a fairly safe assumption. Should the SM get join requests for both applications that join the group on the same host? Or only the first one? Only the first join request should make it to the SA. The second join request is fulfilled by ib_multicast. This is what makes ib_multicast suspect. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 4/7] IB/ipath - performance improvements via mmap of queues
On Mon, 2006-08-14 at 12:37 -0700, Roland Dreier wrote: ralphc This doesn't break compatibility. uverbs_cmd.c ralphc ib_uverbs_resize_cq() allocates a struct ralphc ib_uverbs_resize_cq_resp on the stack but only reads the ralphc first element in. The structure change isn't really ralphc needed at all since the INIT_UDATA() macro gets the start ralphc of driver_data from the struct ib_uverbs_resize_cq. The ralphc change to ib_uverbs_resize_cq_resp just matches the ralphc structure change used by libipathverbs to initialize ralphc ib_uverbs_resize_cq.response. Am I missing something? Think about the case of old libibverbs, new kernel. libibverbs allocates a 4-byte response structure and passes the pointer to that to the kernel. The kernel allocates an 8-byte response structure and copies it back to userspace. And the 4 bytes after the userspace response structure get zeroed. - R. No, I was missing something :-) You are correct. The structure change was needed to get the alignment correct for returning a u64 after the struct ib_uverbs_resize_cq_resp. I can avoid the incompatibility a number of ways: 1) change ib_uverbs_resize_cq() to only copyout resp.cqe. 2) change ib_ipath to ALIGN the udata-outbuf address. 3) define two resp structures, check the version, copyout the right one. Seems like #1 is the simplest to me. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Yes, or ib_mad_, or ibmad_. And same for other IB_ and ib_ names there. We really need to do something about names like ib_attr_t. I like to move away from each library re-defining common IB data types. Something like ibv_gid should be picked up from libibverbs. IMO, the core of the problem is that opensm include files carry too many legacy typedefs. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Quoting r. Sean Hefty [EMAIL PROTECTED]: This could eliminate the warning, and remove an if statement from executing on each iteration. We still need to test size0 to set size0 = size. So we just reuse the extra branch, and I agree with Roland this way code is clearer. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
To answer your questions: I'm not totally sure about your application but it seems to me to fall in the category being discussed. Not all of the definitions in ib_types.h are elsewhere. I am working on a patch to get you past this issue. -- Hal From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Mon 8/14/2006 4:35 PM To: Hal Rosenstock; Michael S. Tsirkin Cc: [EMAIL PROTECTED]; openib-general@openib.org Subject: Re: Conflicting typedefs for ib_gid_t I don't think the way forward is using iba/ in all applications. I see it mostly as a legacy header for opensm and related apps that want their own layer for portability between stacks. I agree with your view on iba/ib_types.h Does that imply that the definitions in iba/ib_types.h are not expected to be required or used by user-space applications other than those categories mentioned above? If iba/ib_types.h is only a legacy header, are the definitions also present in another header file? Madhu ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] sanity check on datapath
Roland, do we really need code like
if (wr-opcode = sizeof mthca_opcode / sizeof mthca_opcode[0])
{
ret = -1;
*bad_wr = wr;
goto out;
}
in mthca on data path? Should this be put within ifdef DEBUG or something?
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Michael S. Tsirkin wrote: We still need to test size0 to set size0 = size. So we just reuse the extra branch, and I agree with Roland this way code is clearer. You're right; I missed where size0 was used below the loop. Then I think we can also do without initializing op0 = 0, and we can eliminate the size0 initialization by changing if (!size0) to if (!nreq). - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
I concur with your categorization of the application I mentioned. I was curious about the way going forward, as to whether anyone anticipated this to be a more commonly recurring issue. Thanks in advance for the patch. -- Madhu -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Monday, August 14, 2006 4:39 PM To: Lakshmanan, Madhu; Michael S. Tsirkin Cc: openib-general@openib.org Subject: RE: Conflicting typedefs for ib_gid_t To answer your questions: I'm not totally sure about your application but it seems to me to fall in the category being discussed. Not all of the definitions in ib_types.h are elsewhere. I am working on a patch to get you past this issue. -- Hal From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Mon 8/14/2006 4:35 PM To: Hal Rosenstock; Michael S. Tsirkin Cc: [EMAIL PROTECTED]; openib-general@openib.org Subject: Re: Conflicting typedefs for ib_gid_t I don't think the way forward is using iba/ in all applications. I see it mostly as a legacy header for opensm and related apps that want their own layer for portability between stacks. I agree with your view on iba/ib_types.h Does that imply that the definitions in iba/ib_types.h are not expected to be required or used by user-space applications other than those categories mentioned above? If iba/ib_types.h is only a legacy header, are the definitions also present in another header file? Madhu ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Mon, 2006-08-14 at 13:33 -0700, Sean Hefty wrote: Steve Wise wrote: So is this replicating done in the mthca hca? As just an FYI, I didn't see anything wrong in the mthca driver either when I was looking at this problem. Ok. I added printks in the mcast attach/detach and they're firing as expected: vic18:/home/swise/zip # dmesg mthca_multicast_attach qp_num 406 gid ff124001:000a0aff lid c003 mthca_multicast_attach qp_num 407 gid ff124001:000a0aff lid c003 mthca_multicast_detach qp_num 406 gid ff124001:000a0aff lid c003 mthca_multicast_detach qp_num 407 gid ff124001:000a0aff lid c003 Should the SM get join requests for both applications that join the group on the same host? Or only the first one? Only the first join request should make it to the SA. The second join request is fulfilled by ib_multicast. This is what makes ib_multicast suspect. I'll look into this module... Thanks, Stevo. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Quoting r. Sean Hefty [EMAIL PROTECTED]: Subject: RE: [openib-general] Conflicting typedefs for ib_gid_t Yes, or ib_mad_, or ibmad_. And same for other IB_ and ib_ names there. We really need to do something about names like ib_attr_t. I like to move away from each library re-defining common IB data types. There are not that many common IB types. verbs and management for example are more or less isolated. CM users mostly don't care about SMPs. etc. What's common? GID? It does not seem worth it for a free format 16 byte network endianness type. I don't necessary see a problem with what we have. Such defines mirror IB spec so are static and there is no overhead maintaining them. And this simplifies dependencies no end - think about testing tens of dependent libraries for breakage just 'cause you removed an used line in this included by everyone header. Something like ibv_gid should be picked up from libibverbs. Hmm. User might not have uverbs even loaded, so adding verbs as a dependency to e.g. SM seems like a bad idea. IMO, the core of the problem is that opensm include files carry too many legacy typedefs. opensm really tries to be stack-agnostic, so it does need its own layer for things. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Hi Sean, If we had all the IBTA related defines and structs in a well defined and tracked location there was no reason for OpenSM to have this header. To my best recollection I could not get any agreement to such common IBTA specifications file. Can I assume you are willing to have such common set of header files??? I would propose having a set of files (one common and one for each class) under include/infiniband/iba I like to move away from each library re-defining common IB data types. Something like ibv_gid should be picked up from libibverbs. IMO, the core of the problem is that opensm include files carry too many legacy typedefs. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Quoting r. Sean Hefty [EMAIL PROTECTED]: Subject: Re: [PATCH] mthca: make IB_SEND_FENCE work Michael S. Tsirkin wrote: We still need to test size0 to set size0 = size. So we just reuse the extra branch, and I agree with Roland this way code is clearer. You're right; I missed where size0 was used below the loop. Then I think we can also do without initializing op0 = 0, and we can eliminate the size0 initialization by changing if (!size0) to if (!nreq). - Sean There's the ee_nds line above that tests size0 and should be changed then as well. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Only the first join request should make it to the SA. The second join request is fulfilled by ib_multicast. This is what makes ib_multicast suspect. I'll look into this module... ib_multicast takes care of sending the join/leave info to the SA, right? It keeps track of _when_ to leave, for instance. So since opensm -is- getting the join and setting up the group, and the mcast packet is being passed to the first member who joined, then I don't think ib_multicast can mess up the subsequent members, can it? I confirmed that mthca was called to attach both qps to the mgid/mlid, so this makes me think ib_multicast worked ok. I'm new to IB mcast, so I'm learning, but it seems like the mthca firmware maybe isn't doing the right thing here. Any suggestions on how to further debug this? BTW my HCAs are at the latest firmware. I just had them upgraded. Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Quoting r. Eitan Zahavi [EMAIL PROTECTED]: If we had all the IBTA related defines and structs in a well defined and tracked location there was no reason for OpenSM to have this header. Define well defined :) Seriously, won't opensm need a portability layer to be stack agnostic anyway? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Michael S. Tsirkin wrote: There are not that many common IB types. verbs and management for example are more or less isolated. CM users mostly don't care about SMPs. etc. What's common? GID? It does not seem worth it for a free format 16 byte network endianness type. Verbs and management are not isolated. Establishing a connection, joining a multicast group, or acquiring a path record are essential for actually using verbs correctly. I don't necessary see a problem with what we have. Such defines mirror IB spec so are static and there is no overhead maintaining them. And this simplifies dependencies no end - think about testing tens of dependent libraries for breakage just 'cause you removed an used line in this included by everyone header. Having umad_gid, ibv_gid, ibv_sa_gid, mad_gid, cm_gid, some_other_random_library_gid is goofy. (Sorry, I'm completely ranting now.) Hmm. User might not have uverbs even loaded, so adding verbs as a dependency to e.g. SM seems like a bad idea. We only need the include file, not the library. opensm really tries to be stack-agnostic, so it does need its own layer for things. Then those things should be completely internal to opensm. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work
Michael Maybe compile with -Wno-uninitialized? This is discussed periodically on lkml. The problem with -Wno-uninitialized is that it shuts up the good is used uninitialized warnings also (in addition to the may be used warnings, which are often bogus). - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
OpenSM ib_types.h had nothing to do with stack implementation. It carries the constants and structures (MAD formats) defined in IBTA specification 1.2 (+ errata). Any application that needs to send MADs and decode their status etc will need to use similar headers. ib_gid_t is just another example for a struct defined by IBTA. It has nothing to do with specific stack implementation. Eitan Zahavi Senior Engineering Director, Software Architect Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -Original Message- From: Michael S. Tsirkin Sent: Tuesday, August 15, 2006 12:17 AM To: Eitan Zahavi Cc: Sean Hefty; Hal Rosenstock; openib-general@openib.org Subject: Re: [openib-general] Conflicting typedefs for ib_gid_t Quoting r. Eitan Zahavi [EMAIL PROTECTED]: If we had all the IBTA related defines and structs in a well defined and tracked location there was no reason for OpenSM to have this header. Define well defined :) Seriously, won't opensm need a portability layer to be stack agnostic anyway? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve So is this replicating done in the mthca hca? Yes, it should be. There may be a bug in the mthca kernel multicast code for handling multiple QPs attached to the same group. Steve Since one app is getting the mcast packet, can I assume the Steve opensm code is doing the right thing switch/port wise? Yep. Steve Should the SM get join requests for both applications that Steve join the group on the same host? Or only the first one? No there should only be one join request for a given port. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] sanity check on datapath
Michael if (wr-opcode = sizeof mthca_opcode / sizeof mthca_opcode[0])
Michael {
Michaelret = -1;
Michael*bad_wr = wr;
Michaelgoto out;
Michael }
Michael in mthca on data path? Should this be put within ifdef
Michael DEBUG or something?
Probably not needed -- I guess we can trust what the consumer gives
us.
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve Wise wrote: ib_multicast takes care of sending the join/leave info to the SA, right? It keeps track of _when_ to leave, for instance. So since opensm -is- getting the join and setting up the group, and the mcast packet is being passed to the first member who joined, then I don't think ib_multicast can mess up the subsequent members, can it? It theory, it shouldn't mess up subsequent members. While the first join is active, subsequent join / leave requests to that same group should be queued. After the first join completes, subsequent joins should get a copy of the MCMemberRecord that was returned by the SA. (This is a slight simplification, with the actual operation determined by the type of join operation that occurs. But for the RDMA CM, this is what should happen.) I'm new to IB mcast, so I'm learning, but it seems like the mthca firmware maybe isn't doing the right thing here. This was my suspicion, but I couldn't be certain. It would help if anyone can say that they've successfully tested this sort of multicast configuration. I.e. two QPs from the same HCA in the same group. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] (no subject)
Hi Roland, In order to support High-Availability in OFED 1.1, we need more functionality to the srp daemon. Based on your code I implemented a new srp daemon (I listed its new features below) . I put the code in https://openib.org/svn/trunk/contrib/mellanox/gen2/src/userspace/srptools/srp_daemon and I'm in an initialtesting stage (there are some known bugs). Since I think that people may still want to use your original ibsrpdm, I think we should keep your version and start a new tool from my code. What do you think? Ishai The new tool main features: 1) Register to Traps 64 and 144. 2) Can ask ib_srp toconnect tothe targets it finds. 3) Can check if the target is already connectedby ib_srp from the same port. 4) Can perform rescan of the fabric every X seconds. 5) Identify SA changes and other events and act accordingly. 6) Can get an hca name and a port number as input (not only a umad device). 7) Uses the umad package. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Sean, I think it was agreed a long time ago on OpenIB to have duplicated definitions for some of the ib_xxx things. The specific issue here is that the one in the gen2 user libraries/verbs is different from the one which OpenSM uses. If they both were the same, this would work, right ? -- Hal From: [EMAIL PROTECTED] on behalf of Sean Hefty Sent: Mon 8/14/2006 5:15 PM To: Michael S. Tsirkin Cc: openib-general@openib.org Subject: Re: [openib-general] Conflicting typedefs for ib_gid_t Michael S. Tsirkin wrote: There are not that many common IB types. verbs and management for example are more or less isolated. CM users mostly don't care about SMPs. etc. What's common? GID? It does not seem worth it for a free format 16 byte network endianness type. Verbs and management are not isolated. Establishing a connection, joining a multicast group, or acquiring a path record are essential for actually using verbs correctly. I don't necessary see a problem with what we have. Such defines mirror IB spec so are static and there is no overhead maintaining them. And this simplifies dependencies no end - think about testing tens of dependent libraries for breakage just 'cause you removed an used line in this included by everyone header. Having umad_gid, ibv_gid, ibv_sa_gid, mad_gid, cm_gid, some_other_random_library_gid is goofy. (Sorry, I'm completely ranting now.) Hmm. User might not have uverbs even loaded, so adding verbs as a dependency to e.g. SM seems like a bad idea. We only need the include file, not the library. opensm really tries to be stack-agnostic, so it does need its own layer for things. Then those things should be completely internal to opensm. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] A new version for srp daemon
Adding a subject From: Ishai Rabinovitz Sent: Tuesday, August 15, 2006 12:32 AMTo: 'Roland Dreier'Cc: 'openib-general@openib.org'; Tziporet KorenSubject: Hi Roland, In order to support High-Availability in OFED 1.1, we need more functionality to the srp daemon. Based on your code I implemented a new srp daemon (I listed its new features below) . I put the code in https://openib.org/svn/trunk/contrib/mellanox/gen2/src/userspace/srptools/srp_daemon and I'm in an initialtesting stage (there are some known bugs). Since I think that people may still want to use your original ibsrpdm, I think we should keep your version and start a new tool from my code. What do you think? Ishai The new tool main features: 1) Register to Traps 64 and 144. 2) Can ask ib_srp toconnect tothe targets it finds. 3) Can check if the target is already connectedby ib_srp from the same port. 4) Can perform rescan of the fabric every X seconds. 5) Identify SA changes and other events and act accordingly. 6) Can get an hca name and a port number as input (not only a umad device). 7) Uses the umad package. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
This is not the main issue (the lack of replication is) but I don't think a subsequent join from the same port does any harm but ib_multicast shouldn't be doing this. It would matter in terms of the leave though. -- Hal From: [EMAIL PROTECTED] on behalf of Roland Dreier Sent: Mon 8/14/2006 5:24 PM To: Steve Wise Cc: openib-general Subject: Re: [openib-general] IB mcast question Steve So is this replicating done in the mthca hca? Yes, it should be. There may be a bug in the mthca kernel multicast code for handling multiple QPs attached to the same group. Steve Since one app is getting the mcast packet, can I assume the Steve opensm code is doing the right thing switch/port wise? Yep. Steve Should the SM get join requests for both applications that Steve join the group on the same host? Or only the first one? No there should only be one join request for a given port. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 00:38 +0300, Hal Rosenstock wrote: This is not the main issue (the lack of replication is) but I don't think a subsequent join from the same port does any harm but ib_multicast shouldn't be doing this. It would matter in terms of the leave though. The osm logs seem to show only one join_mgrp request, when the first app joins, and one leave_mgrp when the 2nd app exits. So I think the interaction with OSM is okeydokey. Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Quoting r. Eitan Zahavi [EMAIL PROTECTED]: ib_gid_t is just another example for a struct defined by IBTA. IBTA only defines a wire protocol. It's often wrong for applications work in terms of raw packets- we should have libraries to encapsulate typical usage. While using on-the-wire values for library API sometimes makes sense, this is not necessarily always the right way and e.g. efficiency concerns might dictate otherwise. It has nothing to do with specific stack implementation. The name is stack specific. Whether it's best as an array or a union or a struct is specific for an app. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [openfabrics-ewg] OFED 1.1-rc1 is available
Why is the OFED 1.1-rc1 source tar ball missing files when compared with the 1.1 branch? Of specific question is the absence of autogen.sh in libibverbs. Ira On Sun, 13 Aug 2006 16:14:10 +0300 Tziporet Koren [EMAIL PROTECTED] wrote: Hal Rosenstock wrote: Target release date: 12-Sep Intermediate milestones: 1. Create 1.1 branch of user level: 27-Jul - done 2. RC1: 8-Aug - done 3. Feature freeze (RC2): 17-Aug What is the start build date for RC2 ? When do developers need to have their code in by to make RC2 ? We will start on Tue 15-Aug. Is this OK with you? 4. Code freeze (rc-x): 6-Sep Is this 1 or 2 RCs beyond RC2 in order to make this ? I hope one but I guess it will be two more RCs. Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/srp: add port/device attributes
Michael Hi, Roland! There does not, at the moment, seem to exist Michael a way to find out which HCA port the specific SRP host is Michael connected through. Seems OK -- although I wonder about the names srp_port and srp_device. I think ib_port and ib_device would make more sense (or perhaps local_ib_port and local_ib_device although I don't think that's really required). Michael While not really a bugfix, maybe the following is small Michael enough for 2.6.18? We will use it in srptools that will Michael ship with OFED. No, I think this is purely a new feature and I'll queue it for 2.6.19. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
I added some debug printks in mthca_multicast_attach(). Roland, does this look ok to you? It seems correct to me: # dmesg mthca_multicast_attach qp_num 406 gid ff124001:000a0a0a lid c003 mthca_multicast_attach line 167 - found mgm, hash a20, prev , index a20 mthca_multicast_attach line 197 - updated mgm gid: mgm gid ff124001:000a0a0a mthca_multicast_attach line 219 - writing mgm: mgm-qp[0] 8406 (BE) mthca_multicast_attach qp_num 407 gid ff124001:000a0a0a lid c003 mthca_multicast_attach line 167 - found mgm, hash a20, prev , index a20 mthca_multicast_attach line 197 - updated mgm gid: mgm gid ff124001:000a0a0a mthca_multicast_attach line 219 - writing mgm: mgm-qp[1] 8407 (BE) On Mon, 2006-08-14 at 14:30 -0700, Sean Hefty wrote: Steve Wise wrote: ib_multicast takes care of sending the join/leave info to the SA, right? It keeps track of _when_ to leave, for instance. So since opensm -is- getting the join and setting up the group, and the mcast packet is being passed to the first member who joined, then I don't think ib_multicast can mess up the subsequent members, can it? It theory, it shouldn't mess up subsequent members. While the first join is active, subsequent join / leave requests to that same group should be queued. After the first join completes, subsequent joins should get a copy of the MCMemberRecord that was returned by the SA. (This is a slight simplification, with the actual operation determined by the type of join operation that occurs. But for the RDMA CM, this is what should happen.) I'm new to IB mcast, so I'm learning, but it seems like the mthca firmware maybe isn't doing the right thing here. This was my suspicion, but I couldn't be certain. It would help if anyone can say that they've successfully tested this sort of multicast configuration. I.e. two QPs from the same HCA in the same group. - SeanR ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Do you have a proposal for how to get to where you think this needs to be ? Have you looked at this ? I think you are proposing OpenSM include verbs.h. I think that's only part of what would need to be done (and has some other side effects). -- Hal From: [EMAIL PROTECTED] on behalf of Sean Hefty Sent: Mon 8/14/2006 5:15 PM To: Michael S. Tsirkin Cc: openib-general@openib.org Subject: Re: [openib-general] Conflicting typedefs for ib_gid_t Michael S. Tsirkin wrote: There are not that many common IB types. verbs and management for example are more or less isolated. CM users mostly don't care about SMPs. etc. What's common? GID? It does not seem worth it for a free format 16 byte network endianness type. Verbs and management are not isolated. Establishing a connection, joining a multicast group, or acquiring a path record are essential for actually using verbs correctly. I don't necessary see a problem with what we have. Such defines mirror IB spec so are static and there is no overhead maintaining them. And this simplifies dependencies no end - think about testing tens of dependent libraries for breakage just 'cause you removed an used line in this included by everyone header. Having umad_gid, ibv_gid, ibv_sa_gid, mad_gid, cm_gid, some_other_random_library_gid is goofy. (Sorry, I'm completely ranting now.) Hmm. User might not have uverbs even loaded, so adding verbs as a dependency to e.g. SM seems like a bad idea. We only need the include file, not the library. opensm really tries to be stack-agnostic, so it does need its own layer for things. Then those things should be completely internal to opensm. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
Hal Rosenstock wrote: Do you have a proposal for how to get to where you think this needs to be ? Have you looked at this ? I think defining include files similar to what we have for the kernel make sense. The problem is more complex than what definitions an application gets from which include file. The data in these structures are also exchanged between userspace and the kernel. For example, there's an sa.h file as part of libibverbs, since it marshals parameters from kernel to userspace. Wire structure definitions ended up working well for me for user to kernel transitions. I think you are proposing OpenSM include verbs.h. I think that's only part of what would need to be done (and has some other side effects). I would also have definitions in a libibsa and libibcm. The relevant CM definitions are there. (I don't see a need to expose most of the CM wire definition outside of the kernel.) I can't think of a reason why OpenSM would need any CM definitions, so I would remove those from from any OpenSM include files. SA attribute structures are also needed by libibsa, but since it seems overkill to include OpenSM on all nodes, I would define the SA attribute structures as part of libibsa, and let OpenSM use its include files. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Conflicting typedefs for ib_gid_t
I agree that SM (OpenSM) doesn't need CM definitions. ib_types.h was for convenience of other apps and having all the definitions in one place. It includes other definitions for other classes also unused by OpenSM. Once libibsa is present and supports (at least) all the SA attributes that OpenSM does, we could then talk about moving this over. However, at that point, things will be quite different between Linux and Windows versions of OpenSM unless Windows adopted more of what going on in Linux. -- Hal From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Mon 8/14/2006 7:44 PM To: Hal Rosenstock Cc: Michael S. Tsirkin; openib-general@openib.org Subject: Re: [openib-general] Conflicting typedefs for ib_gid_t Hal Rosenstock wrote: Do you have a proposal for how to get to where you think this needs to be ? Have you looked at this ? I think defining include files similar to what we have for the kernel make sense. The problem is more complex than what definitions an application gets from which include file. The data in these structures are also exchanged between userspace and the kernel. For example, there's an sa.h file as part of libibverbs, since it marshals parameters from kernel to userspace. Wire structure definitions ended up working well for me for user to kernel transitions. I think you are proposing OpenSM include verbs.h. I think that's only part of what would need to be done (and has some other side effects). I would also have definitions in a libibsa and libibcm. The relevant CM definitions are there. (I don't see a need to expose most of the CM wire definition outside of the kernel.) I can't think of a reason why OpenSM would need any CM definitions, so I would remove those from from any OpenSM include files. SA attribute structures are also needed by libibsa, but since it seems overkill to include OpenSM on all nodes, I would define the SA attribute structures as part of libibsa, and let OpenSM use its include files. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
I added some debug printks in mthca_multicast_attach(). Roland, does this look ok to you? It seems correct to me: # dmesg mthca_multicast_attach qp_num 406 gid ff124001:000a0a0a lid c003 mthca_multicast_attach line 167 - found mgm, hash a20, prev , index a20 mthca_multicast_attach line 197 - updated mgm gid: mgm gid ff124001:000a0a0a mthca_multicast_attach line 219 - writing mgm: mgm-qp[0] 8406 (BE) mthca_multicast_attach qp_num 407 gid ff124001:000a0a0a lid c003 mthca_multicast_attach line 167 - found mgm, hash a20, prev , index a20 mthca_multicast_attach line 197 - updated mgm gid: mgm gid ff124001:000a0a0a mthca_multicast_attach line 219 - writing mgm: mgm-qp[1] 8407 (BE) You're two steps ahead. Yeah, that looks fine to me. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/srp: add port/device attributes
Quoting r. Roland Dreier [EMAIL PROTECTED]: Subject: Re: [PATCH] IB/srp: add port/device attributes Michael Hi, Roland! There does not, at the moment, seem to exist Michael a way to find out which HCA port the specific SRP host is Michael connected through. Seems OK -- although I wonder about the names srp_port and srp_device. I think ib_port and ib_device would make more sense (or perhaps local_ib_port and local_ib_device although I don't think that's really required). OK. Want a patch like that or will you fix it up? Michael While not really a bugfix, maybe the following is small Michael enough for 2.6.18? We will use it in srptools that will Michael ship with OFED. No, I think this is purely a new feature and I'll queue it for 2.6.19. Fair enough. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
