Re: [openib-general] IB mcast question
Hi guys. On Monday 14 August 2006 23:33, Sean Hefty wrote: Steve Wise wrote: So is this replicating done in the mthca hca? As just an FYI, I didn't see anything wrong in the mthca driver either when I was looking at this problem. Since one app is getting the mcast packet, can I assume the opensm code is doing the right thing switch/port wise? That seems like a fairly safe assumption. Should the SM get join requests for both applications that join the group on the same host? Or only the first one? Only the first join request should make it to the SA. The second join request is fulfilled by ib_multicast. This is what makes ib_multicast suspect. What is exactly the scenario that you are doing? We have a test (over the verbs) that have 1 server and n clients. All of the clients create a QPs and attaches them to the (same) multicast group (without any join). The server sends m messages and all of the clients get those messages in every QP. This test passes when it being executed in one HCA, in two HCAs (without any switch in the middle). Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] why sdp connections cost so much memory
(1) ibv_devinfo HCA: MHES18-XTC FW: 1.1.0 OFED: OFED-1.1-rc1 (2) Test Bed On Client: ib0: 193.12.10.24 test command: LD_PRELOAD=/usr/local/ofed/lib64/libsdp.so SIMPLE_LIBSDP=1 ab -c m -n m -X 193.12.10.14:3129 http://www.sse.com.cn/sseportal/ps/zhs/home.shtml The web page is about 68K. On Server: ib0: 193.12.10.14 squid.sdp -d 10 -f squid2.conf (I have changed squid-cache to support listening on SDP port 3129) The test result is : Concurrent Conns(=m) Free MemoryRequests completed 0 926980 0 100 712508 100 200 497372 200 300 282636 256 400 52868 256 500 kernel crashed because of out of memory From above, every about 100 concurrent SDP connections will cost 210M memory. It's too vast for large scale applications. TCP costs very lower memory than SDP. The max concurrent connections completed successfully is 256. it is some bad limit. Who knows how and when will solve the problem? I'll test the performance of sdp connection and compare it with TCP further. tks zhu --- [EMAIL PROTECTED] wrote: Send openib-general mailing list submissions to openib-general@openib.org To subscribe or unsubscribe via the World Wide Web, visit http://openib.org/mailman/listinfo/openib-general or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than Re: Contents of openib-general digest... Today's Topics: 1. Re: Conflicting typedefs for ib_gid_t (Sean Hefty) 2. Re: [PATCH] mthca: make IB_SEND_FENCE work (Michael S. Tsirkin) 3. Re: Conflicting typedefs for ib_gid_t (Hal Rosenstock) 4. sanity check on datapath (Michael S. Tsirkin) 5. Re: [PATCH] mthca: make IB_SEND_FENCE work (Sean Hefty) 6. Re: Conflicting typedefs for ib_gid_t (Lakshmanan, Madhu) 7. Re: IB mcast question (Steve Wise) 8. Re: Conflicting typedefs for ib_gid_t (Michael S. Tsirkin) -- Message: 1 Date: Mon, 14 Aug 2006 13:38:42 -0700 From: Sean Hefty [EMAIL PROTECTED] Subject: Re: [openib-general] Conflicting typedefs for ib_gid_t To: 'Michael S. Tsirkin' [EMAIL PROTECTED], Hal Rosenstock [EMAIL PROTECTED] Cc: openib-general@openib.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=us-ascii Yes, or ib_mad_, or ibmad_. And same for other IB_ and ib_ names there. We really need to do something about names like ib_attr_t. I like to move away from each library re-defining common IB data types. Something like ibv_gid should be picked up from libibverbs. IMO, the core of the problem is that opensm include files carry too many legacy typedefs. - Sean -- Message: 2 Date: Mon, 14 Aug 2006 23:41:54 +0300 From: Michael S. Tsirkin [EMAIL PROTECTED] Subject: Re: [openib-general] [PATCH] mthca: make IB_SEND_FENCE work To: Sean Hefty [EMAIL PROTECTED] Cc: Roland Dreier [EMAIL PROTECTED], openib-general@openib.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=us-ascii Quoting r. Sean Hefty [EMAIL PROTECTED]: This could eliminate the warning, and remove an if statement from executing on each iteration. We still need to test size0 to set size0 = size. So we just reuse the extra branch, and I agree with Roland this way code is clearer. -- MST -- Message: 3 Date: Mon, 14 Aug 2006 23:39:00 +0300 From: Hal Rosenstock [EMAIL PROTECTED] Subject: Re: [openib-general] Conflicting typedefs for ib_gid_t To: [EMAIL PROTECTED],Michael S. Tsirkin [EMAIL PROTECTED] Cc: openib-general@openib.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=iso-8859-1 To answer your questions: I'm not totally sure about your application but it seems to me to fall in the category being discussed. Not all of the definitions in ib_types.h are elsewhere. I am working on a patch to get you past this issue. -- Hal From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Mon 8/14/2006 4:35 PM To: Hal Rosenstock; Michael S. Tsirkin Cc: [EMAIL PROTECTED]; openib-general@openib.org Subject: Re: Conflicting typedefs for ib_gid_t I don't think the way forward is using iba/ in all applications. I see it mostly as a legacy header for opensm and related apps that want their own layer for portability between stacks. I agree with your view on iba/ib_types.h Does that imply that the definitions in iba/ib_types.h are not expected to be required or used by user-space applications other than those categories
Re: [openib-general] why sdp connections cost so much memory
Quoting r. zhu shi song [EMAIL PROTECTED]: Subject: why sdp connections cost so much memory (1) ibv_devinfo HCA: MHES18-XTC FW: 1.1.0 OFED: OFED-1.1-rc1 (2) Test Bed On Client: ib0: 193.12.10.24 test command: LD_PRELOAD=/usr/local/ofed/lib64/libsdp.so SIMPLE_LIBSDP=1 ab -c m -n m -X 193.12.10.14:3129 http://www.sse.com.cn/sseportal/ps/zhs/home.shtml The web page is about 68K. On Server: ib0: 193.12.10.14 squid.sdp -d 10 -f squid2.conf (I have changed squid-cache to support listening on SDP port 3129) The test result is : Concurrent Conns(=m) Free MemoryRequests completed 0 926980 0 100 712508 100 200 497372 200 300 282636 256 400 52868 256 500 kernel crashed because of out of memory From above, every about 100 concurrent SDP connections will cost 210M memory. It's too vast for large scale applications. TCP costs very lower memory than SDP. The max concurrent connections completed successfully is 256. it is some bad limit. Who knows how and when will solve the problem? I'll test the performance of sdp connection and compare it with TCP further. tks zhu Most memory in SDP goes into pre-posted receive buffers. Currently SDP pre-posts a fixed 64 32K buffers per connection, that is 2M per connection. To verify that's the issue, try opening drivers/infiniband/ulp/sdp/sdp.h and changing SDP_RX_SIZE from 0x40 to a smaller value. If this helps, as a quick work-around I can make this value globally configurable. TCP on the other hand scales down more gracefully, and so should SDP longer-term. --- [EMAIL PROTECTED] wrote: Send openib-general mailing list submissions to openib-general@openib.org To subscribe or unsubscribe via the World Wide Web, visit http://openib.org/mailman/listinfo/openib-general or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than Re: Contents of openib-general digest... Is this relevant somehow? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] A new version for srp daemon
See Below -Original Message- From: Roland Dreier [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 15, 2006 1:48 AM To: Ishai Rabinovitz Cc: openib-general@openib.org; Tziporet Koren Subject: Re: A new version for srp daemon I put the code in https://openib.org/svn/trunk/contrib/mellanox/gen2/src/userspace/srptool s/srp_daemon Seems like a bizarre place for it -- a package inside of the srptools package?? [ishai] This is another tool for srp. I thought that we want to put several tools in srptools directory. I will put it in https://openib.org/svn/gen2/trunk/src/userspace/srp_daemon 7) Uses the umad package. This seems like it adds a fairly complicated dependency (since umad depends on something else, etc) for minimal gain. Was it really worth it in terms of your code? I found the umad API more trouble than it was worth for the original srptools. [ishai] You may be right, but this way I can gain from future improvements in the umad package (I'm optimistic) - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [opensm] the default behavior of the openSM causes problems (configure the PKey table)
On Mon, 2006-08-14 at 07:36, Dotan Barak wrote: Hi. I noticed that the behavior of the openSM was changed in the latest driver: in the past, every HCA was configured (by the FW) with 0x in the first entry. today, Just as an FYI: I think that Anafas have this in the second entry on port 0. -- Hal the PKey table is being configured by the openSM: the first entry is being set to 0x7fff (except for the host that the SM is being executed from) This behavior is very problemtic because not all of the users would like to change the default PKey table (for example: MPI users). Users that will try to use OFED 1.1 (in the same way they used OFED 1.0) will get unexplained failures, because the connectivity because the nodes will be broken. (even the perfquery started to fail after executing the SM) I think that the default behavior of the openSM should be: not to change the PKey table, unless the user provided a PKey table policy file. Here are the props of the machines * Host Architecture : x86_64 Linux Distribution: Red Hat Enterprise Linux AS release 4 (Nahant Update 3) Kernel Version: 2.6.9-34.ELsmp GCC Version : gcc (GCC) 3.4.5 20051201 (Red Hat 3.4.5-2) Memory size : 4039892 kB Driver Version: gen2_linux-20060813-1905 (REV=8916) HCA ID(s) : mthca0 HCA model(s) : 23108 FW version(s) : 3.4.927 Board(s) : MT_003001 * thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
can you send me this code? I suspect the main difference is that I'm using librdmacm to join and leave mcast groups. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [openfabrics-ewg] OFED 1.1-rc1 is available
The OFED-1.1-rc1 source tar ball (openib-1.1.tgz ) created by build_ofed.sh script (from https://openib.org/svn/gen2/branches/1.1/ofed/build) build_ofed.sh script takes userspace libraries/binaries after executing: autogen.sh configure make dist Therefor, autogen.sh is not a part of it and also it is the reason that you see Makefiles there. Regards, Vladimir Ira Weiny wrote: Why is the OFED 1.1-rc1 source tar ball missing files when compared with the 1.1 branch? Of specific question is the absence of autogen.sh in libibverbs. Ira On Sun, 13 Aug 2006 16:14:10 +0300 Tziporet Koren [EMAIL PROTECTED] wrote: Hal Rosenstock wrote: Target release date: 12-Sep Intermediate milestones: 1. Create 1.1 branch of user level: 27-Jul - done 2. RC1: 8-Aug - done 3. Feature freeze (RC2): 17-Aug What is the start build date for RC2 ? When do developers need to have their code in by to make RC2 ? We will start on Tue 15-Aug. Is this OK with you? 4. Code freeze (rc-x): 6-Sep Is this 1 or 2 RCs beyond RC2 in order to make this ? I hope one but I guess it will be two more RCs. Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openfabrics-ewg mailing list [EMAIL PROTECTED] http://openib.org/mailman/listinfo/openfabrics-ewg ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Where are running examples using libibcm in OFED-1.0.1?
Hi, I'm just trying to port my IB gen1 application to gen2 in order to get it running on a SLES 9 SP3 x86_64 Linux workstation. I got stuck in porting the usage of the libcm in gen1 to libibcm in gen2. My code fails on using the ib_cm_create_id call. Looked around for some running examples using the libibcm but could not find them. Got a pointer to: https://openib.org/svn/gen2/branches/1.0/src/userspace/libibcm/examples/ but those wont compile. Can someone help me out here? Here is some background about my application: Out application is running between a PowerPC Linux gen1 IB source and an x86 Linux PC data destination. All is running fine with gen1 drivers from Mellanox IBGD-1.8.2. In order to get our data destination application running on 64-Bit Workstations with SLES 9 SP3 and above as well I have to port it gen2. Thanks in advance. Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] A new version for srp daemon
Ishai [ishai] This is another tool for srp. I thought that Ishai we want to put several tools in srptools directory. I will Ishai put it in Ishai https://openib.org/svn/gen2/trunk/src/userspace/srp_daemon It would be fine to have the srp daemon be built as part of srptools. But putting another package with its own Makefile etc as a subdirectory of srptools is just weird. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] DAPL and local_iov in RDMA RR/RW mode
Hi Arlin, Thank you for your quick reply. Both dat_ep_post_rdma_read nad dat_ep_post_rdma_write return DAT_SUCCESS. When I read a field 'transfered_length' from DAT_DTO_COMPLETION_EVENT_DATA after calling a post function I receive the correct value which equals num_segs*seg_size. Unfortunately, when I read a content of a local buffer, only first segment is filled by appropriete data. I have tried to set up debug switch (by export DAPL_DBG_TYPE=0x before running my application) but unfortunately this does not produce any additional output for post functions. Do you have any other ideas? I did not mention before, but the case with num_segments1 works fine with a send/recv type of transmision. Best regards, Ryszard. - Original Message - From: Arlin Davis [EMAIL PROTECTED] To: Ryszard Jurga [EMAIL PROTECTED] Cc: openib openib-general@openib.org Sent: Friday, August 11, 2006 10:14 PM Subject: Re: [openib-general] DAPL and local_iov in RDMA RR/RW mode Ryszard Jurga wrote: Hi everybody, I have one question about a number of segments in local_iov when using RDMA Write and Read mode. Is it possible to have num_segments1? I am asking, because when I try to set up num_segments to a value 1, then I can still only read/write one segment, even though I have an appropriate remote buffer already reserved. The size of transfered buffer is 10bytes, num_segs=2. The information, which is printed below, was obrained from network devices with one remark - I have set up manualy max_rdma_read_iov=10 and max_rdma_write_iov=10. Thank you in advance for your help. Yes, uDAPL will support num_segments up to the max counts returned on the ep_attr. Can you be more specific? Does the post return immediate errors or are you simply missing data on the remote node? Can you turn up the uDAPL debug switch (export DAPL_DBG_TYPE=0x) and send output of the post call? -arlin Best regards, Ryszard. EP_ATTR: the same for both nodes: -- max_message_size=2147483648 max_rdma_size=2147483648 max_recv_dtos=16 max_request_dtos=16 max_recv_iov=4 max_request_iov=4 max_rdma_read_in=4 max_rdma_read_out=4 srq_soft_hw=0 max_rdma_read_iov=10 max_rdma_write_iov=10 ep_transport_specific_count=0 ep_provider_specific_count=0 -- IA_ATTR: different for nodes -- IA Info: max_eps=64512 max_dto_per_ep=65535 max_rdma_read_per_ep_in=4 max_rdma_read_per_ep_out=1610616831 max_evds=65408 max_evd_qlen=131071 max_iov_segments_per_dto=28 max_lmrs=131056 max_lmr_block_size=18446744073709551615 max_pzs=32768 max_message_size=2147483648 max_rdma_size=2147483648 max_rmrs=0 max_srqs=0 max_ep_per_srq=0 max_recv_per_srq=143263 max_iov_segments_per_rdma_read=1073741824 max_iov_segments_per_rdma_write=0 max_rdma_read_in=0 max_rdma_read_out=65535 max_rdma_read_per_ep_in_guaranteed=7286 max_rdma_read_per_ep_out_guaranteed=0 IA Info: max_eps=64512 max_dto_per_ep=65535 max_rdma_read_per_ep_in=4 max_rdma_read_per_ep_out=0 max_evds=65408 max_evd_qlen=131071 max_iov_segments_per_dto=28 max_lmrs=131056 max_lmr_block_size=18446744073709551615 max_pzs=32768 max_message_size=2147483648 max_rdma_size=2147483648 max_rmrs=0 max_srqs=0 max_ep_per_srq=0 max_recv_per_srq=142247 max_iov_segments_per_rdma_read=1073741824 max_iov_segments_per_rdma_write=0 max_rdma_read_in=0 max_rdma_read_out=65535 max_rdma_read_per_ep_in_guaranteed=7286 max_rdma_read_per_ep_out_guaranteed=28 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Just throwing out ideas here:
Maybe something in the ib_sa_mcmember_rec is prohibiting replication on
the HCA? And maybe ib_multicast is incorrectly building this record...
struct ib_sa_mcmember_rec {
union ib_gid mgid;
union ib_gid port_gid;
__be32 qkey;
__be16 mlid;
u8 mtu_selector;
u8 mtu;
u8 traffic_class;
__be16 pkey;
u8 rate_selector;
u8 rate;
u8 packet_life_time_selector;
u8 packet_life_time;
u8 sl;
__be32 flow_label;
u8 hop_limit;
u8 scope;
u8 join_state;
int proxy_join;
};
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve Just throwing out ideas here: Maybe something in the Steve ib_sa_mcmember_rec is prohibiting replication on the HCA? Steve And maybe ib_multicast is incorrectly building this Steve record... Shouldn't make a difference -- if one copy of the packet arrives at the HCA then none of the SA stuff matters as far as replicating it to multiple QPs. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] IB/core: fix SM LID/LID change with client reregister set
Hi, Roland!
Please consider the following patch for 2.6.18 - this fixes a regression
from 2.6.17 for us.
After commit 12bbb2b7be7f5564952ebe0196623e97464b8ac5, when
SM LID change or LID change MAD also has a client reregistration bit
set, only CLIENT_REREGISTER event is generated.
As a result, the sa_query module and the cache module don't update
the port information, and ULPs (e.g. IPoIB) stop working.
This is the regression we observe as compared to 2.6.17.
Rather than generate multiple events (which would have negative performance
impact), let us simply let cache and sa query respond to reregister event
in the same way as to LID and SM change events.
---
IB/core: fix SM LID/LID change with client reregister set
If PortInfo set (e.g. LID change) MAD has the reregister bit set,
IB_EVENT_LID_CHANGE event is no longer generated.
So sa_query and cache must respond to IB_EVENT_CLIENT_REREGISTER event
in the same way as to IB_EVENT_LID_CHANGE.
Signed-off-by: Jack Morgenstein [EMAIL PROTECTED]
Signed-off-by: Michael S. Tsirkin [EMAIL PROTECTED]
Index: ofed_1_1/drivers/infiniband/core/cache.c
===
--- ofed_1_1.orig/drivers/infiniband/core/cache.c 2006-08-03
14:30:20.0 +0300
+++ ofed_1_1/drivers/infiniband/core/cache.c2006-08-15 16:31:36.880294000
+0300
@@ -301,7 +301,8 @@ static void ib_cache_event(struct ib_eve
event-event == IB_EVENT_PORT_ACTIVE ||
event-event == IB_EVENT_LID_CHANGE ||
event-event == IB_EVENT_PKEY_CHANGE ||
- event-event == IB_EVENT_SM_CHANGE) {
+ event-event == IB_EVENT_SM_CHANGE ||
+ event-event == IB_EVENT_CLIENT_REREGISTER) {
work = kmalloc(sizeof *work, GFP_ATOMIC);
if (work) {
INIT_WORK(work-work, ib_cache_task, work);
Index: ofed_1_1/drivers/infiniband/core/sa_query.c
===
--- ofed_1_1.orig/drivers/infiniband/core/sa_query.c2006-08-03
14:30:20.0 +0300
+++ ofed_1_1/drivers/infiniband/core/sa_query.c 2006-08-15 16:32:35.100728000
+0300
@@ -405,7 +405,8 @@ static void ib_sa_event(struct ib_event_
event-event == IB_EVENT_PORT_ACTIVE ||
event-event == IB_EVENT_LID_CHANGE ||
event-event == IB_EVENT_PKEY_CHANGE ||
- event-event == IB_EVENT_SM_CHANGE) {
+ event-event == IB_EVENT_SM_CHANGE ||
+ event-event == IB_EVENT_CLIENT_REREGISTER) {
struct ib_sa_device *sa_dev;
sa_dev = container_of(handler, typeof(*sa_dev), event_handler);
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
How about qp attributes? pkeys? qkeys? On Tue, 2006-08-15 at 07:15 -0700, Roland Dreier wrote: Steve Just throwing out ideas here: Maybe something in the Steve ib_sa_mcmember_rec is prohibiting replication on the HCA? Steve And maybe ib_multicast is incorrectly building this Steve record... Shouldn't make a difference -- if one copy of the packet arrives at the HCA then none of the SA stuff matters as far as replicating it to multiple QPs. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve How about qp attributes? pkeys? qkeys? Good question -- yes, the QPs will need be to set up with the right keys for packets to appear. It's definitely something to check. If different mcmembers are used for the first join of the group and subsequent joins by another QP, that could explain the problem. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
When I run: # ip maddres add 224.0.0.9 dev ib0 I got (dmesg): ib0: joining MGID :::::7eb5:25c0:40f1 ib0: multicast join failed for: :::::7eb5:25c0:40f1, status -22 And from the SM side: ERR 1B01: Wrong MGID Prefix 0x00 must be 0xFF Regards, Vladimir Roland Dreier wrote: Vladimir Currently there is an issue with join to IPoIB multicast Vladimir group using both ip and ipmaddr utilities. What's the issue? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCHv2] IB/srp: add port/device attributes
Quoting r. Roland Dreier [EMAIL PROTECTED]:
Michael Hi, Roland! There does not, at the moment, seem to exist
Michael a way to find out which HCA port the specific SRP host is
Michael connected through.
Here's the updated version of the patch. Pls queue for 2.6.19.
Add local_ib_device/local_ib_port attributes to srp scsi_host.
Needed for when we want to connect to the same target through multiple distinct
ports.
Signed-off-by: Ishai Rabinovitz [EMAIL PROTECTED]
Signed-off-by: Michael S. Tsirkin [EMAIL PROTECTED]
Index: last_stable/drivers/infiniband/ulp/srp/ib_srp.c
===
--- last_stable.orig/drivers/infiniband/ulp/srp/ib_srp.c2006-08-15
16:55:55.0 +0300
+++ last_stable/drivers/infiniband/ulp/srp/ib_srp.c 2006-08-15
16:58:22.0 +0300
@@ -1467,12 +1467,29 @@ static ssize_t show_zero_req_lim(struct
return sprintf(buf, %d\n, target-zero_req_lim);
}
-static CLASS_DEVICE_ATTR(id_ext, S_IRUGO, show_id_ext, NULL);
-static CLASS_DEVICE_ATTR(ioc_guid, S_IRUGO, show_ioc_guid, NULL);
-static CLASS_DEVICE_ATTR(service_id, S_IRUGO, show_service_id, NULL);
-static CLASS_DEVICE_ATTR(pkey, S_IRUGO, show_pkey, NULL);
-static CLASS_DEVICE_ATTR(dgid, S_IRUGO, show_dgid, NULL);
-static CLASS_DEVICE_ATTR(zero_req_lim, S_IRUGO, show_zero_req_lim, NULL);
+static ssize_t show_local_ib_port(struct class_device *cdev, char *buf)
+{
+ struct srp_target_port *target = host_to_target(class_to_shost(cdev));
+
+ return sprintf(buf, %d\n, target-srp_host-port);
+}
+
+static ssize_t show_local_ib_device(struct class_device *cdev, char *buf)
+{
+ struct srp_target_port *target = host_to_target(class_to_shost(cdev));
+
+ return sprintf(buf, %s\n, target-srp_host-dev-dev-name);
+}
+
+
+static CLASS_DEVICE_ATTR(id_ext, S_IRUGO, show_id_ext, NULL);
+static CLASS_DEVICE_ATTR(ioc_guid, S_IRUGO, show_ioc_guid,NULL);
+static CLASS_DEVICE_ATTR(service_id, S_IRUGO, show_service_id, NULL);
+static CLASS_DEVICE_ATTR(pkey, S_IRUGO, show_pkey,NULL);
+static CLASS_DEVICE_ATTR(dgid, S_IRUGO, show_dgid,NULL);
+static CLASS_DEVICE_ATTR(zero_req_lim, S_IRUGO, show_zero_req_lim,NULL);
+static CLASS_DEVICE_ATTR(local_ib_port, S_IRUGO, show_local_ib_port, NULL);
+static CLASS_DEVICE_ATTR(local_ib_device, S_IRUGO, show_local_ib_device, NULL);
static struct class_device_attribute *srp_host_attrs[] = {
class_device_attr_id_ext,
@@ -1481,6 +1498,8 @@ static struct class_device_attribute *sr
class_device_attr_pkey,
class_device_attr_dgid,
class_device_attr_zero_req_lim,
+ class_device_attr_local_ib_port,
+ class_device_attr_local_ib_device,
NULL
};
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID change with client reregister set
On Tue, 2006-08-15 at 10:20, Michael S. Tsirkin wrote:
Hi, Roland!
Please consider the following patch for 2.6.18 - this fixes a regression
from 2.6.17 for us.
After commit 12bbb2b7be7f5564952ebe0196623e97464b8ac5, when
SM LID change or LID change MAD also has a client reregistration bit
set, only CLIENT_REREGISTER event is generated.
As a result, the sa_query module and the cache module don't update
the port information, and ULPs (e.g. IPoIB) stop working.
This is the regression we observe as compared to 2.6.17.
Rather than generate multiple events (which would have negative performance
impact), let us simply let cache and sa query respond to reregister event
in the same way as to LID and SM change events.
Are these two events equivalent ? e.g. does LID change require
reregistration ? (That's a potential overhead as well).
What about deregistration of the old registrations when this occurs ?
Is that handled ?
-- Hal
---
IB/core: fix SM LID/LID change with client reregister set
If PortInfo set (e.g. LID change) MAD has the reregister bit set,
IB_EVENT_LID_CHANGE event is no longer generated.
So sa_query and cache must respond to IB_EVENT_CLIENT_REREGISTER event
in the same way as to IB_EVENT_LID_CHANGE.
Signed-off-by: Jack Morgenstein [EMAIL PROTECTED]
Signed-off-by: Michael S. Tsirkin [EMAIL PROTECTED]
Index: ofed_1_1/drivers/infiniband/core/cache.c
===
--- ofed_1_1.orig/drivers/infiniband/core/cache.c 2006-08-03
14:30:20.0 +0300
+++ ofed_1_1/drivers/infiniband/core/cache.c 2006-08-15 16:31:36.880294000
+0300
@@ -301,7 +301,8 @@ static void ib_cache_event(struct ib_eve
event-event == IB_EVENT_PORT_ACTIVE ||
event-event == IB_EVENT_LID_CHANGE ||
event-event == IB_EVENT_PKEY_CHANGE ||
- event-event == IB_EVENT_SM_CHANGE) {
+ event-event == IB_EVENT_SM_CHANGE ||
+ event-event == IB_EVENT_CLIENT_REREGISTER) {
work = kmalloc(sizeof *work, GFP_ATOMIC);
if (work) {
INIT_WORK(work-work, ib_cache_task, work);
Index: ofed_1_1/drivers/infiniband/core/sa_query.c
===
--- ofed_1_1.orig/drivers/infiniband/core/sa_query.c 2006-08-03
14:30:20.0 +0300
+++ ofed_1_1/drivers/infiniband/core/sa_query.c 2006-08-15
16:32:35.100728000 +0300
@@ -405,7 +405,8 @@ static void ib_sa_event(struct ib_event_
event-event == IB_EVENT_PORT_ACTIVE ||
event-event == IB_EVENT_LID_CHANGE ||
event-event == IB_EVENT_PKEY_CHANGE ||
- event-event == IB_EVENT_SM_CHANGE) {
+ event-event == IB_EVENT_SM_CHANGE ||
+ event-event == IB_EVENT_CLIENT_REREGISTER) {
struct ib_sa_device *sa_dev;
sa_dev = container_of(handler, typeof(*sa_dev), event_handler);
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v2 1/2] sa_query: add generic query interfaces capable of supporting RMPP
On Thu, 2006-08-03 at 13:37, Sean Hefty wrote:
The following patch adds a generic interface to send MADs to the SA.
The primary motivation of adding these calls is to expand the SA query
interface to include RMPP responses for users wanting more than a
single attribute returned from a query (e.g. multipath record queries).
Do you mean multiple path records rather than multipath record queries
here ?
The implementation of existing SA query routines was layered on top of
the generic query interface.
Signed-off-by: Sean Hefty [EMAIL PROTECTED]
[snip...]
+/* Return size of SA attributes on the wire. */
+static int sa_mad_attr_size(int attr_id)
+{
+ int size;
+
+ switch (attr_id) {
+ case IB_SA_ATTR_SERVICE_REC:
+ size = 176;
+ break;
+ case IB_SA_ATTR_PATH_REC:
+ size = 64;
+ break;
+ case IB_SA_ATTR_MC_MEMBER_REC:
+ size = 52;
You probably already found this but this should be 56 as SA attributes
are required to be modulo 8 in size.
+ break;
+ default:
+ size = 0;
+ break;
+ }
+ return size;
+}
+
-- Hal
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
On Tue, 2006-08-15 at 10:30, Vladimir Sokolovsky wrote: When I run: # ip maddres add 224.0.0.9 dev ib0 I got (dmesg): ib0: joining MGID :::::7eb5:25c0:40f1 ib0: multicast join failed for: :::::7eb5:25c0:40f1, status -22 And from the SM side: ERR 1B01: Wrong MGID Prefix 0x00 must be 0xFF That means that something is forming the MGID improperly (from the IPmc address 224.0.0.9). Specifically, the MGID must start with 0xFF in the first byte. Is the MGID displayed slightly earlier in the log ? If no, can you run with -V and it should be there to see what the SM is seeing for the invalid MGID. -- Hal Regards, Vladimir Roland Dreier wrote: Vladimir Currently there is an issue with join to IPoIB multicast Vladimir group using both ip and ipmaddr utilities. What's the issue? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID change with client reregisterset
Quoting r. Hal Rosenstock [EMAIL PROTECTED]: Are these two events equivalent ? e.g. does LID change require reregistration ? (That's a potential overhead as well). Client reregistration support is an optional SM feature What about deregistration of the old registrations when this occurs ? Is that handled ? ??? sa_query and cache do not have any old registrations -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
MGID from the log: Aug 15 17:57:20 737657 [40796BB0] - MCMember Record dump: MGID0x : 0x7eb525c040f1 PortGid.0xfe80 : 0x0002c9020020d8ed qkey0xB1B mlid0x0 mtu.0x0 TClass..0x0 pkey0x rate0x0 pkt_life0x0 SLFlowLabelHopLimit.0x0 ScopeState..0x1 ProxyJoin...0x0 Aug 15 17:57:20 737735 [40796BB0] - osm_mcmr_rcv_create_new_mgrp: [ Aug 15 17:57:20 737755 [40796BB0] - __get_new_mlid: [ Aug 15 17:57:20 737776 [40796BB0] - __get_new_mlid: Found mgrp with lid:0xC000 MGID: 0xff12401b : 0x Aug 15 17:57:20 737797 [40796BB0] - __get_new_mlid: Found mgrp with lid:0xC001 MGID: 0xff12401b : 0x0001 Aug 15 17:57:20 737817 [40796BB0] - __get_new_mlid: Found mgrp with lid:0xC002 MGID: 0xff12601b : 0x0001ff20d8ee Aug 15 17:57:20 737836 [40796BB0] - __get_new_mlid: Found mgrp with lid:0xC003 MGID: 0xff12601b : 0x0001 Aug 15 17:57:20 737856 [40796BB0] - __get_new_mlid: Found mgrp with lid:0xC004 MGID: 0xff12601b : 0x0001ff20d8ed Aug 15 17:57:20 737876 [40796BB0] - __get_new_mlid: Found mgrp with lid:0xC005 MGID: 0xff12401b : 0x00fb Aug 15 17:57:20 737895 [40796BB0] - __get_new_mlid: Found mgrp with lid:0xC006 MGID: 0xff12601b : 0x0001ff20d901 Aug 15 17:57:20 737916 [40796BB0] - __get_new_mlid: Found available mlid:0xC007 at idx:7 Aug 15 17:57:20 737936 [40796BB0] - __get_new_mlid: ] Regards, Vladimir Hal Rosenstock wrote: On Tue, 2006-08-15 at 10:30, Vladimir Sokolovsky wrote: When I run: # ip maddres add 224.0.0.9 dev ib0 I got (dmesg): ib0: joining MGID :::::7eb5:25c0:40f1 ib0: multicast join failed for: :::::7eb5:25c0:40f1, status -22 And from the SM side: ERR 1B01: Wrong MGID Prefix 0x00 must be 0xFF That means that something is forming the MGID improperly (from the IPmc address 224.0.0.9). Specifically, the MGID must start with 0xFF in the first byte. Is the MGID displayed slightly earlier in the log ? If no, can you run with -V and it should be there to see what the SM is seeing for the invalid MGID. -- Hal Regards, Vladimir Roland Dreier wrote: Vladimir Currently there is an issue with join to IPoIB multicast Vladimir group using both ip and ipmaddr utilities. What's the issue? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] IPoIB high availability daemon
Vladimir When I run: # ip maddres add 224.0.0.9 dev ib0 It's poorly documented, but ip maddr add needs a link level address, not an IP address. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID change with client reregister set
On Tuesday 15 August 2006 17:42, Hal Rosenstock wrote: Are these two events equivalent ? e.g. does LID change require reregistration ? (That's a potential overhead as well). Before the change in mthca_mad.c (sm_snoop), the LID CHANGE event was generated in all cases where a SET port-info MAD was received. After the change, either LID_CHANGE -- or -- CLIENT_REREGISTER is generated. (i.e., CLIENT REREGISTER replaced LID_CHANGE in those cases where the reregistration bit was set in the MAD). This patch simply restores the previous behavior of the sa_query and cache modules in responding to events. No reregistration is required in these modules. What about deregistration of the old registrations when this occurs ? Is that handled ? There is no deregistration involvement. - Jack ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID change with client reregisterset
On Tue, 2006-08-15 at 10:59, Michael S. Tsirkin wrote: Quoting r. Hal Rosenstock [EMAIL PROTECTED]: Are these two events equivalent ? e.g. does LID change require reregistration ? (That's a potential overhead as well). Client reregistration support is an optional SM feature I don't think that answers my question. Yes, client reregistration is optional but when it is supported, are the two events equivalent ? What about deregistration of the old registrations when this occurs ? Is that handled ? ??? What's the ??? for ? sa_query and cache do not have any old registrations Currently; it could be an issue in the future. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] A new version for srp daemon
Tziporet So do you suggest we change the current srptools Tziporet Makefile to include this daemon too? Sure, that would be fine. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] A new version for srp daemon
Roland Dreier wrote: Ishai[ishai] This is another tool for srp. I thought that Ishai we want to put several tools in srptools directory. I will Ishai put it in Ishai https://openib.org/svn/gen2/trunk/src/userspace/srp_daemon It would be fine to have the srp daemon be built as part of srptools. But putting another package with its own Makefile etc as a subdirectory of srptools is just weird. - R. So do you suggest we change the current srptools Makefile to include this daemon too? Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] osm: Dynamic verbosity control per file
Hi Yevgeny, On Wed, 2006-08-02 at 11:16, Yevgeny Kliteynik wrote: Hi Hal This patch adds new verbosity functionality. I finally got some time to look this over and have a few comments on it (see embedded text). Generally, this looks good. One other thing that should be added to this patch is an update to the opensm man page for this functionality. So can you address the comment above as well as the embedded ones and resubmit ? Also, is this functionality needed for OFED 1.1 or is this trunk only ? Thanks. -- Hal 1. Verbosity configuration file --- The user is able to set verbosity level per source code file by supplying verbosity configuration file using the following command line arguments: -b filename --verbosity_file filename By default, the OSM will use the following file: /etc/opensmlog.conf Nit: For consistency in naming, this would be better as osmlog.conf (or osm-log.conf) rather than opensmlog.conf Verbosity configuration file should contain zero or more lines of the following pattern: filename verbosity_level where 'filename' is the name of the source code file that the 'verbosity_level' refers to, and the 'verbosity_level' itself should be specified as an integer number (decimal or hexadecimal). One reserved filename is 'all' - it represents general verbosity level, that is used for all the files that are not specified in the verbosity configuration file. If 'all' is not specified, the verbosity level set in the command line will be used instead. Note: The 'all' file verbosity level will override any other general level that was specified by the command line arguments. Sending a SIGHUP signal to the OSM will cause it to reload the verbosity configuration file. 2. Logging source code filename and line number --- If command line option -S or --log_source_info is specified, OSM will add source code filename and line number to every log message that is written to the log file. By default, the OSM will not log this additional info. Yevgeny Signed-off-by: Yevgeny Kliteynik [EMAIL PROTECTED] Index: include/opensm/osm_subnet.h === --- include/opensm/osm_subnet.h(revision 8614) +++ include/opensm/osm_subnet.h(working copy) @@ -285,6 +285,8 @@ typedef struct _osm_subn_opt osm_qos_options_tqos_sw0_options; osm_qos_options_tqos_swe_options; osm_qos_options_tqos_rtr_options; + boolean_tsrc_info; + char * verbosity_file; } osm_subn_opt_t; /* * FIELDS @@ -463,6 +465,27 @@ typedef struct _osm_subn_opt *qos_rtr_options *QoS options for router ports * +*src_info +*If TRUE - the source code filename and line number will be +* added to each log message. +*Default value - FALSE. +* +*verbosity_file +* OSM log configuration file - the file that describes +* verbosity level per source code file. +* The file may containg zero or more lines of the following +* pattern: +* filename verbosity_level +* where 'filename' is the name of the source code file that +* the 'verbosity_level' refers to. +* Filename all represents general verbosity level, that is +* used for all the files that are not specified in the +* verbosity file. +* If all is not specified, the general verbosity level will +* be used instead. +* Note: the all file verbosity level will override any other +* general level that was specified by the command line arguments. +* * SEE ALSO *Subnet object */ Index: include/opensm/osm_base.h === --- include/opensm/osm_base.h(revision 8614) +++ include/opensm/osm_base.h(working copy) @@ -222,6 +222,22 @@ BEGIN_C_DECLS #endif /***/ +/d* OpenSM: Base/OSM_DEFAULT_VERBOSITY_FILE +* NAME +*OSM_DEFAULT_VERBOSITY_FILE +* +* DESCRIPTION +*Specifies the default verbosity config file name +* +* SYNOPSIS +*/ +#ifdef __WIN__ +#define OSM_DEFAULT_VERBOSITY_FILE strcat(GetOsmPath(), opensmlog.conf) +#else +#define OSM_DEFAULT_VERBOSITY_FILE /etc/opensmlog.conf +#endif +/***/ + /d* OpenSM: Base/OSM_DEFAULT_PARTITION_CONFIG_FILE * NAME *OSM_DEFAULT_PARTITION_CONFIG_FILE Index: include/opensm/osm_log.h === --- include/opensm/osm_log.h(revision 8652) +++ include/opensm/osm_log.h(working copy) @@ -57,6 +57,7 @@ #include complib/cl_log.h #include complib/cl_spinlock.h #include opensm/osm_base.h +#include opensm/st.h #include iba/ib_types.h #include stdio.h @@ -123,9 +124,45 @@ typedef struct
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID change with client reregister set
Jack Morgenstein wrote: Before the change in mthca_mad.c (sm_snoop), the LID CHANGE event was generated in all cases where a SET port-info MAD was received. After the change, either LID_CHANGE -- or -- CLIENT_REREGISTER is generated. (i.e., CLIENT REREGISTER replaced LID_CHANGE in those cases where the reregistration bit was set in the MAD). This patch simply restores the previous behavior of the sa_query and cache modules in responding to events. I wish to emphases that without this fix opensm handover is NOT working now in OFED (and in kernel 2.6.18) and this is a must feature. Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID change with client reregister set
On Tue, 2006-08-15 at 11:06, Jack Morgenstein wrote: On Tuesday 15 August 2006 17:42, Hal Rosenstock wrote: Are these two events equivalent ? e.g. does LID change require reregistration ? (That's a potential overhead as well). Before the change in mthca_mad.c (sm_snoop), the LID CHANGE event was generated in all cases where a SET port-info MAD was received. After the change, either LID_CHANGE -- or -- CLIENT_REREGISTER is generated. (i.e., CLIENT REREGISTER replaced LID_CHANGE in those cases where the reregistration bit was set in the MAD). This patch simply restores the previous behavior of the sa_query and cache modules in responding to events. Understood but doesn't it also has the effect of doing more than this for certain cases of Set PortInfo ? No reregistration is required in these modules. Currently but I think it is a possibility in the future. -- Hal What about deregistration of the old registrations when this occurs ? Is that handled ? There is no deregistration involvement. - Jack ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v2 1/2] sa_query: add generic query interfaces capableof supporting RMPP
Do you mean multiple path records rather than multipath record queries here ? I meant multiple path records (GET_TABLE), but the interface is designed to handle both. The current implementation doesn't handle multipath record queries with more than 1 SGID and 1 DGID. (I.e. it uses a size of 56 bytes.) It shouldn't be hard to calculate the correct size based on the S/DGID counts; I just didn't do it yet. You probably already found this but this should be 56 as SA attributes are required to be modulo 8 in size. Isn't this an attribute offset issue, rather than a size issue? (I did have to account for the offset when walking attributes.) - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v2 1/2] sa_query: add generic query interfaces capableof supporting RMPP
On Tue, 2006-08-15 at 11:52, Sean Hefty wrote: Do you mean multiple path records rather than multipath record queries here ? I meant multiple path records (GET_TABLE), but the interface is designed to handle both. The current implementation doesn't handle multipath record queries with more than 1 SGID and 1 DGID. (I.e. it uses a size of 56 bytes.) It shouldn't be hard to calculate the correct size based on the S/DGID counts; I just didn't do it yet. I only saw SA SRs, PRs. and MCMs not MPRs. Did I miss the MPR support ? You probably already found this but this should be 56 as SA attributes are required to be modulo 8 in size. Isn't this an attribute offset issue, rather than a size issue? (I did have to account for the offset when walking attributes.) Yes. -- Hal - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v2 1/2] sa_query: add generic query interfacescapableof supporting RMPP
I only saw SA SRs, PRs. and MCMs not MPRs. Did I miss the MPR support ? Ah - this is in my local copy. I defined all attribute lengths (in an enum). I wanted to get the libibsa support completed to make sure that the SA interfaces provided what I needed. I will need to extract this patch, and SA registration back out from my other changes and re-submit the patches. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM
Can you see if this patch lets you build compost?
Signed-off-by: Sean Hefty [EMAIL PROTECTED]
---
Index: examples/cmpost.c
===
--- examples/cmpost.c (revision 8215)
+++ examples/cmpost.c (working copy)
@@ -614,6 +614,7 @@ out:
static int query_for_path(char *dst)
{
+ struct rdma_event_channel *channel;
struct rdma_cm_id *id;
struct sockaddr_in addr_in;
struct rdma_cm_event *event;
@@ -623,15 +624,19 @@ static int query_for_path(char *dst)
if (ret)
return ret;
- ret = rdma_create_id(id, NULL);
+ channel = rdma_create_event_channel();
+ if (!channel)
+ return -1;
+
+ ret = rdma_create_id(channel, id, NULL, RDMA_PS_TCP);
if (ret)
- return ret;
+ goto destroy_channel;
ret = rdma_resolve_addr(id, NULL, (struct sockaddr *) addr_in, 2000);
if (ret)
goto out;
- ret = rdma_get_cm_event(event);
+ ret = rdma_get_cm_event(channel, event);
if (!ret event-event != RDMA_CM_EVENT_ADDR_RESOLVED)
ret = event-status;
rdma_ack_cm_event(event);
@@ -642,7 +647,7 @@ static int query_for_path(char *dst)
if (ret)
goto out;
- ret = rdma_get_cm_event(event);
+ ret = rdma_get_cm_event(channel, event);
if (!ret event-event != RDMA_CM_EVENT_ROUTE_RESOLVED)
ret = event-status;
rdma_ack_cm_event(event);
@@ -652,6 +657,8 @@ static int query_for_path(char *dst)
test.path_rec = id-route.path_rec[0];
out:
rdma_destroy_id(id);
+destroy_channel:
+ rdma_destroy_event_channel(channel);
return ret;
}
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve How about qp attributes? pkeys? qkeys? Good question -- yes, the QPs will need be to set up with the right keys for packets to appear. It's definitely something to check. The qkeys used by the RDMA CM sound like they may be the problem. I'll verify this and see how to fix it if so. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID change withclient reregister set
Quoting r. Hal Rosenstock [EMAIL PROTECTED]: This patch simply restores the previous behavior of the sa_query and cache modules in responding to events. Understood but doesn't it also has the effect of doing more than this for certain cases of Set PortInfo ? Not that I can see. No reregistration is required in these modules. Currently but I think it is a possibility in the future. So then we'll have to do it. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
The qkeys used by the RDMA CM sound like they may be the problem. I'll verify this and see how to fix it if so. If I set the qkeys for the QPs and MCMemberRecord to 0, I can get this to work now. The RDMA CM uses a qkey = port number for UD QPs, and a qkey = IPv4 address for MCMemberRecords. A potential fix I see for this is to use the same qkey for all UD QPs and multicast groups created by the RDMA CM. Otherwise we restrict UD QPs to using a single destination (remote UD QP or multicast group.) - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 09:58 -0700, Sean Hefty wrote: The qkeys used by the RDMA CM sound like they may be the problem. I'll verify this and see how to fix it if so. If I set the qkeys for the QPs and MCMemberRecord to 0, I can get this to work now. The RDMA CM uses a qkey = port number for UD QPs, and a qkey = IPv4 address for MCMemberRecords. A potential fix I see for this is to use the same qkey for all UD QPs and multicast groups created by the RDMA CM. Otherwise we restrict UD QPs to using a single destination (remote UD QP or multicast group.) I was marching to the same tune! But I have a few points needing clarification. In my IP-centric mind, the sender specifies the ip mcast address and a remote port. All hosts with subscribers to the ip mcast address get the packet, and all sockets on those hosts who are bound to the dst_port receive a copy. Other sockets on those hosts that joined the ipmcast group but are bound to different ports will _not_ get a copy of the packet. In addition, the sender's local port number doesn't matter at all in the equation. Now how does that translate to qkeys, udqops, and ib mcast? It sounds to me like the remote_qkey is used to identify the mcast group when sending a mcast -and- to identify the set of qps on each host that should receive the incoming mcast packets. Is this true? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 12:58, Sean Hefty wrote: The qkeys used by the RDMA CM sound like they may be the problem. I'll verify this and see how to fix it if so. If I set the qkeys for the QPs and MCMemberRecord to 0, I can get this to work now. The RDMA CM uses a qkey = port number for UD QPs, and a qkey = IPv4 address for MCMemberRecords. A potential fix I see for this is to use the same qkey for all UD QPs and multicast groups created by the RDMA CM. Otherwise we restrict UD QPs to using a single destination (remote UD QP or multicast group.) Doesn't the QKey need to be the same as the one used for the IPoIB broadcast group (for the partition in question) per IPoIB RFC ? It should also be the one returned in the SA MCMemberRecord response. -- Hal - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID changewithclient reregister set
Quoting r. Hal Rosenstock [EMAIL PROTECTED]: Subject: Re: [PATCH] IB/core: fix SM LID/LID changewithclient reregister set On Tue, 2006-08-15 at 12:45, Michael S. Tsirkin wrote: Quoting r. Hal Rosenstock [EMAIL PROTECTED]: This patch simply restores the previous behavior of the sa_query and cache modules in responding to events. Understood but doesn't it also has the effect of doing more than this for certain cases of Set PortInfo ? Not that I can see. It's more based on what the driver(s) is/are doing which is not your change but is the one (back on May 31) which caused this change to be needed. Client reregister takes precedence over LID change so client reregister needs to do everything LID change does and possibly more. Yes, I dislike this too - this seems to set policy in low level driver, of all places. In hindsight, it seems that we should have had a single PORT_INFO_SET event with additional bits marking which fields were set. Something along the lines of IB_EVENT_PORTINFO_SET = 0x100, IB_EVENT_LID_CHANGE= 0x101, IB_EVENT_PKEY_CHANGE = 0x102, IB_EVENT_SM_CHANGE = 0x104, IB_EVENT_CLIENT_REREGISTER = 0x108 and now each event can pass full information to users, and you can test specific bits to figure out if you are interested, or just IB_EVENT_PORTINFO_SET to handle all such events identically. Clearly not 2.6.18, but - how does this sound? Your change looks fine to me. What about local_sa ? Does it need this change too ? No idea. I'm focusing on 2.6.18 mainly. local sa revocation policy is not really clear to me, maybe it's already covered? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID changewithclient reregister set
On Tue, 2006-08-15 at 13:53, Michael S. Tsirkin wrote: Quoting r. Hal Rosenstock [EMAIL PROTECTED]: Subject: Re: [PATCH] IB/core: fix SM LID/LID changewithclient reregister set On Tue, 2006-08-15 at 12:45, Michael S. Tsirkin wrote: Quoting r. Hal Rosenstock [EMAIL PROTECTED]: This patch simply restores the previous behavior of the sa_query and cache modules in responding to events. Understood but doesn't it also has the effect of doing more than this for certain cases of Set PortInfo ? Not that I can see. It's more based on what the driver(s) is/are doing which is not your change but is the one (back on May 31) which caused this change to be needed. Client reregister takes precedence over LID change so client reregister needs to do everything LID change does and possibly more. Yes, I dislike this too - this seems to set policy in low level driver, of all places. In hindsight, it seems that we should have had a single PORT_INFO_SET event with additional bits marking which fields were set. Something along the lines of IB_EVENT_PORTINFO_SET = 0x100, IB_EVENT_LID_CHANGE= 0x101, IB_EVENT_PKEY_CHANGE = 0x102, IB_EVENT_SM_CHANGE = 0x104, IB_EVENT_CLIENT_REREGISTER = 0x108 and now each event can pass full information to users, and you can test specific bits to figure out if you are interested, or just IB_EVENT_PORTINFO_SET to handle all such events identically. Clearly not 2.6.18, but - how does this sound? A bit mask sounds better. Is there any other info along with the events ? Your change looks fine to me. What about local_sa ? Does it need this change too ? No idea. I'm focusing on 2.6.18 mainly. local sa revocation policy is not really clear to me, maybe it's already covered? Sean ? -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] DAPL and local_iov in RDMA RR/RW mode
Ryszard Jurga wrote: Hi Arlin, Thank you for your quick reply. Both dat_ep_post_rdma_read nad dat_ep_post_rdma_write return DAT_SUCCESS. When I read a field 'transfered_length' from DAT_DTO_COMPLETION_EVENT_DATA after calling a post function I receive the correct value which equals num_segs*seg_size. Unfortunately, when I read a content of a local buffer, only first segment is filled by appropriete data. I have tried to set up debug switch (by export DAPL_DBG_TYPE=0x before running my application) but unfortunately this does not produce any additional output for post functions. Do you have any other ideas? I did not mention before, but the case with num_segments1 works fine with a send/recv type of transmision. You have to configure --enable-debug to get the debug information. You may want to pick up the latest dapl/test/dtest/dtest.c and take a look at the rdma write section for a simple multi-segment uDAPL example. I recently made a few modifications to include multiple segments in the test. You can also use dapltest to verify that RDMA with multple segments are working properly. Look at cl.sh for example script and dapltest -TT --help for usage. -arlin ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] IB/mthca: recover from device errors
Hello, Roland!
The following makes it possible to recover from catastrophic
errors through device reset. Could you please queue it for 2.6.19?
Implementation detail:
Catastrophic event device reset. Implemented via a fatal list, in
which device objects are queued for resetting. A spinlock guarantees
list insertion/deletion protection, while a mutex guarantees
that we don't perform device resets while a device
add/remove operation is in progress (and vice versa).
Added a workqueue to the mthca driver to perform the reset in a
thread context.
--
Trigger devie remove and then add once a catastrophic error was
detected in hardware. This, in turn, will cause a device
reset typically recovering from the catastrophic condition.
Since this might interefere with debugging the error root
cause, add a module option to suppress this behaviour.
Signed-off-by: Jack Morgenstein [EMAIL PROTECTED]
Signed-off-by: Michael S. Tsirkin [EMAIL PROTECTED]
Index: ofed_1_1/drivers/infiniband/hw/mthca/mthca_catas.c
===
--- ofed_1_1.orig/drivers/infiniband/hw/mthca/mthca_catas.c 2006-08-03
14:30:21.645701000 +0300
+++ ofed_1_1/drivers/infiniband/hw/mthca/mthca_catas.c 2006-08-10
16:46:57.418864000 +0300
@@ -34,6 +34,7 @@
#include linux/jiffies.h
#include linux/timer.h
+#include linux/workqueue.h
#include mthca_dev.h
@@ -48,9 +49,42 @@ enum {
static DEFINE_SPINLOCK(catas_lock);
+static struct workqueue_struct *catas_wq;
+static struct list_head catas_list;
+static struct work_struct catas_work;
+
+static int catas_reset_disable = 0;
+module_param_named(catas_reset_disable, catas_reset_disable, int, 0644);
+MODULE_PARM_DESC(catas_reset_disable, disable reset on catastrophic event if
0);
+
+static void catas_reset(void *work_ptr)
+{
+ struct mthca_dev *dev, *tmpdev;
+ LIST_HEAD(local_catas);
+ unsigned long flags;
+ int rc;
+
+ mutex_lock(mthca_device_mutex);
+
+ spin_lock_irqsave(catas_lock, flags);
+ list_for_each_entry_safe(dev, tmpdev, catas_list, catas_err.list)
+ list_move_tail(dev-catas_err.list, local_catas);
+ spin_unlock_irqrestore(catas_lock, flags);
+
+ list_for_each_entry_safe(dev, tmpdev, local_catas, catas_err.list) {
+ rc = mthca_restart_one(dev-pdev);
+ if (rc)
+ mthca_err(dev, Reset failed (%d)\n, rc);
+ else
+ mthca_dbg(dev, Reset succeeded\n);
+ }
+ mutex_unlock(mthca_device_mutex);
+}
+
static void handle_catas(struct mthca_dev *dev)
{
struct ib_event event;
+ unsigned long flags;
const char *type;
int i;
@@ -82,6 +116,14 @@ static void handle_catas(struct mthca_de
for (i = 0; i dev-catas_err.size; ++i)
mthca_err(dev, buf[%02x]: %08x\n,
i, swab32(readl(dev-catas_err.map + i)));
+
+ if (catas_reset_disable)
+ return;
+
+ spin_lock_irqsave(catas_lock, flags);
+ list_add(dev-catas_err.list, catas_list);
+ queue_work(catas_wq, catas_work);
+ spin_unlock_irqrestore(catas_lock, flags);
}
static void poll_catas(unsigned long dev_ptr)
@@ -135,11 +177,14 @@ void mthca_start_catas_poll(struct mthca
dev-catas_err.timer.data = (unsigned long) dev;
dev-catas_err.timer.function = poll_catas;
dev-catas_err.timer.expires = jiffies + MTHCA_CATAS_POLL_INTERVAL;
+ INIT_LIST_HEAD(dev-catas_err.list);
add_timer(dev-catas_err.timer);
}
void mthca_stop_catas_poll(struct mthca_dev *dev)
{
+ unsigned long flags;
+
spin_lock_irq(catas_lock);
dev-catas_err.stop = 1;
spin_unlock_irq(catas_lock);
@@ -153,4 +198,23 @@ void mthca_stop_catas_poll(struct mthca_
dev-catas_err.addr),
dev-catas_err.size * 4);
}
+
+ spin_lock_irqsave(catas_lock, flags);
+ list_del(dev-catas_err.list);
+ spin_unlock_irqrestore(catas_lock, flags);
+}
+
+int __init mthca_catas_init(void)
+{
+ INIT_LIST_HEAD(catas_list);
+ INIT_WORK(catas_work, catas_reset, NULL);
+ catas_wq = create_singlethread_workqueue(mthcacatas);
+ if (!catas_wq)
+ return -ENOMEM;
+ return 0;
+}
+
+void mthca_catas_cleanup(void)
+{
+ destroy_workqueue(catas_wq);
}
Index: ofed_1_1/drivers/infiniband/hw/mthca/mthca_main.c
===
--- ofed_1_1.orig/drivers/infiniband/hw/mthca/mthca_main.c 2006-08-03
14:30:21.747701000 +0300
+++ ofed_1_1/drivers/infiniband/hw/mthca/mthca_main.c 2006-08-10
16:46:16.770946000 +0300
@@ -80,6 +80,8 @@ static int tune_pci = 0;
module_param(tune_pci, int, 0444);
MODULE_PARM_DESC(tune_pci, increase PCI burst from the default set by BIOS if
nonzero);
+struct mutex mthca_device_mutex;
+
static
Re: [openib-general] [PATCH] IB/core: fix SM LID/LID changewithclient reregister set
What about local_sa ? Does it need this change too ? No idea. I'm focusing on 2.6.18 mainly. local sa revocation policy is not really clear to me, maybe it's already covered? ib_multicast is okay, but ib_local_sa could probably use the change. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 14:18, Sean Hefty wrote: A potential fix I see for this is to use the same qkey for all UD QPs and multicast groups created by the RDMA CM. Otherwise we restrict UD QPs to using a single destination (remote UD QP or multicast group.) Doesn't the QKey need to be the same as the one used for the IPoIB broadcast group (for the partition in question) per IPoIB RFC ? It should also be the one returned in the SA MCMemberRecord response. It shouldn't. The RDMA CM multicast groups are separate from those used by ipoib. Is the IP address only used locally to construct the MGID ? What does the MGID look like ? What signature does it use if any ? -- Hal - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
In my IP-centric mind, the sender specifies the ip mcast address and a remote port. All hosts with subscribers to the ip mcast address get the packet, and all sockets on those hosts who are bound to the dst_port receive a copy. Other sockets on those hosts that joined the ipmcast group but are bound to different ports will _not_ get a copy of the packet. In addition, the sender's local port number doesn't matter at all in the equation. Now how does that translate to qkeys, udqops, and ib mcast? Currently, the IP address is mapped to an MGID. Senders and receivers are required to subscribe to the multicast group in order to receive packets from the multicast group. (The UD QPs must be attached to the group to get the packet.) The port number is not used. Is it possible for an IP socket to receive packets from multiple multicast groups? It sounds to me like the remote_qkey is used to identify the mcast group when sending a mcast -and- to identify the set of qps on each host that should receive the incoming mcast packets. Is this true? I think the QKey usage in the RDMA CM needs to be redone. If we look at just UD QP transfers, in order to support one-to-many data transfers, all of the QPs need to have the same QKey. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Is the IP address only used locally to construct the MGID ? What does the MGID look like ? What signature does it use if any ? The IP address may also used be used to lookup routing information in order to bind to a local device. The address is then used locally construct the MGID. The MGID looks a lot like the ipoib MGIDs, with a byte 8 being 0x01. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 14:33, Sean Hefty wrote: Is the IP address only used locally to construct the MGID ? What does the MGID look like ? What signature does it use if any ? The IP address may also used be used to lookup routing information in order to bind to a local device. The address is then used locally construct the MGID. The MGID looks a lot like the ipoib MGIDs, with a byte 8 being 0x01. One of the reserved bytes in the MGID is 1 rather than 0 and it's using an IPv4 signature (0x401b) ? Where does the qkey come from on the creation of the group ? -- Hal - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
One of the reserved bytes in the MGID is 1 rather than 0 and it's using an IPv4 signature (0x401b) ? It uses a signature of 0x4001 to avoid conflicts with ipoib groups. Where does the qkey come from on the creation of the group ? The qkey is the same as the IPv4 address. I need to spend some time looking at the QKeys of the QPs and the multicast group to understand how one of the receivers worked. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [Fwd: RE: IB mcast question]
accidentally dropped off the list... Forwarded Message From: Sean Hefty [EMAIL PROTECTED] To: 'Steve Wise' [EMAIL PROTECTED] Subject: RE: [openib-general] IB mcast question Date: Tue, 15 Aug 2006 11:59:12 -0700 FYI - your reply dropped off the list. For type SOCK_DGRAM (UDP), the socket will receive packets from multiple subscribed ip mcast groups iff the dst_port of the incoming packet matches the port to which the socket is bound... This is what I was referring to. I'm really not familiar with IP multicast beyond what I read in a book while coding the RDMA CM. It sounds like we might be able to use the QKey as the port number for the QP to mimic the behavior. The RDMA CM sets the QKey for UD QPs to the port number, but sets the QKey of a multicast group to the IPv4 address. NOTE: I'm just trying to understand how this works in IB. I'm not necessarily advocating it should behave exactly like ip mcast/udp. Clients need to create an UD QP. When they join a multicast group, they get an MGID, MLID, and QKey. The UD QP needs to attach to the MGID / MLID, and have a matching QKey. Today, the RDMA CM assigns a QKey to a UD QP when it's created; it doesn't know if it will join a multicast group or not. And if you want to support a single UD QP receiving from multiple subscribed groups, you'll have to have the same Qkey for all the groups and QPs. Right? I believe that this will be the case. A send can specify the remote QKey, so it may be that UD QP transfers are okay, and only multicast has an issue. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
[adding back to list] On Tue, 2006-08-15 at 11:59 -0700, Sean Hefty wrote: For type SOCK_DGRAM (UDP), the socket will receive packets from multiple subscribed ip mcast groups iff the dst_port of the incoming packet matches the port to which the socket is bound... This is what I was referring to. I'm really not familiar with IP multicast beyond what I read in a book while coding the RDMA CM. It sounds like we might be able to use the QKey as the port number for the QP to mimic the behavior. The RDMA CM sets the QKey for UD QPs to the port number, but sets the QKey of a multicast group to the IPv4 address. NOTE: I'm just trying to understand how this works in IB. I'm not necessarily advocating it should behave exactly like ip mcast/udp. Clients need to create an UD QP. When they join a multicast group, they get an MGID, MLID, and QKey. The UD QP needs to attach to the MGID / MLID, and have a matching QKey. Today, the RDMA CM assigns a QKey to a UD QP when it's created; it doesn't know if it will join a multicast group or not. Looking at the mckey code, I see that the code calls rdma_get_dst_attr() to get the remote qpn/qkey + the ah_attrs for the mcast group (which is the dst addr in this case). Then it creates an ibv_ah. Later when sending, the SEND WR contains both the ah and the remote qpn/qkey. Why are these separated? Isn't an address handle needed for each destination QP? If so, then why is the remote qpn/qkey also needed to transmit a datagram? Trying to understand how ah's relate to qpn/qkeys... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Why are these separated? Isn't an address handle needed for each destination QP? If so, then why is the remote qpn/qkey also needed to transmit a datagram? The address handle doesn't include QPN/QKey information. Maybe think of them more as specifying the path to some port. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
This is what I was referring to. I'm really not familiar with IP multicast beyond what I read in a book while coding the RDMA CM. It sounds like we might be able to use the QKey as the port number for the QP to mimic the behavior. I don't see how this could work-- to mimic IP, an app has to be able to multicast a datagram to multiple destination ports using the same multicast group. Since IB multicast groups have a Q_Key associated, the only way this could work is if all QPs and MCGs share one Q_Key. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] question: ib_umem page_size
Michael Roland, could you please clarify what does the page_size Michael field in struct ib_mem do? It gives the page size for the user memory described by the struct. The idea was that if/when someone tries to optimize for huge pages, then the low-level driver can know that a region is using huge pages without having to walk through the page list and search for the minimum physically contiguous size. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 12:39 -0700, Sean Hefty wrote: Why are these separated? Isn't an address handle needed for each destination QP? If so, then why is the remote qpn/qkey also needed to transmit a datagram? The address handle doesn't include QPN/QKey information. Maybe think of them more as specifying the path to some port. Ok. From what I can tell via experimentation, the qkey of the mcast group doesn't need to have any relation to the qkeys of the qps. I was able to create a mcast group with the mc qkey==0xe00a0a0a, and 3 apps joined this group, but their qp qkeys were 0 (I changed ucma_init_ud_qp() to set the qp qkey to 0). One app sent to the mcgroup ah/qkey/qpn and the other two received the packet. Does that make sense? So maybe all we need is the concept of REUSE_PORT to allow multiple librdma users to create cm_ids with the same local port. currently this isn't allowed. If we do this, then all processes that want to exchange mcast packets would create cm_ids and do rdma_resolve_addr() with the same src port number on all systems. Senders send to the ah/remote_qpn/remote_qkey of the mcast group. This routes packets to all IB ports that have subscribers. Then since the sender's qp has the same qkey as all the group participants each qp will receive a copy of the packet. The mcast setup code in librdma doesn't need to change. IE the qkey can remain the ip mcast address. I think this will work. It is similar to UDP/IP/MCAST... Or am I all wet? whatchathink? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] question: ib_umem page_size
Quoting r. Roland Dreier [EMAIL PROTECTED]: Subject: Re: question: ib_umem page_size Michael Roland, could you please clarify what does the page_size Michael field in struct ib_mem do? It gives the page size for the user memory described by the struct. The idea was that if/when someone tries to optimize for huge pages, then the low-level driver can know that a region is using huge pages without having to walk through the page list and search for the minimum physically contiguous size. Thoguth though. Cool, that's exactly what I'm trying to do. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 15:59, Roland Dreier wrote: This is what I was referring to. I'm really not familiar with IP multicast beyond what I read in a book while coding the RDMA CM. It sounds like we might be able to use the QKey as the port number for the QP to mimic the behavior. I don't see how this could work-- to mimic IP, an app has to be able to multicast a datagram to multiple destination ports using the same multicast group. Since IB multicast groups have a Q_Key associated, the only way this could work is if all QPs and MCGs share one Q_Key. Isn't it all QPs on the same MCG need to share one Q_Key ? -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 16:12 -0400, Hal Rosenstock wrote: On Tue, 2006-08-15 at 15:59, Roland Dreier wrote: This is what I was referring to. I'm really not familiar with IP multicast beyond what I read in a book while coding the RDMA CM. It sounds like we might be able to use the QKey as the port number for the QP to mimic the behavior. I don't see how this could work-- to mimic IP, an app has to be able to multicast a datagram to multiple destination ports using the same multicast group. Since IB multicast groups have a Q_Key associated, the only way this could work is if all QPs and MCGs share one Q_Key. Isn't it all QPs on the same MCG need to share one Q_Key ? From my experimentation, only the qkeys of the QPs need match. The MCG qkey is only needed for sending to the group. Am I confused? (often the case :) ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve I was able to create a mcast group with the mc Steve qkey==0xe00a0a0a, and 3 apps joined this group, but their Steve qp qkeys were 0 (I changed ucma_init_ud_qp() to set the qp Steve qkey to 0). One app sent to the mcgroup ah/qkey/qpn and Steve the other two received the packet. Does that make sense? In theory the Q_Key of a multicast group record is the Q_Key you're supposed to use when sending to the group. Of course nothing enforces this but I don't really like abusing things this way. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 13:17 -0700, Roland Dreier wrote: Steve I was able to create a mcast group with the mc Steve qkey==0xe00a0a0a, and 3 apps joined this group, but their Steve qp qkeys were 0 (I changed ucma_init_ud_qp() to set the qp Steve qkey to 0). One app sent to the mcgroup ah/qkey/qpn and Steve the other two received the packet. Does that make sense? In theory the Q_Key of a multicast group record is the Q_Key you're supposed to use when sending to the group. Of course nothing enforces this but I don't really like abusing things this way. I _did_ send the message to the qkey of the mcg. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
On Tue, 2006-08-15 at 16:07, Steve Wise wrote: On Tue, 2006-08-15 at 12:39 -0700, Sean Hefty wrote: Why are these separated? Isn't an address handle needed for each destination QP? If so, then why is the remote qpn/qkey also needed to transmit a datagram? The address handle doesn't include QPN/QKey information. Maybe think of them more as specifying the path to some port. Ok. From what I can tell via experimentation, the qkey of the mcast group doesn't need to have any relation to the qkeys of the qps. That may be what is happening but I don't think that is correct (per the IBA spec). I was able to create a mcast group with the mc qkey==0xe00a0a0a, Don't we need to be careful about controlled Q_Keys as well ? -- Hal and 3 apps joined this group, but their qp qkeys were 0 (I changed ucma_init_ud_qp() to set the qp qkey to 0). One app sent to the mcgroup ah/qkey/qpn and the other two received the packet. Does that make sense? So maybe all we need is the concept of REUSE_PORT to allow multiple librdma users to create cm_ids with the same local port. currently this isn't allowed. If we do this, then all processes that want to exchange mcast packets would create cm_ids and do rdma_resolve_addr() with the same src port number on all systems. Senders send to the ah/remote_qpn/remote_qkey of the mcast group. This routes packets to all IB ports that have subscribers. Then since the sender's qp has the same qkey as all the group participants each qp will receive a copy of the packet. The mcast setup code in librdma doesn't need to change. IE the qkey can remain the ip mcast address. I think this will work. It is similar to UDP/IP/MCAST... Or am I all wet? whatchathink? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB mcast question
Steve I was able to create a mcast group with the mc Steve qkey==0xe00a0a0a, and 3 apps joined this group, but their Steve qp qkeys were 0 (I changed ucma_init_ud_qp() to set the qp Steve qkey to 0). One app sent to the mcgroup ah/qkey/qpn and Steve the other two received the packet. Does that make sense? In theory the Q_Key of a multicast group record is the Q_Key you're supposed to use when sending to the group. Of course nothing enforces this but I don't really like abusing things this way. I _did_ send the message to the qkey of the mcg. I didn't think that this was supposed to work. Is the QKey going out on the wire the QKey from the send WR, or that associated with the QP? I think the QKey going out on the wire is the latter, which just happens to make it work. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 13:55 -0700, Roland Dreier wrote: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. - R. You guys are confusing me. sends to multicast group but with QKey=22... Does that mean you posted a SEND with the remote_qkey=22? According to the spec, C10-15 sez the qkey in the outgoing packet will be the qkey from the QP context IF the high order bit of the qkey is set. If the high order bit is _not_ set, then the outgoing packet will contain the qkey from the WR. (why? why?) Now, in my experiment, my mcast qkey was 0xe00a0a0a and the qp qkeys were zero. So when the sender posted a SEND with remote_qkey=0xe00a0a0a, the interface placed the qkey from sender's qp, which was zero, in the packet. That's why it worked I guess... Now my head hurts... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RFC: [PATCH untested] IB/uverbs: optimize registration for huge pages
Quoting r. Roland Dreier [EMAIL PROTECTED]:
Subject: Re: question: ib_umem page_size
Michael Roland, could you please clarify what does the page_size
Michael field in struct ib_mem do?
It gives the page size for the user memory described by the struct.
The idea was that if/when someone tries to optimize for huge pages,
then the low-level driver can know that a region is using huge pages
without having to walk through the page list and search for the
minimum physically contiguous size.
OK, so here's a patch [warning: untested] that attempts to do this - we have
customers that run out of resources when they register lots of huge pages,
and this will help.
How does this look? Is this the intended usage?
uverbs_mem.c | 14 +-
1 files changed, 13 insertions(+), 1 deletion(-)
--
Optimize memory registration for huge pages, by walking through
the page list and searching for the minimum physically contiguous
size.
Signed-off-by: Michael S. Tsirkin [EMAIL PROTECTED]
diff --git a/drivers/infiniband/core/uverbs_mem.c
b/drivers/infiniband/core/uverbs_mem.c
index efe147d..f750652 100644
--- a/drivers/infiniband/core/uverbs_mem.c
+++ b/drivers/infiniband/core/uverbs_mem.c
@@ -73,6 +73,8 @@ int ib_umem_get(struct ib_device *dev, s
unsigned long lock_limit;
unsigned long cur_base;
unsigned long npages;
+ dma_addr_t a, seg_end;
+ u32 mask = 0;
int ret = 0;
int off;
int i;
@@ -87,7 +89,6 @@ int ib_umem_get(struct ib_device *dev, s
mem-user_base = (unsigned long) addr;
mem-length= size;
mem-offset= (unsigned long) addr ~PAGE_MASK;
- mem-page_size = PAGE_SIZE;
mem-writable = write;
INIT_LIST_HEAD(mem-chunk_list);
@@ -149,6 +150,15 @@ int ib_umem_get(struct ib_device *dev, s
goto out;
}
+ for (i = 0; i chunk-nents; ++i) {
+ a = sg_dma_adress(chunk-page_list[i]);
+ if ((i || off) a != seg_end) {
+ mask |= seg_end;
+ mask |= a;
+ }
+ seg_end = a + sg_dma_len(chunk-page_list[i]);
+ }
+
ret -= chunk-nents;
off += chunk-nents;
list_add_tail(chunk-list, mem-chunk_list);
@@ -157,6 +167,8 @@ int ib_umem_get(struct ib_device *dev, s
ret = 0;
}
+ mem-page_size = ffs(mask) ? 1 (ffs(mask) - 1) : (1 31);
+
out:
if (ret 0)
__ib_umem_release(dev, mem, 0);
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 14:18 -0700, Sean Hefty wrote: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. I'll admit that this definitely seems like a hack, but I haven't found where it is violation. (I'll keep looking.) From 10.5.2.1, a QP doesn't need to be attached to a multicast group to initiate a send, so it seems like any potential violation is only that the QKey and MLID wouldn't match. - Sean And I can't find in the IBTA spec where they talk about the qkey field of a mcast group at all. qkeys are used to validate an incoming datagram. If the qkey in the packet doesn't match the qkey of the qp, then its dropped. This is an HCA ingress issue, not a mcast issue really. From what I can gather, the sender: 1) Doesn't need to be in the group (as sean pointed out) 2) posts a SEND WR with an address handle for the the mcast group, the remote_qpn == 0xf, and the remote_qkey == to remote qp's qkey. Now why C10-15 is there, I dunno...but i'm sure somebody around here does... :) ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:18, Sean Hefty wrote: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. I'll admit that this definitely seems like a hack, but I haven't found where it is violation. (I'll keep looking.) The first occurrence that I find is the following: p.145 13) where it states that Further, in addition to having the same MGID, all members of the multicast group must share the same P_Key and Q_Key. That is compliance per C4-3 on p. 143. There may be others. -- Hal From 10.5.2.1, a QP doesn't need to be attached to a multicast group to initiate a send, so it seems like any potential violation is only that the QKey and MLID wouldn't match. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:13, Steve Wise wrote: On Tue, 2006-08-15 at 13:55 -0700, Roland Dreier wrote: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. - R. You guys are confusing me. sends to multicast group but with QKey=22... Does that mean you posted a SEND with the remote_qkey=22? According to the spec, C10-15 sez the qkey in the outgoing packet will be the qkey from the QP context IF the high order bit of the qkey is set. If the high order bit is _not_ set, then the outgoing packet will contain the qkey from the WR. (why? why?) The high order bit is for controlled QKeys which are only supposed to be sent by privileged consumers (e.g. MAD layer for QP1 handing). Also, the C10-15 requirement does not speak to additional requirements on the Q_Keys themselves like the one I cited to Sean in a previous email in terms of multicast group addressing requirements. Now, in my experiment, my mcast qkey was 0xe00a0a0a I don't think that should be allowed from user space. -- Hal and the qp qkeys were zero. So when the sender posted a SEND with remote_qkey=0xe00a0a0a, the interface placed the qkey from sender's qp, which was zero, in the packet. That's why it worked I guess... Now my head hurts... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
p.145 13) where it states that Further, in addition to having the same MGID, all members of the multicast group must share the same P_Key and Q_Key. That is compliance per C4-3 on p. 143. Thanks - my take is that the compliance breaks when the QPs are first created with the wrong QKey. Then, as Roland said, in order to support a QP using multiple multicast groups, we need all QKeys to be the same (for the RDMA CM). If we want to do anything with port numbers, we probably need to fold those into the MGID. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:29 -0400, Hal Rosenstock wrote: On Tue, 2006-08-15 at 17:18, Sean Hefty wrote: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. I'll admit that this definitely seems like a hack, but I haven't found where it is violation. (I'll keep looking.) The first occurrence that I find is the following: p.145 13) where it states that Further, in addition to having the same MGID, all members of the multicast group must share the same P_Key and Q_Key. Where does it say that a mcast group is associated with any qkey at all? 13) just sez that all qp's that want to exchange datagrams must have the same pkey/qkey. That's the normal UD paradigm. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:26, Steve Wise wrote: On Tue, 2006-08-15 at 14:18 -0700, Sean Hefty wrote: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. I'll admit that this definitely seems like a hack, but I haven't found where it is violation. (I'll keep looking.) From 10.5.2.1, a QP doesn't need to be attached to a multicast group to initiate a send, so it seems like any potential violation is only that the QKey and MLID wouldn't match. - Sean And I can't find in the IBTA spec where they talk about the qkey field of a mcast group at all. Not so fast... I know the requirements are spread out through the spec and it's hard to piece them all together. I cited where this one is located. qkeys are used to validate an incoming datagram. If the qkey in the packet doesn't match the qkey of the qp, then its dropped. This is an HCA ingress issue, not a mcast issue really. From what I can gather, the sender: 1) Doesn't need to be in the group (as sean pointed out) Yes it does. There are send only (non) members in the JoinState component of the MCMemberRecord. This can be used to optimize the MC routing (as that port never needs to receive but needs to be able to send on the MLID). -- Hal 2) posts a SEND WR with an address handle for the the mcast group, the remote_qpn == 0xf, and the remote_qkey == to remote qp's qkey. Now why C10-15 is there, I dunno...but i'm sure somebody around here does... :) ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] debug version
Is there a way to use the build.sh script that comes with OFED to build a debug version of all the things listed in the ofed.conf file? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:40, Sean Hefty wrote: p.145 13) where it states that Further, in addition to having the same MGID, all members of the multicast group must share the same P_Key and Q_Key. That is compliance per C4-3 on p. 143. Thanks - my take is that the compliance breaks when the QPs are first created with the wrong QKey. Yes, I think so too. Then, as Roland said, in order to support a QP using multiple multicast groups, we need all QKeys to be the same (for the RDMA CM). Does 1 QP need to support multiple multicast groups ? If so, I agree. That was the point I missed before. If we want to do anything with port numbers, we probably need to fold those into the MGID. I'm not quite following you yet on this. -- Hal - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:41, Steve Wise wrote: On Tue, 2006-08-15 at 17:29 -0400, Hal Rosenstock wrote: On Tue, 2006-08-15 at 17:18, Sean Hefty wrote: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. I'll admit that this definitely seems like a hack, but I haven't found where it is violation. (I'll keep looking.) The first occurrence that I find is the following: p.145 13) where it states that Further, in addition to having the same MGID, all members of the multicast group must share the same P_Key and Q_Key. Where does it say that a mcast group is associated with any qkey at all? In order to create a multicast group, the Q_Key needs to be supplied. See p. 912-3 o15-0.2.3 -- Hal 13) just sez that all qp's that want to exchange datagrams must have the same pkey/qkey. That's the normal UD paradigm. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
1) Doesn't need to be in the group (as sean pointed out) Yes it does. There are send only (non) members in the JoinState component of the MCMemberRecord. This can be used to optimize the MC routing (as that port never needs to receive but needs to be able to send on the MLID). I believe that the local port needs to be at least a SendOnly member (for routing purposes), but the QP does not need to be attached to the group. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
According to the spec, C10-15 sez the qkey in the outgoing packet will be the qkey from the QP context IF the high order bit of the qkey is set. If the high order bit is _not_ set, then the outgoing packet will contain the qkey from the WR. (why? why?) The high order bit is for controlled QKeys which are only supposed to be sent by privileged consumers (e.g. MAD layer for QP1 handing). I see. Also, the C10-15 requirement does not speak to additional requirements on the Q_Keys themselves like the one I cited to Sean in a previous email in terms of multicast group addressing requirements. Now, in my experiment, my mcast qkey was 0xe00a0a0a I don't think that should be allowed from user space. Ah. So rdma cm needs to not use these special qkeys... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ib_get_dma_mr and remote access
Hi there, I would like to know if any application today uses ib_get_dma_mr verb with remote access flag(s). It seems to me that such a dependency could first, create a security hole and second, make this verb hard to implement for some RNICs. If only local access is required for this special memory region, can it be implemented with the Reserved LKey or STag0, whichever way it's called? Thanks, Louis ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:49, Steve Wise wrote: According to the spec, C10-15 sez the qkey in the outgoing packet will be the qkey from the QP context IF the high order bit of the qkey is set. If the high order bit is _not_ set, then the outgoing packet will contain the qkey from the WR. (why? why?) The high order bit is for controlled QKeys which are only supposed to be sent by privileged consumers (e.g. MAD layer for QP1 handing). I see. Also, the C10-15 requirement does not speak to additional requirements on the Q_Keys themselves like the one I cited to Sean in a previous email in terms of multicast group addressing requirements. Now, in my experiment, my mcast qkey was 0xe00a0a0a I don't think that should be allowed from user space. Ah. So rdma cm needs to not use these special qkeys... Yes, that's what I think. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] ib_get_dma_mr and remote access
Louis Hi there, I would like to know if any application today Louis uses ib_get_dma_mr verb with remote access flag(s). Yes, the SRP initiator does. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] ib_get_dma_mr and remote access
On Tue, 2006-08-15 at 15:18 -0700, Roland Dreier wrote: Louis Hi there, I would like to know if any application today Louis uses ib_get_dma_mr verb with remote access flag(s). Yes, the SRP initiator does. So does NFSoRDMA... - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
