RE: Write Packets to InfiniBand HCA
That would depend on which HCA you are using. I know that you can use QLogic's iPath interface to do what you want, but I don't think it is possible through the verbs interface available through stock OFED. -Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Greg I Kerr Sent: Monday, January 02, 2012 9:50 PM To: linux-rdma@vger.kernel.org Subject: Write Packets to InfiniBand HCA Hi, Does anyone know if it is or isn't possible to put the HCA in some kind of raw mode where I can compose a packet in software and write it to the card? This is obviously in comparison to calling ibv_post_send and having a work request converted into a packet. Thanks, Greg Kerr -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [OMPI users] Problems when running open-MPI on OFED
On 29-Dec-11 7:10 PM, Venkateswara Rao Dokku wrote: Yes, Previously we could run those tests, but after adding some changes we see those errors,,... can you please suggest why those errors will occur? I would strongly suggest to have the usual IB tests running before even trying to run OMPI - these tests are going to be way more easy to debug. OMPI have lots of components and layers, and debugging driver with OMPI won't be pretty. Please address your questions to Linux RDMA mailing list:linux-rdma@vger.kernel.org You can find all the relevant experts there. -- YK On Thu, Dec 29, 2011 at 4:06 PM, Yevgeny Kliteynik klit...@dev.mellanox.co.il mailto:klit...@dev.mellanox.co.il wrote: Hi, Does OMPI with IMP work OK on the official OFED release? Do the usual ibv performance tests (ibv_rc_*) work on your customized OFED? -- YK On 29-Dec-11 9:34 AM, Venkateswara Rao Dokku wrote: Hi, We tried running the Intel Benchmarks(IMB_3.2) on the customized OFED(that was build by us).. While executing the tests we are getting the following errors 1. -- The OpenFabrics stack has reported a network error event. Open MPI will try to continue, but your job may end up failing. Local host:localhost.localdomain MPI process PID: 5259 Error number: 2 (IBV_EVENT_QP_REQ_ERR) This error may indicate connectivity problems within the fabric; please contact your system administrator. -- 2. While executing the Reduce_scatter benchmark.. [localhost.localdomain:04437] *** An error occurred in MPI_Reduce_scatter [localhost.localdomain:04437] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0 [localhost.localdomain:04437] *** MPI_ERR_TRUNCATE: message truncated [localhost.localdomain:04437] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) -- What is the root cause for those errors??, and any suggestions on how to remove those errors..!!! Thanks for your reply.. -- Thanks Regards, D.Venkateswara Rao, ___ users mailing list us...@open-mpi.org mailto:us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Thanks Regards, D.Venkateswara Rao, Software Engineer,One Convergence Devices Pvt Ltd., Jubille Hills,Hyderabad. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On 1/2/2012 7:34 PM, Hefty, Sean wrote: this field is a -- union -- how would that work if more than one extension is to be applied for a structure? The fields at the end of the structure should only be accessed if the structure is of the correct type. In this case, ext.xrc_recv is only available if the qp type is xrc recv. Maybe I wasn't clear enough, I was thinking on a case where one extends a structure and later on a 2nd extension is applied to the same structure, but the functionality / list of use cased related to the 1st extension isn't disjoint with that of the 2nd, e.g the second extends the first... maybe generally we can have a union named ext and on rare cases added ext2, etc? 2. indeed, reality wise, new features, much of the time will also interact with existing data structures... so what happens if we extend a structure but the the extended strucutre is actually a field within another existing structure e.g suppose we want to extend ibv_ah_attr which is within ibv_qp_attr e.g to be used for the RAW Ethernet QPs. I don't see how we can be backward compatible with apps linked against libibverbs with the internal structure size being smaller, correct? so extended fields must be on the END always - in the actual structure they are added and if this structure is a field of another structure then we can't just extend it and need to come up with new structure which is in turn used as the field? New features want to interact with existing structures and functions, which is what makes providing a clean separation difficult. We can extend the structures using the above method as long as we have some sort type field available. Where one is not available, we need to add one. See the proposed struct ibv_srq for an example. The extended SRQ type is only available by calling ibv_create_xsrq(), since ibv_create_srq() cannot know whether the user supports the extended ibv_srq_init_attr or not. Yes, I understand that, in 2nd thought, for the case of extending a structure which sits within another structure, e.g ibv_ah_attr within ibv_qp_attr, maybe we don't have much choice and rather add ib_ah_ext_attr and place it in the end of ibv_qp_attr --- and here's a case for double extensions... as we need the ext union of ibv_qp_attr to contain struct ibv_ah_ext_attr prim , struct ibv_ah_ext_attr alt and possibly also struct qp_attr_ext qp_ext... ibv_wc and ibv_send_wr are allocated by the caller, so those are more difficult to deal with. I agree that the size of those structures cannot change. It may be possible that some of the features you mentioned could be set as part of the qp attributes (ibv_modify_qp), and for the others, I'm not sure. Run time checks shouldn't be a big deal, since we already have to check things like ibv_wr_opcode and ibv_send_flags anyway. But it could be that we require a new function, similar to ibv_create_xsrq. I'd like to better understand the allocated by the caller ... are more difficult to deal with part of your response - for ibv_send_wr - if the caller have set a new IBV_WR_NEW_FEATURE value for the wr type, they surely aware to the new fields and actually the size of the structure can change as of structs allocated by the library. As for ibv_wc, yep, looks like we can't change the size unless we want to write a copatility layer that also comes into play in fast path calls, specifically ibv_poll_cq and translates from the new ibv_wc to the old ibv_wc structure. Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Send with immediate data completion
Hi all, I have a question about a completion for a send with immediate data. The IB spec (1.2.1) only mentions that the WC's immediate data be present at the receiver. It is silent on the value on the sender at completion. It does say that it is only valid if the WC's immediate data indicator is set. When I test using a 2.6.38 kernel with the kernel.org libibverbs git tree, I see a send completion's wc_flags set with IBV_WC_WITH_IMM yet the imm_data is not what I passed in. Since the spec is silent on setting imm_data on the sender, I assume that I should not rely on looking at the imm_data on a send completion. Given that, should IBV_WC_WITH_IMM ever be set on the sender? Thanks, Scott - Scott Atchley HPC Systems Engineer Center for Computational Sciences Oak Ridge National Laboratory atchle...@ornl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
I'd like to better understand the allocated by the caller ... are more difficult to deal with part of your response - for ibv_send_wr - if the caller have set a new IBV_WR_NEW_FEATURE value for the wr type, they surely aware to the new fields and actually the size of the structure can change as of structs allocated by the library. As for ibv_wc, yep, looks like we can't change the size unless we want to write a copatility layer that also comes into play in fast path calls, specifically ibv_poll_cq and translates from the new ibv_wc to the old ibv_wc structure. You're right. I was thinking more of ibv_wc, which has issues, since it may be used as an array. ibv_send_wr is probably okay, since we walk a list using pointers. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Write Packets to InfiniBand HCA
Yes I should have mentioned that I am using a Mellanox Connect-X adapter. Do you know where I can find documentation for QLogic's iPath? A quick google search didn't seem to turn anything up. Thanks for the information. - Greg Kerr On Tue, Jan 3, 2012 at 9:53 AM, Mike Heinz michael.he...@qlogic.com wrote: That would depend on which HCA you are using. I know that you can use QLogic's iPath interface to do what you want, but I don't think it is possible through the verbs interface available through stock OFED. -Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Greg I Kerr Sent: Monday, January 02, 2012 9:50 PM To: linux-rdma@vger.kernel.org Subject: Write Packets to InfiniBand HCA Hi, Does anyone know if it is or isn't possible to put the HCA in some kind of raw mode where I can compose a packet in software and write it to the card? This is obviously in comparison to calling ibv_post_send and having a work request converted into a packet. Thanks, Greg Kerr -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Send with immediate data completion
I have a question about a completion for a send with immediate data. The IB spec (1.2.1) only mentions that the WC's immediate data be present at the receiver. It is silent on the value on the sender at completion. It does say that it is only valid if the WC's immediate data indicator is set. Can you provide a section reference to the spec on the areas that you're looking at? Looking quickly, section 11.4.2.1 reads like immediate data should be available in either case. I've never checked imm data on the send wc. I'm just trying to determine if there's an issue in the spec that should be addressed, or if this is simply a bug in the hca/driver. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Send with immediate data completion
On Jan 3, 2012, at 11:55 AM, Hefty, Sean wrote: I have a question about a completion for a send with immediate data. The IB spec (1.2.1) only mentions that the WC's immediate data be present at the receiver. It is silent on the value on the sender at completion. It does say that it is only valid if the WC's immediate data indicator is set. Can you provide a section reference to the spec on the areas that you're looking at? Looking quickly, section 11.4.2.1 reads like immediate data should be available in either case. I've never checked imm data on the send wc. I'm just trying to determine if there's an issue in the spec that should be addressed, or if this is simply a bug in the hca/driver. For the record, I am using: hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.7.626 node_guid: 0002:c903:000b:64e8 sys_image_guid: 0002:c903:000b:64eb vendor_id: 0x02c9 vendor_part_id: 26428 hw_ver: 0xB0 board_id: MT_0D90110009 Scott-- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] please pull infiniband.git
On Mon, Dec 19, 2011 at 5:39 PM, Roland Dreier rol...@kernel.org wrote: Roland Dreier (2): IB/mlx4: Fix shutdown crash accessing a non-existent bitmap Hi Roland, As far as I can see this fix is upstream as commit 4af3ce0de0c12e5c17811eaefad36ab8e146c0fd but is not yet included in v3.1.7. Shouldn't this patch be sent to stable such that it gets included in v3.1.x too ? Thanks, Bart. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [GIT PULL] please pull infiniband.git
On Tue, Jan 3, 2012 at 11:18 AM, Bart Van Assche bvanass...@acm.org wrote: As far as I can see this fix is upstream as commit 4af3ce0de0c12e5c17811eaefad36ab8e146c0fd but is not yet included in v3.1.7. Shouldn't this patch be sent to stable such that it gets included in v3.1.x too ? The upstream commit has Cc: stable tagged so it should all happen automatically. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Write Packets to InfiniBand HCA
It is possible with ConnectX (cf MLX QPs in the kernel driver). However I don't know what documentation is available and some hacking would be needed to use this for something more general than sending MADs on special QPs. On Tue, Jan 3, 2012 at 8:43 AM, Greg I Kerr ker...@husky.neu.edu wrote: Yes I should have mentioned that I am using a Mellanox Connect-X adapter. Do you know where I can find documentation for QLogic's iPath? A quick google search didn't seem to turn anything up. Thanks for the information. - Greg Kerr On Tue, Jan 3, 2012 at 9:53 AM, Mike Heinz michael.he...@qlogic.com wrote: That would depend on which HCA you are using. I know that you can use QLogic's iPath interface to do what you want, but I don't think it is possible through the verbs interface available through stock OFED. -Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Greg I Kerr Sent: Monday, January 02, 2012 9:50 PM To: linux-rdma@vger.kernel.org Subject: Write Packets to InfiniBand HCA Hi, Does anyone know if it is or isn't possible to put the HCA in some kind of raw mode where I can compose a packet in software and write it to the card? This is obviously in comparison to calling ibv_post_send and having a work request converted into a packet. Thanks, Greg Kerr -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] ib/core: Protect QP mcast list
applied, thanks -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/4] ib/mlx4: fix 8021q vlan priority-bits used under IBoE
On Sun, Dec 11, 2011 at 4:38 PM, Or Gerlitz ogerl...@mellanox.com wrote: This series deals with fixes related to Ethernet 8021q vlan priority-bits used by mlx4 code under IBoE. It is made of one kernel/mlx4_ib patch and three user-space/libmlx4 patches. Hi Roland, Just a reminder... did you had the chance to look on that? Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] ib/cm: fix layout of APR message
applied, thanks -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
