The hardware fetches two SGEs anyway, since the QP was created with the
number of send s/g entries = 2.
It fetches both of the SGEs as part of the WR, but only requires a
single fetch for the data, versus two.
Sean, I wouldnt worry about that.
Hardware can (and does) pipeline these two
Quoting r. Sean Hefty [EMAIL PROTECTED]:
I did consider this, and I don't know when we'll start hitting issues
allocating
a single data buffer. But we're going to ask for 10,000 256-byte buffers -
over
2.5 MB of kernel memory in order to perform this single data transfer. Is it
likely
Hi Hal,
Currently, the OpenSM sends PortInfo with ClientReRegistration bit
turned on only during the first sweep after becoming Master.
This doesn't cover all cases where ClientReRegistration should be
turned on. OpenSM should turn on this bit also on new ports it
discovers (in cases of subnet
test, please ignore
--
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit
Hi,
This will check config dir existance and create it if neccessary. As
previous one but for libraries and diags.
Sasha.
This will check config dir existance and create it if neccessary.
Signed-off-by: Sasha Khapyorsky [EMAIL PROTECTED]
---
diags/autogen.sh |3 +++
Sean,
Can you see if/what is there a way for a CMA consumer to set the QP
timeout? Reading the .h files and the ib_cm/cma code the best i managed
to find is the following setting in cm_init_qp_rts_attr
qp_attr-timeout = cm_id_priv-local_ack_timeout;
is cm_id_priv-local_ack_timeout
On 2/8/06, Michael Di Domenico [EMAIL PROTECTED] wrote:
FYI...
Changed over to RHEL4 IA32 w/ SilverStorm Host Stack v3.2.0.0.21 and
now i get the below info and a working infiniband setup...
Since I have two servers, I'm going to leave this one working and try
openib on the second machine...
Title: core kernel changes for query SRQ
Add support to uverbs to handle querying userspace SRQs (Shared
Receive Queue), including adding an ABI for marshalling requests and
responses. The kernel midlayer already has ib_query_srq().
Signed-off-by: Dotan Barak [EMAIL PROTECTED]
Index:
Title: libibverbs + libmthca changes for query SRQ
libibverbs and libmthca changes to handle quering SRQs. Essentially just adding
API and support for passing the call through to provider plug-ins.
Signed-off-by: Dotan Barak [EMAIL PROTECTED]
Index:
Michael My problems if you recall with the mellanox cards seem to
Michael be related to Linux Kernel 2.6.15.3... Installed RHEL4
Michael instead of Fedora Core and get the same results using
Michael kernel 2.6.15.3 and RHEL4...
No, I think the problem is that something about your
Hi Yael,
On Mon, 2006-02-06 at 03:41, Yael Kalka wrote:
Hi Hal,
The Patch Michael Tsirkin suggested for fixing the OSM_LOG_ENTER
problem works fine both for windows and for linux.
Here is the patch for this, instead of the previous one I sent.
Thanks. Applied.
-- Hal
Thanks,
Yael
Hi Yael,
On Mon, 2006-02-06 at 03:53, Yael Kalka wrote:
Hi Hal,
The following patch adds the casting done in a clearer way - to avoid
compilation errors in windows. Also - added a clear message if the
timeout was trimmed (due to the casting).
Thanks,
Yael
Signed-off-by: Yael Kalka
Hi:
For a switch only one port_priv object is created in mad.c. As a result, it
appears like the process_mad function is always called with a port number of
zero.
And the return path is always filled with zero as well (in smi.c). Should
not this be the physical port number from which the mad
Suresh For a switch only one port_priv object is created in
Suresh mad.c. As a result, it appears like the process_mad
Suresh function is always called with a port number of zero. And
Suresh the return path is always filled with zero as well (in
Suresh smi.c). Should not this
On Mon, 2006-02-06 at 04:03, Yael Kalka wrote:
Hi Hal,
The following patch adds some changes in osm_db_file.c to match the
windows stack.
Thanks. Applied with some commentary difference and error number change.
-- Hal
Thanks,
Yael
Signed-off-by: Yael Kalka [EMAIL PROTECTED]
Index:
Hi all,
I have setup libsdp and it works quite well except if I try to send
buffer with a size 5100 bytes I get this kind of kernel messages :
Unable to handle kernel paging request for data at address
0xd80080085cc0
Faulting instruction address:
0xd01dd3b4
Can you send me the output of objdump -d on ib_mthca.ko? I'd like
to try to figure out where in mthca_tavor_map_phys_fmr() the oops is
happening, and so I'd like to see the compiled code.
There's probably no need to spam the list with a huge assembly
language listing.
Thanks,
Roland
On Mon, 2006-02-06 at 04:27, Yael Kalka wrote:
Hi Hal,
The following patch adds some missing casts and fixes object types to
fix compilation errors in the windows stack, aadds some changes in
osm_db_file.c to match the windows stack.
Thanks. Applied.
-- Hal
This reminds me of something I've been meaning to ask: what is the
attr_mask of the query QP function used for? The verbs definition in
the IB spec does not include an attribute mask. Obviously we don't
have to follow the spec precisely but I'm wondering what the advantage
is in having an
Hi Suresh,
On Thu, 2006-02-09 at 10:40, Suresh Shelvapille wrote:
Hi:
For a switch
Note that this is some new ground for this code.
only one port_priv object is created in mad.c.
A switch only has one port through which all MADs are processed (switch
port 0 (either base or enhanced)) so
On Thu, 2006-02-09 at 10:55, Roland Dreier wrote:
Suresh For a switch only one port_priv object is created in
Suresh mad.c. As a result, it appears like the process_mad
Suresh function is always called with a port number of zero. And
Suresh the return path is always filled with
Roland Dreier wrote:
My rule of thumb is that we shouldn't rely on being able to allocate a
contiguous buffer bigger than 4 KB, but assuming we can allocate 4 KB
is fine. 4 KB is the lowest page size of any real architecture, and
if the kernel is out of free pages then any allocation is likely
Or Gerlitz wrote:
Can you see if/what is there a way for a CMA consumer to set the QP
timeout? Reading the .h files and the ib_cm/cma code the best i managed
to find is the following setting in cm_init_qp_rts_attr
qp_attr-timeout = cm_id_priv-local_ack_timeout;
Local ACK timeout is
On Mon, 2006-02-06 at 05:00, Yael Kalka wrote:
Hi Hal,
There was an issue discussed a while ago regarding declaration of
several variables inside the function, in the code handling path
record for multicast. Declaration in the middle of the function
doesn't compile on windows, and in the
[EMAIL PROTECTED] wrote:
Roland Dreier wrote:
This reminds me of something I've been meaning to ask: what is the
attr_mask of the query QP function used for? The verbs definition in
the IB spec does not include an attribute mask. Obviously we don't
have to follow the spec precisely but I'm
On Thu, Feb 09, 2006 at 07:32:31AM -0800, Roland Dreier wrote:
Michael My problems if you recall with the mellanox cards seem to
Michael be related to Linux Kernel 2.6.15.3... Installed RHEL4
Michael instead of Fedora Core and get the same results using
Michael kernel 2.6.15.3
On Mon, 2006-02-06 at 05:05, Yael Kalka wrote:
Hi Hal,
The original static allocation doesn't compile in Windows.
The attached patch replaces it with dynamic allocation.
Thanks. Applied.
-- Hal
___
openib-general mailing list
But why define an IB specific feature when a transport neutral feature
can be defined?
Viewing the operation as Write with following Send maintains transport
neutral semantics AND allows IB to encode it as a Write with Immediate.
That avoids IB to use the silicon that already exists to support
Attached is a user-mode program, called rping, that uses librdmacm and
libibverbs to implement a ping-pong program over an RC connection. The
program utilizes SEND, RECV, RDMA READ, and WRITE ops, as well as cq
channels to get cq events, and rdma_get_event() to detect CMA events.
It is
[EMAIL PROTECTED] wrote:
Attached is a user-mode program, called rping, that uses librdmacm
and libibverbs to implement a ping-pong program over an RC
connection. The program utilizes SEND, RECV, RDMA READ, and WRITE
ops, as well as cq channels to get cq events, and rdma_get_event()
to
Sounds like the connection didn't get setup...
By default it should run continually ping/ponging until you hit ctrl-c.
Without -d or -v, you won't see any output. If it exited, then something
must have failed. -d should show more info. The -vV flags will display
the ping/pong messages and
On Mon, 2006-02-06 at 07:43, Yael Kalka wrote:
Hi Hal,
Currently SYSLOG prints are not executed under Windows.
The following patch adds these printings to the Windows stack as well.
Thanks. Applied.
-- Hal
___
openib-general mailing list
Sounds like the connection didn't get setup...
That was my assumption as well. Okay - I'll start debugging this. Nothing in
the code jumped out at me as incorrect. Have you tested this on IB, or only
iWarp?
By default it should run continually ping/ponging until you hit ctrl-c.
Without -d or
Roland Dreier wrote:
Hmm. Can you put a number on how much better RDMA write with
immediate is on current HCA hardware? How does using the underlying
OpenIB verbs ability to post a list of work requests compare (ie
posting an RDMA write followed by a send in one verbs call)?
Maybe post
Hi Yael,
On Mon, 2006-02-06 at 07:39, Yael Kalka wrote:
Hi Hal,
Currently in osm_vendor_mlx_sa.c the sent context is saved arbitrarily
as nodeInfo_context. This results in need for strange castings from
long to pointer and vice-versa. The following patch adds another
possible context -
On Thu, 2006-02-09 at 11:16 -0800, Sean Hefty wrote:
Sounds like the connection didn't get setup...
That was my assumption as well. Okay - I'll start debugging this. Nothing in
the code jumped out at me as incorrect. Have you tested this on IB, or only
iWarp?
I tested it over IB only,
[EMAIL PROTECTED] wrote:
Roland Dreier wrote:
Hmm. Can you put a number on how much better RDMA write with
immediate is on current HCA hardware? How does using the underlying
OpenIB verbs ability to post a list of work requests compare (ie
posting an RDMA write followed by a send in one
At 03:36 PM 2/8/2006, Arlin Davis wrote:
Roland Dreier wrote:
Michael So,
here we have a long discussion on attempting to
Michael perpetuate a concept that is not universal
across
Michael transports and was deemed to have minimal value
that most
Michael wanted to see removed from the
Hmm. Can you put a number on how much better RDMA write with
immediate is on current HCA hardware? How does using the underlying
OpenIB verbs ability to post a list of work requests compare (ie
posting an RDMA write followed by a send in one verbs call)?
Maybe post multiple is a better
Quoting r. Xavier Grave [EMAIL PROTECTED]:
Subject: libsdp running nearly fine
Hi all,
I have setup libsdp and it works quite well except if I try to send
buffer with a size 5100 bytes I get this kind of kernel messages :
Unable to handle kernel paging request for data at address
I'm looking to export a filesystem from each of four linux 64bit boxes to a
single Windows server 2003 64bit Ed.
Has anyone achieved this already using an IB transport? Can I use NFS over
IPoIB cross platform? i.e. do both ends support a solution?
Is NFS over RDMA compatible with Windows
Quoting r. Roland Dreier [EMAIL PROTECTED]:
Subject: Re: 2/2 libibverbs + libmthca changes for query QP
This reminds me of something I've been meaning to ask: what is the
attr_mask of the query QP function used for? The verbs definition in
the IB spec does not include an attribute mask.
Why both Immediate Data and the Stag which was used for RDMA Write?
Immediate data already contains info in response to what operation
the RDMA Write has completed locally.
Stag would make sence if Stag invalidation also put in the mix.
But for MPI RMR_context have a long lifecycle so not clear
Caitlin,
can you clarify this.
Are you proposing that Consumer encode a bit of Immediate Data to
specify that it is immediate data?
iWARP will pass it in Send message and IB in Immediate Data.
Arkady Kanevsky email: [EMAIL PROTECTED]
Network Appliance Inc.
[EMAIL PROTECTED] wrote:
Hmm. Can you put a number on how much better RDMA write with
immediate is on current HCA hardware? How does using the
underlying OpenIB verbs ability to post a list of work requests
compare (ie posting an RDMA write followed by a send in one verbs
call)? Maybe post
[EMAIL PROTECTED] wrote:
Caitlin,
can you clarify this.
Are you proposing that Consumer encode a bit of Immediate
Data to specify that it is immediate data?
iWARP will pass it in Send message and IB in Immediate Data.
If we agreed that there was some accute need for this 33rd
bit coming
Mike,
but then the combined operation can as easily be handle
by a "multiple post operation".
What is the need specific transport-independent RDMA
Write with immediate data.
I am still concern over the need of Consumer Recv side
to separate recv of Immediate Data
from "regular" Recv.
Roy,
and if tomorrow iWARP decides to support Immediate data with variable
length. API does not changes. Semantic does not changes and IB
will not be able to support it.
I am trying to define the semantic and API which will not have to be
modified for each rev of the transport.
Arkady Kanevsky
test, please ignore
--
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit
At 03:17 PM 2/9/2006, Paul Baxter wrote:
I'm looking to export a filesystem from each of four linux 64bit boxes to a
single Windows server 2003 64bit Ed.
Has anyone achieved this already using an IB transport? Can I use NFS over
IPoIB cross platform? i.e. do both ends support a solution?
Is
All we're asking is that a write/send combined API not be
called immediate data unless it fits the semantics of
immediate data. I am puzzled at the resistance this is
getting. There is a standards body specification for
immediate data. If it is not followed, don't call it
immediate data.
Larsen, Roy K wrote:
Even on iWARP transports small send data can be in-lined, avoiding
the need for buffers to be registered. A special API where the
length of the send buffer is known in advance makes this even
easier.
Ah, I wasn't aware iWARP could carry inline data. I take it
Roy,
and if tomorrow iWARP decides to support Immediate data with variable
length. API does not changes. Semantic does not changes and IB
will not be able to support it.
I am trying to define the semantic and API which will not have to be
modified for each rev of the transport.
Arkady,
Simply
Hi Paul,
I'm looking to export a filesystem from each of four linux
64bit boxes to a single Windows server 2003 64bit Ed.
Has anyone achieved this already using an IB transport? Can
I use NFS over IPoIB cross platform? i.e. do both ends
support a solution?
IPoIB will interoperate cross
On 2/9/06, Grant Grundler [EMAIL PROTECTED] wrote:
On Thu, Feb 09, 2006 at 07:32:31AM -0800, Roland Dreier wrote:
Michael My problems if you recall with the mellanox cards seem to
Michael be related to Linux Kernel 2.6.15.3... Installed RHEL4
Michael instead of Fedora Core and
Michael I'm still curious why the reset caused an issue and
Michael whether it is thought that this is a random occurance
Michael because of the hardware i have?
It is definitely something to do with the particular setup you have.
Hence my previous question: do you know the speed of
On 2/9/06, Roland Dreier [EMAIL PROTECTED] wrote:
Michael I'm still curious why the reset caused an issue and
Michael whether it is thought that this is a random occurance
Michael because of the hardware i have?
It is definitely something to do with the particular setup you have.
Hal:
I am only addressing the process_mad issue here:
As a result, it appears like the process_mad function is always called
with a port number of
zero.
That should be OK. The switch external port it was received on should be
in the ib_wc as follows:
struct ib_wc {
u64
Hi Suresh,
On Thu, 2006-02-09 at 17:22, Suresh Shelvapille wrote:
Hal:
I am only addressing the process_mad issue here:
As a result, it appears like the process_mad function is always called
with a port number of
zero.
That should be OK. The switch external port it was
Here is some help from my side. I'm not so familiar with NFS but I'll
try to help.
IPoIB works well between Linux and windows, so running some NFS client
should work well if both sides uses IPoIB.
If, and this is a big if, both the NFS client on windows and the Linux
sides are using (TCP)
Unfortunately the sysfs PMA counters support will not be that useful,
since it will only show counters for port 0; but that's a different issue.
- R.
[SS] That's a bummer, so don't we need to create as many file descriptors as
the number of physical ports on a switch, so that we can
Michael Krause wrote:
RDMA Write with Immediate is part of the IB Extended Transport
Header. It is a fixed-sized quantity and not one subject to change,
i.e. increasing its size.
Your argument above reinforces that the particular application need is
IB-specific and thus should not be part
Arlin,
This can be done.
But I have an issue that extension call violate Transport Requirement.
Currently, the matching semantic is well-defined since
Recv only matches Send. Since Spec does not have any idea what
operations are defined in extension(s) there is a problem
with the transport
Hal and folks:
For switches, the one from the WC needs to be filled in and passed so
that sounds wrong and needs fixing. Do you want to take a crack at this
or should I ?
[SS] I am not as IB core stack savvy as you guys are, so I would appreciate
it if you can provide the changes...I will
On Thu, 2006-02-09 at 17:56, Suresh Shelvapille wrote:
Unfortunately the sysfs PMA counters support will not be that useful,
since it will only show counters for port 0; but that's a different issue.
- R.
[SS] That's a bummer, so don't we need to create as many file descriptors as
Hi Suri,
On Thu, 2006-02-09 at 18:06, Suresh Shelvapille wrote:
Hal and folks:
For switches, the one from the WC needs to be filled in and passed so
that sounds wrong and needs fixing. Do you want to take a crack at this
or should I ?
[SS] I am not as IB core stack savvy as you
Quoting r. Sean Hefty [EMAIL PROTECTED]:
Subject: Re: FW: [PATCH 1 of 3] mad: large RMPP support
Roland Dreier wrote:
My rule of thumb is that we shouldn't rely on being able to allocate a
contiguous buffer bigger than 4 KB, but assuming we can allocate 4 KB
is fine. 4 KB is the lowest
Michael S. Tsirkin wrote:
Given that the last segment has a different size, it seems cleaner
to just keep the segment size part of ib_mad_segment structure.
The last segment should provide any necessary padding, so that the resulting MAD
is 256-bytes. Segments 2 through n should be the same
Sean, at least I am a bit confused at this point. Let's go back and
summarize the list of issues you see with the last patchset, OK?
As far as I can see, we decided that the list of segments is the right approach.
It also seems you are also inclining towards uniform handling of the first
segment
Michael S. Tsirkin wrote:
As far as I can see, we decided that the list of segments is the right approach.
Agreed.
It also seems you are also inclining towards uniform handling of the first
segment and the rest of them, so I hope this means the simplification
achieved by always using an s/g
Michael S. Tsirkin wrote:
The last segment should provide any necessary padding, so that the
resulting MAD is 256-bytes. Segments 2 through n should be the same size.
Uh, right. So segment size could just be a define.
To be clear, the segment size still varies by class, but would be
Sean The problem is that it results in walking the list every
Sean time the user wants to see the next segment. This is an
Sean O(n^2) operation. In any case I can think of, all that's
Sean needed is a single walk through the list to copy the
Sean userspace MAD data into the
Here's an updated version of rping. I restructured to code to make it
more modular, reduce the size of some of the functions, simplify some
areas, and make it more consistent. The updated version worked for my
limited testing. Please review the changes to see if I changed any of
the intended
Quoting r. Sean Hefty [EMAIL PROTECTED]:
Externally, a user should have access to the segment list
directly, rather than through a function call.
Why not a function call?
The latest idea that I was suggesting was to separate the MAD header from
the data segments for multi-segment/RMPP
Quoting r. Roland Dreier [EMAIL PROTECTED]:
If we want to keep the internals hidden, then we could expose some
sort of iterator object that clients can use to walk the list in O(n).
AFAIK thats what Jack is working on. Expect results by Monday.
--
Michael S. Tsirkin
Staff Engineer, Mellanox
On Thu, 2006-02-09 at 17:25 -0800, Hefty, Sean wrote:
Here's an updated version of rping. I restructured to code to make it
more modular, reduce the size of some of the functions, simplify some
areas, and make it more consistent. The updated version worked for my
limited testing. Please
Steve So what's up with these access flags on qp? You cannot set
Steve them on qp creation...only on qp modify. That seems
Steve strange. Since rdma read/writes work without these
Steve attributes set, I'm wondering what they really do?
I'm not sure how things interact with the
On Thu, 2006-02-09 at 18:42 -0800, Roland Dreier wrote:
Steve So what's up with these access flags on qp? You cannot set
Steve them on qp creation...only on qp modify. That seems
Steve strange. Since rdma read/writes work without these
Steve attributes set, I'm wondering
Fab:
As you point out, we've been focused on the main trunk and or target
test platforms are Linux based. That said, we actually had Beta versions
of NDIS and Winsock Direct drivers for the AMSO adapter, so we know this
works and we know where the dead are buried.
It probably makes sense to
On Mon, 2006-02-06 at 21:46, Sean Hubbell wrote:
-- Original message --
Date: 05 Feb 2006 11:44:52 -0500
From: Hal Rosenstock [EMAIL PROTECTED]
Reply-To: Hal Rosenstock [EMAIL PROTECTED]
To: Sean Hubbell [EMAIL PROTECTED]
Subject: Re: [openib-general] relocation error / link
If we want to keep the internals hidden, then we could expose some
sort of iterator object that clients can use to walk the list in O(n).
AFAIK thats what Jack is working on. Expect results by Monday.
Why not just use the list functionality that ships with the kernel? This is
what the
The latest idea that I was suggesting was to separate the MAD header from
the data segments for multi-segment/RMPP active MADs. Let *mad reference
the header, and rmpp_list reference all data segments. Non-RMPP MADs would
use *mad to reference the entire buffer. For an RMPP MAD (one with
Hi:
I want to compile openIB with 2.6.9-22EL kernel version. It is giving me
error with respect to iSER module.
After browsing through openib.org website, I came to know that iSER is
supported from 2.6.11 onwards but some work is going on for the patch
with 2.6.9-22 kernel version.
Just
I like what you did on making it more modular! I don't see any real
functional changes, but you removed the simple private data exchange.
Any particular reason?
I didn't see that the private data was being checked. I dropped it from one of
the function calls when I was merging some of the calls
84 matches
Mail list logo
