RE: ib mad definitions
Right now there are 3 headers I find path record in. libibverbs: sa.h This isn't a MAD path record, this is the kernel version, which is unpacked. What we really needs is MAD 2 kernel and vice versa conversion in a library. I already have code that does this in several places :( This contains both. A wire structure was added to libibverbs to support librdmacm, which doesn't depend on umad. There are probably only a few of structures that make sense as part of ibverbs - path record and maybe multicast record. But, we'd have to get agreement that umad should depend on ibverbs, or we'll always have duplicate definitions. Jason, can you share a portion of the final output looks like for your codegen? Currently, I'd be happy just starting by having things like the mgmt classes, methods, attribute IDs, status values, common mad header, rmpp header, and sa header listed in a single place. Eventually, I need the CM MADs defined, which aren't available AFAICT. ib_types uses a lot of typedefs and #defines, so we may be able to use those to provide backwards compatability. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, 19 Oct 2010 20:28:18 -0700 Jason Gunthorpe jguntho...@obsidianresearch.com wrote: On Tue, Oct 19, 2010 at 06:12:56PM -0700, Ira Weiny wrote: I am all for cleaner, more capable... but why incompatible? If we want to start fresh and then convert OpenSM later, fine. But _don't_ forget to go back and convert OpenSM, because if you leave ib_types.h out there someone is going to use it and we are back to where we started... :-( Same for ibmad, when these definitions become available in umad, mad can be simplified. ib_types.h should not be installed in /usr/include, stop doing that and that risk goes away. That is one way to solve it, but only after those definitions (or equivalent) are available in /usr/include. ibmad can't really be changed, it is system library with a defined API. Maybe ibmad.2 or something, I don't know. I tried to use some of the PR APIs in it, and I've found them not useful :( ibmad.2 is the way to go. I have expressed my distaste for ibmad in the past and I agree, it is too established to be changed. But I have not had the time. Mainly I am trying to support Sean who has the time. Ira For instance we can't just abandon the mad_get_fields approach because we have real, usuable field access in umad, it has to stay. Right now there are 3 headers I find path record in. libibverbs: sa.h This isn't a MAD path record, this is the kernel version, which is unpacked. What we really needs is MAD 2 kernel and vice versa conversion in a library. I already have code that does this in several places :( libibmad: mad.h You mean mad_get_fields IB_SA_PR_DGID_F, etc? It doesn't even have all the fields :( opensm: ib_types.h Yep. Node type is defined in: libibverbs: verbs.h opensm: ib_types.h libibmad: mad.h I could go on. Keep in mind that for the most part libibmad is someones attempt to make a set of accessors and structures for mads. It is incomplete. It is largely unusable. I certainly haven't been able to use its PR structure parsing functions for any real app. Was it just pulled out of opensm? I don't know, I'd just as soon see that part of it be discarded, and a complete set of structures added to umad. opensm has unique problems because they want to remain independent of the OFA stack, I don't think they have a choice but to duplicate. Jason -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Wed, Oct 20, 2010 at 09:06:54AM -0700, Hefty, Sean wrote:
This contains both. A wire structure was added to libibverbs to
support librdmacm, which doesn't depend on umad. There are probably
only a few of structures that make sense as part of ibverbs - path
record and maybe multicast record. But, we'd have to get agreement
that umad should depend on ibverbs, or we'll always have duplicate
definitions.
Duplication doesn't concern me too much, as long as the structures are
cast compatible it doesn't matter too much, there is nothing in verbs
that uses the type, for instance?
I think umad should depend on ibverbs (and use ibv_context to open the
device, but that is another subject), and I think the *structures*
should not have a ABI dependency on umad - static inlines and
structure definitions. So you can compile against it without a link
time dependency. This would let rdmacm use the structures, just have
to compile libumad first.
Jason, can you share a portion of the final output looks like for your
codegen?
Well, my codegen is tailored to what I've been doing. I would not use
this approach to target C code, you need to use macros instead of
templates, etc, etc. But that doesn't matter too much, you can codegen
whatever you want, be it mad_get_field, or pack/unpack or the
non-bitfield approach in ib_types.h.
I have something in mind for C that is a combination of bitfields and
the 1-byte=1-bit structure map technique. I'd codegen both host-endian
overlays and network endian overlays. There is good reason for both, IMHO.
[This email is so long already, I can outline my thoughts on this
seperately if you are interested]
What I've done has great properties, it is very difficult to make an
endian error using it, and it has zero overhead on BE machines.
/* [15.2.5.16] Information on paths through the subnet */
struct SAPathRecord {
enum { attributeID = 0x35 };
uint32_t rsv0;
uint32_t rsv1;
struct HdrIPv6Addr DGID;
struct HdrIPv6Addr SGID;
uint16_t DLID;
uint16_t SLID;
uint32_t rawTraffic:1;
uint32_t rsv2:3;
uint32_t flowLabel:20;
uint32_t hopLimit:8;
uint8_t TClass;
uint8_t reversible:1;
uint8_t numbPath:7;
uint16_t PKey;
uint16_t rsv3:12;
uint16_t SL:4;
uint8_t MTUSelector:2;
uint8_t MTU:6;
uint8_t rateSelector:2;
uint8_t rate:6;
uint8_t packetLifeTimeSelector:2;
uint8_t packetLifeTime:6;
uint8_t preference;
uint16_t rsv4;
uint32_t rsv5;
};
This is a host-endian bitfield overlay. Two are output, this is the
host == BE version, the host == LE version is re-ordered:
uint32_t hopLimit:8;
uint32_t flowLabel:20;
uint32_t rsv2:3;
uint32_t rawTraffic:1;
etc. The neat thing about the codegen is that is analyzes the
arrangement to choose an optimal basic type. Ie TClass is not a
bitfield, but hopLimit is because flowLabel spans a 16 bit quantity.
This helps avoid any bit slicing code overheads by letting the
compiler produce short and byte loads whenever possible.
Use is sort of like:
uint32_t madBuf[256/4];
for (int I = 0; I != 64; I++)
madBuf[I] = ntohl(madBuf[I]);
SAPathRecord *pr = madBuf;
pr.DLID = 1;
pr.SLID = 2;
pr.TClss = 3;
for (int I = 0; I != 64; I++)
madBuf[I] = htonl(madBuf[I]);
This avoids all ntoh/hton on types = 32, and completely avoids all
overhead on BE machines (which is important for me). In most of my
code I stick the swap in the packet RX/TX path.
The down side is that byte arrays, 64 bits and GIDs get messed up, but
those are compartively rare. With the network endian overlays those 3
things are OK but lot of stuff like flowlabel is hoplessly messed up
(need a ntoh20bits() function). I think both arrangements are useful..
The codgen uses C++ templates and anonymous unions to fix up some of
the ugly stuff in the spec, like the byte arrays and the different
labels for the MADHeaders in some cases.. These are special cases in
the script.
/* [14.2.1.2] SMP Format - Direct Routed */
struct SMPFormatDirected {
enum { mgmtClass = 0x81 };
union {
struct MADHeader MADHeader;
struct {
uint32_t MADHeader1;
uint16_t D:1;
uint16_t status:15;
uint8_t hopPointer;
uint8_t hopCount;
uint32_t MADHeader2[4];
};
};
struct IBA64bit MKey;
uint16_t drSLID;
uint16_t drDLID;
uint32_t rsv0[7];
uint32_t data[16];
IBABitFieldArray8,64 initialPath;
IBABitFieldArray8,64 returnPath;
};
Or the 3 bit wide arrays in PMPortSamplesCtl (so gross):
/* [16.1.3.2] Port Performance Data Sampling Control */
struct PMPortSamplesCtl {
enum { attributeID = 0x10 };
uint8_t opCode;
uint8_t portSelect;
uint8_t tick;
uint8_t rsv0:5;
uint8_t counterWidth:3;
union {
IBABitFieldArray3,15,CounterMaskTraits counterMask;
struct {
uint32_t rsv1:2;
uint32_t counterMask0:3;
RE: ib mad definitions
Works for me. -Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Hefty, Sean Sent: Monday, October 18, 2010 6:25 PM To: linux-rdma@vger.kernel.org; Sasha Khapyorsky Subject: ib mad definitions This has probably been discussed before, but is there a strong reason why ib_types.h can't be moved from opensm/include/iba to libibumad/include/infiniband? This appears to be the only place where IB MAD definitions are available for user space applications, and having them available at the libibumad level makes sense to me. (I'm trying to port madeye to user space as a diag, and want all IB MAD definitions.) - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Mon, Oct 18, 2010 at 6:24 PM, Hefty, Sean sean.he...@intel.com wrote: This has probably been discussed before, Yes, several times AFAIR. but is there a strong reason why ib_types.h can't be moved from opensm/include/iba to libibumad/include/infiniband? Why does this need to be moved ? This appears to be the only place where IB MAD definitions are available for user space applications, and having them available at the libibumad level makes sense to me. (I'm trying to port madeye to user space as a diag, and want all IB MAD definitions.) There already are diags including ib_types.h (saquery for one). -- Hal - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ib mad definitions
but is there a strong reason why ib_types.h can't be moved from
opensm/include/iba to libibumad/include/infiniband?
Why does this need to be moved ?
The dependency should be on libibumad, not opensm. libibumad is pretty much
useless without these definitions. Why wouldn't you move them?
There already are diags including ib_types.h (saquery for one).
Yes, but we're either stuck with everything that needs ib_types.h to be part of
the management.git tree, or the app needs to depend on opensm. Currently,
ibacm duplicates definitions because they aren't available anywhere else.
N�r��yb�X��ǧv�^�){.n�+{��ٚ�{ay� ʇڙ�,j��f���h���z� �w���
���j:+v���w�j�mzZ+�ݢj��!�i
Re: ib mad definitions
On Tue, Oct 19, 2010 at 11:28 AM, Hefty, Sean sean.he...@intel.com wrote: but is there a strong reason why ib_types.h can't be moved from opensm/include/iba to libibumad/include/infiniband? Why does this need to be moved ? The dependency should be on libibumad, not opensm. libibumad is pretty much useless without these definitions. Why wouldn't you move them? Off the top of my head, OpenSM is layered on top of libibumad but doesn't need/use libibmad. I think that was the main reason although that could be changed if ib_types.h were to be moved. I'm not sure what other reasons came up in the previous discussions. There already are diags including ib_types.h (saquery for one). Yes, but we're either stuck with everything that needs ib_types.h to be part of the management.git tree, or the app needs to depend on opensm. Currently, ibacm duplicates definitions because they aren't available anywhere else. I agree ib_types.h is more generic than opensm. Moving to libibmad and making opensm depend on this is probably better than all the duplication. There have been viewpoints that libibumad and libibmad shouldn't be separate (as they are small) but they were never combined into a single library. -- Hal -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ib mad definitions
I agree ib_types.h is more generic than opensm. Moving to libibmad and making opensm depend on this is probably better than all the duplication. There have been viewpoints that libibumad and libibmad shouldn't be separate (as they are small) but they were never combined into a single library. My motivation with these changes is for ibacm to receive and use notification of CM timeouts to update its path record cache. ibacm already defines the basic mad structure, multicast record, and path record. It would also need the CM mad format. I'd happily remove these definitions if they were already available. Porting madeye to user space is a side benefit to the proposed kernel changes. ibacm only depends on libibumad. The madeye port also only depends on libibumad. Honestly, I find the libibmad APIs confusing. I'd much rather libibumad provide mad definitions. Sasha/Ira, do either of you have opinions on this?
Re: ib mad definitions
On Tue, Oct 19, 2010 at 12:48 PM, Hefty, Sean sean.he...@intel.com wrote: I agree ib_types.h is more generic than opensm. Moving to libibmad and making opensm depend on this is probably better than all the duplication. There have been viewpoints that libibumad and libibmad shouldn't be separate (as they are small) but they were never combined into a single library. The other thing I just recalled was the OpenSM portability issue. ib_types.h is needed here and libibmad/libibumad is not in all those environments. As you''re all too well aware, this was even the case in Windows until very recently. There may still be others we care about where moving ib_types.h might be problematic. -- Hal My motivation with these changes is for ibacm to receive and use notification of CM timeouts to update its path record cache. ibacm already defines the basic mad structure, multicast record, and path record. It would also need the CM mad format. I'd happily remove these definitions if they were already available. Porting madeye to user space is a side benefit to the proposed kernel changes. ibacm only depends on libibumad. The madeye port also only depends on libibumad. Honestly, I find the libibmad APIs confusing. I'd much rather libibumad provide mad definitions. Sasha/Ira, do either of you have opinions on this? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, 19 Oct 2010 08:43:22 -0700 Hal Rosenstock hal.rosenst...@gmail.com wrote: On Tue, Oct 19, 2010 at 11:28 AM, Hefty, Sean sean.he...@intel.com wrote: but is there a strong reason why ib_types.h can't be moved from opensm/include/iba to libibumad/include/infiniband? Why does this need to be moved ? The dependency should be on libibumad, not opensm. libibumad is pretty much useless without these definitions. Why wouldn't you move them? Off the top of my head, OpenSM is layered on top of libibumad but doesn't need/use libibmad. I think that was the main reason although that could be changed if ib_types.h were to be moved. I'm not sure what other reasons came up in the previous discussions. I think ib_types.h should be part of ibumad. Everything depends on libibumad at some point.[*] Therefore common mad definitions should be in ib_types.h and packaged with libibumad. [*] ok OpenSM does not strictly, see below. There already are diags including ib_types.h (saquery for one). Yes, but we're either stuck with everything that needs ib_types.h to be part of the management.git tree, or the app needs to depend on opensm. Currently, ibacm duplicates definitions because they aren't available anywhere else. I agree ib_types.h is more generic than opensm. Moving to libibmad and making opensm depend on this is probably better than all the duplication. There have been viewpoints that libibumad and libibmad shouldn't be separate (as they are small) but they were never combined into a single library. The opposing view is that libibumad is only an interface to the kernel umad module, where libibmad is more abstract. As far as moving ib_types, I suggested this a while back. http://www.mail-archive.com/gene...@lists.openfabrics.org/msg27439.html Let's see if I can summarize the thread. - Sean was workiong on libibacm and redefined ib_types.h definitions. - I suggested moving ib_types.h to umad so he would not have a dependancy on OpenSM. - Sean brought up that ib_types.h is large and probably should be split - I agreed, and asked Sasha if such a patch would be acceptable, or create a new library to deal with the inline functions in ib_types.h - Hal said that ibutils requires ib_types.h but does not want a dependancy on libibumad... - I suggested a separate library to solve this problem. - Hal corrected himself saying that ibutils requires osm_vendor_ibumad. However, OpenSM does not always use libibumad (depending on the underlying stack) so it would need to get ib_types somewhere else. Hal was also concerned about a library with little more than a header file in it. - Jason chimed in with Please no more libraries... :-) (and digressed with Sean in to PR queries, MPI, and other useful, but unrelated, stuff) - Sean says libibumad is pretty useless without some network structure definitions. - I state that it looks like ibutils dependancy is on the static functions in ib_types.h only. - Hal says yes ibutils depends on OpenSM for the vendor layer and that Mellanox is better able to answer questions regarding ibutils support. - Hal says he thinks ib_types is more akin to what is in libibmad rather than libibumad - Sean finds that ib_types.h includes complib headers. - I submit a rough hack to remove complib headers. - Jason, Sean, and myself discuss ugly byteswapping functions. - Sasha agrees that he is not sure that umad is the right place for ib_types - Sean says we should split the file up and at least some of the definitions should be in umad... We all get busy... I think we need to move ib_types (mad definitions to umad). Basic MAD definitions should be provided at the lowest possible level so all software can use them. The issues (solutions) are: ib_types depends on complib at the moment (fixable) ibutils depends on OpenSM (it will anyway -- non-issue) somethings in ib_types are ugly, byteswapping (non-issue; deal with it later) OpenSM may _not_ include umad and therefore miss defines. (fixable?) As for this last item, would it be a big deal to require umad for the header only? Does umad not compile somewhere that other vendor layers are used? I think it is much better for OpenSM to require umad than for other MAD processing software to require OpenSM. Also, would splitting ib_types help this at all? Ira -- Hal -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://BLOCKEDvger.kernel.org/majordomo-info.html -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ib mad definitions
ib_types depends on complib at the moment (fixable) ibutils depends on OpenSM (it will anyway -- non-issue) somethings in ib_types are ugly, byteswapping (non-issue; deal with it later) OpenSM may _not_ include umad and therefore miss defines. (fixable?) As for this last item, would it be a big deal to require umad for the header only? Does umad not compile somewhere that other vendor layers are used? I think it is much better for OpenSM to require umad than for other MAD processing software to require OpenSM. Also, would splitting ib_types help this at all? I'll propose the following: 1. Add to libibumad/include/infiniband: umad_types.h - basic mad, rmpp headers umad_sa.h- SA attributes umad_cm.h- CM messages 2. Include umad_types.h and umad_sa.h from ib_types.h 3. Include umad_cm.h from ib_cm_types.h We start with a minimal set of definitions to umad and add/move other definitions later as needed, creating new header files where appropriate (umad_smi.h, umad_pm.h, etc.) If we can get some basic agreement on this, I'll start on the patches immediately. In an ideal world, the new header files would work on any platform. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, 19 Oct 2010 11:50:46 -0700 Hefty, Sean sean.he...@intel.com wrote: ib_types depends on complib at the moment (fixable) ibutils depends on OpenSM (it will anyway -- non-issue) somethings in ib_types are ugly, byteswapping (non-issue; deal with it later) OpenSM may _not_ include umad and therefore miss defines. (fixable?) As for this last item, would it be a big deal to require umad for the header only? Does umad not compile somewhere that other vendor layers are used? I think it is much better for OpenSM to require umad than for other MAD processing software to require OpenSM. Also, would splitting ib_types help this at all? I'll propose the following: 1. Add to libibumad/include/infiniband: umad_types.h - basic mad, rmpp headers umad_sa.h- SA attributes umad_cm.h- CM messages 2. Include umad_types.h and umad_sa.h from ib_types.h 3. Include umad_cm.h from ib_cm_types.h We start with a minimal set of definitions to umad and add/move other definitions later as needed, creating new header files where appropriate (umad_smi.h, umad_pm.h, etc.) If we can get some basic agreement on this, I'll start on the patches immediately. In an ideal world, the new header files would work on any platform. I agree, Ira - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://BLOCKEDvger.kernel.org/majordomo-info.html -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, Oct 19, 2010 at 11:50:46AM -0700, Hefty, Sean wrote:
We start with a minimal set of definitions to umad and add/move
other definitions later as needed, creating new header files where
appropriate (umad_smi.h, umad_pm.h, etc.)
If we can get some basic agreement on this, I'll start on the
patches immediately. In an ideal world, the new header files would
work on any platform.
Can we at least agree on the usage of these structures first? Are the
constants going to be in host or network byte order?
Are you going to make something like the kernel where there is a
native structure and pack/unpack function set?
Something macro-based like foo = GET_MEMBER(*pr,preference)
Network byte order casting structures?
Host byte order casting structures? (my favorite)
bitfields?
For years now I've had a set of data files that describe all the IB
structures bitfield layouts. I think I can contribute the data files
but not the generator script.
Since they all have various merits, maybe the smartest thing is to just
codegen all of the above permutations from single data source?
ie
// network endian bitfield casting structure
struct MADHeader_NE x = {};
x.status = htons(1);
// host endian bitfield casting structure
struct MADHeader_HE x = {};
x.status = 1
to_network(x,sizeof(x)); // x[i] = htonl(x[i]) for i in len/4
/* Non-bitfield macro access structure
(using the 1 byte = 1 bit helper structure technique) */
struct MADHeader_M x = {}
SET_MEMBER(x,status,1);
// Pack/unpack function structure
struct MADHeader_UP x = {};
x.status = htons(1);
pack_MADHeader(x,mad_buf,sizeof(mad_buf));
I'd like to think we don't need the last one, but people seem to like
that scheme ..
I also like to codegen structure printing functions, that is
surprisingly useful - and implements a good chunk of madeye.
What do you think?
I've also very recently been thinking that I'd like python bindings
for MADs for some projects. I was planning on building it out with the
code gen scheme.
Ira, I think the cleanest answer is that OSM keeps its type file, and
umad gets a new one that is cleaner, more capable and probably
incompatible. I'd hate to see us stick to the OSM scheme for umad just
for code compatability.
Jason
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ib mad definitions
Ira Weiny wrote: On Tue, 19 Oct 2010 11:50:46 -0700 Hefty, Sean sean.he...@intel.com wrote: ib_types depends on complib at the moment (fixable) ibutils depends on OpenSM (it will anyway -- non-issue) somethings in ib_types are ugly, byteswapping (non-issue; deal with it later) OpenSM may _not_ include umad and therefore miss defines. (fixable?) As for this last item, would it be a big deal to require umad for the header only? Does umad not compile somewhere that other vendor layers are used? I think it is much better for OpenSM to require umad than for other MAD processing software to require OpenSM. Also, would splitting ib_types help this at all? I'll propose the following: 1. Add to libibumad/include/infiniband: umad_types.h - basic mad, rmpp headers umad_sa.h- SA attributes umad_cm.h- CM messages 2. Include umad_types.h and umad_sa.h from ib_types.h 3. Include umad_cm.h from ib_cm_types.h We start with a minimal set of definitions to umad and add/move other definitions later as needed, creating new header files where appropriate (umad_smi.h, umad_pm.h, etc.) If we can get some basic agreement on this, I'll start on the patches immediately. In an ideal world, the new header files would work on any platform. I agree, Ira Just to be painfully clear ... A user-mode application would then only need to include ib_types.h + CM flavor of choice .h files ? - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://BLOCKEDvger.kernel.org/majordomo-info.html -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ib mad definitions
Just to be painfully clear ... A user-mode application would then only need to include ib_types.h + CM flavor of choice .h files ? For compatibility, ib_types.h would include whatever files any definitions were moved to. An application that includes ib_types.h today wouldn't need additional includes. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ib mad definitions
Can we at least agree on the usage of these structures first? Are the constants going to be in host or network byte order? I was simply suggesting to 'move' some of the existing structures and defines. Are you going to make something like the kernel where there is a native structure and pack/unpack function set? This would not be my preference. Something macro-based like foo = GET_MEMBER(*pr,preference) Network byte order casting structures? Host byte order casting structures? (my favorite) bitfields? again - not my preference Ira, I think the cleanest answer is that OSM keeps its type file, and umad gets a new one that is cleaner, more capable and probably incompatible. I'd hate to see us stick to the OSM scheme for umad just for code compatability. Whatever is done must fit within the windows development framework that we use. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, Oct 19, 2010 at 06:00:51PM -0700, Hefty, Sean wrote: Can we at least agree on the usage of these structures first? Are the constants going to be in host or network byte order? I was simply suggesting to 'move' some of the existing structures and defines. But they are horrible and little used outside opensm right now, you really want to commit to that forever? Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, 19 Oct 2010 18:00:51 -0700 Hefty, Sean sean.he...@intel.com wrote: Can we at least agree on the usage of these structures first? Are the constants going to be in host or network byte order? I was simply suggesting to 'move' some of the existing structures and defines. Are you going to make something like the kernel where there is a native structure and pack/unpack function set? This would not be my preference. Something macro-based like foo = GET_MEMBER(*pr,preference) Network byte order casting structures? Host byte order casting structures? (my favorite) bitfields? again - not my preference Ira, I think the cleanest answer is that OSM keeps its type file, and umad gets a new one that is cleaner, more capable and probably incompatible. I'd hate to see us stick to the OSM scheme for umad just for code compatability. Whatever is done must fit within the windows development framework that we use. I am all for cleaner, more capable... but why incompatible? If we want to start fresh and then convert OpenSM later, fine. But _don't_ forget to go back and convert OpenSM, because if you leave ib_types.h out there someone is going to use it and we are back to where we started... :-( Same for ibmad, when these definitions become available in umad, mad can be simplified. What I would like right now is to get the definitions in 1 place! Right now there are 3 headers I find path record in. libibverbs: sa.h libibmad: mad.h opensm: ib_types.h Node type is defined in: libibverbs: verbs.h opensm: ib_types.h libibmad: mad.h I could go on. What Sean is offering to do is move ib_types to umad. From there I can use those definitions in mad (thus removing them from mad and consolidating at least 2 of the 3 above). Perhaps use them in ibverbs as well? As a first step I think we should take Sean up on his offer to start cleaning things up. But we have to remove stuff as we go or we will just be defining yet another place to look for these. After this we can look at making things cleaner (perhaps even combining mad and umad, and including some of the ideas you have above). As Sean said in another email, after this change; including ib_types.h will be the same for anyone using it. The exception is that we have simplified the code. I think this is a win-win with minimal work. Ira -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, 19 Oct 2010 18:09:58 -0700 Jason Gunthorpe jguntho...@obsidianresearch.com wrote: On Tue, Oct 19, 2010 at 06:00:51PM -0700, Hefty, Sean wrote: Can we at least agree on the usage of these structures first? Are the constants going to be in host or network byte order? I was simply suggesting to 'move' some of the existing structures and defines. But they are horrible and little used outside opensm right now, you really want to commit to that forever? Not everything is horrible. And if it is we can fix it. But I think defining yet another header with the same functionality is worse. Like it or not ib_types is there. If you don't remove/fix it, someone will find it and use it. How does that make things cleaner just because there is something clean somewhere else? Someone will find ib_types use it. I still feel this is the best first step at getting rid of ib_types.h (at least as it currently stands). Ira Jason -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, Oct 19, 2010 at 06:32:57PM -0700, Ira Weiny wrote: On Tue, 19 Oct 2010 18:09:58 -0700 Jason Gunthorpe jguntho...@obsidianresearch.com wrote: On Tue, Oct 19, 2010 at 06:00:51PM -0700, Hefty, Sean wrote: Can we at least agree on the usage of these structures first? Are the constants going to be in host or network byte order? I was simply suggesting to 'move' some of the existing structures and defines. But they are horrible and little used outside opensm right now, you really want to commit to that forever? Not everything is horrible. And if it is we can fix it. But I think defining yet another header with the same functionality is worse. Like it or libibumad is a system library. It needs to have a stable ABI, low churn and ideally be 'complete'. My database of IB structs has 117 structures, all with wakky alignment and all manner of strangeness. IMHO, it is infeasible to keep with the ad hoc approach in ibtypes.h and generate a complete header set without a lot of churn. This is why it is horrible. There are things worse than 'yet another' header - for instance a system library being churned again and again for cleanups. Figure out what you want, do it once, do it right, be done. If we could all agree what these structs should look like I can provide my database and someone can write the codegen AND WE CAN BE DONE FOREVER. How is this not much better?? Don't treat the API of a system library as some casual thing. :( Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ib mad definitions
On Tue, Oct 19, 2010 at 06:12:56PM -0700, Ira Weiny wrote: I am all for cleaner, more capable... but why incompatible? If we want to start fresh and then convert OpenSM later, fine. But _don't_ forget to go back and convert OpenSM, because if you leave ib_types.h out there someone is going to use it and we are back to where we started... :-( Same for ibmad, when these definitions become available in umad, mad can be simplified. ib_types.h should not be installed in /usr/include, stop doing that and that risk goes away. ibmad can't really be changed, it is system library with a defined API. Maybe ibmad.2 or something, I don't know. I tried to use some of the PR APIs in it, and I've found them not useful :( For instance we can't just abandon the mad_get_fields approach because we have real, usuable field access in umad, it has to stay. Right now there are 3 headers I find path record in. libibverbs: sa.h This isn't a MAD path record, this is the kernel version, which is unpacked. What we really needs is MAD 2 kernel and vice versa conversion in a library. I already have code that does this in several places :( libibmad: mad.h You mean mad_get_fields IB_SA_PR_DGID_F, etc? It doesn't even have all the fields :( opensm: ib_types.h Yep. Node type is defined in: libibverbs: verbs.h opensm: ib_types.h libibmad: mad.h I could go on. Keep in mind that for the most part libibmad is someones attempt to make a set of accessors and structures for mads. It is incomplete. It is largely unusable. I certainly haven't been able to use its PR structure parsing functions for any real app. Was it just pulled out of opensm? I don't know, I'd just as soon see that part of it be discarded, and a complete set of structures added to umad. opensm has unique problems because they want to remain independent of the OFA stack, I don't think they have a choice but to duplicate. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
