Re: [OMPI devel] Compile-time MPI_Datatype checking
Ralph, Indeed, some of us are using clang (and other llvm front ends) for JIT on our hetero HPC clusters for amorphous problem spaces. Obviously, I don't work for a National Lab. But I do mod/sim/vis for quantum, nano, and meso-physics. Just wanted you to be aware. Ken From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] On Behalf Of Paul Hargrove Sent: Wednesday, October 31, 2012 12:48 PM To: Open MPI Developers Subject: Re: [OMPI devel] Compile-time MPI_Datatype checking Note that with Apple's latest versions of Xcode (4.2 and higher, IIRC) Clang is now the default C compiler. I am told that Clang is the ONLY bundled compiler for OSX 10.8 (Mountain Lion) unless you take extra steps to install gcc (which is actually llvm-gcc and cross-compiles for OSX 10.7). So, Clang *is* gaining some "market share", though not yet in major HPC systems. -Paul On Wed, Oct 31, 2012 at 11:40 AM, Ralph Castain <r...@open-mpi.org> wrote: If it's only on for Clang, I very much doubt anyone will care - I'm unaware of any of our users that currently utilize that compiler, and certainly not on the clusters in the national labs (gcc, Intel, etc. - but I've never seen them use Clang). Not saying anything negative about Clang - just noting it isn't much used in our current community that I've heard. On Oct 31, 2012, at 11:19 AM, Dmitri Gribenko <griboz...@gmail.com> wrote: > On Wed, Oct 31, 2012 at 5:04 PM, Jeff Squyres <jsquy...@cisco.com> wrote: >> On Oct 31, 2012, at 9:38 AM, Dmitri Gribenko wrote: >> >>>> The rationale here is that correct MPI applications should not need to add any extra compiler files to compile without warnings. >>> >>> I would disagree with this. Compiler warnings are most useful when >>> they are on by default. Only a few developers will turn on a warning >>> because warnings are hard to discover and enabling a warning requires >>> an explicit action from the developer. >> >> Understood, but: >> >> a) MPI explicitly allows this kind of deliberate mismatch. It does not make sense to warn for things that are correct in MPI. > > I don't think it is MPI. It is the C standard that allows one to > store any pointer in void* and char*. But C standard also considers > lots of other weird things to be valid, see below. > >> b) Warnings are significantly less useful if the user looks at them and says, "the compiler is wrong; I know that MPI says that this deliberate mismatch in my code is ok." > > Well, one can also argue that since the following is valid C, the > warning in question should not be implemented at all: > > long *b = malloc(sizeof(int)); > MPI_Recv(b, 1, MPI_INT, ...); > > But this code is clearly badly written, so we are left with a question > about where to draw the line. > >> c) as such, these warnings are really only useful for the application where type/MPI_Datatype matching is expected/desired. > > Compilers already warn about valid C code. Silencing many warnings > relies on conventions that are derived from best practices of being > explicit about something unusual. For example: > > $ cat /tmp/aaa.c > void foo(void *a) { > for(int i = a; i < 10; i++) > { >if(i = 5) > return; > } > } > $ clang -fsyntax-only -std=c99 /tmp/aaa.c > /tmp/aaa.c:2:11: warning: incompatible pointer to integer conversion > initializing 'int' with an expression of type 'void *' > [-Wint-conversion] > for(int i = a; i < 10; i++) > ^ ~ > /tmp/aaa.c:4:10: warning: using the result of an assignment as a > condition without parentheses [-Wparentheses] >if(i = 5) > ~~^~~ > /tmp/aaa.c:4:10: note: place parentheses around the assignment to > silence this warning >if(i = 5) > ^ > () > /tmp/aaa.c:4:10: note: use '==' to turn this assignment into an > equality comparison >if(i = 5) > ^ > == > 2 warnings generated. > > According to C standard this is valid C code, but clang emits two > warnings on this. > >> Can these warnings be enabled as part of the warnings rollup -Wall option? That would be an easy way to find/enable these warnings. > > IIRC, -Wall warning set is frozen in clang. -Wall is misleading in > that it does not turn on all warnings implemented in the compiler. > Clang has -Weverything to really turn on all warnings. But > -Weverything is very noisy (by design, not to be fixed) unless one > also turns off all warnings that are not interesting for the project > with -Wno-foo. > > I don't think it is possible to disable this warning by default > because off-by-default warnings are discouraged in Clang. There is no > formal policy, but the ru
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Wed, Oct 31, 2012 at 9:04 PM, Ralph Castainwrote: > Understood, but also remember that the national labs don't have Mac clusters > - and so they couldn't care less about Clang. Clang is also the new system compiler for FreeBSD. But there are not many FreeBSD clusters either. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
Note that with Apple's latest versions of Xcode (4.2 and higher, IIRC) Clang is now the default C compiler. I am told that Clang is the ONLY bundled compiler for OSX 10.8 (Mountain Lion) unless you take extra steps to install gcc (which is actually llvm-gcc and cross-compiles for OSX 10.7). So, Clang *is* gaining some "market share", though not yet in major HPC systems. -Paul On Wed, Oct 31, 2012 at 11:40 AM, Ralph Castainwrote: > If it's only on for Clang, I very much doubt anyone will care - I'm > unaware of any of our users that currently utilize that compiler, and > certainly not on the clusters in the national labs (gcc, Intel, etc. - but > I've never seen them use Clang). > > Not saying anything negative about Clang - just noting it isn't much used > in our current community that I've heard. > > > On Oct 31, 2012, at 11:19 AM, Dmitri Gribenko wrote: > > > On Wed, Oct 31, 2012 at 5:04 PM, Jeff Squyres > wrote: > >> On Oct 31, 2012, at 9:38 AM, Dmitri Gribenko wrote: > >> > The rationale here is that correct MPI applications should not need > to add any extra compiler files to compile without warnings. > >>> > >>> I would disagree with this. Compiler warnings are most useful when > >>> they are on by default. Only a few developers will turn on a warning > >>> because warnings are hard to discover and enabling a warning requires > >>> an explicit action from the developer. > >> > >> Understood, but: > >> > >> a) MPI explicitly allows this kind of deliberate mismatch. It does not > make sense to warn for things that are correct in MPI. > > > > I don't think it is MPI. It is the C standard that allows one to > > store any pointer in void* and char*. But C standard also considers > > lots of other weird things to be valid, see below. > > > >> b) Warnings are significantly less useful if the user looks at them and > says, "the compiler is wrong; I know that MPI says that this deliberate > mismatch in my code is ok." > > > > Well, one can also argue that since the following is valid C, the > > warning in question should not be implemented at all: > > > > long *b = malloc(sizeof(int)); > > MPI_Recv(b, 1, MPI_INT, ...); > > > > But this code is clearly badly written, so we are left with a question > > about where to draw the line. > > > >> c) as such, these warnings are really only useful for the application > where type/MPI_Datatype matching is expected/desired. > > > > Compilers already warn about valid C code. Silencing many warnings > > relies on conventions that are derived from best practices of being > > explicit about something unusual. For example: > > > > $ cat /tmp/aaa.c > > void foo(void *a) { > > for(int i = a; i < 10; i++) > > { > >if(i = 5) > > return; > > } > > } > > $ clang -fsyntax-only -std=c99 /tmp/aaa.c > > /tmp/aaa.c:2:11: warning: incompatible pointer to integer conversion > > initializing 'int' with an expression of type 'void *' > > [-Wint-conversion] > > for(int i = a; i < 10; i++) > > ^ ~ > > /tmp/aaa.c:4:10: warning: using the result of an assignment as a > > condition without parentheses [-Wparentheses] > >if(i = 5) > > ~~^~~ > > /tmp/aaa.c:4:10: note: place parentheses around the assignment to > > silence this warning > >if(i = 5) > > ^ > > () > > /tmp/aaa.c:4:10: note: use '==' to turn this assignment into an > > equality comparison > >if(i = 5) > > ^ > > == > > 2 warnings generated. > > > > According to C standard this is valid C code, but clang emits two > > warnings on this. > > > >> Can these warnings be enabled as part of the warnings rollup -Wall > option? That would be an easy way to find/enable these warnings. > > > > IIRC, -Wall warning set is frozen in clang. -Wall is misleading in > > that it does not turn on all warnings implemented in the compiler. > > Clang has -Weverything to really turn on all warnings. But > > -Weverything is very noisy (by design, not to be fixed) unless one > > also turns off all warnings that are not interesting for the project > > with -Wno-foo. > > > > I don't think it is possible to disable this warning by default > > because off-by-default warnings are discouraged in Clang. There is no > > formal policy, but the rule of thumb is: either make the warning good > > enough for everyone or don't implement it; if some particular app does > > something strange, it can disable this warning. > > > >>> The pattern you described is an important one, but most MPI > >>> applications will have matching buffer types/type tags. > >> > >> I agree that most applications *probably* don't do this. But > significant developer in this community (i.e., Sandia) has at least > multiple applications that *do* do it. I can't ignore that. :-( > > > > Here are a few approaches to solving this in order of preference: > > > > 0. Is this really a concern for Sandia? (I.e., do they target Clang?) > > > > 1.
Re: [OMPI devel] Compile-time MPI_Datatype checking
If it's only on for Clang, I very much doubt anyone will care - I'm unaware of any of our users that currently utilize that compiler, and certainly not on the clusters in the national labs (gcc, Intel, etc. - but I've never seen them use Clang). Not saying anything negative about Clang - just noting it isn't much used in our current community that I've heard. On Oct 31, 2012, at 11:19 AM, Dmitri Gribenkowrote: > On Wed, Oct 31, 2012 at 5:04 PM, Jeff Squyres wrote: >> On Oct 31, 2012, at 9:38 AM, Dmitri Gribenko wrote: >> The rationale here is that correct MPI applications should not need to add any extra compiler files to compile without warnings. >>> >>> I would disagree with this. Compiler warnings are most useful when >>> they are on by default. Only a few developers will turn on a warning >>> because warnings are hard to discover and enabling a warning requires >>> an explicit action from the developer. >> >> Understood, but: >> >> a) MPI explicitly allows this kind of deliberate mismatch. It does not make >> sense to warn for things that are correct in MPI. > > I don't think it is MPI. It is the C standard that allows one to > store any pointer in void* and char*. But C standard also considers > lots of other weird things to be valid, see below. > >> b) Warnings are significantly less useful if the user looks at them and >> says, "the compiler is wrong; I know that MPI says that this deliberate >> mismatch in my code is ok." > > Well, one can also argue that since the following is valid C, the > warning in question should not be implemented at all: > > long *b = malloc(sizeof(int)); > MPI_Recv(b, 1, MPI_INT, ...); > > But this code is clearly badly written, so we are left with a question > about where to draw the line. > >> c) as such, these warnings are really only useful for the application where >> type/MPI_Datatype matching is expected/desired. > > Compilers already warn about valid C code. Silencing many warnings > relies on conventions that are derived from best practices of being > explicit about something unusual. For example: > > $ cat /tmp/aaa.c > void foo(void *a) { > for(int i = a; i < 10; i++) > { >if(i = 5) > return; > } > } > $ clang -fsyntax-only -std=c99 /tmp/aaa.c > /tmp/aaa.c:2:11: warning: incompatible pointer to integer conversion > initializing 'int' with an expression of type 'void *' > [-Wint-conversion] > for(int i = a; i < 10; i++) > ^ ~ > /tmp/aaa.c:4:10: warning: using the result of an assignment as a > condition without parentheses [-Wparentheses] >if(i = 5) > ~~^~~ > /tmp/aaa.c:4:10: note: place parentheses around the assignment to > silence this warning >if(i = 5) > ^ > () > /tmp/aaa.c:4:10: note: use '==' to turn this assignment into an > equality comparison >if(i = 5) > ^ > == > 2 warnings generated. > > According to C standard this is valid C code, but clang emits two > warnings on this. > >> Can these warnings be enabled as part of the warnings rollup -Wall option? >> That would be an easy way to find/enable these warnings. > > IIRC, -Wall warning set is frozen in clang. -Wall is misleading in > that it does not turn on all warnings implemented in the compiler. > Clang has -Weverything to really turn on all warnings. But > -Weverything is very noisy (by design, not to be fixed) unless one > also turns off all warnings that are not interesting for the project > with -Wno-foo. > > I don't think it is possible to disable this warning by default > because off-by-default warnings are discouraged in Clang. There is no > formal policy, but the rule of thumb is: either make the warning good > enough for everyone or don't implement it; if some particular app does > something strange, it can disable this warning. > >>> The pattern you described is an important one, but most MPI >>> applications will have matching buffer types/type tags. >> >> I agree that most applications *probably* don't do this. But significant >> developer in this community (i.e., Sandia) has at least multiple >> applications that *do* do it. I can't ignore that. :-( > > Here are a few approaches to solving this in order of preference: > > 0. Is this really a concern for Sandia? (I.e., do they target Clang?) > > 1. Ask the developer to silence the warning with a cast to 'void *' or > -Wno-type-safety. Rationale: compilers already do warn about valid > but suspicious code. > > 2. Turn off checking for char* just like for void*. Rationale: C > standard allows char* to alias a pointer of any type. Note that char* > is special in this regard (strict aliasing rules). > > 3. Turn off annotations by default in mpi.h. > > Dmitri > > -- > main(i,j){for(i=2;;i++){for(j=2;j (j){printf("%d\n",i);}}} /*Dmitri Gribenko */ > ___ >
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Wed, Oct 31, 2012 at 5:04 PM, Jeff Squyreswrote: > On Oct 31, 2012, at 9:38 AM, Dmitri Gribenko wrote: > >>> The rationale here is that correct MPI applications should not need to add >>> any extra compiler files to compile without warnings. >> >> I would disagree with this. Compiler warnings are most useful when >> they are on by default. Only a few developers will turn on a warning >> because warnings are hard to discover and enabling a warning requires >> an explicit action from the developer. > > Understood, but: > > a) MPI explicitly allows this kind of deliberate mismatch. It does not make > sense to warn for things that are correct in MPI. I don't think it is MPI. It is the C standard that allows one to store any pointer in void* and char*. But C standard also considers lots of other weird things to be valid, see below. > b) Warnings are significantly less useful if the user looks at them and says, > "the compiler is wrong; I know that MPI says that this deliberate mismatch in > my code is ok." Well, one can also argue that since the following is valid C, the warning in question should not be implemented at all: long *b = malloc(sizeof(int)); MPI_Recv(b, 1, MPI_INT, ...); But this code is clearly badly written, so we are left with a question about where to draw the line. > c) as such, these warnings are really only useful for the application where > type/MPI_Datatype matching is expected/desired. Compilers already warn about valid C code. Silencing many warnings relies on conventions that are derived from best practices of being explicit about something unusual. For example: $ cat /tmp/aaa.c void foo(void *a) { for(int i = a; i < 10; i++) { if(i = 5) return; } } $ clang -fsyntax-only -std=c99 /tmp/aaa.c /tmp/aaa.c:2:11: warning: incompatible pointer to integer conversion initializing 'int' with an expression of type 'void *' [-Wint-conversion] for(int i = a; i < 10; i++) ^ ~ /tmp/aaa.c:4:10: warning: using the result of an assignment as a condition without parentheses [-Wparentheses] if(i = 5) ~~^~~ /tmp/aaa.c:4:10: note: place parentheses around the assignment to silence this warning if(i = 5) ^ () /tmp/aaa.c:4:10: note: use '==' to turn this assignment into an equality comparison if(i = 5) ^ == 2 warnings generated. According to C standard this is valid C code, but clang emits two warnings on this. > Can these warnings be enabled as part of the warnings rollup -Wall option? > That would be an easy way to find/enable these warnings. IIRC, -Wall warning set is frozen in clang. -Wall is misleading in that it does not turn on all warnings implemented in the compiler. Clang has -Weverything to really turn on all warnings. But -Weverything is very noisy (by design, not to be fixed) unless one also turns off all warnings that are not interesting for the project with -Wno-foo. I don't think it is possible to disable this warning by default because off-by-default warnings are discouraged in Clang. There is no formal policy, but the rule of thumb is: either make the warning good enough for everyone or don't implement it; if some particular app does something strange, it can disable this warning. >> The pattern you described is an important one, but most MPI >> applications will have matching buffer types/type tags. > > I agree that most applications *probably* don't do this. But significant > developer in this community (i.e., Sandia) has at least multiple applications > that *do* do it. I can't ignore that. :-( Here are a few approaches to solving this in order of preference: 0. Is this really a concern for Sandia? (I.e., do they target Clang?) 1. Ask the developer to silence the warning with a cast to 'void *' or -Wno-type-safety. Rationale: compilers already do warn about valid but suspicious code. 2. Turn off checking for char* just like for void*. Rationale: C standard allows char* to alias a pointer of any type. Note that char* is special in this regard (strict aliasing rules). 3. Turn off annotations by default in mpi.h. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Oct 31, 2012, at 9:38 AM, Dmitri Gribenko wrote: >> The rationale here is that correct MPI applications should not need to add >> any extra compiler files to compile without warnings. > > I would disagree with this. Compiler warnings are most useful when > they are on by default. Only a few developers will turn on a warning > because warnings are hard to discover and enabling a warning requires > an explicit action from the developer. Understood, but: a) MPI explicitly allows this kind of deliberate mismatch. It does not make sense to warn for things that are correct in MPI. b) Warnings are significantly less useful if the user looks at them and says, "the compiler is wrong; I know that MPI says that this deliberate mismatch in my code is ok." c) as such, these warnings are really only useful for the application where type/MPI_Datatype matching is expected/desired. Can these warnings be enabled as part of the warnings rollup -Wall option? That would be an easy way to find/enable these warnings. > The pattern you described is an important one, but most MPI > applications will have matching buffer types/type tags. I agree that most applications *probably* don't do this. But significant developer in this community (i.e., Sandia) has at least multiple applications that *do* do it. I can't ignore that. :-( > Applications > that use code like that can disable the warning or silence it. > > If the pattern you describe is extremely frequent, we could disable > checking for 'char *' buffer type, just like for 'void *'. > > Dmitri > > -- > main(i,j){for(i=2;;i++){for(j=2;j (j){printf("%d\n",i);}}} /*Dmitri Gribenko*/ > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Oct 31, 2012, at 3:45 AM, Dmitri Gribenko wrote: >> With this patch, they'd get warnings about these uses, even though they are >> completely valid according to MPI. >> >> A suggestion was that this functionality could be disabled by default, and >> enabled with a magic macro. Perhaps something like: > > This is a vaild concern and a good idea for a FAQ entry. Can we flip the orientation such that -Wmpi-type-safety needs to be specified to enable this behavior, rather than -Wno-mpi-type-safety is required to disable it? (or the equivalent -D's) The rationale here is that correct MPI applications should not need to add any extra compiler files to compile without warnings. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Wed, Oct 31, 2012 at 4:25 AM, Jeff Squyreswrote: > On Oct 28, 2012, at 10:28 AM, Dmitri Gribenko wrote: > >> Thank you for the feedback! Hopefully the attached patch fixes both of >> these. >> >> 1. There are two helper structs with complex numbers. I predicated >> the struct declarations and use to appear only in C99. >> >> 2. These macros were indeed missing. > > I did a few tests and this now looks good; no more warnings. > > I brought up this functionality on the weekly OMPI dev telecon today and got > an important piece of feedback: apparently there are a large class of apps > that wrap their messages as transparent blobs, and then use either > non-blob-like or derived MPI datatypes. (I said something similar to this > earlier in the thread, but I didn't know that there was a large class of apps > that actually did it) > > A very simple example is: > > char *foo = malloc(...); > // ...fill foo... > MPI_Send(foo, x, MPI_INT, ...); > > Another not-uncommon example is: > > char *foo = malloc(...); > // Receive some INTEGERs from a Fortran sender > MPI_Recv(foo, x, MPI_INTEGER, ...); > > With this patch, they'd get warnings about these uses, even though they are > completely valid according to MPI. > > A suggestion was that this functionality could be disabled by default, and > enabled with a magic macro. Perhaps something like: > > mpicc -DOMPI_DDT_CHECKING ... > > or something like that. > > Thoughts? This is a vaild concern and a good idea for a FAQ entry. Q: How to silence warnings about buffer type/type tag mismatch? A: There are multiple ways for an MPI application to silence these warnings. To silence a particular warning, change the type of the buffer to 'void *' or explicitly cast it to 'void *': void *foo = malloc(...); // ...fill foo... MPI_Send(foo, x, MPI_INT, ...); or: char *foo = malloc(...); // ...fill foo... MPI_Send((void *) foo, x, MPI_INT, ...); To turn off all warnings of this kind in clang, use the -Wno-type-safety command line option. It is also possible to completely remove annotations from mpi.h by defining a macro OMPI_NO_ATTR_TYPE_TAGS in the source code or on the compiler's command line. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Oct 28, 2012, at 10:28 AM, Dmitri Gribenko wrote: > Thank you for the feedback! Hopefully the attached patch fixes both of these. > > 1. There are two helper structs with complex numbers. I predicated > the struct declarations and use to appear only in C99. > > 2. These macros were indeed missing. I did a few tests and this now looks good; no more warnings. I brought up this functionality on the weekly OMPI dev telecon today and got an important piece of feedback: apparently there are a large class of apps that wrap their messages as transparent blobs, and then use either non-blob-like or derived MPI datatypes. (I said something similar to this earlier in the thread, but I didn't know that there was a large class of apps that actually did it) A very simple example is: char *foo = malloc(...); // ...fill foo... MPI_Send(foo, x, MPI_INT, ...); Another not-uncommon example is: char *foo = malloc(...); // Receive some INTEGERs from a Fortran sender MPI_Recv(foo, x, MPI_INTEGER, ...); With this patch, they'd get warnings about these uses, even though they are completely valid according to MPI. A suggestion was that this functionality could be disabled by default, and enabled with a magic macro. Perhaps something like: mpicc -DOMPI_DDT_CHECKING ... or something like that. Thoughts? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Thu, Oct 18, 2012 at 7:32 PM, Jeff Squyreswrote: > On Oct 7, 2012, at 2:25 PM, Dmitri Gribenko wrote: >> I tried to follow your advice about Fortran datatypes and updated the >> patch accordingly (attached). This patch is against OpenMPI 1.9. >> Please review. > > Comments in the attached. Look for "*** JMS". Thank you for the review! Please find attached an updated patch. *** JMS The thought occurs to me that Fortran's built-in types CHARACTER, COMPLEX, LOGICAL, INTEGER, REAL, DOUBLE COMPLEX, and DOUBLE PRECISION will *always* exist if we are including Fortran support. The other (LOGICAL1, LOGICAL4, ...etc.) are optional/compiler extensions and may or may not exist. Regardless, the point is that I wonder if we don't need these extra #defines (E.g., OMPI_HAVE_FORTRAN_CHARACTER) -- we could just use #if (OMPI_BUILD_FORTRAN_MPIFH_BINDINGS || OMPI_BUILD_FORTRAN_USEMPIF08_BINDINGS || OMPI_BUILD_FORTRAN_USEMPI_BINDINGS) (or have an abbreviation for that)... Thank you for the suggestion, I introduced OMPI_BUILD_FORTRAN_BINDINGS into mpi.h.in for this. *** JMS Per my above ramble, CHARACTER and INTEGER (and others) are built-in to Fortran, so we'll always have these if we're building Fortran at all. So how about simplifying these cases a little; perhaps something like: *** JMS I think these "2foo" types might be able to be simplified per my above ramble...? Simplified thanks to OMPI_BUILD_FORTRAN_BINDINGS. *** JMS Can we do anything with ## to simplify these macros a bit, perchance? (I haven't thought this through) +OMPI_DECLSPEC extern struct ompi_predefined_datatype_t ompi_mpi_logical1 +#if OMPI_HAVE_FORTRAN_LOGICAL1 + OMPI_ATTR_TYPE_TAG(ompi_fortran_logical1_t) +#endif + ; I could not think of anything that could help to simplify that part. All of these attributes are predicated on different conditions. typedef struct { ompi_fortran_real_t real; ompi_fortran_real_t imag; -} ompi_fortran_complex_t; +} ompi_fortran_complex_struct_t; #endif *** JMS I was initially curious as to why you renamed these, but now I see: it's because we added the same names into mpi.h.in. Do we really still need them here in ompi_config.h.in? I.e., aren't the definitions the same? They are different because macro names in mpi.h.in are defined to expand standard complex types' names (like float _Complex); there is special syntax to access real and imaginary parts of these. Types in ompi_config.h.in are custom structs that are layout-compatible with standard complex types; real and imaginary parts can be accessed with usual member access operators. So, while we could change op_base_functions.c that uses these structs to use standard syntax (and remove struct declarations), I doubt that it is desirable. As far as I understand, one should be able to build OpenMPI with a compiler that does not support standard complex types. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/ ompi-v5.patch Description: Binary data
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Thu, Oct 18, 2012 at 7:32 PM, Jeff Squyreswrote: > On Oct 7, 2012, at 2:25 PM, Dmitri Gribenko wrote: > Hmm. I'm not sure how to do that -- I don't know of any C compiler that has > built-in #defines for what Fortran types exist. > > I'm open to suggestions, though -- can you suggest an easier solution? Sorry, that was unclear. I meant just not annotating Fortran datatypes -- they will not get any checking, so we will not get false positives, but such a change is much simpler to review for 1.7. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Oct 7, 2012, at 2:25 PM, Dmitri Gribenko wrote: >> Ok, this might get a little complicated. You'll probably need to use a pair >> of them (this is trunk only; it's different in v1.6 because we wholly >> revamped the trunk's Fortran support recently): >> >> 1. You can see all the OMPI_HAVE_FORTRAN_'s at the top of mpi.h.in. >> These indicate whether the Fortran compiler supports these types or not. >> >> 2. We currently define *one* Fortran type in mpi.h.in: >> ompi_fortran_integer_t. It looks like we need to add the rest of them: >> ompi_fortran__t (these are all in opal/include/opal_config.h, but >> mpi.h is a standalone, user-includeable file, which is why it replicates a >> subset of all the configure-generated results). Here's a first stab at what >> I think will be needed in mpi.h.in: > > Hello Jeff, I'm *so* sorry for the delay; I'm literally buried in SC deadlines. It's that time of year again... :-( > I would like to continue this discussion. Corresponding changes in > Clang are already in SVN and the feature should be released with the > upcoming Clang 3.2. Great! > I tried to follow your advice about Fortran datatypes and updated the > patch accordingly (attached). This patch is against OpenMPI 1.9. > Please review. Comments in the attached. Look for "*** JMS". > Is there any chance we can get a less invasive (header-only, without > autotools magic for Fortran datatypes support) change in the OpenMPI > 1.7? Hmm. I'm not sure how to do that -- I don't know of any C compiler that has built-in #defines for what Fortran types exist. I'm open to suggestions, though -- can you suggest an easier solution? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ ompi-v4-jms.patch Description: Binary data
Re: [OMPI devel] Compile-time MPI_Datatype checking
Dear OpenMPI developers, Could someone please review the attached patch? Dmitri On Sun, Oct 7, 2012 at 9:25 PM, Dmitri Gribenkowrote: > On Thu, May 31, 2012 at 2:38 PM, Jeff Squyres wrote: >> On May 31, 2012, at 7:29 AM, Jeff Squyres wrote: >> > We should have AC macros for all of these already. OK, I'll try find them to support (1) usecase described below. >>> >>> No, I'll find them -- sorry, I meant to look them up before I sent the last >>> mail. Let me look them up and get back to you. Our configury is quite >>> complicated, and I know the right places to look. :-) >> >> Ok, this might get a little complicated. You'll probably need to use a pair >> of them (this is trunk only; it's different in v1.6 because we wholly >> revamped the trunk's Fortran support recently): >> >> 1. You can see all the OMPI_HAVE_FORTRAN_'s at the top of mpi.h.in. >> These indicate whether the Fortran compiler supports these types or not. >> >> 2. We currently define *one* Fortran type in mpi.h.in: >> ompi_fortran_integer_t. It looks like we need to add the rest of them: >> ompi_fortran__t (these are all in opal/include/opal_config.h, but >> mpi.h is a standalone, user-includeable file, which is why it replicates a >> subset of all the configure-generated results). Here's a first stab at what >> I think will be needed in mpi.h.in: > > Hello Jeff, > > I would like to continue this discussion. Corresponding changes in > Clang are already in SVN and the feature should be released with the > upcoming Clang 3.2. > > I tried to follow your advice about Fortran datatypes and updated the > patch accordingly (attached). This patch is against OpenMPI 1.9. > Please review. > > Is there any chance we can get a less invasive (header-only, without > autotools magic for Fortran datatypes support) change in the OpenMPI > 1.7? > >> Does clang link together with gfortran? I.e., does the following work: >> >> ./configure CC=clang CXX=clang++ FC=gfortran ... > > Seems like it works. > > Dmitri > > -- > main(i,j){for(i=2;;i++){for(j=2;j (j){printf("%d\n",i);}}} /*Dmitri Gribenko */ -- main(i,j){for(i=2;;i++){for(j=2;j*/ ompi-v4.patch Description: Binary data
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Thu, May 31, 2012 at 2:38 PM, Jeff Squyreswrote: > On May 31, 2012, at 7:29 AM, Jeff Squyres wrote: > We should have AC macros for all of these already. >>> >>> OK, I'll try find them to support (1) usecase described below. >> >> No, I'll find them -- sorry, I meant to look them up before I sent the last >> mail. Let me look them up and get back to you. Our configury is quite >> complicated, and I know the right places to look. :-) > > Ok, this might get a little complicated. You'll probably need to use a pair > of them (this is trunk only; it's different in v1.6 because we wholly > revamped the trunk's Fortran support recently): > > 1. You can see all the OMPI_HAVE_FORTRAN_'s at the top of mpi.h.in. > These indicate whether the Fortran compiler supports these types or not. > > 2. We currently define *one* Fortran type in mpi.h.in: > ompi_fortran_integer_t. It looks like we need to add the rest of them: > ompi_fortran__t (these are all in opal/include/opal_config.h, but mpi.h > is a standalone, user-includeable file, which is why it replicates a subset > of all the configure-generated results). Here's a first stab at what I think > will be needed in mpi.h.in: Hello Jeff, I would like to continue this discussion. Corresponding changes in Clang are already in SVN and the feature should be released with the upcoming Clang 3.2. I tried to follow your advice about Fortran datatypes and updated the patch accordingly (attached). This patch is against OpenMPI 1.9. Please review. Is there any chance we can get a less invasive (header-only, without autotools magic for Fortran datatypes support) change in the OpenMPI 1.7? > Does clang link together with gfortran? I.e., does the following work: > > ./configure CC=clang CXX=clang++ FC=gfortran ... Seems like it works. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/ ompi-v4.patch Description: Binary data
Re: [OMPI devel] Compile-time MPI_Datatype checking
On May 31, 2012, at 7:29 AM, Jeff Squyres wrote: >>> We should have AC macros for all of these already. >> >> OK, I'll try find them to support (1) usecase described below. > > No, I'll find them -- sorry, I meant to look them up before I sent the last > mail. Let me look them up and get back to you. Our configury is quite > complicated, and I know the right places to look. :-) Ok, this might get a little complicated. You'll probably need to use a pair of them (this is trunk only; it's different in v1.6 because we wholly revamped the trunk's Fortran support recently): 1. You can see all the OMPI_HAVE_FORTRAN_'s at the top of mpi.h.in. These indicate whether the Fortran compiler supports these types or not. 2. We currently define *one* Fortran type in mpi.h.in: ompi_fortran_integer_t. It looks like we need to add the rest of them: ompi_fortran__t (these are all in opal/include/opal_config.h, but mpi.h is a standalone, user-includeable file, which is why it replicates a subset of all the configure-generated results). Here's a first stab at what I think will be needed in mpi.h.in: - /* A bogus type that allows us to have sentinel type values that are still valid */ #define ompi_fortran_bogus_type_t int /* C type corresponding to Fortran CHARACTER */ #undef ompi_fortran_character_t /* C type corresponding to Fortran COMPLEX*16 */ #undef ompi_fortran_complex16_t /* C type corresponding to Fortran COMPLEX*32 */ #undef ompi_fortran_complex32_t /* C type corresponding to Fortran COMPLEX*4 */ #undef ompi_fortran_complex4_t /* C type corresponding to Fortran COMPLEX*8 */ #undef ompi_fortran_complex8_t /* C type corresponding to Fortran COMPLEX */ #undef ompi_fortran_complex_t /* C type corresponding to Fortran DOUBLE COMPLEX */ #undef ompi_fortran_double_complex_t /* C type corresponding to Fortran DOUBLE PRECISION */ #undef ompi_fortran_double_precision_t /* C type corresponding to Fortran INTEGER*16 */ #undef ompi_fortran_integer16_t /* C type corresponding to Fortran INTEGER*1 */ #undef ompi_fortran_integer1_t /* C type corresponding to Fortran INTEGER*2 */ #undef ompi_fortran_integer2_t /* C type corresponding to Fortran INTEGER*4 */ #undef ompi_fortran_integer4_t /* C type corresponding to Fortran INTEGER*8 */ #undef ompi_fortran_integer8_t long /* C type corresponding to Fortran INTEGER */ #undef ompi_fortran_integer_t /* JMS -- ^^ might as well put the existing ompi_fortran_integer_t in the middle of the list here, with all the rest */ /* C type corresponding to Fortran LOGICAL*1 */ #undef ompi_fortran_logical1_t /* C type corresponding to Fortran LOGICAL*2 */ #undef ompi_fortran_logical2_t /* C type corresponding to Fortran LOGICAL*4 */ #undef ompi_fortran_logical4_t /* C type corresponding to Fortran LOGICAL*8 */ #undef ompi_fortran_logical8_t long /* C type corresponding to Fortran LOGICAL */ #undef ompi_fortran_logical_t /* C type corresponding to Fortran REAL*16 */ #undef ompi_fortran_real16_t long /* C type corresponding to Fortran REAL*2 */ #undef ompi_fortran_real2_t /* C type corresponding to Fortran REAL*4 */ #undef ompi_fortran_real4_t /* C type corresponding to Fortran REAL*8 */ #undef ompi_fortran_real8_t /* C type corresponding to Fortran REAL */ #undef ompi_fortran_real_t - Does clang link together with gfortran? I.e., does the following work: ./configure CC=clang CXX=clang++ FC=gfortran ... -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Thu, May 31, 2012 at 2:29 PM, Jeff Squyreswrote: > Ah, good point. But in those cases, MPI specifically defines that those > arguments are wholly ignored on non-root processes. So you could still even > pass <, MPI_DATATYPE_NULL>, or , and it would be ok. I see. If this warning is imposing some coding style guideline, then we would better not annotate MPI_DATATYPE_NULL or some users will find these diagnostics to be too noisy. >>> We should have AC macros for all of these already. >> >> OK, I'll try find them to support (1) usecase described below. > > No, I'll find them -- sorry, I meant to look them up before I sent the last > mail. Let me look them up and get back to you. Our configury is quite > complicated, and I know the right places to look. :-) Thanks in advance. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On May 31, 2012, at 7:22 AM, Dmitri Gribenko wrote: >> I can't think of a case offhand where it's acceptable to pass >> MPI_DATATYPE_NULL to an MPI function. I could be missing something, >> though... (actually, I guess I can think of some cases -- we have some >> regression test programs that specifically pass MPI_DATATYPE_NULL, just to >> ensure that it properly invokes an MPI exception) > > I usually passas arguments which are > "significant only at root" and that function call is not in the root > process code path. Ah, good point. But in those cases, MPI specifically defines that those arguments are wholly ignored on non-root processes. So you could still even pass <, MPI_DATATYPE_NULL>, or , and it would be ok. >> But even so, if one passes MPI_DATATYPE_NULL, the buffer and count arguments >> will be ignored. So I don't think that we want to require that >> MPI_DATATYPE_NULL *requires* a NULL pointer. > > It just doesn't make sense to pass non-null pointer and > MPI_DATATYPE_NULL (because null pointer will be ignored). Hence the > warning. Hmm. I'm still a bit torn about this. >>> *** JMS I'm a bit confused as to why 2int got a tag, but the others >>>did not. We do have C equivalents for all types -- do you need a >>>listing of the configure results where we figured out all the C >>>equivalents? (i.e., I can look them up for you -- our configury >>>is a bit... deep :-) ) >>> >>> I did not annotate them because those are Fortran types. If there are >>> some known C equivalents in OpenMPI, I will happily add them. But >>> please note that if these equivalents are discovered during configure, >>> we can not hardcode them into mpi.h.in. Some autoconf macros will be >>> needed probably. >> >> We should have AC macros for all of these already. > > OK, I'll try find them to support (1) usecase described below. No, I'll find them -- sorry, I meant to look them up before I sent the last mail. Let me look them up and get back to you. Our configury is quite complicated, and I know the right places to look. :-) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Thu, May 31, 2012 at 2:07 PM, Jeff Squyreswrote: > On May 30, 2012, at 7:04 PM, Dmitri Gribenko wrote: >> +must_be_null specifies that the expression should be a null >> +pointer constant, for example: >> + >> + >> + >> +/* In mpi.h */ >> +extern struct mpi_datatype mpi_datatype_null >> + __attribute__(( type_tag_for_datatype(mpi, void, must_be_null) )); >> + >> +#define MPI_DATATYPE_NULL ((MPI_Datatype) mpi_datatype_null) >> + >> +/* In user code */ >> +MPI_Send(buffer, 1, MPI_DATATYPE_NULL /*, ... */); // warning: >> MPI_DATATYPE_NULL >> + // was specified but >> buffer >> + // is not a null pointer > > I'm not sure that this is a warning we want to set. > > MPI__NULL constants are defined as "invalid handles" -- in most cases, > it's an error to pass them to an MPI function (e.g., the MPI_Send example, > above, would generate an MPI exception). They're usually used for comparison. > > I can't think of a case offhand where it's acceptable to pass > MPI_DATATYPE_NULL to an MPI function. I could be missing something, > though... (actually, I guess I can think of some cases -- we have some > regression test programs that specifically pass MPI_DATATYPE_NULL, just to > ensure that it properly invokes an MPI exception) I usually pass as arguments which are "significant only at root" and that function call is not in the root process code path. > But even so, if one passes MPI_DATATYPE_NULL, the buffer and count arguments > will be ignored. So I don't think that we want to require that > MPI_DATATYPE_NULL *requires* a NULL pointer. It just doesn't make sense to pass non-null pointer and MPI_DATATYPE_NULL (because null pointer will be ignored). Hence the warning. >> *** JMS I'm a bit confused as to why 2int got a tag, but the others >> did not. We do have C equivalents for all types -- do you need a >> listing of the configure results where we figured out all the C >> equivalents? (i.e., I can look them up for you -- our configury >> is a bit... deep :-) ) >> >> I did not annotate them because those are Fortran types. If there are >> some known C equivalents in OpenMPI, I will happily add them. But >> please note that if these equivalents are discovered during configure, >> we can not hardcode them into mpi.h.in. Some autoconf macros will be >> needed probably. > > We should have AC macros for all of these already. OK, I'll try find them to support (1) usecase described below. > The point of this is because all MPI datatypes are available in all > languages, meaning that I could MPI_Send an MPI_INTEGER from fortran and > receive it in C code (that MPI_Recv's an MPI_INTEGER). FWIW, I've seen apps > do this in two general cases: > > 1. Fortran sends to C, C just holds the blob without looking at/understanding > the value, C eventually sends the blob back to Fortran. > > 2. Fortran sends to C, and C interprets the value because it knows the > interoperable type (e.g., sends MPI_INTEGER, receives into a C int). > > If the app doesn't know the exact equivalent C type corresponding to the > Fortran type (e.g., case 1), they may need one of the examples I provided > above (e.g., cast the buffer to (void*)). Agreed. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On May 30, 2012, at 7:04 PM, Dmitri Gribenko wrote: > +must_be_null specifies that the expression should be a null > +pointer constant, for example: > + > + > + > +/* In mpi.h */ > +extern struct mpi_datatype mpi_datatype_null > +__attribute__(( type_tag_for_datatype(mpi, void, must_be_null) )); > + > +#define MPI_DATATYPE_NULL ((MPI_Datatype) mpi_datatype_null) > + > +/* In user code */ > +MPI_Send(buffer, 1, MPI_DATATYPE_NULL /*, ... */); // warning: > MPI_DATATYPE_NULL > + // was specified but > buffer > + // is not a null pointer I'm not sure that this is a warning we want to set. MPI__NULL constants are defined as "invalid handles" -- in most cases, it's an error to pass them to an MPI function (e.g., the MPI_Send example, above, would generate an MPI exception). They're usually used for comparison. I can't think of a case offhand where it's acceptable to pass MPI_DATATYPE_NULL to an MPI function. I could be missing something, though... (actually, I guess I can think of some cases -- we have some regression test programs that specifically pass MPI_DATATYPE_NULL, just to ensure that it properly invokes an MPI exception) But even so, if one passes MPI_DATATYPE_NULL, the buffer and count arguments will be ignored. So I don't think that we want to require that MPI_DATATYPE_NULL *requires* a NULL pointer. > *** JMS What happens if the argument type is (void*)? Does that match >the tag type? E.g.: > > [...snipped...] > If a function is passed a type tag (that is, MPI_Datatype) that does > not have an annotation, then that function call is not checked. That > is, in MPI_Send(buf, 1, MPI_BYTE, ...) buf can be of any pointer type. All the above sounds good. > *** JMS I'm a bit confused as to why 2int got a tag, but the others >did not. We do have C equivalents for all types -- do you need a >listing of the configure results where we figured out all the C >equivalents? (i.e., I can look them up for you -- our configury >is a bit... deep :-) ) > > I did not annotate them because those are Fortran types. If there are > some known C equivalents in OpenMPI, I will happily add them. But > please note that if these equivalents are discovered during configure, > we can not hardcode them into mpi.h.in. Some autoconf macros will be > needed probably. We should have AC macros for all of these already. The point of this is because all MPI datatypes are available in all languages, meaning that I could MPI_Send an MPI_INTEGER from fortran and receive it in C code (that MPI_Recv's an MPI_INTEGER). FWIW, I've seen apps do this in two general cases: 1. Fortran sends to C, C just holds the blob without looking at/understanding the value, C eventually sends the blob back to Fortran. 2. Fortran sends to C, and C interprets the value because it knows the interoperable type (e.g., sends MPI_INTEGER, receives into a C int). If the app doesn't know the exact equivalent C type corresponding to the Fortran type (e.g., case 1), they may need one of the examples I provided above (e.g., cast the buffer to (void*)). > *** JMS Per my question on MPI_Alltoallw, I'm not quite sure how these >tags work with arrays of datatypes...? > > I removed the incorrect attribute on PMPI_Alltoallw. Cool. > *** What happens if we're compiling C and std::complex isn't >defined? I see that is only defined above #if __cplusplus. > > Then OMPI_ATTR_TYPE_TAG_CXX(type) is defined to be empty and these > type tags are not checked. Ah! I missed that it was a different macro (OMPI_ATTR_TYPE_TAG_CPP/CXX). Got it. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Compile-time MPI_Datatype checking
Hi Jeff, On Thu, May 31, 2012 at 12:57 AM, Jeff Squyreswrote: > I've reviewed the patch. Good stuff! Thank you very much for the review. Answers to comments below. Updated patch attached. *** JMS What do the 3-argument forms of type_tag_for_datatype() do? They aren't described in http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20120521/057992.html My bad: that page has documentation for the previous version of clang patch. Below is a relevant excerpt of current docs. Basically, the third argument states additional instructions on how to compare types. Please excuse me for a long prose copypaste. +type_tag_for_datatype(...) + +Clang supports annotating type tags of two forms. + + +Type tag that is an expression containing a reference to some declared +identifier. Use __attribute__((type_tag_for_datatype(kind, type))) +on a declaration with that identifier: + + + +extern struct mpi_datatype mpi_datatype_int +__attribute__(( type_tag_for_datatype(mpi,int) )); +#define MPI_INT ((MPI_Datatype) mpi_datatype_int) + + + +Type tag that is an integral literal. Introduce a static +const variable with a corresponding initializer value and attach +__attribute__((type_tag_for_datatype(kind, type))) on that +declaration, for example: + + + +#define MPI_INT ((MPI_Datatype) 42) +static const MPI_Datatype mpi_datatype_int +__attribute__(( type_tag_for_datatype(mpi,int) )) = 42 + + + + +The attribute also accepts an optional third argument that determines how +the expression is compared to the type tag. There are two supported flags: + +layout_compatible will cause types to be compared according to +layout-compatibility rules (C++11 [class.mem] p17, 18). This is +implemented to support annotating types like MPI_DOUBLE_INT. + +For example: + + +/* In mpi.h */ +struct internal_mpi_double_int { double d; int i; }; +extern struct mpi_datatype mpi_datatype_double_int +__attribute__(( type_tag_for_datatype(mpi, struct internal_mpi_double_int, + layout_compatible) )); + +#define MPI_DOUBLE_INT ((MPI_Datatype) mpi_datatype_double_int) + +/* In user code */ +struct my_pair { double a; int b; }; +struct my_pair *buffer; +MPI_Send(buffer, 1, MPI_DOUBLE_INT /*, ... */); // no warning + +struct my_int_pair { int a; int b; } +struct my_int_pair *buffer2; +MPI_Send(buffer2, 1, MPI_DOUBLE_INT /*, ... */); // warning: actual buffer element + // type 'struct my_int_pair' + // doesn't match specified MPI_Datatype + + + + +must_be_null specifies that the expression should be a null +pointer constant, for example: + + + +/* In mpi.h */ +extern struct mpi_datatype mpi_datatype_null +__attribute__(( type_tag_for_datatype(mpi, void, must_be_null) )); + +#define MPI_DATATYPE_NULL ((MPI_Datatype) mpi_datatype_null) + +/* In user code */ +MPI_Send(buffer, 1, MPI_DATATYPE_NULL /*, ... */); // warning: MPI_DATATYPE_NULL + // was specified but buffer + // is not a null pointer + + + + *** JMS What happens if the argument type is (void*)? Does that match the tag type? E.g.: char recvbuf; void *foo = MPI_Send(foo, 1, MPI_CHAR, ...) If buffer has type void* then no warning is emitted. *** JMS What if I deliberately want to receive into the wrong type (and for some reason, I know it's ok)? Is casting ok enough to defeat the tag testing? E.g.: char recvbuf[100]; MPI_Recv((int*) recvbuf, 1, MPI_INT, ...); Explicit casts are always honored. In this case buffer type is thought to be int*, hence no warning is emitted. *** JMS Random suggestion: how about CXX instead of CPP? CPP could also mean C preprocessor. Right, CXX is better. Done. *** JMS Why doesn't ompi_mpi_byte have an attribute? And what does it mean to not have an attribute? Will the type checking be silently ignored at compile-time if there's no matching (MPI,foo) type_tag_for_datatype? (same question applies to other instances with no tags, like ompi_mpi_packed, but I won't bother repeating it everywhere) ompi_mpi_byte doesn't have an attr because I annotated only those types that have C or C++ equivalents stated in MPI 2.2 specification. If OpenMPI provides additional guarantees, for example that Fortran types always have some known equivalent C types, we can annotate them, too. If a function is passed a type tag (that is, MPI_Datatype) that does not have an annotation, then that function call is not checked. That is, in MPI_Send(buf, 1, MPI_BYTE, ...) buf can be of any pointer type. *** JMS I'm a bit confused as to why 2int got a tag, but the others did not. We do have C equivalents for all types -- do you need a listing of the configure results where we figured out all the C equivalents? (i.e., I can
Re: [OMPI devel] Compile-time MPI_Datatype checking
I've reviewed the patch. Good stuff! I annotated your patch file -- see attached for my comments. Search for "*** JMS" (my initials). On May 29, 2012, at 11:08 AM, Dmitri Gribenko wrote: > Hello, > > I've implemented a patch for clang that enables compile-time checking > ofarguments to functions. When applied to MPI, > this enables the compiler to check that buffer type and MPI_Datatype > match. > > Latest version of clang patch can be found here. [1] Please note that > clang patch was not yet accepted to clang trunk. > > On the OpenMPI side we need to: > * add attributes to MPI functions to mark them as accepting pointers > with type tags; > * add attributes to ompi_mpi_* declarations to mark them as type tags. > > All in all, the changes boil down to: > 1. Annotate type tags: > OMPI_DECLSPEC extern struct ompi_predefined_datatype_t >ompi_mpi_float OMPI_ATTR_TYPE_TAG(float); > > 2. Annotate functions: > OMPI_DECLSPEC int MPI_Send(void *buf, int count, MPI_Datatype > datatype, int dest, >int tag, MPI_Comm comm) >OMPI_ATTR_POINTER_WITH_TYPE_TAG(1,3); > > > where OMPI_ATTR* are macros that are defined to attributes when > compiling under clang with this feature. > > Attached is the OpenMPI patch I've arrived to. Although I tried to be > attentive, changes to mpi.h are very repetitive and error-prone, so > please review them closely. > > I've implemented a similar patch for MPICH2. [2] OpenMPI developers > might want to follow that discussion, too. > > Dmitri > > [1] > http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20120521/058137.html > [2] http://lists.mcs.anl.gov/pipermail/mpich2-dev/2012-May/000938.html > > -- > main(i,j){for(i=2;;i++){for(j=2;j (j){printf("%d\n",i);}}} /*Dmitri Gribenko */ > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ ompi-v2-annotated.diff Description: Binary data
Re: [OMPI devel] Compile-time MPI_Datatype checking
Sweet! I downloaded clang 3.1 and added it to my nightly regression testing for the OMPI SVN trunk and v1.6 branch. We'd only be able to accept this patch for the SVN trunk (i.e., what will become v1.7) -- the v1.6 series is closed for new features. I'm pretty fried right now (end of a long day); I'll review your patch tomorrow. On May 29, 2012, at 11:08 AM, Dmitri Gribenko wrote: > Hello, > > I've implemented a patch for clang that enables compile-time checking > ofarguments to functions. When applied to MPI, > this enables the compiler to check that buffer type and MPI_Datatype > match. > > Latest version of clang patch can be found here. [1] Please note that > clang patch was not yet accepted to clang trunk. > > On the OpenMPI side we need to: > * add attributes to MPI functions to mark them as accepting pointers > with type tags; > * add attributes to ompi_mpi_* declarations to mark them as type tags. > > All in all, the changes boil down to: > 1. Annotate type tags: > OMPI_DECLSPEC extern struct ompi_predefined_datatype_t >ompi_mpi_float OMPI_ATTR_TYPE_TAG(float); > > 2. Annotate functions: > OMPI_DECLSPEC int MPI_Send(void *buf, int count, MPI_Datatype > datatype, int dest, >int tag, MPI_Comm comm) >OMPI_ATTR_POINTER_WITH_TYPE_TAG(1,3); > > > where OMPI_ATTR* are macros that are defined to attributes when > compiling under clang with this feature. > > Attached is the OpenMPI patch I've arrived to. Although I tried to be > attentive, changes to mpi.h are very repetitive and error-prone, so > please review them closely. > > I've implemented a similar patch for MPICH2. [2] OpenMPI developers > might want to follow that discussion, too. > > Dmitri > > [1] > http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20120521/058137.html > [2] http://lists.mcs.anl.gov/pipermail/mpich2-dev/2012-May/000938.html > > -- > main(i,j){for(i=2;;i++){for(j=2;j (j){printf("%d\n",i);}}} /*Dmitri Gribenko */ > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[OMPI devel] Compile-time MPI_Datatype checking
Hello, I've implemented a patch for clang that enables compile-time checking ofarguments to functions. When applied to MPI, this enables the compiler to check that buffer type and MPI_Datatype match. Latest version of clang patch can be found here. [1] Please note that clang patch was not yet accepted to clang trunk. On the OpenMPI side we need to: * add attributes to MPI functions to mark them as accepting pointers with type tags; * add attributes to ompi_mpi_* declarations to mark them as type tags. All in all, the changes boil down to: 1. Annotate type tags: OMPI_DECLSPEC extern struct ompi_predefined_datatype_t ompi_mpi_float OMPI_ATTR_TYPE_TAG(float); 2. Annotate functions: OMPI_DECLSPEC int MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) OMPI_ATTR_POINTER_WITH_TYPE_TAG(1,3); where OMPI_ATTR* are macros that are defined to attributes when compiling under clang with this feature. Attached is the OpenMPI patch I've arrived to. Although I tried to be attentive, changes to mpi.h are very repetitive and error-prone, so please review them closely. I've implemented a similar patch for MPICH2. [2] OpenMPI developers might want to follow that discussion, too. Dmitri [1] http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20120521/058137.html [2] http://lists.mcs.anl.gov/pipermail/mpich2-dev/2012-May/000938.html -- main(i,j){for(i=2;;i++){for(j=2;j*/ ompi-v2.patch Description: Binary data
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Thu, Feb 2, 2012 at 7:31 AM, Christopher Samuelwrote: > On 29/01/12 10:07, Dmitri Gribenko wrote: >> My colleague and I want to implement a compile-time check (warning) >> for clang compiler that specified buffer type matches passed >> MPI_Datatype. > > Interesting, is it possible to do the same for GCC with its plugin > architecture ? I haven't worked with GCC plugins, but after looking at the documentation I think it is possible. Dmitri Gribenko -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/01/12 10:07, Dmitri Gribenko wrote: > My colleague and I want to implement a compile-time check (warning) > for clang compiler that specified buffer type matches passed > MPI_Datatype. Interesting, is it possible to do the same for GCC with its plugin architecture ? cheers! Chris - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8qH8cACgkQO2KABBYQAh/3WQCfXO4KaN5wVGsMSOCHaWFdiZH1 CDcAnRlYS+Xr1s38uiZB2QcH1B8Afhh1 =EprZ -END PGP SIGNATURE-
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Feb 1, 2012, at 5:36 PM, Dmitri Gribenko wrote: >> Depending on the patch, however, we might need a signed Open MPI >> contribution agreement from you/your employer > > That's fine for us. Excellent. :-) It all sounds good to me! -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Wed, Feb 1, 2012 at 10:01 PM, Jeff Squyreswrote: > On Jan 28, 2012, at 6:07 PM, Dmitri Gribenko wrote: >> Here's what is required on the OpenMPI part: all MPI functions that >> accept a buffer and MPI_Datatype should be annotated with >> mpi_typed_arg(buffer-arg-index, type-arg-index) attribute. For >> example: >> >> int MPI_Send(void *buf, ...etc...) __attribute__(( mpi_typed_arg(1,3) )); > > This isn't terrible; we do similar things throughout the code base. > > Will you be able to do this for MPI functions that take 2 choice buffers? > E.g., MPI_Gather. Yes, just put two attributes: __attribute__(( mpi_typed_arg(1,3), mpi_typed_arg(4,6) )) >> Predefined datatypes should be annotated with corresponding C types: >> mpi_datatype(var-decl-with-corresponding-type): >> >> extern int ompi_mpi_int_dummy; >> extern struct ompi_predefined_datatype_t ompi_mpi_int >> __attribute__(( mpi_datatype(ompi_mpi_int_dummy) )); > > Just to make sure I understand this syntax: is ompi_mpi_int_dummy just to > indicate the *type* that is congruent with the ompi_predefined_datatype_t? Yes, I did it that way because parsing a type-name as attribute argument is not possible currently in clang, but I wanted to check how it will work as quickly as possible. But I'm now hacking the parser to support the cleaner syntax. > I.e., I'd do something like this for MPI_DOUBLE: > > extern double ompi_mpi_double_dummy; > extern struct ompi_predefined_datatype_t ompi_mpi_double > __attribute__(( mpi_datatype(ompi_mpi_double_dummy) )); > > ? Right, but I'm going for the cleaner syntax anyway. >> Users can annotate their types too: >> int my_int_dummy; >> MPI_Datatype my_int_type __attribute__(( mpi_datatype(my_int_dummy) )); > > More importantly, can they do: > > struct my_struct_t my_struct_dummy; > MPI_Datatype my_struct_type __attribute__(( mpi_datatype(my_struct_dummy) )); > > ? Yes, specifying any type is possible. >> I would like to hear if there is any issue with this idea or >> implementation design that could prevent the corresponding patch for >> mpi.h to be accepted. > > I'm supportive of this (mpi.h.in, you mean :-) ). > > Depending on the patch, however, we might need a signed Open MPI contribution > agreement from you/your employer That's fine for us. Dmitri Gribenko -- main(i,j){for(i=2;;i++){for(j=2;j*/
Re: [OMPI devel] Compile-time MPI_Datatype checking
On Jan 28, 2012, at 6:07 PM, Dmitri Gribenko wrote: > My colleague and I want to implement a compile-time check (warning) > for clang compiler that specified buffer type matches passed > MPI_Datatype. Interesting! > Here's what is required on the OpenMPI part: all MPI functions that > accept a buffer and MPI_Datatype should be annotated with > mpi_typed_arg(buffer-arg-index, type-arg-index) attribute. For > example: > > int MPI_Send(void *buf, ...etc...) __attribute__(( mpi_typed_arg(1,3) )); This isn't terrible; we do similar things throughout the code base. Will you be able to do this for MPI functions that take 2 choice buffers? E.g., MPI_Gather. > Predefined datatypes should be annotated with corresponding C types: > mpi_datatype(var-decl-with-corresponding-type): > > extern int ompi_mpi_int_dummy; > extern struct ompi_predefined_datatype_t ompi_mpi_int >__attribute__(( mpi_datatype(ompi_mpi_int_dummy) )); Just to make sure I understand this syntax: is ompi_mpi_int_dummy just to indicate the *type* that is congruent with the ompi_predefined_datatype_t? I.e., I'd do something like this for MPI_DOUBLE: extern double ompi_mpi_double_dummy; extern struct ompi_predefined_datatype_t ompi_mpi_double __attribute__(( mpi_datatype(ompi_mpi_double_dummy) )); ? > Users can annotate their types too: > int my_int_dummy; > MPI_Datatype my_int_type __attribute__(( mpi_datatype(my_int_dummy) )); More importantly, can they do: struct my_struct_t my_struct_dummy; MPI_Datatype my_struct_type __attribute__(( mpi_datatype(my_struct_dummy) )); ? > This design is not final, I'd really like to have mpi_datatype(type) > (e.g., mpi_datatype(long long)) but clang doesn't currently have > support for parsing that. (But I think that clang developers won't > accept the patch unless we implement that). So type annotations will > probably look like: > > extern struct ompi_predefined_datatype_t ompi_mpi_int >__attribute__(( mpi_datatype(int) )); That would certainly be better. > The attributes can be hidden with macros from compilers that don't > support them, so compatibility should not be a problem. Yep -- we have a bunch of those already. > I would like to hear if there is any issue with this idea or > implementation design that could prevent the corresponding patch for > mpi.h to be accepted. I'm supportive of this (mpi.h.in, you mean :-) ). Depending on the patch, however, we might need a signed Open MPI contribution agreement from you/your employer (it's essentially the Apache Foundation agreement with s/Apache/Open MPI/g). It's annoying, I know :-(, but we have to keep the intellectual property of Open MPI "clean" so that it can guaranteed to remain BSD: http://www.open-mpi.org/community/contribute/ -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[OMPI devel] Compile-time MPI_Datatype checking
Hello, My colleague and I want to implement a compile-time check (warning) for clang compiler that specified buffer type matches passed MPI_Datatype. For example: MPI_Send(long_buf, 1, MPI_INT, 1, 1, MPI_COMM_WORLD); // expected-warning {{actual buffer element type 'long' doesn't match specified MPI_Datatype}} We've hacked a prototype patch for clang, but first we wanted to discuss this idea with OpenMPI developers. Here's what is required on the OpenMPI part: all MPI functions that accept a buffer and MPI_Datatype should be annotated with mpi_typed_arg(buffer-arg-index, type-arg-index) attribute. For example: int MPI_Send(void *buf, ...etc...) __attribute__(( mpi_typed_arg(1,3) )); Predefined datatypes should be annotated with corresponding C types: mpi_datatype(var-decl-with-corresponding-type): extern int ompi_mpi_int_dummy; extern struct ompi_predefined_datatype_t ompi_mpi_int __attribute__(( mpi_datatype(ompi_mpi_int_dummy) )); Users can annotate their types too: int my_int_dummy; MPI_Datatype my_int_type __attribute__(( mpi_datatype(my_int_dummy) )); This design is not final, I'd really like to have mpi_datatype(type) (e.g., mpi_datatype(long long)) but clang doesn't currently have support for parsing that. (But I think that clang developers won't accept the patch unless we implement that). So type annotations will probably look like: extern struct ompi_predefined_datatype_t ompi_mpi_int __attribute__(( mpi_datatype(int) )); The attributes can be hidden with macros from compilers that don't support them, so compatibility should not be a problem. See attached test and clang output to see what will it look like. Any comments are welcome. Please also tell me if you have any ideas for additional compile-time checks. I would like to hear if there is any issue with this idea or implementation design that could prevent the corresponding patch for mpi.h to be accepted. Dmitri Gribenko -- main(i,j){for(i=2;;i++){for(j=2;j*/ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:10:21: error: attribute requires exactly 2 arguments __attribute__(( mpi_typed_arg )); // expected-error {{attribute requires exactly 2 arguments}} ^ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:13:21: error: 'mpi_typed_arg' attribute parameter 1 is out of bounds __attribute__(( mpi_typed_arg(0,7) )); // expected-error {{attribute parameter 1 is out of bounds}} ^ ~ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:16:21: error: 'mpi_typed_arg' attribute parameter 1 is out of bounds __attribute__(( mpi_typed_arg(5,7) )); // expected-error {{attribute parameter 1 is out of bounds}} ^ ~ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:19:21: error: 'mpi_typed_arg' attribute parameter 2 is out of bounds __attribute__(( mpi_typed_arg(3,0) )); // expected-error {{attribute parameter 2 is out of bounds}} ^ ~ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:22:21: error: 'mpi_typed_arg' attribute parameter 2 is out of bounds __attribute__(( mpi_typed_arg(3,5) )); // expected-error {{attribute parameter 2 is out of bounds}} ^ ~ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:27:21: error: 'mpi_typed_arg' attribute requires parameter 1 to be an integer constant __attribute__(( mpi_typed_arg((x+1),7) )); // expected-error {{attribute requires parameter 1 to be an integer constant}} ^ ~ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:30:21: error: 'mpi_typed_arg' attribute requires parameter 2 to be an integer constant __attribute__(( mpi_typed_arg(3,x) )); // expected-error {{attribute requires parameter 2 to be an integer constant}} ^ ~ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:32:34: error: 'mpi_typed_arg' attribute only applies to functions and methods int MPI_Wrong8() __attribute__(( mpi_typed_arg(1,3) )); // expected-error {{attribute only applies to functions and methods}} ^ MPITypedArgAttr1 3 ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:45:74: error: attribute takes one argument extern struct ompi_predefined_datatype_t ompi_mpi_wrong1 __attribute__(( mpi_datatype )); // expected-error {{attribute takes one argument}} ^ ../../../llvm/tools/clang/test/Sema/warn-mpi-type-mismatch.c:46:74: error: attribute takes one argument extern struct ompi_predefined_datatype_t ompi_mpi_wrong2 __attribute__(( mpi_datatype(1,2) )); // expected-error {{attribute takes one argument}}