Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
On Nov 17, 2010, at 6:58 AM, Ricardo Reis wrote: > Thanks for the explanation. Then this should be updated in the spec no...? You have no idea. :-) The MPI Forum has been debating about exactly this issue for over a year. It turns out to be a surprisingly complex, subtle issue (i.e., it's not easy to just "upgrade" the type used to pass counts around in MPI functions). The Forum has not resolved this issue yet; a small subset of the issues are described in the SC MPI Forum BOF slides that were presented last week. Rich Graham is going to post those slides on the web somewhere, but I don't think he has posted them yet. As Gus points out, the workaround is to use MPI datatypes so that your count can still be "low" (i.e., still positive for an INTEGER or int), even though the total amount of data being written/sent/received/whatever is larger. MPI should do the Right Thing under the covers in this case. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
Ricardo Reis wrote: On Wed, 17 Nov 2010, Gus Correa wrote: For what is worth, the MPI addresses (a.k.a. pointers) in the Fortran bindings are integers, of standard size 4 bytes, IIRR. Take a look at mpif.h, mpi.h and their cousins to make sure. Unlike the Fortran FFTW "plans", you don't declare MPI addresses as big as you want, MPI chooses their size when it is built, right? As Pascal pointed out, 4-byte integers would flip sign at around 2GB, and even unsigned integers won't go beyond 4GB. Would this be part of the problem? yes, I think is the most probable explanation. I've solved it by using several processes to write the file (after all I just didn't want to program a bunch of checkups required for spanning several processes for such a simple thing...) I guess all the OpenMPI pros and developers are busy now in Bourbon Street, New Orleans, I mean, at Supercomputer 2010. Hard to catch their attention right now, but eventually somebody will clarify this. oh, just a small grain of sand... doesn't seems worth to stop the full machine for it... :) many thanks all Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 > ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users Caro Ricardo Pascal hit the nail on the head. Counting with (4-byte) integers seems to be an MPI thing, written in stone standard perhaps. In any case, here is an old thread, discussing a related problem, namely the number of items (count) in MPI_Send/Recv messages, which is again an integer, hence has the same 2GB limitation: http://www.open-mpi.org/community/lists/users/2009/02/8100.php Note that Jeff's workaround suggestion was to declare a user defined MPI type (or perhaps a hierarchy of types), then concatenate as much data as needed in a message. Granted that my knowledge of mpi-io is nil, I wonder if an approach like this would allow you to get around the count limit of mpi-io functions, which sounds no different from the count limit of other MPI functions. Say, you could use MPI_TYPE_CONTIGUOUS or MPI_TYPE_VECTOR, to aggregate big chunks of data (but still smaller than 2GB), then write a modest number of these chunks/types to the file, I suppose. Abrac,o, Gus
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
On Wed, 17 Nov 2010, Gus Correa wrote: For what is worth, the MPI addresses (a.k.a. pointers) in the Fortran bindings are integers, of standard size 4 bytes, IIRR. Take a look at mpif.h, mpi.h and their cousins to make sure. Unlike the Fortran FFTW "plans", you don't declare MPI addresses as big as you want, MPI chooses their size when it is built, right? As Pascal pointed out, 4-byte integers would flip sign at around 2GB, and even unsigned integers won't go beyond 4GB. Would this be part of the problem? yes, I think is the most probable explanation. I've solved it by using several processes to write the file (after all I just didn't want to program a bunch of checkups required for spanning several processes for such a simple thing...) I guess all the OpenMPI pros and developers are busy now in Bourbon Street, New Orleans, I mean, at Supercomputer 2010. Hard to catch their attention right now, but eventually somebody will clarify this. oh, just a small grain of sand... doesn't seems worth to stop the full machine for it... :) many thanks all Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 >
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
Ricardo Reis wrote: On Wed, 17 Nov 2010, Pascal Deveze wrote: I think the limit for a write (and also for a read) is 2^31-1 (2G-1). In a C program, after this value, an integer becomes negative. I suppose this is also true in Fortran. The solution, is to make a loop of writes (reads) of no more than this value. Is that MPI-IO specific? I remember that when using FFTW they ask for using INTEGER(8) for the returning handle. This is used has a pointer interface with the library and (8) will be equivalent to a 64 bit pointer (sort of, sorry if I am not being exact). Anyway, if I have no problems writing Big files with normal Fortran shouldn't this behaviour be found with MPI-IO? And, more to the point, if not, shouldn't it be documented somewhere? Does anyone knows if this carries over to other MPI implementations (or the answer is "download, try it and tell us?") best, For what is worth, the MPI addresses (a.k.a. pointers) in the Fortran bindings are integers, of standard size 4 bytes, IIRR. Take a look at mpif.h, mpi.h and their cousins to make sure. Unlike the Fortran FFTW "plans", you don't declare MPI addresses as big as you want, MPI chooses their size when it is built, right? As Pascal pointed out, 4-byte integers would flip sign at around 2GB, and even unsigned integers won't go beyond 4GB. Would this be part of the problem? I guess all the OpenMPI pros and developers are busy now in Bourbon Street, New Orleans, I mean, at Supercomputer 2010. Hard to catch their attention right now, but eventually somebody will clarify this. Gus Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 > ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
On Wed, 17 Nov 2010, Pascal Deveze wrote: This is due to the interface defined for MPI_File_write that specifies an integer for the length. The positive value of an integer are coded in hexadecimal from to 7FFF FFF and negative values are coded from 8000 to . (7FFF is exactly 2^31-1). Thanks for the explanation. Then this should be updated in the spec no...? cheers! Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 >___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
This is due to the interface defined for MPI_File_write that specifies an integer for the length. The positive value of an integer are coded in hexadecimal from to 7FFF FFF and negative values are coded from 8000 to . (7FFF is exactly 2^31-1). Pascal Ricardo Reis a écrit : On Wed, 17 Nov 2010, Pascal Deveze wrote: I think the limit for a write (and also for a read) is 2^31-1 (2G-1). In a C program, after this value, an integer becomes negative. I suppose this is also true in Fortran. The solution, is to make a loop of writes (reads) of no more than this value. Is that MPI-IO specific? I remember that when using FFTW they ask for using INTEGER(8) for the returning handle. This is used has a pointer interface with the library and (8) will be equivalent to a 64 bit pointer (sort of, sorry if I am not being exact). Anyway, if I have no problems writing Big files with normal Fortran shouldn't this behaviour be found with MPI-IO? And, more to the point, if not, shouldn't it be documented somewhere? Does anyone knows if this carries over to other MPI implementations (or the answer is "download, try it and tell us?") best, Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 > ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
On Tue, 16 Nov 2010, Gus Correa wrote: Ricardo Reis wrote: and sorry to be such a nuisance... but any motive for an MPI-IO "wall" between the 2.0 and 2.1 Gb? Salve Ricardo Reis! Is this "wall" perhaps the 2GB Linux file size limit on 32-bit systems? No. This is a 64bit machine and if I write the file normaly (like just using Fortran open/write/close) it works without problem (I just got a 3.5Gb file this way). If try to use 2 processes it will also spit a 3.5 Gb file (because each will be writing a chunk smaller than the 2.1 Gb "limit" ?) I haven't tried to replicate the same program in C, I must admit best, Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 >
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
Ricardo Reis wrote: and sorry to be such a nuisance... but any motive for an MPI-IO "wall" between the 2.0 and 2.1 Gb? Salve Ricardo Reis! Is this "wall" perhaps the 2GB Linux file size limit on 32-bit systems? Gus (1 mpi process) best, Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 > ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] mpi-io, fortran, going crazy... (ADENDA)
and sorry to be such a nuisance... but any motive for an MPI-IO "wall" between the 2.0 and 2.1 Gb? (1 mpi process) best, Ricardo Reis 'Non Serviam' PhD candidate @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt Cultural Instigator @ Rádio Zero http://www.radiozero.pt Keep them Flying! Ajude a/help Aero Fénix! http://www.aeronauta.com/aero.fenix http://www.flickr.com/photos/rreis/ contacts: gtalk: kyriu...@gmail.com skype: kyriusan < sent with alpine 2.00 >
