let me amend my last email by making clear that I do not recommend using NFS for parallel I/O. But if you have to, make sure your code does not do things like read-after-write, or multiple processes writing data that ends up in the same file system block (can often be avoided by using collective I/O for example).
-----Original Message----- From: users <users-boun...@lists.open-mpi.org> On Behalf Of Gabriel, Edgar via users Sent: Thursday, September 23, 2021 5:31 PM To: Eric Chamberland <eric.chamberl...@giref.ulaval.ca>; Open MPI Users <users@lists.open-mpi.org> Cc: Gabriel, Edgar <egabr...@central.uh.edu>; Louis Poirel <louis.poi...@michelin.com>; Vivien Clauzon <vivien.clau...@michelin.com> Subject: Re: [OMPI users] Status of pNFS, CephFS and MPI I/O -----Original Message----- From: Eric Chamberland <eric.chamberl...@giref.ulaval.ca> Thanks for your answer Edgard! In fact, we are able to use NFS and certainly any POSIX file system on a single node basis. I should have been asking for: What are the supported file systems for *multiple nodes* read/write access to files? -> We have tested it on BeeGFS, GPFS, Lustre, PVFS2/OrangeFS, and NFS, but again, if a parallel file system has POSIX functions would expect it to work (and yes, I am aware of the strict POSIX semantics are not necessarily available in parallel file systems. Internally, we are using open, close, (p)readv, (pwritev), lock, unlock, seek). For nfs, MPI I/O is known to *not* work on NFS when using multiple nodes ... except for NFS v3 with "noac" mount option (we are about to test with "actimeo=0" option to see if it works). -> well, it depends on how you define that. I would suspect that our largest user base is actually using NFS v3/v4 with the noac option + nfslock server. Most I/O patterns will probably work in this scenario, and in fact, we are actually passing our entire testsuite on multi-node NFS setup (which does some nasty things). However, it is true that there could be corner cases that fail. In addition, parallel I/O on multi-node NFS can be outrageously slow, since we lock the *entire* file before every operation ( in contrary to ROMIO, which only locks the file range that is currently being accessed). Btw, is OpenMPI MPI I/O having some "hidden" (mca?) options to make a multiple nodes NFS cluster to work? -> OMPO recognizes the NFS file system automatically, without requiring an mca parameter. I usually recommend to users to try to relax the locking options and see whether they still produce correct data in order to improve performance of their code, since most I/O patterns do not require this super-strict locking behavior. This is the fs_ufs_lock_algorithm parameter. Thanks Edgar Thanks, Eric On 2021-09-23 1:57 p.m., Gabriel, Edgar wrote: > Eric, > > generally speaking, ompio should be able to operate correctly on all file > systems that have support for POSIX functions. The generic ufs component is > for example being used on BeeGFS parallel file systems without problems, we > are using that on a daily basis. For GPFS, the only reason we handle that > file system separately is because of some custom info objects that can be > used to configure the file during file_open. If one would not use these info > objects the generic ufs component would be as good as the GPFS specific > component. > > Note, the generic ufs component is also being used for NFS, it has logic > built in to recognize an NFS file system and handle some operations slightly > differently (but still relying on POSIX functions). The one big exception is > Lustre: due its different file locking strategy we are required to use a > different collective I/O component (dynamic_gen2 vs. vulcan). Generic ufs > would work on Lustre, too, but it would be horribly slow. > > I cannot comment on CephFS and pNFS since I do not have access to those file > systems, it would come down to test them. > > Thanks > Edgar > > > -----Original Message----- > From: users <users-boun...@lists.open-mpi.org> On Behalf Of Eric > Chamberland via users > Sent: Thursday, September 23, 2021 9:28 AM > To: Open MPI Users <users@lists.open-mpi.org> > Cc: Eric Chamberland <eric.chamberl...@giref.ulaval.ca>; Vivien > Clauzon <vivien.clau...@michelin.com> > Subject: [OMPI users] Status of pNFS, CephFS and MPI I/O > > Hi, > > I am looking around for information about parallel filesystems supported for > MPI I/O. > > Clearly, GFPS, Lustre are fully supported, but what about others? > > - CephFS > > - pNFS > > - Other? > > when I "grep" for "pnfs\|cephfs" into ompi source code, I found nothing... > > Otherwise I found this into ompi/mca/common/ompio/common_ompio.h : > > enum ompio_fs_type > { > NONE = 0, > UFS = 1, > PVFS2 = 2, > LUSTRE = 3, > PLFS = 4, > IME = 5, > GPFS = 6 > }; > > Does that mean that other fs types (pNFS, CephFS) does not need special > treatment or are not supported or not optimally supported? > > Thanks, > > Eric > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42 > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42