Blaise A Bourdin <bour...@lsu.edu> writes: > I am not sure if Exodus has a good solution here. As far as I > understand, exodus is inherently sequential, even when implemented > with HDF5 instead of netcdf. I would also worry about third party > support for exodus files using HDF5 as their storage format. Exodus > has an parallel extension called nemesis, but I can’t figure out how > how their concept of ghost point and cells works. The documentation on > this point is really unclear.
PyLith accesses Exodus files directly via the NetCDF interface rather than through the Exodus library. I don't recall if it has true parallel IO that way, but it's certainly possible. > Considering the dismal state of parallel FE formats and libraries, it > seems to me that one needs to chose between two options: > > a. scatter back to a single I/O node and use sequential I/O using the > ordering of the original (exodus) mesh. This allows reading and > writing on an arbitrary number of processors, but has potential > memory footprint and performance issues. How large a mesh can we > reasonably expect to be able to handle this way? > > b. Do “poor man” parallel I/O where each CPU does its own I/O, and This "independent" IO leads to atrocious performance on parallel file systems. You can manually aggregate to a smaller number of processes and have those write, but collective IO is simpler and more sustainable. > possibly create interface matching files à la nemesis or SILO. Maybe, > we can save enough information on the parallel layout in order to > easily write an un-partitionner as a post-processor. > > Unless one can come up with a better solution than a or b, I’d like to > start by writing a very simple ASCII viewer demonstrating the > communication involved, The VTS viewer is mode (a), but could simply be extended to support writing all the heavy data using MPI-IO collectives. > then modify it to use exodus, SILO or HDF5 format. Is SILO expressive enough for high-order, DG, H(div), etc?
pgp7QEaMITSNa.pgp
Description: PGP signature