On Tue, May 7, 2013 at 8:57 PM, Henning Glawe <glawe at mpi-halle.mpg.de> wrote: > Moin, > we have recently discovered performance problems on our parallel scratch file > system (fhgfs) in a 400-node-cluster. > Much of the blame went to my quantum espresso jobs (custom-patched 4.3.2). I > am performing calculations on thousands of small crystals, most having no > symmetry, leading to a huge number of irreducable k-points. > At least in this version, QE/iotk creates one directory per k-point, with only > one small file (eigenval.xml) contained in it. > Even if I set '&system::disk_io="low"', this behaviour seems to have a major > impact on the overall performance of the file system.
have you tried using a ramdisk? if your machine is linux based, then you can usually access up to half of the physical memory via /dev/shm. given the ever increasing amount of RAM that current many-core machine have, this has become a viable alternative to modifying the source code to keep data in core. > Are there any plans to move away from the 'many-small-files' paradigm? Or is > there any way/switch to make QE use another scheme for the scratch files? whatever you do, it will require some programming effort. this can be done from inside of QE (tedious) or on the outside by using a ramdisk (as mentioned above) or faking it, e.g. by wrapping the stdio library to write to huge RAM buffers for all files matching a given pattern. cray used to provide such a library on their XT series machines to work around the same kind of problems that you are describing on their lustre scratch file systems. if there are some people here on the list that would be interested in having such a wrapper and would be volunteering some time testing and providing feedback, i might be tempted to give it a try and write such a wrapper. axel. > -- > Mit freundlichen Gr??en > Henning Glawe > > Max-Planck-Institut f?r Mikrostrukturphysik > Weinberg 2, 06120 Halle (Saale), Germany http://www.mpi-halle.de/~theory > Phone: +49-345-5582-613 Fax: +49-345-5511223 Email: glawe at mpi-halle.de > _______________________________________________ > Pw_forum mailing list > Pw_forum at pwscf.org > http://pwscf.org/mailman/listinfo/pw_forum -- Dr. Axel Kohlmeyer akohlmey at gmail.com http://goo.gl/1wk0 International Centre for Theoretical Physics, Trieste. Italy.