12.01.2018, 12:58, "Alan W. Irwin" <[email protected]>: > I am looking into the practicality of using clusters containing ~5 > cheap ARM 8-core computers (such as the Banana Pi M3) to rapidly build > and test software (since even with ccache I am currently spending a > lot of my time waiting for tests to complete as I develop my > software). Such clusters would give you ~40 cores which is a lot of > computer power for ~$500 or so. But, of course, the issue with > clusters is how to use them efficiently?
1. For compilation you often need good single-core performance more than parallelism, and I doubt that embedded-grade ARMv8 can provide it I guess exception is compiling projects written in C which don't have huge source files. Still, overhead of passing data between nodes may be substanital. 2. You should probably use icecc or distcc, unless you have MPI as a requirement. > > For one 8-core M3 you would expect > > make -j<NNN> target > > to scale linearly with NNN up to the saturation limit of roughly NNN=8 > (where target depends on ~hundreds of different and mostly independent > tests). But my understanding from recent Google searching is that the > -j option for GNU Make is not MPI-aware, i.e., only works for the > cores for a single computer (node) in a cluster. So in a cluster you > would have to assign the above task to just one node and simply accept > that you cannot scale NNN up to the total number of cores in that > cluster. > > I did find one reference to an effort (called the pmake project at > Lawrence Livermore, see <https://computing.llnl.gov/code/pmake.html>) > to modify GNU make to become MPI-aware so the above command would > roughly scale NNN up to the number of cores in the cluster. But the > single developer of the project seemed unable to overcome a bug in the > result so that parallel make project was abandoned. > > I hope someone here can point me to a make variant (does Ninja > qualify?) that does have the capability of scaling NNN up to the > number of cores in the cluster since that would make clusters with a > large number of cores become extremely useful for the purpose of quick > testing by software developers. > > Alan > __________________________ > Alan W. Irwin > > Astronomical research affiliation with Department of Physics and Astronomy, > University of Victoria (astrowww.phys.uvic.ca). > > Programming affiliations with the FreeEOS equation-of-state > implementation for stellar interiors (freeeos.sf.net); the Time > Ephemerides project (timeephem.sf.net); PLplot scientific plotting > software package (plplot.sf.net); the libLASi project > (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); > and the Linux Brochure Project (lbproject.sf.net). > __________________________ > > Linux-powered Science > __________________________ > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > https://cmake.org/mailman/listinfo/cmake -- Regards, Konstantin -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake
