Hi Gary, Do you see either of these two warning messages show up:
"warning: -jN forced in submake: disabling jobserver mode." or "warning: jobserver unavailable: using -j1. Add `+' to parent make rule." These warnings often accompany the forced serialization of a parallel make build, although usually they show up regardless of the launch method, i.e. ctests vs manual make. These warnings are often indicative of a particular problem in super-build scenarios.. - Chuck On Thu, Sep 11, 2014 at 2:29 PM, Hennigan, Gary L <glhe...@sandia.gov> wrote: > I have a strange, and very frustrating, problem. I have a pretty large > piece of software that I build nightly as part of regression testing of my > own software. All of the software uses CMake and I use a ctest script, via > “ctest –S [script file]”, for my nightly regression testing . As I stated, > this is a pretty large collection of software but during development it’s > not a huge issue because the build is quite parallelizable via GNU make’s > “-j N” option. On my nightly test platform, a 64-core machine, I can build > the whole thing in about an hour. A nice manageable amount of time for a > nightly regression test. Unfortunately when I run the build process via > ctest something is causing the parallel make to fail and I’m lucky if the > build takes under 15 hours. Barely practical for a nightly test. > > I’m not sure how to find out what’s going on. After the ctest build I can > go into the build directory, do a “make clean” and then a “make –j 12”, for > example, and the build flies. Of course I can build the software entirely > outside of ctest and it too flies. Only when the build happens as part of > ctest does it seem to revert to, essentially, a “make –j 1” and slow to a > crawl. > > I can look at the process tree, via “ps –ef”, during the ctest build and I > see the root invocation of gmake and it’s fine. For example, it typically > looks something like: > > PID PPID C STIME TTY TIME CMD > 6141 6283 96 11:41 pts/0 00:24:40 ctest -VV -S ctest_nightly.cmake > -DPROCESSORCOUNT=12 > 8032 6141 0 11:42 pts/0 00:00:00 /usr/bin/gmake -i *-j 12* > 8035 8032 0 11:42 pts/0 00:00:00 /usr/bin/gmake -f > CMakeFiles/Makefile2 all > 851 8035 0 11:52 pts/0 00:00:00 /usr/bin/gmake -f > packages/ml/src/CMakeFiles/ml.dir/build.make > 27797 8035 0 11:46 pts/0 00:00:00 /usr/bin/gmake -f > packages/moocho/src/CMakeFiles/moocho.dir/build.make > > You can see that the parent make, PID 8032 which is started via ctest (PID > 6141), has the appropriate flag, “-j 12”, but at this point in the build > it’s compiling one file at a time. Another odd thing is that I think the > build starts out fine, invoking multiple file compilations simultaneously, > but after a couple of minutes it reverts to essentially the “make –j 1” > behavior. It’s like the GNU make jobserver is failing, but I’m not getting > any error messages from GNU make to that affect. > > If anyone has any suggestions on how I can figure this out I’d appreciate > it. > > Apologies for the lengthy explanation. I’ve been pulling my hair out > trying to figure out how to solve this issue. > > Thanks in advance, > Gary > > > > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > http://public.kitware.com/mailman/listinfo/cmake >
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake