Re: [cmake-developers] Parallel ctest issues on the MinGW-w64/MSYS2 platform

[Bill Hoffman]

On 3/29/2016 5:34 PM, Alan W. Irwin wrote:

In any case, whatever the issue is, Arjen appears (so far) to have
found a reliable reproducer with the -j option for ctest.  So I hope
others here try the -j option for ctest on MinGW-w64/MSYS2 platforms
for their favorite projects to get a better idea of how widespread
this issue is.
I would recommend building ctest with debug.  Then running it with -j 
and then use gdb to figure out where it is stuck.   Until we know 
exactly where it is stuck, the rest is pure speculation


-Bill


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Parallel ctest issues on the MinGW-w64/MSYS2 platform

[Nils Gladitz]

On 29.03.2016 23:34, Alan W. Irwin wrote:

On 2016-03-29 22:46+0200 Nils Gladitz wrote:


On 29.03.2016 21:39, Alan W. Irwin wrote:

One of the PLplot developers, Arjen Markus, has just reported to me
that ctest -j4 for PLplot hangs on his MinGW-w64/MSYS2 platform while
ctest works fine. This issue occurred for the
mingw64/mingw-w64-x86_64-cmake 3.4.1-1 package version of ctest.exe,
and the PLplot build was done with the "MSYS Makefiles" generator. In
contrast to the MinGW-w64/MSYS2 platform case, ctest -j4 works fine
for the Linux case.

Has anyone else here had similar difficulties OR success with the
parallel ctest option on the MinGW-w64/MSYS2 platform?


Afair there are sporadic issues with the cygwin runtime (on which 
msys/msys2 is based) in the process handling code of kwsys (used by 
cmake).
The code is used by e.g. execute_process() and also ctest for test 
execution.


I think the issue is due to how pipes are implemented by the runtime.
From what I remember rsync also suffers from this.

Also having Kaspersky installed (even when paused) seems to make 
these hangs more likely.


Thanks, Nils, for that information.  Could this issue be related to
the Cygwin unreliability mentioned in
 where they say "In
summary, current Windows implementations make it impossible to
implement a perfectly reliable fork, and occasional fork failures are
inevitable."?


Afair the issue was a wait or select on the pipes used to connect to the 
spawned process rather than fork but I haven't debugged into it myself.
I think there was a cygwin mailing list post on this too but I don't 
seem to be able to find it.


Nils
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Parallel ctest issues on the MinGW-w64/MSYS2 platform

[Alan W. Irwin]

On 2016-03-29 22:46+0200 Nils Gladitz wrote:


On 29.03.2016 21:39, Alan W. Irwin wrote:

One of the PLplot developers, Arjen Markus, has just reported to me
that ctest -j4 for PLplot hangs on his MinGW-w64/MSYS2 platform while
ctest works fine. This issue occurred for the
mingw64/mingw-w64-x86_64-cmake 3.4.1-1 package version of ctest.exe,
and the PLplot build was done with the "MSYS Makefiles" generator. In
contrast to the MinGW-w64/MSYS2 platform case, ctest -j4 works fine
for the Linux case.

Has anyone else here had similar difficulties OR success with the
parallel ctest option on the MinGW-w64/MSYS2 platform?


Afair there are sporadic issues with the cygwin runtime (on which msys/msys2 
is based) in the process handling code of kwsys (used by cmake).

The code is used by e.g. execute_process() and also ctest for test execution.

I think the issue is due to how pipes are implemented by the runtime.
From what I remember rsync also suffers from this.

Also having Kaspersky installed (even when paused) seems to make these hangs 
more likely.


Thanks, Nils, for that information.  Could this issue be related to
the Cygwin unreliability mentioned in
 where they say "In
summary, current Windows implementations make it impossible to
implement a perfectly reliable fork, and occasional fork failures are
inevitable."?

In any case, whatever the issue is, Arjen appears (so far) to have
found a reliable reproducer with the -j option for ctest.  So I hope
others here try the -j option for ctest on MinGW-w64/MSYS2 platforms
for their favorite projects to get a better idea of how widespread
this issue is.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Parallel ctest issues on the MinGW-w64/MSYS2 platform

[Nils Gladitz]

On 29.03.2016 21:39, Alan W. Irwin wrote:

One of the PLplot developers, Arjen Markus, has just reported to me
that ctest -j4 for PLplot hangs on his MinGW-w64/MSYS2 platform while
ctest works fine. This issue occurred for the
mingw64/mingw-w64-x86_64-cmake 3.4.1-1 package version of ctest.exe,
and the PLplot build was done with the "MSYS Makefiles" generator. In
contrast to the MinGW-w64/MSYS2 platform case, ctest -j4 works fine
for the Linux case.

Has anyone else here had similar difficulties OR success with the
parallel ctest option on the MinGW-w64/MSYS2 platform?


Afair there are sporadic issues with the cygwin runtime (on which 
msys/msys2 is based) in the process handling code of kwsys (used by cmake).
The code is used by e.g. execute_process() and also ctest for test 
execution.


I think the issue is due to how pipes are implemented by the runtime.
From what I remember rsync also suffers from this.

Also having Kaspersky installed (even when paused) seems to make these 
hangs more likely.


Nils
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] Parallel ctest issues on the MinGW-w64/MSYS2 platform

[Alan W. Irwin]

One of the PLplot developers, Arjen Markus, has just reported to me
that ctest -j4 for PLplot hangs on his MinGW-w64/MSYS2 platform while
ctest works fine. This issue occurred for the
mingw64/mingw-w64-x86_64-cmake 3.4.1-1 package version of ctest.exe,
and the PLplot build was done with the "MSYS Makefiles" generator. In
contrast to the MinGW-w64/MSYS2 platform case, ctest -j4 works fine
for the Linux case.

Has anyone else here had similar difficulties OR success with the
parallel ctest option on the MinGW-w64/MSYS2 platform?

Previously, Arjen had discovered that the parallel build -j4 option
for make.exe (from the msys/make 4.1-4 package) was not reliable on
this platform.  I have forgotten all the details for that issue, but
we didn't pursue it further at that time because Cygwin also had the
same parallel make trouble so we wrote it off to some make.exe bug for
Cygwin that had been copied to MinGW-w64/MSYS2 platform (since the two
platforms are so closely related).

Of course, it always possible on both platforms that some make race
condition caused by a badly implemented CMake-based build system for
PLplot could be the source of these parallel make issues.  However,
make -j4 works fine for the Linux case so if there is such a make race
issue it is a subtle one. To eliminate this possibility does there
exist a definitive way to search for make race conditions for
CMake-based builds?  (I am thinking along the lines of looking for
duplicate "Building ..." lines in the output from "make all", but
there may be some better method that someone here would like to
suggest.)

Assuming make race conditions can be eliminated as the cause of the
issue with the make parallel build option, is there any obvious
connection between that issue and the parallel ctest issue on this
platform? For example, is the ctest parallel build option implemented
using generator facilities (e.g., the parallel build option for
make.exe) or do both the ctest and make implementations of their
respective parallel options use some common library that could be
screwing up on the MinGW-w64/MSYS2 platform?

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers