Hi,
I'm doing research on scalable performance tools at Lawrence Livermore and my
group is very enthusiastic about switching to CMake for most of our development.
I've been trying to get a proof of concept port from autotools to CMake done
for a simple project, and things seem to work ok for our Linux systems but not
for our BlueGene/P systems. I'm trying to get a BlueGene/P platform file
together, and while there seem to be posts on this subject to this list before,
I couldn't find anything that worked. BG/P supports shared libraries with two
compiler toolchains, so this is a bit more involved than the BlueGene/L file
that's already in CMake. BG/P also has a crazier system library layout, and
I'm not sure that there is sufficient support in CMake for this or for the way
we typically install packages for the BG compute nodes.
I'm listing the issues I've discovered below. Are there ways around these that
I'm missing?
1. Bootstrap script doesn't seem to work quite right on the frontend.
Building CMake was a bit hairy on the Power6 front end node on our BlueGene
systems. The machine runs Linux and has both GNU and IBM XL toolchains for the
frontend itself. The CMake bootstrap script detected cc for the C compiler and
xlC for the C++ compiler, and linking failed on something 50% through the build
because it was passing -dynamic to xlC, which is the wrong flag. It would be
nice if CMake would prefer one or the other tool chain.
I was able to build with either if I just manually specified the compilers and
linker like this:
> env CC=gcc C++=g++ LD=ld ./bootstrap --prefix=${CMAKE_HOME}
> env CC=xlc C++=xlC LD=xlC ./bootstrap --prefix=${CMAKE_HOME}
But It would've been nice if I didn't have to do all that. Is PPC Linux not
used many other places anymore? I guess this is a sles10/ppc64 environment, so
maybe that is throwing the script off.
2. The FindMPI module's library search breaks because the the cross-compiler
name on BlueGene/P contains -l. Specifically, the GNU compute node compilers on
BlueGene/P are named like this:
> powerpc-bgp-linux-g++
And the FindMPI module claims that it can't find the '-inux-g++' library, or
something similar. I was able to fix this by modifying FindMPI's search to
look for -l only when it's followed by a space. This is a little worrisome
because I think that is a standard gnu naming convention, so other platforms
with compilers named blah-linux are going to have the same bug.
3. Finding system libraries in directories not named 'lib'
CMake cross compile support doesn't seem robust enough for platforms where
there is more than a simple system environment directory (specified with
CMAKE_FIND_ROOT_PATH) and a custom user directory where platform binaries will
go (also included in CMAKE_FIND_ROOT_PATH). All the online documentation and
the latest edition of the book seem to assume a system where there are a very
limited number of places where backend libraries might be.
To start my port, I've tried to set up a BlueGene/P GNU tool chain file like
this, based on the advice on the webpage:
> # the name of the target operating system
> set(CMAKE_SYSTEM_NAME BlueGeneP)
>
> # set the compiler
> set(CMAKE_C_COMPILER
> /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc )
> set(CMAKE_CXX_COMPILER
> /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-g++ )
>
> # set the search path for the environment coming with the compiler
> # and a directory where you can install your own compiled software
> set(CMAKE_FIND_ROOT_PATH
> /bgsys/drivers/ppcfloor/
> /bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/
> /bgsys/drivers/ppcfloor/runtime/SPI
> /bgsys/drivers/ppcfloor/comm/
> /bgsys/drivers/ppcfloor/comm/default/
> /bgsys/drivers/ppcfloor/comm/sys/
> )
>
> # adjust the default behaviour of the FIND_XXX() commands:
> # search headers and libraries in the target environment, search
> # programs in the host environment
> set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
> set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
> set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
There are three problems here.
3a. First, using this setup, FindMPI fails because the last library it needs is
in /bgsys/drivers/ppcfloor/runtime/SPI, not
/bgsys/drivers/ppcfloor/runtime/SPI/lib. CMAKE_FIND_ROOT_PATH seems to assume
that its elements are just above a lib directory, but that's not how things are
structured on BG/P (you can blame IBM). So, if I just use the root path for
searching, find_library fails for libs in runtime/SPI.
3b. I tried passing the root path elements as locations in a HINTS to
find_library command instead. Well, the first problem with this is that if you
try this with CMAKE_FIND_ROOT_PATH_MODE_LIBRARY set to ONLY as advised, it just
ignores the HINTS parameters. I then tried BOTH, and when I do that,
find_library seems to find the libraries in /bgsys/drivers/ppcfloor/runtime/SPI
correctly. However, it fails to find other libraries correctly because now it
searches system directories on the front end node in addition to the backend
library paths I've given above. The result is that other libraries (like
librt) resolve to their frontend versions when they should resolve to the
backend versions. If I build this way, I'll either get me a link error or a
nasty runtime error.
3c. The third problem I see here is that while this particular project is
self-contained, we have a lot of preinstalled libraries for the BlueGene
backend that don't live in one nice, self-contained place. The webpage for
cross compiling with CMake suggests that I have *one* directory where all my
binaries for the backend should go, and that I put this directory last in my
CMAKE_FIND_ROOT_PATH.
This isn't realistic for any of the large supercomputer installations I know
of. LLNL, Argonne, and ORNL all have prebuilt packages for the backends in a
lot of different places, and I need a mechanism to pass those locations to the
find_library command. If I could somehow use CMAKE_FIND_ROOT_PATH and HINTS at
the same time, this would be a start. I'm assuming that PATHS is also ignored
when CMAKE_FIND_ROOT_PATH_LIBRARY is set to ONLY.
Finally, is there some way to globally set NO_CMAKE_SYSTEM_PATH and
NO_SYSTEM_ENVIRONMENT_PATH for every find_library call, while setting
CMAKE_FIND_ROOT_PATH_MODE_LIBRARY to BOTH? This might get me the behavior I
want.
Please get back to me on these issues, as I'm really enjoying building with
CMake so far, despite the BG/P headaches. I'd like to help out by contributing
a decent platform file for BlueGene/P once I'm done, and hopefully I'll get my
group using CMake for most of our projects.
Thanks!
Todd Gamblin
Center for Applied Scientific Computing
Lawrence Livermore National Laboratory
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
