Folks,
I've spent a while looking at the BuildSystem code, and I think this is going
to take me more time than I have available right now to figure it out on my
own. Someone more familiar with BuildSystem needs to give me some hints --
soon, if possible, as I really think that building with non-GCC compilers and
CUDA should be supported in the upcoming release.
What I want to do is to add a test inside cuda.py that checks to see if
something like
nvcc --compiler-option=<compiler options PETSc has identified> <CUDAFLAGS>
hello.c
will return successfully.
What I wasn't sure about was how to get at the values for a bunch of the above
variables within the cuda.py code. After deciding I couldn't really follow
everything that is happening in the code buy just looking at it, I used the
'pdb' python debugger to stick a breakpoint in the configureLibrary() method in
cuda.py so I could poke around.
**** Aside: Looking at contents of configure objects? ****
I had hoped I could look at everything that is stashed in the different objects
by doing things like
(Pdb) p dir(self.compilers)
But this doesn't actually list everything in there. There is no 'CUDAC'
attribute listed, for instance, but it is there for me to print:
(Pdb) p self.compilers.CUDAC
'nvcc'
Is there a good way for me to actually see all the attributes in something like
the self.compilers object? Sorry, my Python skills are very rusty -- haven't
written much Python in about a decade.
**** End aside ****
It appears that what I need to construct my command line is then available in
self.compilers.CUDAC -- The invocation for the CUDA compiler
self.compilers.CXXFLAGS -- The flags passed to the C++ compiler (our "host")
self.compilers.CUDAFLAGS -- The flags like "-ccbin pgc++" being passed to nvcc
or whatever CUDAC is
I could use these to construct a command that I then pass to the command shell,
and maybe I should just do this, but this doesn't seem to follow the
BuildSystem paradigm. It seems like I should be able to run this test by doing
something like
self.pushLanguage('CUDA')
self.checkCompile(cuda_test)
which is, in fact, invoked in checkCUDAVersion(). But the command put together
by checkCompile() does not include "--compiler-option=<compiler options PETSc
has identified>". Should I be modifying the code the code somewhere so that
this argument goes into the compiler invocation constructed in
self.checkCompile? If so, where should I be doing this?
--Richard
On 3/22/19 10:24 PM, Mills, Richard Tran wrote:
On 3/22/19 3:28 PM, Mills, Richard Tran wrote:
On 3/22/19 12:13 PM, Balay, Satish wrote:
Is there currently an existing check like this somewhere? Or will things just
fail when running 'make' right now?
Most likely no. Its probably best to attempt the error case - and
figure-out how to add a check.
I gave things a try and verified that there is no check for this anywhere in
configure -- things just fail at 'make' time. I think that all we need is a
test that will try to compile any simple, valid C program using "nvcc
--compiler-options=<compiler options PETSc has identified> <CUDAFLAGS>". If the
test fails, it should report something like "Compiler flags do not work with
CUDA compiler; perhaps you need to provide to use -ccbin in CUDAFLAGS to
specify the intended host compiler".
I'm not sure where this test should go. Does it make sense for this to go in
cuda.py with the other checks like checkNVCCDoubleAlign()? If so, how do I get
at the values of <compiler options PETSc has identified> and <CUDAFLAGS>? I'm
not sure what modules I need to import from BuildSystem...
OK, answering part of my own question here: Re-familiarizing myself with how
the configure packages work, and then looking through the makefiles, I see that
the argument to "--compiler-options" is filled in by the makefile variables
${PCC_FLAGS} ${CFLAGS} ${CCPPFLAGS}
and it appears that this partly maps to self.compilers.CFLAGS in BuildSystem.
But so far I've not managed to employ the right combination of find and grep to
figure out where PCC_FLAGS and CCPPFLAGS come from.
--Richard
--Richard
Satish
On Fri, 22 Mar 2019, Mills, Richard Tran via petsc-dev wrote:
On 3/18/19 7:29 PM, Balay, Satish wrote:
On Tue, 19 Mar 2019, Mills, Richard Tran via petsc-dev wrote:
Colleagues,
It took me a while to get PETSc to build at all with anything on Summit other
than the GNU compilers, but, once this was accomplished, editing out the
isGNU() test and then passing something like
'--with-cuda=1',
'--with-cudac=nvcc -ccbin pgc++',
Does the following also work?
--with-cuda=1 --with-cudac=nvcc CUDAFLAGS='-ccbin pgc++'
Yes, using CUDAFLAGS as above also works, and that does seem to be a better way
to do things.
After experimenting with a lot of different builds on Summit, and doing more
reading about how CUDA compilation works on different platforms, I'm now
thinking that perhaps configure.py should *avoid* doing anything clever to try
figure out what the value of "-ccbin" should be. For one, this is not anything
that NVIDIA's toolchain does for the user in the first place: If you want to
use nvcc with a host compiler that isn't whatever NVIDIA considers the default
(g++ on Linux, clang on Mac OS, MSVC on Windows), NVIDIA expects you to provide
the appropriate '-ccbin' argument. Second, nvcc isn't the only CUDA compiler
that a user might want to use: some people use Clang directly to compile CUDA
code. Third, which host compilers are supported appears to be platform
independent; for example, GCC is the default/preferred host compiler on Linux,
but isn't even supported on Mac OS! Figuring out what is supported is very
convoluted, and I think that trying to get configure to determine this may be
more trouble than it is worth. I think we should instead let the user try
whatever, and print out a helpful message how they "may need to specify host
compiler to nvcc with -ccbin" if the CUDA compiler doesn't seem to work. Also,
I'll put something about this in the CUDA configure examples. Any objections?
Sometimes we have extra options in configure for specific features for
ex: --with-pic --with-visibility etc.
But that gets messy. On cuda side - we've have --with-cuda-arch and at
some point elimiated it [so CUDAFLAGS is now the interface for this
flag]. We could add --with-cuda-internal-compiler option to petsc
configure - but it will again have similar drawbacks. I personally
think most users will gravitate towards specifying such option via
CUDAFLAGS
to configure works fine. So, I should make a change to the BuildSystem cuda.py
along these lines. I'm wondering exactly how I should make this work. I could
just remove the check,
sure
but I think that maybe the better thing to do is to check isGNU(), then if the
compiler is *not* GNU, configure should add the appropriate '-ccbin' argument
to "--with-cudac", unless the user has specified '-ccbin' in their
'--with-cudac' already. Or do we need to get this fancy?
The check should be: do --compiler-options= constructed by PETSc configure
work with CUDAC
Is there currently an existing check like this somewhere? Or will things just
fail when running 'make' right now?
[or perhaps we should - just trim the --compiler-options to only -I flags?]
I think we should avoid explict check for a compiler type [i.e isGNU() check]
as much as possible.
CUDA is only supposed to work with certain compilers, but there doesn't seem to
be a correct official list (for instance, it supposedly won't work with the IBM
XL compilers, but they certainly *are* actually supported on Summit). Heck, the
latest GCC suite won't even work right now. Since what compilers are supported
seems to be in flux, I suggest we just let the user try anything and then let
things fail if it doesn't work.
I suspec the list is dependent on the install [for ex: linux vs Windows vs
somthing else?] and version of cuda [for ex: each version of cuda supports only
specific versions of gcc]
Yes, you are correct about this, as I detailed above.
Satish
--Richard
On 3/12/19 8:45 PM, Smith, Barry F. wrote:
Richard,
You need to remove the isGNU() test and then experiment with getting the
Nvidia tools to use the compiler you want it to use.
No one has made a serious effort to use any other compilers but Gnu (at
least not publicly).
Barry
On Mar 12, 2019, at 10:40 PM, Mills, Richard Tran via petsc-dev
<petsc-dev@mcs.anl.gov><mailto:petsc-dev@mcs.anl.gov><mailto:petsc-dev@mcs.anl.gov><mailto:petsc-dev@mcs.anl.gov><mailto:petsc-dev@mcs.anl.gov><mailto:petsc-dev@mcs.anl.gov><mailto:petsc-dev@mcs.anl.gov><mailto:petsc-dev@mcs.anl.gov>
wrote:
Fellow PETSc developers,
If I try to configure PETSc with CUDA support on the ORNL Summit system using
non-GNU compilers, I run into an error due to the following code in
packages/cuda.py:
def configureTypes(self):
import config.setCompilers
if not config.setCompilers.Configure.isGNU(self.setCompilers.CC, self.log):
raise RuntimeError('Must use GNU compilers with CUDA')
...
Is this just because this code predates support for other host compilers with
nvcc, or is there perhaps some more subtle reason that I, with my inexperience
using CUDA, don't know about? I'm guessing that I just need to add support for
using '-ccbin' appropriately to set the location of the non-GNU host compiler,
but maybe there is something that I'm missing. I poked around in the petsc-dev
mailing list archives and can find a few old threads on using non-GNU
compilers, but I'm not sure what conclusions were reached.
Best regards,
Richard
