ML with OpenMPI
The MPI standard does not specify that MPI_Comm = int and in fact OpenMPI uses a pointer value which lets the compiler do slightly more type checking. This type checking recently caused me trouble when building with-download-ml. There is a line in ml_comm.h which defines their communicator to be an int when ML_MPI is not defined. It was not immediately clear why this is not defined, but result is that the compiler chokes when building PETSc. This patch fixes the problem as long as sizeof(int) = sizeof(MPI_Comm), but this is *not* the case on x86_64 with OpenMPI. It's not clear to me how much of this is an upstream issue and how much is a configuration issue. Jed diff -r c074838b79ed src/ksp/pc/impls/ml/ml.c --- a/src/ksp/pc/impls/ml/ml.c Thu Mar 20 17:04:05 2008 -0500 +++ b/src/ksp/pc/impls/ml/ml.c Fri Mar 21 22:50:32 2008 +0100 @@ -815,7 +815,7 @@ PetscErrorCode MatWrapML_SHELL(ML_Operat MLcomm = mlmat-comm; ierr = PetscNew(Mat_MLShell,shellctx);CHKERRQ(ierr); - ierr = MatCreateShell(MLcomm-USR_comm,m,n,PETSC_DETERMINE,PETSC_DETERMINE,shellctx,newmat);CHKERRQ(ierr); + ierr = MatCreateShell((MPI_Comm)MLcomm-USR_comm,m,n,PETSC_DETERMINE,PETSC_DETERMINE,shellctx,newmat);CHKERRQ(ierr); ierr = MatShellSetOperation(*newmat,MATOP_MULT,(void(*)(void))MatMult_ML);CHKERRQ(ierr); ierr = MatShellSetOperation(*newmat,MATOP_MULT_ADD,(void(*)(void))MatMultAdd_ML);CHKERRQ(ierr); shellctx-A = *newmat; @@ -844,7 +844,7 @@ PetscErrorCode MatWrapML_MPIAIJ(ML_Opera n = mlmat-invec_leng; if (m != n) SETERRQ2(PETSC_ERR_ARG_OUTOFRANGE,m %d must equal to n %d,m,n); - ierr = MatCreate(mlmat-comm-USR_comm,A);CHKERRQ(ierr); + ierr = MatCreate((MPI_Comm)mlmat-comm-USR_comm,A);CHKERRQ(ierr); ierr = MatSetSizes(A,m,n,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr); ierr = MatSetType(A,MATMPIAIJ);CHKERRQ(ierr); ierr = PetscMalloc3(m,PetscInt,nnzA,m,PetscInt,nnzB,m,PetscInt,nnz);CHKERRQ(ierr); -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20080321/1dc56156/attachment.pgp
ML with OpenMPI
On Fri 2008-03-21 19:31, Lisandro Dalcin wrote: Mmm... I believe this is a configuration issue... if ML_MPI were defined, then ML_USR_COMM should be MPI_Comm. But the problem is perhaps on the ML side, not the PETSc side. ml_common.h #define ML_MPI if macro HAVE_MPI is defined. In turn HAVE_MPI is at ml_config.h, and that file is surelly generated by ML configure script. For some reason ML's configure failed to found MPI with the command line stuff PETSc pass to it. Look at the 'config.log' file inside the ml-5.0 dir to find what happened.
ML with OpenMPI
Give a try. When using MPICH2, PETSc just passes --with-mpi=PATH_TO_MPI and ML get it right. Perhaps ML have some trouble with OpenMPI, I've never tried. If you built OpenMPI yourself and with shared libs, do not forget to define LD_LIBRARY_PATH to point to the dir with the OpenMPI libs. If not, some configure test of ML could fail, and then MPI is assumed to be absent. On 3/21/08, Jed Brown jed at 59a2.org wrote: On Fri 2008-03-21 19:31, Lisandro Dalcin wrote: Mmm... I believe this is a configuration issue... if ML_MPI were defined, then ML_USR_COMM should be MPI_Comm. But the problem is perhaps on the ML side, not the PETSc side. ml_common.h #define ML_MPI if macro HAVE_MPI is defined. In turn HAVE_MPI is at ml_config.h, and that file is surelly generated by ML configure script. For some reason ML's configure failed to found MPI with the command line stuff PETSc pass to it. Look at the 'config.log' file inside the ml-5.0 dir to find what happened. From the ML docs, it looks like ML's configure expects --with-mpi-compilers if it is building with MPI. I modified config/PETSc/packages/ml.py to include this option and I'll let you know if it fixes the problem. Jed -- Lisandro Dalc?n --- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
petscvariables: hardwired build dir instead of install dir
Barry, I noticed that in the 'petscvariables' file, some of the config vars for external packages (XXX_INCLUDE and XXX_LIB) have hardwired the build dir as basepath, instead of the install dir given with --prefix. This smells bad, but I'm not sure. What do you think? I tend to think that every dir/file inside $PETSC_DIR should have its basepath defined in terms of $PETSC_DIR/$PETSC_ARCH/innerdir[/filename.ext]. -- Lisandro Dalc?n --- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
petscvariables: hardwired build dir instead of install dir
On Mar 22, 2008, at 8:28 AM, Lisandro Dalcin wrote: Barry, I noticed that in the 'petscvariables' file, some of the config vars for external packages (XXX_INCLUDE and XXX_LIB) have hardwired the build dir as basepath, instead of the install dir given with --prefix. This smells bad, but I'm not sure. What do you think? I tend to think that every dir/file inside $PETSC_DIR should have its basepath defined in terms of $PETSC_DIR/$PETSC_ARCH/innerdir[/filename.ext]. Lisandro, Specifically what packages? Do you mean that it lists, for example, SUPERLU_DIST_LIB = -Wl,-rpath,/Users/bsmith/Src/petsc-dev/arch-icc- superlu_dist/lib -L/Users/bsmith/Src/petsc-dev/arch-icc-superlu_dist/ lib -lsuperlu_dist_2.2 instead of SUPERLU_DIST_LIB = -Wl,-rpath,$PETSC_DIR/arch-icc-superlu_dist/lib -L $PETSC_DIR/arch-icc-superlu_dist/lib -lsuperlu_dist_2.2 or do you mean some config/PETSc/packages/*.py ignore the --prefix option, which ones? Barry -- Lisandro Dalc?n --- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
petscvariables: hardwired build dir instead of install dir
On 3/22/08, Barry Smith bsmith at mcs.anl.gov wrote:
Do you mean that it lists, for example,
SUPERLU_DIST_LIB = -Wl,-rpath,/Users/bsmith/Src/petsc-dev/arch-icc-
superlu_dist/lib -L/Users/bsmith/Src/petsc-dev/arch-icc-superlu_dist/
lib -lsuperlu_dist_2.2
instead of
SUPERLU_DIST_LIB = -Wl,-rpath,$PETSC_DIR/arch-icc-superlu_dist/lib -L
$PETSC_DIR/arch-icc-superlu_dist/lib -lsuperlu_dist_2.2
exactly that, as an example, I've just built petsc-dev, passing as
prefix the following '--prefix=/usr/local/petsc/2.3.3/linux-gnu' (yes,
I still want to have multiconfig installations of petsc in a central
location, so I add the PETSC_ARCH to the prefix)
But the 'petscvariables' file stills says, for example
ML_INCLUDE = -I. -I/usr/local/mpich2/1.0.6/include
-I/usr/local/mpich2/1.0.6/include
-I/repos/hg/petsc/petsc-dev/linux-gnu/include
The last include dir is the directory were I've built PETSc. I would
love to see that include as -I${PETSC_DIR}/${PETSC_ARCH}/include. And
now, if external packages got installed in $PETSC_DIR/$PETSC_ARCH, we
perhaps could just put nothing, as that location is always taken into
accout for PETSc itself.
The real problem: if I remove the build dir, the 'petscvariables'
point to locations that no longer exists, This is dangerous. I already
had trouble in the past with the old 'bmake' based system. Is there a
chance to 'fix' this, or I'm missing something?
--
Lisandro Dalc?n
---
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
petscvariables: hardwired build dir instead of install dir
Ok, I think I understand your concern,
On Mar 22, 2008, at 9:00 AM, Lisandro Dalcin wrote:
On 3/22/08, Barry Smith bsmith at mcs.anl.gov wrote:
Do you mean that it lists, for example,
SUPERLU_DIST_LIB = -Wl,-rpath,/Users/bsmith/Src/petsc-dev/arch-icc-
superlu_dist/lib -L/Users/bsmith/Src/petsc-dev/arch-icc-superlu_dist/
lib -lsuperlu_dist_2.2
instead of
SUPERLU_DIST_LIB = -Wl,-rpath,$PETSC_DIR/arch-icc-superlu_dist/lib -L
$PETSC_DIR/arch-icc-superlu_dist/lib -lsuperlu_dist_2.2
exactly that, as an example, I've just built petsc-dev, passing as
prefix the following '--prefix=/usr/local/petsc/2.3.3/linux-gnu' (yes,
I still want to have multiconfig installations of petsc in a central
location, so I add the PETSC_ARCH to the prefix)
But the 'petscvariables' file stills says, for example
ML_INCLUDE = -I. -I/usr/local/mpich2/1.0.6/include
-I/usr/local/mpich2/1.0.6/include
-I/repos/hg/petsc/petsc-dev/linux-gnu/include
This is because the place we want ML to install itself has be
passed in as the true path, it cannot be sent as a symbolic PETSC_DIR
since configure of the subpackage doesn't know PETSC_DIR
The last include dir is the directory were I've built PETSc. I would
love to see that include as -I${PETSC_DIR}/${PETSC_ARCH}/include. And
now, if external packages got installed in $PETSC_DIR/$PETSC_ARCH, we
perhaps could just put nothing, as that location is always taken into
accout for PETSc itself.
The real problem: if I remove the build dir, the 'petscvariables'
point to locations that no longer exists, This is dangerous. I already
had trouble in the past with the old 'bmake' based system. Is there a
chance to 'fix' this, or I'm missing something?
make intall: see the rule in makefile is suppose to replace all
uses of the
petsc-dir directory with the final prefix installed location: the lines
sed -i 's?$${PETSC_DIR}?TMP_INSTALL_DIR?g' $
{INSTALL_DIR}/conf/* ;\
sed -i s?TMP_INSTALL_DIR?${INSTALL_DIR}?g ${INSTALL_DIR}/
conf/* ;\
would replace
/repos/hg/petsc/petsc-dev/linux-gnu/include
/usr/local/petsc/2.3.3/linux-gnu/include
Now it could be that the sed is not working when you run install? Can
you do a
config/configure.py with a --prefix then the make then the make
install and see what
goes wrong?
Barry
with
--
Lisandro Dalc?n
---
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
petscvariables: hardwired build dir instead of install dir
OK, it seems my stderr were being sent I do not know were... I get the
following (BTW, I was not aware of the 'sed' trick, but I was not
seeing the error)
[root at trantor petsc-dev]# make install
Installing PETSc at /usr/local/petsc/dev/linux-gnu
sed: can't read s?${PETSC_DIR}?TMP_INSTALL_DIR?g: No such file or directory
sed: can't read s?TMP_INSTALL_DIR?/usr/local/petsc/dev/linux-gnu?g: No
such file or directory
sed: can't read s?/${PETSC_ARCH}??g: No such file or directory
making shared libraries in /usr/local/petsc/dev/linux-gnu//lib
building libpetsc.so
building libpetscvec.so
building libpetscmat.so
building libpetscdm.so
building libpetscksp.so
building libpetscsnes.so
building libpetscts.so
building libpetsccontrib.so
sh/bash: PETSC_DIR=/usr/local/petsc/dev/linux-gnu; export PETSC_DIR
sh/bash: unset PETSC_ARCH
csh/tcsh: setenv PETSC_DIR /usr/local/petsc/dev/linux-gnu
csh/tcsh: unsetenv PETSC_ARCH
Then do make test to verify correct install
And then the 'petscvariables' still have the build dir in many places.
So the 'sed' command is not working for me. I do not know way, I've
never had the time to learn regexps :-(.
BTW, I'm on a Fedora 6 box.
On 3/22/08, Barry Smith bsmith at mcs.anl.gov wrote:
Ok, I think I understand your concern,
On Mar 22, 2008, at 9:00 AM, Lisandro Dalcin wrote:
On 3/22/08, Barry Smith bsmith at mcs.anl.gov wrote:
Do you mean that it lists, for example,
SUPERLU_DIST_LIB = -Wl,-rpath,/Users/bsmith/Src/petsc-dev/arch-icc-
superlu_dist/lib -L/Users/bsmith/Src/petsc-dev/arch-icc-superlu_dist/
lib -lsuperlu_dist_2.2
instead of
SUPERLU_DIST_LIB = -Wl,-rpath,$PETSC_DIR/arch-icc-superlu_dist/lib -L
$PETSC_DIR/arch-icc-superlu_dist/lib -lsuperlu_dist_2.2
exactly that, as an example, I've just built petsc-dev, passing as
prefix the following '--prefix=/usr/local/petsc/2.3.3/linux-gnu' (yes,
I still want to have multiconfig installations of petsc in a central
location, so I add the PETSC_ARCH to the prefix)
But the 'petscvariables' file stills says, for example
ML_INCLUDE = -I. -I/usr/local/mpich2/1.0.6/include
-I/usr/local/mpich2/1.0.6/include
-I/repos/hg/petsc/petsc-dev/linux-gnu/include
This is because the place we want ML to install itself has be
passed in as the true path, it cannot be sent as a symbolic PETSC_DIR
since configure of the subpackage doesn't know PETSC_DIR
The last include dir is the directory were I've built PETSc. I would
love to see that include as -I${PETSC_DIR}/${PETSC_ARCH}/include. And
now, if external packages got installed in $PETSC_DIR/$PETSC_ARCH, we
perhaps could just put nothing, as that location is always taken into
accout for PETSc itself.
The real problem: if I remove the build dir, the 'petscvariables'
point to locations that no longer exists, This is dangerous. I already
had trouble in the past with the old 'bmake' based system. Is there a
chance to 'fix' this, or I'm missing something?
make intall: see the rule in makefile is suppose to replace all
uses of the
petsc-dir directory with the final prefix installed location: the lines
sed -i 's?$${PETSC_DIR}?TMP_INSTALL_DIR?g' $
{INSTALL_DIR}/conf/* ;\
sed -i s?TMP_INSTALL_DIR?${INSTALL_DIR}?g ${INSTALL_DIR}/
conf/* ;\
would replace
/repos/hg/petsc/petsc-dev/linux-gnu/include
/usr/local/petsc/2.3.3/linux-gnu/include
Now it could be that the sed is not working when you run install? Can
you do a
config/configure.py with a --prefix then the make then the make
install and see what
goes wrong?
Barry
with
--
Lisandro Dalc?n
---
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
--
Lisandro Dalc?n
---
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
