Late followup, but people may want to know that a PETSc/OpenFOAM
interface is included in the pending OpenFOAM-v2006 release as a git
submodule:
https://develop.openfoam.com/modules/external-solver
Cheers,
/mark
On 2019-05-24 12:06, Matthew Knepley via petsc-users wrote:
On Thu, May 23, 2019 at 10:41 PM Vu Q. Do via petsc-users
<petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote:
Hi all,
Thanks for your previous suggestion, I have been able to
successfully link Petsc to OpenFOAM. I have written a simple
interface and it works quite well in serial mode, but cannot run in
parallel. I have been thinking about this problem for weeks but
couldn't solve it. So I think maybe you could give me some idea. I
describe my problem below.
My interface is just a class named "petscSolver/"/, which is used to
convert an openfoam's matrix or blocked matrix to Petsc Mat, then
solve the matrix using Petsc's solver.
To use Petsc, an Openfoam's solver need to be recompiled after
adding the following lines to make file:
EXE_INC = \
...
-I$(LIB_SRC)/petscSolver \
-I$(PETSC_DIR)/include \
-I$(PETSC_DIR)/$(PETSC_ARCH)/include
EXE_LIBS = \
...
-L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
To run an openfoam's case in parallel, first I need to discretize
the domain into subdomains (e.g. 2 subdomains ), then use the
following command:
mpirun -np 2 mySolver -parallel
(where mpirun is literally mpiexec) The simulation crashed even
before doing anything and the error message is as in the attached image.
Run in the debugger and see where it is crashing. Its possible to screw
up the MPI linking here, so that you
link OPENFOAM with one MPI and PETSc with another, or you call MPIInit()
after you call PetscInitialize(), etc.
Thanks,
Matt
I have tested and realized that the solver can run in parallel as
normal by removing the two lines:
-I$(PETSC_DIR)/include \
-I$(PETSC_DIR)/$(PETSC_ARCH)/include
But then it is clearly no longer linked to Petsc.
I would appreciate any suggestion.
Screenshot from 2019-05-24 09-22-17.png
On Thu, Apr 11, 2019 at 1:37 PM Smith, Barry F. <bsm...@mcs.anl.gov
<mailto:bsm...@mcs.anl.gov>> wrote:
Mark,
Thanks for the clarifying email. My google searches didn't
locate the rheoTool you mention nor "a PRACE project running via
CINECA (Bologna)".
It would be nice if someday OpenFOAM had (either directly
or somehow with the modules directory) an interface to the PETSc
solvers. This would allow the use of a variety of other solvers
including hypre BoomerAMG, SuperLU_Dist, MUMPS, and even the use
of PETSc/ViennaCL GPU based solvers automatically from OpenFOAM.
Unfortunately the PETSc group doesn't have the resources or
expertise to develop and support such an interface ourselves. We
would, of course, try to answer emails about PETSc usage and
bugs for such an interface.
Barry
If OpenFOAM did have such an interface one thing we could
provide is the CI infrastructure for tracking changes to PETSc
that may effect OpenFOAM. For example we could automatically
build OpenFOAM each day with the latest master of PETSc thus
immediately detecting changes that effect the interface.
> On Apr 10, 2019, at 4:55 PM, Mark Olesen
<mark.ole...@esi-group.com <mailto:mark.ole...@esi-group.com>>
wrote:
>
> The paper that Barry mentioned gives some generalities, but
probably
> won't help much. There are some PETSc/OpenFOAM interfaces in
rheoTool
> that are probably much more helpful.
>
> As Barry also rightly noted, there are some config files in
the OpenFOAM
> tree that were put in some time ago for helping with setting
up PETSc
> and OpenFOAM. Assuming that you have set the appropriate
values in the
> etc/config.sh/petsc <http://config.sh/petsc> file you will be
able to use those when using wmake.
> For running you will still need to ensure that the
LD_LIBARY_PATH is set
> correctly. For example, what some build scripts exhibit:
>
> wmake(petsc) :
> ==> Before running, verify that PETSc libraries can be found
>
> Enable in the OpenFOAM etc/bashrc, define manually or try
with the
> following (POSIX shell):
>
> eval $(foamEtcFile -sh -config petsc -- -force)
>
> ==
>
>
> There is currently a PRACE project running via CINECA
(Bologna) with
> binding in PETSc as a runtime selectable linear solver in
OpenFOAM. This
> is still at the stage of early testing and performance
benchmarking.
>
> Cheers,
> /mark
>
> On 4/10/19 6:37 PM, Smith, Barry F. via petsc-users wrote:
>>
>> We don't know much about OpenFoam but
>>
>> 1) if I do a
>>
>> git grep -i petsc
>>
>> in the
https://develop.openfoam.com/Development/OpenFOAM-plus.git
repository I see various configuration files specifically for PETSc.
>>
>> etc/config.csh/petsc etc/config.sh/petsc
<http://config.sh/petsc> wmake/scripts/have_petsc
>>
>> so it appears that OpenFOAM has the tools to be linked
against PETSc (to me the documentation on how to use them is
rather terse). Are
>> you using these? If you have trouble with them perhaps
you can ask the OpenFOAM user community how to use them.
>>
>>
>> 2) if you are editing the Make/options file directly you
can try changing
>>
>> -L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>>
>> to
>>
>> -Wl,-rpath,$(PETSC_DIR)/$(PETSC_ARCH)/lib
-L$(PETSC_DIR)/$(PETSC_ARCH)/lib -lpetsc
>>
>>
>>
>> Note also that simply including petsc.h into the OpenFoam
source code and linking against -lpetsc will not immediately
allow calling the PETSc solvers from OpenFOAM. One needs to
write all the interface code that sets up and calls the PETSc
solvers from OpenFOAM. There is a paper
https://www.researchgate.net/publication/319045499_Insertion_of_PETSc_in_the_OpenFOAM_Framework
that describes at an abstract level how they wrote code that
calls the PETSc solvers from OpenFOAM but the source code that
actually does the work does not appear to be available.
>>
>> Note that PETSc is now at version 3.11 we recommend
working with that version (unless you already have a lot of code
that calls PETSc written with a previous version of PETSc, for
that we recommend first upgrading to petsc 3.11 and then
continuing to add code).
>>
>> Barry
>>
>>
>>
>>
>>
>>> On Apr 10, 2019, at 8:23 AM, Balay, Satish via petsc-users
<petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote:
>>>
>>> Runtime error? You might have to add the path to
$PETSC_ARCH/lib in LD_LIBRARY_PATH env variable
>>> or - to your link command. If linux/gcc - the linker option
is -Wl,-rpath,$PETSC_ARCH/lib
>>>
>>> If not - send detail logs.
>>>
>>> Satish
>>>
>>> On Wed, 10 Apr 2019, Vu Do Quoc via petsc-users wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am trying to insert Petsc to OpenFOAM opensource software.
>>>> I have been successfully compiling Petsc with an available
solver in
>>>> OpenFOAM by linking it with the shared library
libpetsc.so. However, when I
>>>> call the solver to run a test case, I got an error saying
that:
>>>> "libpetsc.so cannot be found", even though the library
still exists in the
>>>> $PETSC_ARCH/lib folder.
>>>>
>>>> I have been struggling for weeks but still, have not been
able to figure it
>>>> out. Therefore I would be very grateful for any suggestion
to solve this
>>>> problem.
>>>>
>>>> Thanks in advance for your time,
>>>>
>>>> Best regards,
>>>>
>>>> Vu Do
>>>>
--
*Vu Q. Do
*------------------------------------------------
/Student of Aeronautical Engineering/
Programme de Formation d'Ingénieurs d'Excellence au Vietnam /- PFIEV/
School of Transportation Engineering/
/
Hanoi University of Science and Technology
01 Dai Co Viet Avenue, Hanoi, Vietnam
E-mail: vu.doquoch...@gmail.com <mailto:vu.doquoch...@gmail.com>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
--
Dr Mark OLESEN
Principal Engineer, ESI-OpenCFD
Engineering System International GmbH | Einsteinring 24 | 85609 Munich
Mob. +49 171 9710 149
www.openfoam.com | www.esi-group.com | mark.ole...@esi-group.com
