Re: [petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Junchao Zhang
I don't know OpenMP, but I saw these in your configure OMP_PROC_BIND = 'TRUE' OMP_PLACES = '{0:24}' Try not to do any binding and let OS freely schedule threads. --Junchao Zhang On Fri, Apr 7, 2023 at 7:17 PM Astor Piaz wrote: > Thanks for your reply Matt. > > The problem seems to be th

Re: [petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Astor Piaz
I'm sorry I meant OpenMP threads are able to spawn MKL processes On Fri, Apr 7, 2023 at 8:29 PM Dave May wrote: > > > On Fri 7. Apr 2023 at 07:06, Astor Piaz wrote: > >> Hello petsc-users, >> I am trying to use a code that is parallelized with a combination of >> OpenMP and MKL parallelisms, wh

Re: [petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Dave May
On Fri 7. Apr 2023 at 07:06, Astor Piaz wrote: > Hello petsc-users, > I am trying to use a code that is parallelized with a combination of > OpenMP and MKL parallelisms, where OpenMP threads are able to spawn MPI > processes. > Is this really the correct way to go? Would it not be more suitabl

Re: [petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Astor Piaz
Thanks for your reply Matt. The problem seems to be the MKL threads I just realized. Inside the MatShell I call: call omp_set_nested(.true.) call omp_set_dynamic(.false.) call mkl_set_dynamic(0) Then, inside the omp single thread I use: nMkl0 = mkl_set_num_threads_local(nMkl) where nMkl is se

Re: [petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Matthew Knepley
On Fri, Apr 7, 2023 at 2:26 PM Astor Piaz wrote: > Hi Matthew, Jungchau, > Thank you for your advice. The code still does not work, I give more > details about it below, I can specify more about it as you wish. > > I am implementing a spectral method resulting in a block matrix where the > off-di

Re: [petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Junchao Zhang
> OpenMP threads are able to spawn MPI processes I am curious why you have this usage. Is it because that you want a pure OpenMP code (i.e., not MPI capable) to call petsc? --Junchao Zhang On Fri, Apr 7, 2023 at 9:06 AM Astor Piaz wrote: > Hello petsc-users, > I am trying to use a code that

Re: [petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Matthew Knepley
On Fri, Apr 7, 2023 at 10:06 AM Astor Piaz wrote: > Hello petsc-users, > I am trying to use a code that is parallelized with a combination of > OpenMP and MKL parallelisms, where OpenMP threads are able to spawn MPI > processes. > I have carefully scheduled the processes such that the right amoun

[petsc-users] MPI+OpenMP+MKL

2023-04-07 Thread Astor Piaz
Hello petsc-users, I am trying to use a code that is parallelized with a combination of OpenMP and MKL parallelisms, where OpenMP threads are able to spawn MPI processes. I have carefully scheduled the processes such that the right amount is launched, at the right time. When trying to use my code i