Re: [deal.II] Visualizing higher-order cells in 3D (step-67)

['David' via deal.II User Group]

Hi Daniel,

the problem here is independent of any data set. So, you can also see it in 
paraView in the 'solid color' block. It seems to be an artificial curvature 
within the element. 
To my best knowledge it is not required to update any geometry related data.
What do you think?

Regards,
David

d.arnd...@gmail.com schrieb am Freitag, 12. Februar 2021 um 23:09:59 UTC+1:

> David,
>
> For me the most likely suspect without looking into the code at all is a 
> missing update_ghost_values.
>
> Best,
> Daniel
>
> Am Di., 9. Feb. 2021 um 17:04 Uhr schrieb Alexander :
>
>> David
>> i believe that in order to proceed, one will have to simplify this 
>> further.
>>
>> What is the minimum number of processes this happens? Can you reproduce 
>> it for 2 procs? 
>>
>> Additionally, can the mesh size be reduced to a minimum (ideally a 
>> handful of cells, say 4-8)?
>>
>> Alexander
>>
>> On Monday, February 8, 2021 at 10:20:35 AM UTC+1 daschn...@googlemail.com 
>> wrote:
>>
> Hi Wolfgang,
>>>
>>> > David -- can you be more explicit about what "a bit odd" means in 
>>> this case? 
>>> > What are we looking at, and why do you think this is wrong? 
>>>
>>> Certainly. We are looking at solution_000.vtu (zx-plane) of step-67, 
>>> where I changed the testcase to 1 (cylinder in channel) and the 
>>> dimension to 3D, 
>>> 
>>>  
>>> computed on 4 cores.
>>>
>>> What I mean by 'a bit odd' are in particular the wrinkles on the 
>>> surface:
>>> [image: close-up.png]
>>>
>>>
>>> Running the same case in serial looks as follows.
>>> [image: serial.png]
>>>
>>> The surface is smooth as usual. I would expect the visualization of the 
>>> serial and the parallel case to be the same.
>>>
>>> Best regards,
>>> David
>>> Wolfgang Bangerth schrieb am Montag, 8. Februar 2021 um 05:12:42 UTC+1:
>>>
 On 2/7/21 2:12 AM, 'David' via deal.II User Group wrote: 
 > , 
 > 
 > I'm running a 3D case using the 'write-higher-order-cells' flag and 
 the 
 > 'write-vtu-in-parallel' function. The output writing is quite similar 
 to the 
 > way step-67 handles it. However, the output of my 3D data sets looks 
 a bit odd 
 > when running the case in parallel. 

 David -- can you be more explicit about what "a bit odd" means in this 
 case? 
 What are we looking at, and why do you think this is wrong? 

 Best 
 W. 

 -- 
  

 Wolfgang Bangerth email: bang...@colostate.edu 
 www: http://www.math.colostate.edu/~bangerth/ 

 -- 
>> The deal.II project is located at http://www.dealii.org/
>> For mailing list/forum options, see 
>> https://groups.google.com/d/forum/dealii?hl=en
>> --- 
>>
> You received this message because you are subscribed to the Google Groups 
>> "deal.II User Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to dealii+un...@googlegroups.com.
>
>
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/dealii/aaf0032a-63d8-4709-b913-d0932809f3b3n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/0ed9d893-294b-405b-9ab9-a712438e657cn%40googlegroups.com.

Re: [deal.II] Understanding loops in matrix-free

[peter rum]

What Martin is trying to say (correct me if I am wrong) is that you cannot 
rely on any ordering of the call of the cell/inner-face/boundary-face 
functions, since they might be called (synonym: scheduled) in parallel in 
the case of threading. That means that the code snippet you have posted (if 
it is run by multiple threads) might give you nondeterministic results 
(since you have a single counter - i.e. you have side effect). However, 
what you - as user of MatrixFree - can rely on is the face-index (range). 
Your code would look like the following:
```
for (unsigned int face = face_range.first; face < face_range.second; ++face)
{
   my_ordered_boundary_data[face*n_q_points+q]
}
```
in the assumption, that you are accessing the quadrature point data. Of 
course you need to take care of the vectorization and as a consequence have 
to scale the term by VectorizedArray::size() but that is only technical. 

You might want to take look at our two-phase solver 
(https://github.com/kronbichler/adaflo/blob/593b6597a3b1edb5b03903f575411051f5c021a2/include/adaflo/navier_stokes_matrix.h#L281-L291).
 
Here we are doing something similar, but in the context of setting and 
getting densities and viscosities at cell quadrature points.

Hope this helps,
Peter (btw I like that you use lambdas ;) )


On Thursday, 18 February 2021 at 14:35:52 UTC+1 daschn...@googlemail.com 
wrote:

> Hi Martin,
>
> thank you for your reply.
> before going into detail and providing more information let me ask again 
> in order to avoid a misunderstanding. You say
>
> > apart from when exactly the cells and faces are scheduled, i.e., 
> possibly a change in the order of roundoff errors.
>
> As I said, my boundary condition is in an ordered vector (according to the 
> face loop) and in my 'local_apply_boundary_face' I explicitly iterate over 
> the boundary data. In pseudo code, it would look like this:
>
> int iterator = 0
> for (boundary_face_loop)
> {
>  // apply boundary condition contained in
>  my_ordered_boundary_data[ iterator ]
> ++ iterator;
> }
>
> If I change now the order (when faces are scheduled), I would of course 
> apply different data to different faces. I'm not sure if I exactly 
> understand the meaning of 'when exactly cells and faces are scheduled'. 
> But I assume the implementation/ordering is not a problem.
> I have another idea how to check it and will come back to this topic after 
> investigating.
>
> Regards,
> David
>
> Martin Kronbichler schrieb am Samstag, 13. Februar 2021 um 15:53:11 UTC+1:
>
>> Dear David,
>>
>> Indeed the two approaches should give the same answer; apart from when 
>> exactly the cells and faces are scheduled, i.e., possibly a change in the 
>> order of roundoff errors. Thus, I suspect there is something strange going 
>> on and could be a bug either in deal.II or in how you set up some 
>> particular ingredients. Can you share the code or explain a bit what 
>> exactly is used? We have continuous elements, but which degree? Is it a 
>> system? How many cells? Are you using threads, i.e., is any of the task 
>> parallel settings turned on? From what you say I believe you are running in 
>> serial, right? So it seems a pretty small problem, which we should be able 
>> to investigate rapidly.
>>
>> One thing you could do is to look into the "face_range" that you obtain 
>> when the algorithm calls back into local_apply_boundary_face and compare 
>> that with the range that you manually construct in your first version? I 
>> wonder if there are some parts of the loop we are missing or running twice.
>>
>> Best,
>> Martin
>> On 06.02.21 19:18, 'David' via deal.II User Group wrote:
>>
>>
>> Sorry for messing up the topic. I should be Understanding loops in 
>> matrix-free. I wanted to insert a figure of the source code rather than the 
>> google groups formatting and it didn't work for some reason.
>> David schrieb am Samstag, 6. Februar 2021 um 19:15:07 UTC+1:
>>
>>> Hi there, 
>>>
>>> I'm currently trying to pack my cell operations into one of the 
>>> matrix-free loop functions.
>>> In my first version, I implemented the loops manually by using (sorry 
>>> for the odd formatting):
>>> ```
>>> local_apply_cell(*matrix_free,
>>>  system_rhs,
>>>  source_vector,
>>>  std::make_pair(0,
>>> 
>>> matrix_free->n_cell_batches()));
>>> local_apply_boundary_face(
>>>   *matrix_free,
>>>   system_rhs,
>>>   source_vector,
>>>   std::make_pair(mf_data_reference->n_inner_face_batches(),
>>>  mf_data_reference->n_inner_face_batches() +
>>>
>>> mf_data_reference->n_boundary_face_batches()));
>>> ```
>>> which works as desired. I don't have any operations on internal faces 
>>> and ghost exchange/compression doesn't matter for the rest of the question

Re: [deal.II] Tags and Communicators

[Timo Heister]

I see additional problems with separate communicators per object:
1. For me it is unclear how operations that involve more than one
object should communicate. For example, a mat-vec has 2 vectors (src,
dst) and a matrix and as such 3 communicators. Of course you can make
an arbitrary choice here, but this doesn't seem clean to me.
2. MPI_comm_dup can involve communication and as such can be slow for
large processor counts. Not sure you want to always pay the price for
that. I remember at least one MPI implementation that does an
allreduce (or something worse than that).
3. The number of communicators can be quite limited, see [1]

[1] see https://www.mcs.anl.gov/~thakur/papers/mpi-million.pdf

On Thu, Feb 18, 2021 at 12:19 PM Wolfgang Bangerth
 wrote:
>
>
> > One fix that I have found (PETSc does this) is to assign every object its
> > own duplicated communicator which can then keep track of its own tags with
> > MPI's own get and set attributes functions.
>
> Sometime back in the day (in the 2005-2010 range) I spent an inordinate amount
> of time going through every class that receives a communicator in some way or
> other in the constructor or a reinit() call, and had that class duplicate the
> communicator in the way you mention (Utilities::MPI::duplicate_communicator()
> does that). You can probably still find those patches if you look long enough.
>
> The reason I did this was primarily because I wanted to have one added layer
> of security. If for some reason one process does not participate in a
> collective communication, you should get a deadlock, whereas you would either
> get funny errors or just completely unreliable results if that process
> proceeds to the next communication on the same communicator. Right now, in
> practice, all communication happens on MPI_COMM_WORLD.
>
> But, after spending a good amount of time to duplicate all of these
> communicators (probably several days of work), I spent *an even larger amount
> of time* to track down what was quite likely the worst-to-find bug I have ever
> worked on. It only manifested in a crash after a program had been running for
> many hours, and small changes to the program made the bug move to unrelated
> pieces of the code but not make it appear any earlier in the run time of the
> program.
>
> In the end, what it boiled down is that the MPI implementation I was using was
> only able to provide 64k different communicators and if you asked for the
> 64k+1'st communicator, it just crashed. In programs that do thousands of time
> steps and allocate a few vectors for temp operations in each time step, you'd
> get there in a matter of hours.
>
> I had sort of expected that MPI implementations recycle released
> communicators, and I would expect that that's what they do today, but they
> didn't back then. So after all of this time spent, I ripped out the
> duplication of communicators again. You can probably also find that patch in
> the repository and, with some luck, you might even be able to revert it if you
> allow some fuzz in indentation etc. It would certainly be interesting to try
> this again.
>
> I'm still in favor of this approach. It's conceptually the right thing, it
> would help uncover bugs, and it is *necessary* if you want to do things
> multithreaded. (All of the MPI implementations have reentrant public
> interfaces these days, but we can't use them unless you also duplicate
> communicators somewhere.) But I would first want to try with a small test
> program whether that is scalable to very large numbers of communicators -- I
> think we would quite easily run into millions of communicators with this
> approach if programs run long enough, though of course only a rather small
> number would be live at any given time.
>
> Best
>   W.
>
> --
> 
> Wolfgang Bangerth  email: bange...@colostate.edu
> www: http://www.math.colostate.edu/~bangerth/
>
> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see 
> https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups 
> "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to dealii+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/dealii/0785d886-37ad-fde9-9355-9f2a8c56c095%40colostate.edu.



-- 
Timo Heister
http://www.math.clemson.edu/~heister/

On Fri, Feb 19, 2021 at 3:54 PM Wolfgang Bangerth
 wrote:
>
> On 2/19/21 1:31 PM, Wells, David wrote:
> >
> > That's frightening and now I'm not sure how PETSc avoids this problem, and I
> > am somewhat afraid to even look.
>
> It would not greatly surprise me if MPI implementations now recycle
> communicators that have been freed.
>
> I continue to believe that having each object use its ow

Re: [deal.II] Tags and Communicators

[Wolfgang Bangerth]

On 2/19/21 1:31 PM, Wells, David wrote:


That's frightening and now I'm not sure how PETSc avoids this problem, and I 
am somewhat afraid to even look.


It would not greatly surprise me if MPI implementations now recycle 
communicators that have been freed.


I continue to believe that having each object use its own duplicated 
communicator is the way to go. That can of course be done in user code 
whenever you create these objects, but it might be useful to think about 
putting that into the library again as well.


Best
 W.

--

Wolfgang Bangerth  email: bange...@colostate.edu
   www: http://www.math.colostate.edu/~bangerth/

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/7d110bd0-7fa7-f1b3-6c71-11dffa7c1914%40colostate.edu.

Re: [deal.II] Tags and Communicators

[Wells, David]

Hi Wolfgang,

Thanks for writing this up. I think that, in my circumstance, I will be able to 
get away with duplicating communicators since the objects should exist for the 
entire program run (i.e,. I might make a few dozen instances at most) - e.g., 
each object can duplicate a communicator and then set up vectors and whatnot 
with it. I should be able to do all of this in my own code and not in deal.II 
so I don't think I'll have to dig up any old patches to the library.

> It only manifested in a crash after a program had been running for
many hours, and small changes to the program made the bug move to unrelated
pieces of the code but not make it appear any earlier in the run time of the
program.

That's frightening and now I'm not sure how PETSc avoids this problem, and I am 
somewhat afraid to even look.

Best,
David

From: dealii@googlegroups.com  on behalf of Wolfgang 
Bangerth 
Sent: Thursday, February 18, 2021 12:19 PM
To: dealii@googlegroups.com 
Subject: Re: [deal.II] Tags and Communicators


> One fix that I have found (PETSc does this) is to assign every object its
> own duplicated communicator which can then keep track of its own tags with
> MPI's own get and set attributes functions.

Sometime back in the day (in the 2005-2010 range) I spent an inordinate amount
of time going through every class that receives a communicator in some way or
other in the constructor or a reinit() call, and had that class duplicate the
communicator in the way you mention (Utilities::MPI::duplicate_communicator()
does that). You can probably still find those patches if you look long enough.

The reason I did this was primarily because I wanted to have one added layer
of security. If for some reason one process does not participate in a
collective communication, you should get a deadlock, whereas you would either
get funny errors or just completely unreliable results if that process
proceeds to the next communication on the same communicator. Right now, in
practice, all communication happens on MPI_COMM_WORLD.

But, after spending a good amount of time to duplicate all of these
communicators (probably several days of work), I spent *an even larger amount
of time* to track down what was quite likely the worst-to-find bug I have ever
worked on. It only manifested in a crash after a program had been running for
many hours, and small changes to the program made the bug move to unrelated
pieces of the code but not make it appear any earlier in the run time of the
program.

In the end, what it boiled down is that the MPI implementation I was using was
only able to provide 64k different communicators and if you asked for the
64k+1'st communicator, it just crashed. In programs that do thousands of time
steps and allocate a few vectors for temp operations in each time step, you'd
get there in a matter of hours.

I had sort of expected that MPI implementations recycle released
communicators, and I would expect that that's what they do today, but they
didn't back then. So after all of this time spent, I ripped out the
duplication of communicators again. You can probably also find that patch in
the repository and, with some luck, you might even be able to revert it if you
allow some fuzz in indentation etc. It would certainly be interesting to try
this again.

I'm still in favor of this approach. It's conceptually the right thing, it
would help uncover bugs, and it is *necessary* if you want to do things
multithreaded. (All of the MPI implementations have reentrant public
interfaces these days, but we can't use them unless you also duplicate
communicators somewhere.) But I would first want to try with a small test
program whether that is scalable to very large numbers of communicators -- I
think we would quite easily run into millions of communicators with this
approach if programs run long enough, though of course only a rather small
number would be live at any given time.

Best
  W.

--

Wolfgang Bangerth  email: bange...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/0785d886-37ad-fde9-9355-9f2a8c56c095%40colostate.edu.

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and sto

[deal.II] Re: Configuring deal.II with LAPACK

[Bruno Turcksin]

Giselle,

Instead of setting the library yourself can you try:  
-DBLAS_LIBRARY_NAMES:STRING='mkl_core;mkl_sequential'  
-DLAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64 

Don't set BLAS/LAPACK_FOUND/LIBRARIES/LINKER_FLAGS. Let CMake find the 
libraries and the flags it needs to use.

Best,

Bruno


On Thursday, February 18, 2021 at 6:46:57 PM UTC-5 gisel...@gmail.com wrote:

> Hello,
>
> I am trying to configure deal.II with LAPACK using the following command:
>
>  cmake 
> -DCMAKE_INSTALL_PREFIX=/mnt/c/Users/Giselle/Documents/dealii_install 
> /mnt/c/Users/Giselle/Documents/dealii -DDEAL_II_WITH_PETSC=ON 
> -DPETSC_DIR=$PETSC_DIR -DDEAL_II_WITH_UMFPACK=ON -DUMFPACK_DIR=$PETSC_DIR 
> -DDEAL_II_WITH_LAPACK=ON -DLAPACK_FOUND=true 
> -DLAPACK_LIBRARIES="/home/giselle/intel/mkl/lib/intel64/libmkl_blas95_lp64.a;\
>
> /home/giselle/intel/mkl/lib/intel64/libmkl_lapack95_lp64.a;/home/giselle/intel/mkl/lib/intel64/libmkl_gf_lp64.a;\
>
> /home/giselle/intel/mkl/lib/intel64/libmkl_intel_lp64.so;/home/giselle/intel/mkl/lib/intel64/libmkl_sequential.so;\
> /home/giselle/intel/mkl/lib/intel64/libmkl_core.so" 
> -DLAPACK_LINKER_FLAGS="-lgfortran -lm" -DDEAL_II_WITH_BLAS=ON 
> -DBLAS_FOUND=true 
> -DBLAS_LIBRARIES="/home/giselle/intel/mkl/lib/intel64/libmkl_blas95_lp64.a;\
>
> /home/giselle/intel/mkl/lib/intel64/libmkl_lapack95_lp64.a;/home/giselle/intel/mkl/lib/intel64/libmkl_gf_lp64.a;\
>
> /home/giselle/intel/mkl/lib/intel64/libmkl_intel_lp64.so;/home/giselle/intel/mkl/lib/intel64/libmkl_sequential.so;\
> /home/giselle/intel/mkl/lib/intel64/libmkl_core.so" 
> -DBLAS_LINKER_FLAGS="-lgfortran -lm"
>
> I am getting the following message:
>
> Could not find the lapack library!
>
>   Could not find a sufficient BLAS/LAPACK installation:
>
>   BLAS/LAPACK symbol check failed! This usually means that your BLAS/LAPACK
>   installation is incomplete or the link line is broken.
>
> Attached you will also find the CMakeError.log and CMakeOutput.log files. 
> Does anyone have any idea of what could be going on? I think it has 
> something to do with the linker flags but I have no idea and I've tried 
> everything I could think of.
>
> Thank you so much!
>
> Regards,
> Giselle
>
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/c45e31de-651c-4427-83ba-06dc862227b8n%40googlegroups.com.

Re: [deal.II] SLEPc solve PETScWrappers::MPI::SparseMatrix

[Zachary Streeter]

Oh really?  I'm going to check it out right now!  Debugging parallel 
programs is still a bane of my existence so definitely would like to hear 
your take on it!

Cheers,
Zachary

On Friday, February 19, 2021 at 1:29:43 PM UTC-6 Wolfgang Bangerth wrote:

> On 2/19/21 12:23 PM, Zachary Streeter wrote:
> > I have created a PETSc::MPI::SparseMatrix that has a global matrix
> > distributed across all the processes. I then pass it to SLEPc to solve 
> for
> > its spectrum. I get the correct spectrum so I am pretty sure everything 
> is
> > correct. My question is regarding strange behavior with more cores.
> > 
> > For a relatively small problem, my program runs fine with 1 or 2 cores.
> > But the program will hang if I try it with 3 cores.
> > 
> > If I increase the problem (make a denser grid etc.) my program may work
> > with 3 and 4 cores (which is faster) but it hangs now with 5 cores. (Note
> > this is all the test I’v done so far.)
> > 
> > Each time my program hangs on the SLEPc solve part. This leads me to 
> think
> > there are communications in the solver that are slowing everything down.
> > Though I would imagine collective communications to slow down the program
> > and not have it hang indefinitely (at least as long as my patience 
> lasted).
> > I am using SLEPcWrappers::SolverKyrlovSchur, FYI.
> > 
> > Since my program hangs on the Slepc Solver from dealii I wanted to ask
> > everyone here what they think. Have any of you seen this before?
> > 
> > This behavior is a bottleneck on the number of cores I can use and that 
> is
> > not ideal, of course. So any suggestions on how to investigate/ remedy
> > would be appreciated.
>
> Right, hanging programs are a nuisance ;-)
>
> The approach to debug deadlocks is to attach a debugger and see where 
> exactly 
> your program is stuck and why. I believe that there is a video lecture on 
> debugging parallel programs that shows how to use a debugger in such 
> contexts.
>
> Best
> W.
>
> -- 
> 
> Wolfgang Bangerth email: bang...@colostate.edu
> www: http://www.math.colostate.edu/~bangerth/
>
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/1448557b-0c2f-41cb-b400-ffcd8168d30en%40googlegroups.com.

Re: [deal.II] SLEPc solve PETScWrappers::MPI::SparseMatrix

[Wolfgang Bangerth]

On 2/19/21 12:23 PM, Zachary Streeter wrote:

I have created a PETSc::MPI::SparseMatrix that has a global matrix
distributed across all the processes.  I then pass it to SLEPc to solve for
its spectrum.  I get the correct spectrum so I am pretty sure everything is
correct.  My question is regarding strange behavior with more cores.

For a relatively small problem, my program runs fine with 1 or 2 cores.
But the program will hang if I try it with 3 cores.

If I increase the problem (make a denser grid etc.) my program may work
with 3 and 4 cores (which is faster) but it hangs now with 5 cores. (Note
this is all the test I’v done so far.)

Each time my program hangs on the SLEPc solve part.  This leads me to think
there are communications in the solver that are slowing everything down.
Though I would imagine collective communications to slow down the program
and not have it hang indefinitely (at least as long as my patience lasted).
I am using SLEPcWrappers::SolverKyrlovSchur, FYI.

Since my program hangs on the Slepc Solver from dealii I wanted to ask
everyone here what they think.  Have any of you seen this before?

This behavior is a bottleneck on the number of cores I can use and that is
not ideal, of course.  So any suggestions on how to investigate/ remedy
would be appreciated.


Right, hanging programs are a nuisance ;-)

The approach to debug deadlocks is to attach a debugger and see where exactly 
your program is stuck and why. I believe that there is a video lecture on 
debugging parallel programs that shows how to use a debugger in such contexts.


Best
 W.

--

Wolfgang Bangerth  email: bange...@colostate.edu
   www: http://www.math.colostate.edu/~bangerth/

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/38ce4c19-ff03-fab8-3e5d-4f5fc051aff3%40colostate.edu.

[deal.II] SLEPc solve PETScWrappers::MPI::SparseMatrix

[Zachary Streeter]

Hi everyone,

I have created a PETSc::MPI::SparseMatrix that has a global matrix distributed 
across all the processes.  I then pass it to SLEPc to solve for its spectrum.  
I get the correct spectrum so I am pretty sure everything is correct.  My 
question is regarding strange behavior with more cores.

For a relatively small problem, my program runs fine with 1 or 2 cores.  But 
the program will hang if I try it with 3 cores.

If I increase the problem (make a denser grid etc.) my program may work with 3 
and 4 cores (which is faster) but it hangs now with 5 cores. (Note this is all 
the test I’v done so far.)

Each time my program hangs on the SLEPc solve part.  This leads me to think 
there are communications in the solver that are slowing everything down.  
Though I would imagine collective communications to slow down the program and 
not have it hang indefinitely (at least as long as my patience lasted).  I am 
using SLEPcWrappers::SolverKyrlovSchur, FYI.

Since my program hangs on the Slepc Solver from dealii I wanted to ask everyone 
here what they think.  Have any of you seen this before?

This behavior is a bottleneck on the number of cores I can use and that is not 
ideal, of course.  So any suggestions on how to investigate/ remedy would be 
appreciated.

Thank you,

Zachary

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/CD769B03-304C-4D41-B84C-33476CF8E948%40gmail.com.

[deal.II] Re: Need help installing deal.ii with p4est and Trilinos

[Budhyant Venepalli]

Dear Konrad,

Great! thanks for your suggestions, I will get to the bottom of it.

Regards,
Budhyant

On Friday, February 19, 2021 at 3:56:56 AM UTC-6 Konrad Simon wrote:

> Dear Budhyant,, 
>
> Hard, to say what should be the paths on your system. Dealing with all 
> these large dependencies can be quite cumbersome. I would say the way you 
> install Deal.II depends on what you want to do with it. Let me tell you how 
> I would approach this.
>
> 1. If you just want to develop a parallel application (using all the 
> packages that you mention) on your local laptop it is in my opinion usually 
> enough to use the packages that are provided by the OS package manager. 
> Simply use apt install if it is your own machine and you have super user 
> permissions. Then cmake should find these packages in the PATH and you just 
> need to tell cmake to use them through, e.g., DEAL_II_WITH_P4EST = ON etc, 
> for the build of Deal.II. This is usually sufficient for development since 
> for running large 3D problems a single machine is anyway not enough. So no 
> need to optimize a build for speed
>
> 2. If you do not have root permissions (or if you want to do some larger 
> computations on a cluster, or you don't want to change the system packages) 
> installing Deal.II using spack  is quite a nice 
> option. Spack will resolve all dependencies, optimize the builds for your 
> system and can be customized nicely (you can also link against MPI 
> libraries and other stuff provided by the target system's admin). This is 
> the option that I use.
>
> Hope that helps.
> Best regards,
> Konrad
>
>
> On Friday, February 19, 2021 at 10:07:00 AM UTC+1 venepalli...@gmail.com 
> wrote:
>
>> Dear All:
>>
>> I would appreciate it if you could please help me with the installation 
>> on Ubuntu 20. 
>>
>> I have installed p4est, petsc, Trilinos and metis in "~/src/", they are 
>> working individually. 
>>
>> Cmake succeeds in recognizing petsc and metis, however, it fails to 
>> recognize the p4est and Trilinos installations. 
>>
>> *cmake -DDEAL_II_WITH_MPI=ON -DP4EST_DIR=~/src/p4est/build-mpi/ 
>> -DPETSC_DIR=~/src/petsc-3.14.4/ -DPETSC_ARCH=amd64 
>> -DTRILINOS_DIR=~/src/Trilinos/MY_BUILD/ -DMETIS_DIR=~/src/metis-5.1.0/ 
>> -DCMAKE_INSTALL_PREFIX=~/src/dealii-9.2.0/ ..*
>>
>> What should the path be in my case?
>>
>> Thank you.
>>
>> Regards,
>> Budhyant Venepalli
>>
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/10e85c3b-53a0-42b8-b874-9b00649cdaean%40googlegroups.com.

[deal.II] Re: Need help installing deal.ii with p4est and Trilinos

[Konrad Simon]

Dear Budhyant,, 

Hard, to say what should be the paths on your system. Dealing with all 
these large dependencies can be quite cumbersome. I would say the way you 
install Deal.II depends on what you want to do with it. Let me tell you how 
I would approach this.

1. If you just want to develop a parallel application (using all the 
packages that you mention) on your local laptop it is in my opinion usually 
enough to use the packages that are provided by the OS package manager. 
Simply use apt install if it is your own machine and you have super user 
permissions. Then cmake should find these packages in the PATH and you just 
need to tell cmake to use them through, e.g., DEAL_II_WITH_P4EST = ON etc, 
for the build of Deal.II. This is usually sufficient for development since 
for running large 3D problems a single machine is anyway not enough. So no 
need to optimize a build for speed

2. If you do not have root permissions (or if you want to do some larger 
computations on a cluster, or you don't want to change the system packages) 
installing Deal.II using spack  is quite a nice option. 
Spack will resolve all dependencies, optimize the builds for your system 
and can be customized nicely (you can also link against MPI libraries and 
other stuff provided by the target system's admin). This is the option that 
I use.

Hope that helps.
Best regards,
Konrad

On Friday, February 19, 2021 at 10:07:00 AM UTC+1 venepalli...@gmail.com 
wrote:

> Dear All:
>
> I would appreciate it if you could please help me with the installation on 
> Ubuntu 20. 
>
> I have installed p4est, petsc, Trilinos and metis in "~/src/", they are 
> working individually. 
>
> Cmake succeeds in recognizing petsc and metis, however, it fails to 
> recognize the p4est and Trilinos installations. 
>
> *cmake -DDEAL_II_WITH_MPI=ON -DP4EST_DIR=~/src/p4est/build-mpi/ 
> -DPETSC_DIR=~/src/petsc-3.14.4/ -DPETSC_ARCH=amd64 
> -DTRILINOS_DIR=~/src/Trilinos/MY_BUILD/ -DMETIS_DIR=~/src/metis-5.1.0/ 
> -DCMAKE_INSTALL_PREFIX=~/src/dealii-9.2.0/ ..*
>
> What should the path be in my case?
>
> Thank you.
>
> Regards,
> Budhyant Venepalli
>

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/af5964fe-cfb1-4be7-a679-0235cdbf2b08n%40googlegroups.com.

[deal.II] Need help installing deal.ii with p4est and Trilinos

[Budhyant Venepalli]

Dear All:

I would appreciate it if you could please help me with the installation on 
Ubuntu 20. 

I have installed p4est, petsc, Trilinos and metis in "~/src/", they are 
working individually. 

Cmake succeeds in recognizing petsc and metis, however, it fails to 
recognize the p4est and Trilinos installations. 

*cmake -DDEAL_II_WITH_MPI=ON -DP4EST_DIR=~/src/p4est/build-mpi/ 
-DPETSC_DIR=~/src/petsc-3.14.4/ -DPETSC_ARCH=amd64 
-DTRILINOS_DIR=~/src/Trilinos/MY_BUILD/ -DMETIS_DIR=~/src/metis-5.1.0/ 
-DCMAKE_INSTALL_PREFIX=~/src/dealii-9.2.0/ ..*

What should the path be in my case?

Thank you.

Regards,
Budhyant Venepalli

*###*
*#*
*#  deal.II configuration:*
*#CMAKE_BUILD_TYPE:   DebugRelease*
*#BUILD_SHARED_LIBS:  ON*
*#CMAKE_INSTALL_PREFIX:   /home/venepallib/src/dealii-9.2.0*
*#CMAKE_SOURCE_DIR:   /home/venepallib/src/dealii-9.2.0*
*#(version 9.2.0)*
*#CMAKE_BINARY_DIR:   /home/venepallib/src/dealii-9.2.0/build*
*#CMAKE_CXX_COMPILER: GNU 9.3.0 on platform Linux x86_64*
*#/usr/bin/c++*
*#*
*#  Configured Features (DEAL_II_ALLOW_BUNDLED = ON, 
DEAL_II_ALLOW_AUTODETECTION = ON):*
*#  ( DEAL_II_WITH_64BIT_INDICES = OFF )*
*#  ( DEAL_II_WITH_ADOLC = OFF )*
*#  ( DEAL_II_WITH_ARPACK = OFF )*
*#  ( DEAL_II_WITH_ASSIMP = OFF )*
*#DEAL_II_WITH_BOOST set up with external dependencies*
*#DEAL_II_WITH_COMPLEX_VALUES = ON*
*#  ( DEAL_II_WITH_CUDA = OFF )*
*#DEAL_II_WITH_CXX14 = ON*
*#DEAL_II_WITH_CXX17 = ON*
*#  ( DEAL_II_WITH_GINKGO = OFF )*
*#DEAL_II_WITH_GMSH set up with external dependencies*
*#  ( DEAL_II_WITH_GSL = OFF )*
*#  ( DEAL_II_WITH_HDF5 = OFF )*
*#DEAL_II_WITH_LAPACK set up with external dependencies*
*#DEAL_II_WITH_METIS set up with external dependencies*
*#DEAL_II_WITH_MPI set up with external dependencies*
*#DEAL_II_WITH_MUPARSER set up with bundled packages*
*#  ( DEAL_II_WITH_NANOFLANN = OFF )*
*#  ( DEAL_II_WITH_NETCDF = OFF )*
*#  ( DEAL_II_WITH_OPENCASCADE = OFF )*
*#  ( DEAL_II_WITH_P4EST = OFF )*
*#DEAL_II_WITH_PETSC set up with external dependencies*
*#DEAL_II_WITH_SCALAPACK set up with external dependencies*
*#  ( DEAL_II_WITH_SLEPC = OFF )*
*#DEAL_II_WITH_SUNDIALS set up with external dependencies*
*#  ( DEAL_II_WITH_SYMENGINE = OFF )*
*#DEAL_II_WITH_THREADS set up with bundled packages*
*#  ( DEAL_II_WITH_TRILINOS = OFF )*
*#DEAL_II_WITH_UMFPACK set up with external dependencies*
*#DEAL_II_WITH_ZLIB set up with external dependencies*
*#*
*#  Component configuration:*
*#  ( DEAL_II_COMPONENT_DOCUMENTATION = OFF )*
*#DEAL_II_COMPONENT_EXAMPLES*
*#  ( DEAL_II_COMPONENT_PACKAGE = OFF )*
*#  ( DEAL_II_COMPONENT_PYTHON_BINDINGS = OFF )*
*#*
*#  Detailed information (compiler flags, feature configuration) can be 
found in detailed.log*
*#*
*#  Run  $ make info  to print a help message with a list of top level 
targets*
*#*
*###*
*-- Configuring incomplete, errors occurred!*
*See also 
"/home/venepallib/src/dealii-9.2.0/build/CMakeFiles/CMakeOutput.log".*
*See also 
"/home/venepallib/src/dealii-9.2.0/build/CMakeFiles/CMakeError.log".*

-- 
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see 
https://groups.google.com/d/forum/dealii?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dealii+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dealii/06b37693-2427-4928-8438-5006100304ban%40googlegroups.com.