Re: [Libmesh-users] question about distributed mesh

Hi Manav, Extra communication can be significant. It's been awhile (~2 years) since I've run a large problem at scale on DistributedMesh but at that time it was about 30-40% slower than the equivalent problem on ReplicatedMesh. Thinking back, those numbers seem really bad and I wouldn't expect it

Re: [Libmesh-users] question about distributed mesh

Thanks for your comments, Derek. We have a few different machines here, all with about 2GB per compute core. One of the machines has 12 cores per node, and another has 20 cores per node. Why do you say that run time will be saved with ReplicatedMesh? Is this due to the reduced MPI communicati

Re: [Libmesh-users] question about distributed mesh

A little late to the party - but we use all of the options you listed. However: the one I would recommend is using ReplicatedMesh and just use less cores per node in your cluster so that it fits. It will save you an immense amount of development time... and most likely a lot of run time as well.

Re: [Libmesh-users] question about distributed mesh

On Fri, 12 Aug 2016, Manav Bhatia wrote:   The following if block seems to want a serial vector. I did try passing the System::current_local_solution to the mesh function, but it still resets the element to nullptr.  Yell at me about this again next week? I'm almost to the point where I can

Re: [Libmesh-users] question about distributed mesh

David, The following if block seems to want a serial vector. I did try passing the System::current_local_solution to the mesh function, but it still resets the element to nullptr. -Manav // If we have an element, but it's not a local element, then we // either need to have a serialized

Re: [Libmesh-users] question about distributed mesh

On Thu, Aug 11, 2016 at 7:10 PM, Manav Bhatia wrote: > I got this to work!!! > > The error was that I was not using a serialized vector to initialize the > mesh function. So, even though the point locator was finding the elem, the > mesh function was setting it to null due to the vector not being

Re: [Libmesh-users] question about distributed mesh

I got this to work!!! The error was that I was not using a serialized vector to initialize the mesh function. So, even though the point locator was finding the elem, the mesh function was setting it to null due to the vector not being serial. Now that this is working, I will try to get my ot

Re: [Libmesh-users] question about distributed mesh

This is without any refinement, so all elements should be active. Right? In order to find the elements for ghosting, I do the following: — create mesh with build_cube — serialize mesh with MeshSerializer — iterate over all elements and mark specific elements for ghosting — call prepare_for_use

Re: [Libmesh-users] question about distributed mesh

On Thu, 11 Aug 2016, Manav Bhatia wrote: > However, the point locator does not seem to work with the ghost elements. This sounds like a bug, IMHO. The point locator *ought* to find any ghost element accessible, whether it's retained by the library or by user request. I can't guess what the bug

Re: [Libmesh-users] question about distributed mesh

Cody, the 2GB was per core (not per node). Sorry for the poor choice of terms earlier. Maybe I should seek guidance along a different (but related) trajectory... How do folks on this mailing list handle FSI load/displacement mapping for large mesh? I have tried several options at my end,

Re: [Libmesh-users] question about distributed mesh

Oh wow, that is low memory. Is that per core or per node? If it's the latter you can simply run fewer cores if necessary to get your calculation done using ReplicatedMesh. I'm currently running 5 million elements using ReplicatedMesh for my problem and have no plans to switch to DistributedMesh an

Re: [Libmesh-users] question about distributed mesh

About a million elements. I have tried serial mesh in the past, but that seems to be running into memory issues on our local HPC machines. We have about 2GB per node, not sure if that is enough. -Manav > On Aug 11, 2016, at 3:48 PM, Cody Permann wrote: > > No pointers (no pun intended) bu

Re: [Libmesh-users] question about distributed mesh

No pointers (no pun intended) but how big is your mesh? Just curious. On Thu, Aug 11, 2016 at 2:47 PM Manav Bhatia wrote: > So, I did manage to get the combination of > DistributedMesh::add_extra_ghost_elem() and DofMap::augment_send_list() to > work, but it ended up not giving me the result I w

Re: [Libmesh-users] question about distributed mesh

So, I did manage to get the combination of DistributedMesh::add_extra_ghost_elem() and DofMap::augment_send_list() to work, but it ended up not giving me the result I was hoping. Basically, my intent is to use MeshFunction to return solution values on a specified boundary, and I want this to w

Re: [Libmesh-users] question about distributed mesh

Ok. I am trying to do this to a mesh create with build_cube, which calls the prepare_for_use method before returning. I was going to attempt to use MeshSerializer to serialize the mesh, mark the necessary elements for ghosting, then augment_send_list. Would this be acceptable? Or do the eleme

Re: [Libmesh-users] question about distributed mesh

On Thu, 11 Aug 2016, Manav Bhatia wrote: So if I understand correctly, the current implementation of DistributedMesh::add_extra_ghost_elem() only provides the G() version of your new API Not even that. It provides the G() version of the old API - you can flag current elements that should rem

Re: [Libmesh-users] question about distributed mesh

Thanks for pointing me to the discussions, Roy. So if I understand correctly, the current implementation of DistributedMesh::add_extra_ghost_elem() only provides the G() version of your new API without an “evaluable” entity? If my interest is in the E() entity, would it be best to wait for you

Re: [Libmesh-users] question about distributed mesh

On Wed, 10 Aug 2016, Manav Bhatia wrote: > will do. Thanks! If you want to chime in or just read my thoughts on the design questions, the issue where I've been documenting them is https://github.com/libMesh/libmesh/issues/1028 The first third of the design got merged in https://github.com/libMe

Re: [Libmesh-users] question about distributed mesh

will do. Thanks! -Manav > On Aug 10, 2016, at 5:43 PM, John Peterson wrote: > > > > On Wed, Aug 10, 2016 at 4:40 PM, Manav Bhatia > wrote: > Seems like DistributedMesh::add_extra_ghost_elem() does the trick. > > Roy is also in the midst of generalizing the al

Re: [Libmesh-users] question about distributed mesh

On Wed, Aug 10, 2016 at 4:40 PM, Manav Bhatia wrote: > Seems like DistributedMesh::add_extra_ghost_elem() does the trick. > Roy is also in the midst of generalizing the algorithms used to control which elements are ghosted where, so you may want to keep an eye on what he's doing... -- John --

Re: [Libmesh-users] question about distributed mesh

Seems like DistributedMesh::add_extra_ghost_elem() does the trick. -Manav > On Aug 10, 2016, at 5:17 PM, Manav Bhatia wrote: > > Hi, > > Is it possible to tell/force DistributedMesh to maintain a copy of a > selected set of elements across all processors? > > If this is currently is

[Libmesh-users] question about distributed mesh

Hi, Is it possible to tell/force DistributedMesh to maintain a copy of a selected set of elements across all processors? If this is currently is not a part of the functionality, is there a quick and dirty way of accomplishing the same? I would greatly appreciate your comments. Tha