On Thu, Sep 26, 2024 at 7:18 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
> I see, you mean:
>
> Create the ghost particles at the local cell with the same properties as
> particle 1 (duplicate the original particle) but different value
> DMSwarmField_rank. Then, call DMSwarmMigrate(*,PETSC_FALSE) so we do the
> migration and delete the local copies of the particle 1. Right?
>
Yep. I think it will work, from what I know about BASIC.
Thanks,
Matt
> Thanks,
> Miguel
>
> On Sep 26, 2024, at 11:09 PM, Matthew Knepley <knep...@gmail.com> wrote:
>
> On Thu, Sep 26, 2024 at 11:20 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
> wrote:
>
>> Thank you Matt.
>>
>> Okey, let me have a careful look to the DMSwarmMigrate_Push_Basic
>> implementation
>> to see if there is some workaround.
>>
>> The idea of adding new particles is interesting. However, in that case,
>> we need to initialize the new (ghost) particles using the fields of the
>> “real” particle, right? This can be done using something like:
>>
>> VecGhostUpdateBegin
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateBegin/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpmGXFVK-$
>> >(Vec
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpjFc2Mxy$
>> > globalout,InsertMode
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpmLgCZH7$
>> > ADD_VALUES
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpurYQI7-$
>> >, ScatterMode
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpk7OXKyZ$
>> > SCATTER_REVERSE
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpm4lnFqo$
>> >);VecGhostUpdateEnd
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/VecGhostUpdateEnd/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpuKpVNR3$
>> >(Vec
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/Vec/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpjFc2Mxy$
>> > globalout,InsertMode
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/InsertMode/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpmLgCZH7$
>> > ADD_VALUES
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/ADD_VALUES/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpurYQI7-$
>> >, ScatterMode
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/ScatterMode/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpk7OXKyZ$
>> > SCATTER_REVERSE
>> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Vec/SCATTER_REVERSE/__;!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpm4lnFqo$
>> >);
>>
>> for the particle fields (?).
>>
>
> I think we can just copy from the local particle. For example, suppose I
> decide that particle 1 should go to rank 5, 12, and 27. Then
> I first set p1.rank = 5, then I add two new particles with the same values
> as particle 1, but with rank = 12 and 27. Then when I call migrate, it will
> move these three particles to the correct processes, and delete the
> original particles and the copies from the local set.
>
> Thanks,
>
> Matt
>
>
>> Thanks,
>> Miguel
>>
>>
>> On Sep 26, 2024, at 3:53 PM, Matthew Knepley <knep...@gmail.com> wrote:
>>
>> On Thu, Sep 26, 2024 at 6:31 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>> wrote:
>>
>>> Hi Matt et al,
>>>
>>> I’ve been working on the scheme that you proposed to create ghost
>>> particles (atoms in my case), and it works! With a couple of caveats:
>>> -1º In general the overlap particles will be migrate from their own rank
>>> to more than one neighbor rank, this is specially relevant for those
>>> located close to the corners. Therefore, you'll need to call DMSwarmMigrate
>>> several times (27 times for 3D cells), during the migration process.
>>>
>>
>> That is terrible. Let's just fix DMSwarmMigrate to have a mode that sends
>> the particle to all overlapping neighbors at once. It can't be that hard.
>>
>>
>>> -2º You need to set DMSWARM_MIGRATE_BASIC. Otherwise the proposed
>>> algorithm will not work at all!
>>>
>>
>> Oh, I should have thought of that. Sorry.
>>
>> I can help code up that extension. Can you take a quick look at the BASIC
>> code? Right now, we just use the rank attached to the particle
>> to send it. We could have an arrays of ranks, but that seems crazy, and
>> would blow up particle storage. How about just adding new particles
>> with the other ranks right before migration?
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> Hope this helps to other folks!
>>>
>>> I have a follow-up question about periodic bcc on this context, should I
>>> open a new thread of keep posting here?
>>>
>>> Thanks,
>>> Miguel
>>>
>>> On Aug 7, 2024, at 4:22 AM, MIGUEL MOLINOS PEREZ <mmoli...@us.es> wrote:
>>>
>>> Thanks Matt, I think I'll start by making a small program as a proof of
>>> concept. Then, if it works I'll implement it in my code and I'll be happy
>>> to share it too :-)
>>>
>>> Miguel
>>>
>>> On Aug 4, 2024, at 3:30 AM, Matthew Knepley <knep...@gmail.com> wrote:
>>>
>>> On Fri, Aug 2, 2024 at 7:15 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>> wrote:
>>>
>>>> Thanks again Matt, that makes a lot more sense !!
>>>>
>>>> Just to check that we are on the same page. You are saying:
>>>>
>>>> 1. create a field define a field called "owner rank" for each particle.
>>>>
>>>> 2. Identify the phantom particles and modify the internal variable
>>>> defined by the DMSwarmField_rank variable.
>>>>
>>>> 3. Call DMSwarmMigrate(*,PETSC_FALSE), do the calculations using the
>>>> new local vector including the ghost particles.
>>>>
>>>> 4. Then, once the calculations are done, rename the DMSwarmField_rank
>>>> variable using the "owner rank" variable and call
>>>> DMSwarmMigrate(*,PETSC_FALSE) once again.
>>>>
>>>
>>> I don't think we need this last step. We can just remove those ghost
>>> particles for the next step I think.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> Thank you,
>>>> Miguel
>>>>
>>>>
>>>> On Aug 2, 2024, at 5:33 PM, Matthew Knepley <knep...@gmail.com> wrote:
>>>>
>>>> On Fri, Aug 2, 2024 at 11:15 AM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>> wrote:
>>>>
>>>>> Thank you Matt for your time,
>>>>>
>>>>> What you describe seems to me the ideal approach.
>>>>>
>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned
>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST
>>>>>
>>>>> This means, locally, I need to allocate Nlocal + ghost particles
>>>>> (duplicated) for my model?
>>>>>
>>>>
>>>> I would do it another way. I would allocate the particles with no
>>>> overlap and set them up. Then I would identify the halo particles, mark
>>>> them as OVERLAP, call DMSwarmMigrate(), and mark the migrated particles as
>>>> GHOST, then unmark the OVERLAP particles. Shoot! That marking will not work
>>>> since we cannot tell the difference between particles we received and
>>>> particles we sent. Okay, instead of the `ghost` field we need an `owner
>>>> rank` field. So then we
>>>>
>>>> 1) Setup the non-overlapping particles
>>>>
>>>> 2) Identify the halo particles
>>>>
>>>> 3) Change the `rank`, but not the `owner rank`
>>>>
>>>> 4) Call DMSwarmMigrate()
>>>>
>>>> Now we can identify ghost particles by the `owner rank`
>>>>
>>>>
>>>>> If that so, how to do the communication between the ghost particles
>>>>> living in the rank i and their “real” counterpart in the rank j.
>>>>>
>>>>> Algo, as an alternative, what about:
>>>>> 1) Use an IS tag which contains, for each rank, a list of the global
>>>>> index of the neighbors particles outside of the rank.
>>>>> 2) Use VecCreateGhost to create a new vector which contains extra
>>>>> local space for the ghost components of the vector.
>>>>> 3) Use VecScatterCreate, VecScatterBegin, and VecScatterEnd to do the
>>>>> transference of data between a vector obtained with
>>>>> DMSwarmCreateGlobalVectorFromField
>>>>> 4) Do necessary computations using the vectors created with
>>>>> VecCreateGhost.
>>>>>
>>>>
>>>> This is essentially what Migrate() does. I was trying to reuse the code.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Miguel
>>>>>
>>>>> On Aug 2, 2024, at 8:58 AM, Matthew Knepley <knep...@gmail.com> wrote:
>>>>>
>>>>> On Thu, Aug 1, 2024 at 4:40 PM MIGUEL MOLINOS PEREZ <mmoli...@us.es>
>>>>> wrote:
>>>>>
>>>>>> This Message Is From an External Sender
>>>>>> This message came from outside your organization.
>>>>>>
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I am implementing a Molecular Dynamics (MD) code using the DMSWARM
>>>>>> interface. In the MD simulations we evaluate on each particle (atoms)
>>>>>> some kind of scalar functional using data from the neighbouring atoms.
>>>>>> My problem lies in the parallel implementation of the model, because
>>>>>> sometimes, some of these neighbours lie on a different processor.
>>>>>>
>>>>>> This is usually solved by using ghost particles. A similar approach
>>>>>> (with nodes instead) is already implemented for other PETSc mesh
>>>>>> structures like DMPlexConstructGhostCells. Unfortunately, I don't see
>>>>>> this kind of constructs for DMSWARM. Am I missing something?
>>>>>>
>>>>>> I this could be done by applying a buffer region by exploiting the
>>>>>> background DMDA mesh that I already use to do domain decomposition. Then
>>>>>> using the buffer region of each cell to locate the ghost particles and
>>>>>> finally using VecCreateGhost. Is this feasible? Or is there an easier
>>>>>> approach using other PETSc functions.
>>>>>>
>>>>>>
>>>>> This is feasible, but it would be good to develop a set of best
>>>>> practices, since we have been mainly focused on the case of non-redundant
>>>>> particles. Here is how I think I would do what you want.
>>>>>
>>>>> 1) Add a particle field 'ghost' that identifies ghost vs owned
>>>>> particles. I think it needs options OWNED, OVERLAP, and GHOST
>>>>>
>>>>> 2) At some interval identify particles that should be sent to other
>>>>> processes as ghosts. I would call these "overlap particles". The
>>>>> determination
>>>>> seems application specific, so I would leave this determination to
>>>>> the user right now. We do two things to these particles
>>>>>
>>>>> a) Mark chosen particles as OVERLAP
>>>>>
>>>>> b) Change rank to process we are sending to
>>>>>
>>>>> 3) Call DMSwarmMigrate with PETSC_FALSE for the particle deletion flag
>>>>>
>>>>> 4) Mark OVERLAP particles as GHOST when they arrive
>>>>>
>>>>> There is one problem in the above algorithm. It does not allow sending
>>>>> particles to multiple ranks. We would have to do this
>>>>> in phases right now, or make a small adjustment to the interface
>>>>> allowing replication of particles when a set of ranks is specified.
>>>>>
>>>>> THanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> Thank you,
>>>>>> Miguel
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$
>>>>>
>>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpn5mlkJ8$
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$
>>>>
>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpn5mlkJ8$
>>>> >
>>>>
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$
>>>
>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpn5mlkJ8$
>>> >
>>>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$
>>
>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpn5mlkJ8$
>> >
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$
>
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpn5mlkJ8$
> >
>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xptrlWQeo$
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!emloiQzBtgzgIIVq1midB7Rd9bkkr7wwUgGr8mzcWk2AzIkn7e_2Rr5Nkxd2ysaNUXWFp1jLd98xpn5mlkJ8$
>
