Dear Dave and Matt,
I am really dealing with two different use cases in a code that
will compute a levelset function passing through a large set of points.
If I had DMSwarmSetMigrateType() and if it were safe to switch the
migration mode back and forth in the same swarm, this would cover all my
use cases here. Is it safe to add it back to petsc? Details below if you
are curious.
1) During preprocessing I am loading a point cloud from disk (in
whatever order it comes) and need to send the particles to the right
ranks. Since the background DM is a DMDA I can easily figure out the
destination rank. This would be covered by your suggestion not to attach
the DM, except that later I need to locate these points with respect to
the background cells in order to initialize data on the Vecs associated
to the DMDA.
2) Then I need to implement a semilagrangian time evolution scheme. For
this I'd like to send particles around at the "foot of characteristic",
collect data there and then send them back to the originating point. The
first migration would be based on particle coordinates
(DMSwarmMigrate_DMNeighborScatter and the restriction to only
neighbouring ranks is perfect), while for the second move it would be
easier to just send them back to the originating rank, which I can
easily store in an Int field in the swarm. Thus at each timestep I'd
need to swap migrate types in this swarm (DMScatter for moving them to
the feet and BASIC to send them back).
Thanks
Matteo
Il 22/12/22 18:40, Dave May ha scritto:
Hey Matt,
On Thu 22. Dec 2022 at 05:02, Matthew Knepley <knep...@gmail.com> wrote:
On Thu, Dec 22, 2022 at 6:28 AM Matteo Semplice
<matteo.sempl...@uninsubria.it> wrote:
Dear all
please ignore my previous email and read this one: I have
better localized the problem. Maybe DMSwarmMigrate is designed
to migrate particles only to first neighbouring ranks?
Yes, I believe that was the design.
Dave, is this correct?
Correct. DMSwarmMigrate_DMNeighborScatter() only scatter points to the
neighbour ranks - where neighbours are defined by the DM provided to
represent the mesh.
DMSwarmMigrate_DMNeighborScatter() Is selected by default if you
attach a DM.
The scatter method should be over ridden with
DMSwarmSetMigrateType()
however it appears this method no longer exists.
If one can determine the exact rank where points should should be sent
and it is not going to be the neighbour rank (given by the DM), I
would suggest not attaching the DM at all.
However if this is not possible and one wanted to scatter to say the
neighbours neighbours, we will have to add a new interface and
refactor things a little bit.
Cheers
Dave
Thanks,
Matt
Il 22/12/22 11:44, Matteo Semplice ha scritto:
Dear everybody,
I have bug a bit into the code and I am able to add more
information.
Il 02/12/22 12:48, Matteo Semplice ha scritto:
Hi.
I am sorry to take this up again, but further tests show
that it's not right yet.
Il 04/11/22 12:48, Matthew Knepley ha scritto:
On Fri, Nov 4, 2022 at 7:46 AM Matteo Semplice
<matteo.sempl...@uninsubria.it> wrote:
On 04/11/2022 02:43, Matthew Knepley wrote:
On Thu, Nov 3, 2022 at 8:36 PM Matthew Knepley
<knep...@gmail.com> wrote:
On Thu, Oct 27, 2022 at 11:57 AM Semplice Matteo
<matteo.sempl...@uninsubria.it> wrote:
Dear Petsc developers,
I am trying to use a DMSwarm to locate a cloud
of points with respect to a background mesh.
In the real application the points will be
loaded from disk, but I have created a small
demo in which
* each processor creates Npart particles,
all within the domain covered by the mesh,
but not all in the local portion of the mesh
* migrate the particles
After migration most particles are not any
more in the DMSwarm (how many and which ones
seems to depend on the number of cpus, but it
never happens that all particle survive the
migration process).
Thanks for sending this. I found the problem.
Someone has some overly fancy code inside DMDA to
figure out the local bounding box from the
coordinates.
It is broken for DM_BOUNDARY_GHOSTED, but we never
tested with this. I will fix it.
Okay, I think this fix is correct
https://gitlab.com/petsc/petsc/-/merge_requests/5802
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F5802&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C49a33e96a2144e96728008dae44394ca%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073276161185276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sahWnsix2F7anVnUYz2ANyivon%2F6q33geOlaRJD5iEY%3D&reserved=0>
I incorporated your test as
src/dm/impls/da/tests/ex1.c. Can you take a look and
see if this fixes your issue?
Yes, we have tested 2d and 3d, with various
combinations of DM_BOUNDARY_* along different
directions and it works like a charm.
On a side note, neither DMSwarmViewXDMF nor
DMSwarmMigrate seem to be implemented for 1d: I get
[0]PETSC ERROR: No support for this operation for this
object type[0]PETSC ERROR: Support not provided for 1D
However, currently I have no need for this feature.
Finally, if the test is meant to stay in the source,
you may remove the call to
DMSwarmRegisterPetscDatatypeField as in the attached patch.
Thanks a lot!!
Thanks! Glad it works.
Matt
There are still problems when not using 1,2 or 4 cpus. Any
other number of cpus that I've tested does not work corectly.
I have now modified private_DMDALocatePointsIS_2D_Regular to
print out some debugging information. I see that this is
called twice during migration, once before and once after
DMSwarmMigrate_DMNeighborScatter. If I understand correctly,
the second call to private_DMDALocatePointsIS_2D_Regular
should be able to locate all particles owned by the rank but
it fails for some of them because they have been sent to the
wrong rank (despite being well away from process boundaries).
For example, running the example src/dm/impls/da/tests/ex1.c
with Nx=21 (20x20 Q1 elements on [-1,1]X[-1,1]) with 3
processors,
- the particles (-0.191,-0.462) and (0.191,-0.462) are sent
cpu2 instead of cpu0
- those at (-0.287,-0.693)and (0.287,-0.693) are sent to cpu1
instead of cpu0
- those at (0.191,0.462) and (-0.191,0.462) are sent to cpu0
instead of cpu2
(This is 2d and thus not affected by the 3d issue mentioned
yesterday on petsc-dev. Tests were made based on the release
branch pulled out this morning, i.e. on commit bebdc8d016f).
I see: particles are sent "all around" and not only to the
destination rank.
Still however, running the example src/dm/impls/da/tests/ex1.c
with Nx=21 (20x20 Q1 elements on [-1,1]X[-1,1]) with 3
processors, there are 2 particles initially owned by rank2 (at
y=-0.6929 and x=+/-0.2870) that are sent only to rank1 and
never make it to rank0 and are thus lost in the end since
rank1, correctly, discards them.
Thanks
Matteo
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to
which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/
<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C49a33e96a2144e96728008dae44394ca%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073276161185276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JTwzg7jNDdj6y%2BpT0fIaqllAlIvZfxrQ0IVPFRMAW6E%3D&reserved=0>
--
---
Professore Associato in Analisi Numerica
Dipartimento di Scienza e Alta Tecnologia
Università degli Studi dell'Insubria
Via Valleggio, 11 - Como
