Re: [petsc-users] DMSTAG Gathering Vector on single process

[Colton Bryant]

Ok! Thanks for the reply. I'll take a look when I get a chance.

-Colton

On Wed, Dec 13, 2023 at 10:48 AM Matthew Knepley  wrote:

> On Wed, Dec 13, 2023 at 11:22 AM Colton Bryant <
> coltonbryant2...@u.northwestern.edu> wrote:
>
>> Hi,
>>
>> Thanks for the help last week. The suggestions made the implementation I
>> had much cleaner. I had one follow up question. Is there a way to sort of
>> undo this operation? I know the vec scatter can be done backwards to
>> distribute the arrays but I didn't see an easy way to migrate the DMDA
>> vectors back into the DMStag object.
>>
>
> It is not there. However, writing it would be straightforward. I would
>
>   1) Expose DMStagMigrateVecDMDA(), which is not currently public
>
>   2) Add a ScatterMode argument
>
>   3) Put in code that calls DMStagSetValuesStencil(), rather than
> GetValuesStencil()
>
> We would be happy to take on MR on this, and could help in the review.
>
>   Thanks,
>
>  MAtt
>
>
>> Thanks for any advice.
>>
>> -Colton
>>
>> On Wed, Dec 6, 2023 at 8:18 PM Barry Smith  wrote:
>>
>>>
>>>
>>> On Dec 6, 2023, at 8:35 PM, Matthew Knepley  wrote:
>>>
>>> On Wed, Dec 6, 2023 at 8:10 PM Barry Smith  wrote:
>>>

   Depending on the serial library you may not need to split the vector
 into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just
 global to natural and scatter to zero on the full vector, now the full
 vector is on the first rank and you can access what you need in that one
 vector if possible.

>>>
>>> Does DMStag have a GlobalToNatural?
>>>
>>>
>>>Good point, it does not appear to have such a thing, though it could.
>>>
>>> Also, the serial code would have to have identical interleaving.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
 On Dec 6, 2023, at 6:37 PM, Colton Bryant <
 coltonbryant2...@u.northwestern.edu> wrote:

 Ah excellent! I was not aware of the ability to preallocate the objects
 and migrate them each time.

 Thanks!
 -Colton

 On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley 
 wrote:

> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
> coltonbryant2...@u.northwestern.edu> wrote:
>
>> Hello,
>>
>> I am working on a code in which a DMSTAG object is used to solve a
>> fluid flow problem and I need to gather this flow data on a single 
>> process
>> to interact with an existing (serial) library at each timestep of my
>> simulation. After looking around the solution I've tried is:
>>
>> -use DMStagVecSplitToDMDA to extract vectors of each component of the
>> flow
>> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
>> components naturally ordered
>> -use VecScatterCreateToZero to set up and then do the scatter to
>> gather on the single process
>>
>> Unless I'm misunderstanding something this method results in a lot of
>> memory allocation/freeing happening at each step of the evolution and I 
>> was
>> wondering if there is a way to directly perform such a scatter from the
>> DMSTAG object without splitting as I'm doing here.
>>
>
> 1) You can see here:
>
>
> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>
> that this function is small. You can do the DMDA creation manually,
> and then just call DMStagMigrateVecDMDA() each time, which will not
> create anything.
>
> 2) You can create the natural vector upfront, and just scatter each
> time.
>
> 3) You can create the serial vector upfront, and just scatter each
> time.
>
> This is some data movement. You can compress the g2n and 2zero
> scatters using
>
>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>
> as an optimization.
>
>   Thanks,
>
>  Matt
>
>
>> Any advice would be much appreciated!
>>
>> Best,
>> Colton Bryant
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>


>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> 
>>>
>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Matthew Knepley]

On Wed, Dec 13, 2023 at 11:22 AM Colton Bryant <
coltonbryant2...@u.northwestern.edu> wrote:

> Hi,
>
> Thanks for the help last week. The suggestions made the implementation I
> had much cleaner. I had one follow up question. Is there a way to sort of
> undo this operation? I know the vec scatter can be done backwards to
> distribute the arrays but I didn't see an easy way to migrate the DMDA
> vectors back into the DMStag object.
>

It is not there. However, writing it would be straightforward. I would

  1) Expose DMStagMigrateVecDMDA(), which is not currently public

  2) Add a ScatterMode argument

  3) Put in code that calls DMStagSetValuesStencil(), rather than
GetValuesStencil()

We would be happy to take on MR on this, and could help in the review.

  Thanks,

 MAtt


> Thanks for any advice.
>
> -Colton
>
> On Wed, Dec 6, 2023 at 8:18 PM Barry Smith  wrote:
>
>>
>>
>> On Dec 6, 2023, at 8:35 PM, Matthew Knepley  wrote:
>>
>> On Wed, Dec 6, 2023 at 8:10 PM Barry Smith  wrote:
>>
>>>
>>>   Depending on the serial library you may not need to split the vector
>>> into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just
>>> global to natural and scatter to zero on the full vector, now the full
>>> vector is on the first rank and you can access what you need in that one
>>> vector if possible.
>>>
>>
>> Does DMStag have a GlobalToNatural?
>>
>>
>>Good point, it does not appear to have such a thing, though it could.
>>
>> Also, the serial code would have to have identical interleaving.
>>
>>   Thanks,
>>
>>  Matt
>>
>>> On Dec 6, 2023, at 6:37 PM, Colton Bryant <
>>> coltonbryant2...@u.northwestern.edu> wrote:
>>>
>>> Ah excellent! I was not aware of the ability to preallocate the objects
>>> and migrate them each time.
>>>
>>> Thanks!
>>> -Colton
>>>
>>> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley 
>>> wrote:
>>>
 On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
 coltonbryant2...@u.northwestern.edu> wrote:

> Hello,
>
> I am working on a code in which a DMSTAG object is used to solve a
> fluid flow problem and I need to gather this flow data on a single process
> to interact with an existing (serial) library at each timestep of my
> simulation. After looking around the solution I've tried is:
>
> -use DMStagVecSplitToDMDA to extract vectors of each component of the
> flow
> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
> components naturally ordered
> -use VecScatterCreateToZero to set up and then do the scatter to
> gather on the single process
>
> Unless I'm misunderstanding something this method results in a lot of
> memory allocation/freeing happening at each step of the evolution and I 
> was
> wondering if there is a way to directly perform such a scatter from the
> DMSTAG object without splitting as I'm doing here.
>

 1) You can see here:


 https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA

 that this function is small. You can do the DMDA creation manually, and
 then just call DMStagMigrateVecDMDA() each time, which will not create
 anything.

 2) You can create the natural vector upfront, and just scatter each
 time.

 3) You can create the serial vector upfront, and just scatter each time.

 This is some data movement. You can compress the g2n and 2zero scatters
 using

   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/

 as an optimization.

   Thanks,

  Matt


> Any advice would be much appreciated!
>
> Best,
> Colton Bryant
>


 --
 What most experimenters take for granted before they begin their
 experiments is infinitely more interesting than any results to which their
 experiments lead.
 -- Norbert Wiener

 https://www.cse.buffalo.edu/~knepley/
 

>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> 
>>
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Colton Bryant]

Hi,

Thanks for the help last week. The suggestions made the implementation I
had much cleaner. I had one follow up question. Is there a way to sort of
undo this operation? I know the vec scatter can be done backwards to
distribute the arrays but I didn't see an easy way to migrate the DMDA
vectors back into the DMStag object.

Thanks for any advice.

-Colton

On Wed, Dec 6, 2023 at 8:18 PM Barry Smith  wrote:

>
>
> On Dec 6, 2023, at 8:35 PM, Matthew Knepley  wrote:
>
> On Wed, Dec 6, 2023 at 8:10 PM Barry Smith  wrote:
>
>>
>>   Depending on the serial library you may not need to split the vector
>> into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just
>> global to natural and scatter to zero on the full vector, now the full
>> vector is on the first rank and you can access what you need in that one
>> vector if possible.
>>
>
> Does DMStag have a GlobalToNatural?
>
>
>Good point, it does not appear to have such a thing, though it could.
>
> Also, the serial code would have to have identical interleaving.
>
>   Thanks,
>
>  Matt
>
>> On Dec 6, 2023, at 6:37 PM, Colton Bryant <
>> coltonbryant2...@u.northwestern.edu> wrote:
>>
>> Ah excellent! I was not aware of the ability to preallocate the objects
>> and migrate them each time.
>>
>> Thanks!
>> -Colton
>>
>> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley  wrote:
>>
>>> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
>>> coltonbryant2...@u.northwestern.edu> wrote:
>>>
 Hello,

 I am working on a code in which a DMSTAG object is used to solve a
 fluid flow problem and I need to gather this flow data on a single process
 to interact with an existing (serial) library at each timestep of my
 simulation. After looking around the solution I've tried is:

 -use DMStagVecSplitToDMDA to extract vectors of each component of the
 flow
 -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
 components naturally ordered
 -use VecScatterCreateToZero to set up and then do the scatter to gather
 on the single process

 Unless I'm misunderstanding something this method results in a lot of
 memory allocation/freeing happening at each step of the evolution and I was
 wondering if there is a way to directly perform such a scatter from the
 DMSTAG object without splitting as I'm doing here.

>>>
>>> 1) You can see here:
>>>
>>>
>>> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>>>
>>> that this function is small. You can do the DMDA creation manually, and
>>> then just call DMStagMigrateVecDMDA() each time, which will not create
>>> anything.
>>>
>>> 2) You can create the natural vector upfront, and just scatter each time.
>>>
>>> 3) You can create the serial vector upfront, and just scatter each time.
>>>
>>> This is some data movement. You can compress the g2n and 2zero scatters
>>> using
>>>
>>>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>>>
>>> as an optimization.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 Any advice would be much appreciated!

 Best,
 Colton Bryant

>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> 
>>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>
>
>

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Barry Smith]

> On Dec 6, 2023, at 8:35 PM, Matthew Knepley  wrote:
> 
> On Wed, Dec 6, 2023 at 8:10 PM Barry Smith  > wrote:
>> 
>>   Depending on the serial library you may not need to split the vector into 
>> DMDA vectors with DMStagVecSplitToDMDA() for each component. Just global to 
>> natural and scatter to zero on the full vector, now the full vector is on 
>> the first rank and you can access what you need in that one vector if 
>> possible.
> 
> Does DMStag have a GlobalToNatural?

   Good point, it does not appear to have such a thing, though it could.

> Also, the serial code would have to have identical interleaving.
> 
>   Thanks,
> 
>  Matt 
>>> On Dec 6, 2023, at 6:37 PM, Colton Bryant 
>>> >> > wrote:
>>> 
>>> Ah excellent! I was not aware of the ability to preallocate the objects and 
>>> migrate them each time. 
>>> 
>>> Thanks!
>>> -Colton
>>> 
>>> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley >> > wrote:
 On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant 
 >>> > wrote:
> Hello,
> 
> I am working on a code in which a DMSTAG object is used to solve a fluid 
> flow problem and I need to gather this flow data on a single process to 
> interact with an existing (serial) library at each timestep of my 
> simulation. After looking around the solution I've tried is:
> 
> -use DMStagVecSplitToDMDA to extract vectors of each component of the flow
> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the 
> components naturally ordered
> -use VecScatterCreateToZero to set up and then do the scatter to gather 
> on the single process
> 
> Unless I'm misunderstanding something this method results in a lot of 
> memory allocation/freeing happening at each step of the evolution and I 
> was wondering if there is a way to directly perform such a scatter from 
> the DMSTAG object without splitting as I'm doing here.
 
 1) You can see here:
 
 https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
 
 that this function is small. You can do the DMDA creation manually, and 
 then just call DMStagMigrateVecDMDA() each time, which will not create 
 anything.
 
 2) You can create the natural vector upfront, and just scatter each time.
 
 3) You can create the serial vector upfront, and just scatter each time.
 
 This is some data movement. You can compress the g2n and 2zero scatters 
 using
 
   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
 
 as an optimization.
 
   Thanks,
 
  Matt
  
> Any advice would be much appreciated!
> 
> Best,
> Colton Bryant 
 
 
 -- 
 What most experimenters take for granted before they begin their 
 experiments is infinitely more interesting than any results to which their 
 experiments lead.
 -- Norbert Wiener
 
 https://www.cse.buffalo.edu/~knepley/ 
 
>> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Matthew Knepley]

On Wed, Dec 6, 2023 at 8:10 PM Barry Smith  wrote:

>
>   Depending on the serial library you may not need to split the vector
> into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just
> global to natural and scatter to zero on the full vector, now the full
> vector is on the first rank and you can access what you need in that one
> vector if possible.
>

Does DMStag have a GlobalToNatural? Also, the serial code would have to
have identical interleaving.

  Thanks,

 Matt

> On Dec 6, 2023, at 6:37 PM, Colton Bryant <
> coltonbryant2...@u.northwestern.edu> wrote:
>
> Ah excellent! I was not aware of the ability to preallocate the objects
> and migrate them each time.
>
> Thanks!
> -Colton
>
> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley  wrote:
>
>> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
>> coltonbryant2...@u.northwestern.edu> wrote:
>>
>>> Hello,
>>>
>>> I am working on a code in which a DMSTAG object is used to solve a fluid
>>> flow problem and I need to gather this flow data on a single process to
>>> interact with an existing (serial) library at each timestep of my
>>> simulation. After looking around the solution I've tried is:
>>>
>>> -use DMStagVecSplitToDMDA to extract vectors of each component of the
>>> flow
>>> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
>>> components naturally ordered
>>> -use VecScatterCreateToZero to set up and then do the scatter to gather
>>> on the single process
>>>
>>> Unless I'm misunderstanding something this method results in a lot of
>>> memory allocation/freeing happening at each step of the evolution and I was
>>> wondering if there is a way to directly perform such a scatter from the
>>> DMSTAG object without splitting as I'm doing here.
>>>
>>
>> 1) You can see here:
>>
>>
>> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>>
>> that this function is small. You can do the DMDA creation manually, and
>> then just call DMStagMigrateVecDMDA() each time, which will not create
>> anything.
>>
>> 2) You can create the natural vector upfront, and just scatter each time.
>>
>> 3) You can create the serial vector upfront, and just scatter each time.
>>
>> This is some data movement. You can compress the g2n and 2zero scatters
>> using
>>
>>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>>
>> as an optimization.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Any advice would be much appreciated!
>>>
>>> Best,
>>> Colton Bryant
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> 
>>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Barry Smith]

  Depending on the serial library you may not need to split the vector into 
DMDA vectors with DMStagVecSplitToDMDA() for each component. Just global to 
natural and scatter to zero on the full vector, now the full vector is on the 
first rank and you can access what you need in that one vector if possible.
  

> On Dec 6, 2023, at 6:37 PM, Colton Bryant 
>  wrote:
> 
> Ah excellent! I was not aware of the ability to preallocate the objects and 
> migrate them each time.
> 
> Thanks!
> -Colton
> 
> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley  > wrote:
>> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant 
>> > > wrote:
>>> Hello,
>>> 
>>> I am working on a code in which a DMSTAG object is used to solve a fluid 
>>> flow problem and I need to gather this flow data on a single process to 
>>> interact with an existing (serial) library at each timestep of my 
>>> simulation. After looking around the solution I've tried is:
>>> 
>>> -use DMStagVecSplitToDMDA to extract vectors of each component of the flow
>>> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the components 
>>> naturally ordered
>>> -use VecScatterCreateToZero to set up and then do the scatter to gather on 
>>> the single process
>>> 
>>> Unless I'm misunderstanding something this method results in a lot of 
>>> memory allocation/freeing happening at each step of the evolution and I was 
>>> wondering if there is a way to directly perform such a scatter from the 
>>> DMSTAG object without splitting as I'm doing here.
>> 
>> 1) You can see here:
>> 
>> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>> 
>> that this function is small. You can do the DMDA creation manually, and then 
>> just call DMStagMigrateVecDMDA() each time, which will not create anything.
>> 
>> 2) You can create the natural vector upfront, and just scatter each time.
>> 
>> 3) You can create the serial vector upfront, and just scatter each time.
>> 
>> This is some data movement. You can compress the g2n and 2zero scatters using
>> 
>>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>> 
>> as an optimization.
>> 
>>   Thanks,
>> 
>>  Matt
>>  
>>> Any advice would be much appreciated!
>>> 
>>> Best,
>>> Colton Bryant
>> 
>> 
>> --
>> What most experimenters take for granted before they begin their experiments 
>> is infinitely more interesting than any results to which their experiments 
>> lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Colton Bryant]

Ah excellent! I was not aware of the ability to preallocate the objects and
migrate them each time.

Thanks!
-Colton

On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley  wrote:

> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
> coltonbryant2...@u.northwestern.edu> wrote:
>
>> Hello,
>>
>> I am working on a code in which a DMSTAG object is used to solve a fluid
>> flow problem and I need to gather this flow data on a single process to
>> interact with an existing (serial) library at each timestep of my
>> simulation. After looking around the solution I've tried is:
>>
>> -use DMStagVecSplitToDMDA to extract vectors of each component of the flow
>> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
>> components naturally ordered
>> -use VecScatterCreateToZero to set up and then do the scatter to gather
>> on the single process
>>
>> Unless I'm misunderstanding something this method results in a lot of
>> memory allocation/freeing happening at each step of the evolution and I was
>> wondering if there is a way to directly perform such a scatter from the
>> DMSTAG object without splitting as I'm doing here.
>>
>
> 1) You can see here:
>
> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>
> that this function is small. You can do the DMDA creation manually, and
> then just call DMStagMigrateVecDMDA() each time, which will not create
> anything.
>
> 2) You can create the natural vector upfront, and just scatter each time.
>
> 3) You can create the serial vector upfront, and just scatter each time.
>
> This is some data movement. You can compress the g2n and 2zero scatters
> using
>
>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>
> as an optimization.
>
>   Thanks,
>
>  Matt
>
>
>> Any advice would be much appreciated!
>>
>> Best,
>> Colton Bryant
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Matthew Knepley]

On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
coltonbryant2...@u.northwestern.edu> wrote:

> Hello,
>
> I am working on a code in which a DMSTAG object is used to solve a fluid
> flow problem and I need to gather this flow data on a single process to
> interact with an existing (serial) library at each timestep of my
> simulation. After looking around the solution I've tried is:
>
> -use DMStagVecSplitToDMDA to extract vectors of each component of the flow
> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the components
> naturally ordered
> -use VecScatterCreateToZero to set up and then do the scatter to gather on
> the single process
>
> Unless I'm misunderstanding something this method results in a lot of
> memory allocation/freeing happening at each step of the evolution and I was
> wondering if there is a way to directly perform such a scatter from the
> DMSTAG object without splitting as I'm doing here.
>

1) You can see here:

https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA

that this function is small. You can do the DMDA creation manually, and
then just call DMStagMigrateVecDMDA() each time, which will not create
anything.

2) You can create the natural vector upfront, and just scatter each time.

3) You can create the serial vector upfront, and just scatter each time.

This is some data movement. You can compress the g2n and 2zero scatters
using

  https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/

as an optimization.

  Thanks,

 Matt


> Any advice would be much appreciated!
>
> Best,
> Colton Bryant
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/