Re: [petsc-dev] DMplex: Natural Ordering and subDM

[Blaise A Bourdin]

> On Nov 28, 2017, at 12:39 AM, Jed Brown  wrote:
> 
> Blaise A Bourdin  writes:
> 
>> There may be good reasons to want to read / write in a given ordering:  
>> post-processing an existing computation, applying non trivial boundary 
>> conditions that need to be computed separately, or restarting a computation 
>> on a different number of processors.
>> Also, Exodus has restriction on cell ordering (cells in an element block 
>> must be numbered sequentially) so the distributed cell ordering may not be 
>> acceptable.
> 
> Does PETSc have the ability to write Exodus files?  Yes, if this is a
> requirement then that viewer needs to reorder in some way to write.
> 
> For concreteness, what postprocessors are you thinking about here that
> need an ordering that matches that of the input file?  (I ask because I
> usually see the simulation writing a different format from the input
> file, and often viewed using different tools.)

I have a bunch that I wrote myself. For example: computing stress intensity 
factor using the G_theta method. This involves computing integrals but not 
solving linear systems, it can be run on a much smaller system than the 
analysis itself. It would be a pain to have to move back the data file to a 
large syste and requeue in order to get the same parallel layout.

But the main difficulty with exodus is that the distributed cell ordering is 
not necessarily admissible for exodus element blocks, and that exodus cell sets 
are not implemented in most tools... Fine elements file formats are a mess and 
there are very few formats for which mesh generator, visualization, and I/O 
library are available.

Blaise

-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin

Re: [petsc-dev] DMplex: Natural Ordering and subDM

[Jed Brown]

Blaise A Bourdin  writes:

> There may be good reasons to want to read / write in a given ordering:  
> post-processing an existing computation, applying non trivial boundary 
> conditions that need to be computed separately, or restarting a computation 
> on a different number of processors.
> Also, Exodus has restriction on cell ordering (cells in an element block must 
> be numbered sequentially) so the distributed cell ordering may not be 
> acceptable.

Does PETSc have the ability to write Exodus files?  Yes, if this is a
requirement then that viewer needs to reorder in some way to write.

For concreteness, what postprocessors are you thinking about here that
need an ordering that matches that of the input file?  (I ask because I
usually see the simulation writing a different format from the input
file, and often viewed using different tools.)

> Exodus 6 introduced element sets which are free of this limitation, but as 
> far as I know, no mesh generator or post processing deals with elements sets.
>
> I ended up inverting the migration SF, broadcast the distributed section back 
> to the original mesh, then generate SFnatural using 
> DMPlexCreateGlobalToNaturalSF. It is ugly but done only once.
>
> Still about exodus, I now have parallel (MPIIO through parallel netcdf) I/O 
> working for nodal (linear and quadratic Lagrange elements) and zonal fields 
> (in exodus jargon) in natural and standard ordering. I can have pull request 
> and documented examples and tests ready in a few days.

Cool!

Re: [petsc-dev] DMplex: Natural Ordering and subDM

[Jed Brown]

Matthew Knepley  writes:

> On Mon, Nov 27, 2017 at 9:24 PM, Jed Brown  wrote:
>
>> Matthew Knepley  writes:
>>
>> > On Mon, Nov 27, 2017 at 8:08 PM, Jed Brown  wrote:
>> >
>> >> I don't know the answer to your question (Matt?), but do you really need
>> >> to reorder the entire mesh or would it be sufficient to label your
>> >> points with their original numbering?
>> >
>> >
>> > Maybe I am wrong, but I think it amounts to the same thing. If we are
>> going
>> > to output things in parallel,
>> > we would need to communicate to he writing process, which this
>> essentially
>> > does.
>>
>> Writing a label doesn't require redistribution of the mesh.  It's possible
>> to do parallel IO.
>>
>
> My understanding is that 3rd party programs want the mesh in the
> original order, so we want it ordered in the HDF5 file in the original
> order. You could I guess write it in the order, but it seems messy to
> write stuff all over the place in the file. Is that what you mean?

What is the third-party program doing?  It might be easier for it to
apply the permutation than for PETSc to redistribute in order to write
the file in a possibly poor ordering.

Re: [petsc-dev] DMplex: Natural Ordering and subDM

[Matthew Knepley]

On Mon, Nov 27, 2017 at 9:24 PM, Jed Brown  wrote:

> Matthew Knepley  writes:
>
> > On Mon, Nov 27, 2017 at 8:08 PM, Jed Brown  wrote:
> >
> >> I don't know the answer to your question (Matt?), but do you really need
> >> to reorder the entire mesh or would it be sufficient to label your
> >> points with their original numbering?
> >
> >
> > Maybe I am wrong, but I think it amounts to the same thing. If we are
> going
> > to output things in parallel,
> > we would need to communicate to he writing process, which this
> essentially
> > does.
>
> Writing a label doesn't require redistribution of the mesh.  It's possible
> to do parallel IO.
>

My understanding is that 3rd party programs want the mesh in the original
order, so we
want it ordered in the HDF5 file in the original order. You could I guess
write it in the order,
but it seems messy to write stuff all over the place in the file. Is that
what you mean?

  Matt


> > I thought I responded, but what has to happen is that the SF for this
> > reordering must be decimated to
> > produce a subSF for the subDM. Its just not simple, so I have not had the
> > free time to work it out.
> >
> >Matt
> >
> >
> >> Blaise A Bourdin  writes:
> >>
> >> > Hi,
> >> > I am fighting with natural ordering in DMplex with multi-fields /
> multi
> >> components DM.
> >> > I need to read / write files in a distribution-independent ordering.
> The
> >> natural way to do so is to define the natural ordering as that of the
> file,
> >> then call DMPlexGlobalToNaturalBegin/End before VecView:
> >> > The pseudo code would be
> >> >
> >> >   DMPlexCreateFromFile(PETSC_COMM_WORLD,ifilename,
> interpolate,);
> >> >   // create default section associated with a complicated layout of
> >> multiple fields
> >> >   // (U,\alpha,\sigma)
> >> >   DMSetUseNatural(dmUAS,PETSC_TRUE);
> >> >   DMPlexDistribute(dmUAS,0,,);
> >> >
> >> >   DMGetGlobalVector(dmDist,);
> >> >   DMGetGlobalVector(dmDist,);
> >> >   DMPlexGlobalToNaturalBegin(dmDist,UAS,UASNatural);
> >> >   DMPlexGlobalToNaturalEnd(dmDist,UAS,UASNatural);
> >> >   // Do my I/O stuff using UASNatural
> >> >
> >> > If I try to do the same thing one field at a time, things break:
> >> > Say that I get the subDM for the first field (U):
> >> >
> >> >   DMCreateSubDM(dmDist,1,,,);
> >> >
> >> > Then dmU->useNatural is PETSC_FALSE and  I cannot compute
> dmU->sfnatural
> >> using DMPlexCreateGlobalToNaturalSF since I do not have the “sequential”
> >> version of dmU.
> >> >
> >> > What is the proper way to handle this situation? Creating sequential
> >> subDMs and calling DMDistribute on each of them is ugly and possibly
> costly.
> >> >
> >> > Regards,
> >> >
> >> > Blaise
> >> >
> >> > --
> >> > Department of Mathematics and Center for Computation & Technology
> >> > Louisiana State University, Baton Rouge, LA 70803, USA
> >> > Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276
> http://www.math.lsu.edu/~
> >> bourdin
> >>
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> their
> > experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ 
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] DMplex: Natural Ordering and subDM

[Jed Brown]

Matthew Knepley  writes:

> On Mon, Nov 27, 2017 at 8:08 PM, Jed Brown  wrote:
>
>> I don't know the answer to your question (Matt?), but do you really need
>> to reorder the entire mesh or would it be sufficient to label your
>> points with their original numbering?
>
>
> Maybe I am wrong, but I think it amounts to the same thing. If we are going
> to output things in parallel,
> we would need to communicate to he writing process, which this essentially
> does.

Writing a label doesn't require redistribution of the mesh.  It's possible to 
do parallel IO.

> I thought I responded, but what has to happen is that the SF for this
> reordering must be decimated to
> produce a subSF for the subDM. Its just not simple, so I have not had the
> free time to work it out.
>
>Matt
>
>
>> Blaise A Bourdin  writes:
>>
>> > Hi,
>> > I am fighting with natural ordering in DMplex with multi-fields / multi
>> components DM.
>> > I need to read / write files in a distribution-independent ordering. The
>> natural way to do so is to define the natural ordering as that of the file,
>> then call DMPlexGlobalToNaturalBegin/End before VecView:
>> > The pseudo code would be
>> >
>> >   DMPlexCreateFromFile(PETSC_COMM_WORLD,ifilename,interpolate,);
>> >   // create default section associated with a complicated layout of
>> multiple fields
>> >   // (U,\alpha,\sigma)
>> >   DMSetUseNatural(dmUAS,PETSC_TRUE);
>> >   DMPlexDistribute(dmUAS,0,,);
>> >
>> >   DMGetGlobalVector(dmDist,);
>> >   DMGetGlobalVector(dmDist,);
>> >   DMPlexGlobalToNaturalBegin(dmDist,UAS,UASNatural);
>> >   DMPlexGlobalToNaturalEnd(dmDist,UAS,UASNatural);
>> >   // Do my I/O stuff using UASNatural
>> >
>> > If I try to do the same thing one field at a time, things break:
>> > Say that I get the subDM for the first field (U):
>> >
>> >   DMCreateSubDM(dmDist,1,,,);
>> >
>> > Then dmU->useNatural is PETSC_FALSE and  I cannot compute dmU->sfnatural
>> using DMPlexCreateGlobalToNaturalSF since I do not have the “sequential”
>> version of dmU.
>> >
>> > What is the proper way to handle this situation? Creating sequential
>> subDMs and calling DMDistribute on each of them is ugly and possibly costly.
>> >
>> > Regards,
>> >
>> > Blaise
>> >
>> > --
>> > Department of Mathematics and Center for Computation & Technology
>> > Louisiana State University, Baton Rouge, LA 70803, USA
>> > Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~
>> bourdin
>>
>
>
>
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] DMplex: Natural Ordering and subDM

[Matthew Knepley]

On Mon, Nov 27, 2017 at 8:08 PM, Jed Brown  wrote:

> I don't know the answer to your question (Matt?), but do you really need
> to reorder the entire mesh or would it be sufficient to label your
> points with their original numbering?


Maybe I am wrong, but I think it amounts to the same thing. If we are going
to output things in parallel,
we would need to communicate to he writing process, which this essentially
does.

I thought I responded, but what has to happen is that the SF for this
reordering must be decimated to
produce a subSF for the subDM. Its just not simple, so I have not had the
free time to work it out.

   Matt


> Blaise A Bourdin  writes:
>
> > Hi,
> > I am fighting with natural ordering in DMplex with multi-fields / multi
> components DM.
> > I need to read / write files in a distribution-independent ordering. The
> natural way to do so is to define the natural ordering as that of the file,
> then call DMPlexGlobalToNaturalBegin/End before VecView:
> > The pseudo code would be
> >
> >   DMPlexCreateFromFile(PETSC_COMM_WORLD,ifilename,interpolate,);
> >   // create default section associated with a complicated layout of
> multiple fields
> >   // (U,\alpha,\sigma)
> >   DMSetUseNatural(dmUAS,PETSC_TRUE);
> >   DMPlexDistribute(dmUAS,0,,);
> >
> >   DMGetGlobalVector(dmDist,);
> >   DMGetGlobalVector(dmDist,);
> >   DMPlexGlobalToNaturalBegin(dmDist,UAS,UASNatural);
> >   DMPlexGlobalToNaturalEnd(dmDist,UAS,UASNatural);
> >   // Do my I/O stuff using UASNatural
> >
> > If I try to do the same thing one field at a time, things break:
> > Say that I get the subDM for the first field (U):
> >
> >   DMCreateSubDM(dmDist,1,,,);
> >
> > Then dmU->useNatural is PETSC_FALSE and  I cannot compute dmU->sfnatural
> using DMPlexCreateGlobalToNaturalSF since I do not have the “sequential”
> version of dmU.
> >
> > What is the proper way to handle this situation? Creating sequential
> subDMs and calling DMDistribute on each of them is ugly and possibly costly.
> >
> > Regards,
> >
> > Blaise
> >
> > --
> > Department of Mathematics and Center for Computation & Technology
> > Louisiana State University, Baton Rouge, LA 70803, USA
> > Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~
> bourdin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] DMplex: Natural Ordering and subDM

[Jed Brown]

I don't know the answer to your question (Matt?), but do you really need
to reorder the entire mesh or would it be sufficient to label your
points with their original numbering?

Blaise A Bourdin  writes:

> Hi,
> I am fighting with natural ordering in DMplex with multi-fields / multi 
> components DM.
> I need to read / write files in a distribution-independent ordering. The 
> natural way to do so is to define the natural ordering as that of the file, 
> then call DMPlexGlobalToNaturalBegin/End before VecView:
> The pseudo code would be
>
>   DMPlexCreateFromFile(PETSC_COMM_WORLD,ifilename,interpolate,);
>   // create default section associated with a complicated layout of multiple 
> fields 
>   // (U,\alpha,\sigma)
>   DMSetUseNatural(dmUAS,PETSC_TRUE);
>   DMPlexDistribute(dmUAS,0,,);
>
>   DMGetGlobalVector(dmDist,);
>   DMGetGlobalVector(dmDist,);
>   DMPlexGlobalToNaturalBegin(dmDist,UAS,UASNatural);
>   DMPlexGlobalToNaturalEnd(dmDist,UAS,UASNatural);
>   // Do my I/O stuff using UASNatural
>
> If I try to do the same thing one field at a time, things break:
> Say that I get the subDM for the first field (U):
>
>   DMCreateSubDM(dmDist,1,,,);
>
> Then dmU->useNatural is PETSC_FALSE and  I cannot compute dmU->sfnatural 
> using DMPlexCreateGlobalToNaturalSF since I do not have the “sequential” 
> version of dmU.
>
> What is the proper way to handle this situation? Creating sequential subDMs 
> and calling DMDistribute on each of them is ugly and possibly costly.
>
> Regards,
>
> Blaise
>
> -- 
> Department of Mathematics and Center for Computation & Technology
> Louisiana State University, Baton Rouge, LA 70803, USA
> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 
> http://www.math.lsu.edu/~bourdin

[petsc-dev] DMplex: Natural Ordering and subDM

[Blaise A Bourdin]

Hi,
I am fighting with natural ordering in DMplex with multi-fields / multi 
components DM.
I need to read / write files in a distribution-independent ordering. The 
natural way to do so is to define the natural ordering as that of the file, 
then call DMPlexGlobalToNaturalBegin/End before VecView:
The pseudo code would be

  DMPlexCreateFromFile(PETSC_COMM_WORLD,ifilename,interpolate,);
  // create default section associated with a complicated layout of multiple 
fields 
  // (U,\alpha,\sigma)
  DMSetUseNatural(dmUAS,PETSC_TRUE);
  DMPlexDistribute(dmUAS,0,,);

  DMGetGlobalVector(dmDist,);
  DMGetGlobalVector(dmDist,);
  DMPlexGlobalToNaturalBegin(dmDist,UAS,UASNatural);
  DMPlexGlobalToNaturalEnd(dmDist,UAS,UASNatural);
  // Do my I/O stuff using UASNatural

If I try to do the same thing one field at a time, things break:
Say that I get the subDM for the first field (U):

  DMCreateSubDM(dmDist,1,,,);

Then dmU->useNatural is PETSC_FALSE and  I cannot compute dmU->sfnatural using 
DMPlexCreateGlobalToNaturalSF since I do not have the “sequential” version of 
dmU.

What is the proper way to handle this situation? Creating sequential subDMs and 
calling DMDistribute on each of them is ugly and possibly costly.

Regards,

Blaise

-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin